It is now acknowledged that Alzheimer’s Disease (AD) processes are present decades before the onset of clinical symptoms, but it remains unknown whether lifestyle factors can protect against these early AD processes in mid-life. We asked whether modifiable lifestyle activities impact cognition in middle-aged individuals who are cognitively healthy, but at risk for late life AD. Participants (40–59 years) completed cognitive and clinical assessments at baseline (N = 206) and two years follow-up (N = 174). Mid-life activities were measured with the Lifetime of Experiences Questionnaire. We assessed the impact of lifestyle activities, known risk factors for sporadic late-onset AD (Apolipoprotein E □4 allele status, family history of dementia, and the Cardiovascular Risk Factors Aging and Dementia score), and their interactions on cognition. More frequent engagement in physically, socially and intellectually stimulating activities was associated with better cognition (verbal, spatial and relational memory), at baseline and follow-up. Critically, more frequent engagement in these activities was associated with stronger cognition (verbal and visuospatial functions, and conjunctive short-term memory binding) in individuals with family history of dementia. Impaired visuospatial function is one of the earliest cognitive deficits in AD and has previously associated with increased AD risk in this cohort. Additionally, conjunctive memory functions have been found impaired in the pre-symptomatic stages of AD. These findings suggest that modifiable lifestyle activities offset cognitive decrements due to AD risk in mid-life and support the targeting of modifiable lifestyle activities for the prevention of Alzheimer’s Disease.