Sex hormones such as estrogen fluctuate across the female lifespan, with high levels during reproductive years and natural decline during the transition to menopause. Women’s exposure to estrogen may influence their heightened risk of Alzheimer’s disease (AD) relative to men, but little is known about how it affects normal brain aging. Recent findings from the UK Biobank demonstrate less apparent brain aging in women with a history of multiple childbirths, highlighting a potential link between sex‐hormone exposure and brain aging. We investigated endogenous and exogenous sex‐hormone exposure, genetic risk for AD, and neuroimaging‐derived biomarkers for brain aging in 16,854 middle to older‐aged women. The results showed that as opposed to parity, higher cumulative sex‐hormone exposure was associated with more evident brain aging, indicating that i) high levels of cumulative exposure to sex‐hormones may have adverse effects on the brain, and ii) beneficial effects of pregnancies on the female brain are not solely attributable to modulations in sex‐hormone exposure. In addition, for women using hormonal replacement therapy (HRT), starting treatment earlier was associated with less evident brain aging, but only in women with a genetic risk for AD. Genetic factors may thus contribute to how timing of HRT initiation influences women’s brain aging trajectories.