We studied how brain volume loss in old age is affected by age, the APOE gene, sex, and the level of education completed. The quantitative characterization of brain volume loss at an old age relative to a young age requires—at least in principle—two MRI scans, one performed at a young age and one at an old age. There is, however, a way to address this problem when having only one MRI scan obtained at an old age. We computed the total brain losses of elderly subjects as a ratio between the estimated brain volume and the estimated total intracranial volume. Magnetic resonance imaging (MRI) scans of 890 healthy subjects aged 70 to 85 years were assessed. A causal analysis of factors affecting brain atrophy was performed using probabilistic Bayesian modelling and the mathematics of causal inference. We found that both age and sex were causally related to brain atrophy, with women reaching an elderly age with a 1% larger brain volume relative to their intracranial volume than men. How the brain ages and the rationale for sex differences in brain volume losses during the adult lifespan are questions that need to be addressed with causal inference and empirical data. The graphical causal modelling presented here can be instrumental in understanding a puzzling scientific area of study—the biological aging of the brain.