Brain MRI atlases may be used to characterize brain structural changes across the life course. Atlases have important applications in research, e.g., as registration and segmentation targets to underpin image analysis in population imaging studies, and potentially in future in clinical practice, e.g., as templates for identifying brain structural changes out with normal limits, and increasingly for use in surgical planning. However, there are several caveats and limitations which must be considered before successfully applying brain MRI atlases to research and clinical problems. For example, the influential Talairach and Tournoux atlas was derived from a single fixed cadaveric brain from an elderly female with limited clinical information, yet is the basis of many modern atlases and is often used to report locations of functional activation. We systematically review currently available whole brain structural MRI atlases with particular reference to the implications for population imaging through to emerging clinical practice. We found 66 whole brain structural MRI atlases world-wide. The vast majority were based on T1, T2, and/or proton density (PD) structural sequences, had been derived using parametric statistics (inappropriate for brain volume distributions), had limited supporting clinical or cognitive data, and included few younger (>5 and <18 years) or older (>60 years) subjects. To successfully characterize brain structural features and their changes across different stages of life, we conclude that whole brain structural MRI atlases should include: more subjects at the upper and lower extremes of age; additional structural sequences, including fluid attenuation inversion recovery (FLAIR) and T2* sequences; a range of appropriate statistics, e.g., rank-based or non-parametric; and detailed cognitive and clinical profiles of the included subjects in order to increase the relevance and utility of these atlases.