Age-related declines in processing speed are hypothesized to underlie the widespread changes in cognition experienced by older adults. We used a structural covariance approach to identify putative neural networks that underlie age-related structural changes associated with processing speed for 42 adults ranging in age from 19 to 79 years. To characterize a potential mechanism by which age-related gray matter changes lead to slower processing speed, we examined the extent to which cerebral small vessel disease influenced the association between age-related gray matter changes and processing speed. A frontal pattern of gray matter and white matter variation that was related to cerebral small vessel disease, as well as a cerebellar pattern of gray matter and white matter variation were uniquely related to age-related declines in processing speed. These results demonstrate that at least two distinct factors affect age-related changes in processing speed, which might be slowed by mitigating cerebral small vessel disease and factors affecting declines in cerebellar morphology.