Meta-analysis of Montreal cognitive assessment diagnostic accuracy in amnestic mild cognitive impairment

Background The Montreal Cognitive Assessment (MoCA) is one of the most widely-used cognitive screening instruments and has been translated into several different languages and dialects. Although the original validation study suggested to use a cutoff of ≤26, subsequent studies have shown that lower cutoff values may yield fewer false-positive indications of cognitive impairment. The aim of this study was to summarize the diagnostic accuracy and mean difference of the MoCA when comparing cognitively unimpaired (CU) older adults to those with amnestic mild cognitive impairment (aMCI). Methods PubMed and EMBASE databases were searched from inception to 22 February 2022. Meta-analyses for area under the curve (AUC) and standardized mean difference (SMD) values were performed. Results Fifty-five observational studies that included 17,343 CU and 8,413 aMCI subjects were selected for inclusion. Thirty-nine studies were used in the AUC analysis while 44 were used in the SMD analysis. The overall AUC value was 0.84 (95% CI: 0.81, 0.87) indicating good diagnostic accuracy and a large effect size was noted for the SMD analysis (Hedge’s g = 1.49, 95% CI: 1.33, 1.64). Both analyses had high levels of between-study heterogeneity. The median cutoff score for identifying aMCI was <24. Discussion and conclusion The MoCA has good diagnostic accuracy for detecting aMCI across several different languages. The findings of this meta-analysis also support the use of 24 as the optimal cutoff when the MoCA is used to screen for suspected cognitive impairment.


Introduction
Amnestic mild cognitive impairment (aMCI) due to Alzheimer's disease (AD) is a syndrome that is associated with future progression to clinical AD.While not all individuals with aMCI progress to AD, they are thought to be at the highest risk of progression and this classification is often referred to as "MCI due to AD" (Albert et al., 2011;Sperling et al., 2011).The diagnostic criteria for aMCI have remained largely the same since their initial publication (Petersen et al., 1999) and require that an individual's episodic memory performance fall at least 1.5 standard deviations below what would be expected for their age and education level and is accompanied by a self-reported or collateral-reported complaint of cognitive decline.However, the aMCI diagnosis is made only after an extensive neuropsychological examination which prevents the diagnosis from being made in general practice settings where cognitive screening measures are often used to determine if an individual requires a more comprehensive cognitive assessment (Townley et al., 2019).Further refinements to the aMCI diagnostic criteria include the differentiation of those whose impairments are only in the memory domain (single domain) versus those who are impaired in memory and another cognitive domain (multiple domain) (Petersen and Negash, 2008).These classifications also apply for cases where the memory domain is not impaired (non-amnestic MCI), but other domains are (Petersen and Negash, 2008).
For several decades the Mini-Mental State Exam (MMSE) (Folstein et al., 1975) has been the most ubiquitous cognitive screening instrument, however the Montreal Cognitive Assessment (MoCA) (Nasreddine et al., 2005) is now among the most widely-used assessments for cognitive screening in general practice settings.Recent evidence indicates that the MoCA is superior to the MMSE in its ability to differentiate aMCI from normal cognition (Pinto et al., 2019a) as many individuals with aMCI often obtain normal scores on the MMSE (26-30) despite collateral reports of significant cognitive decline.The initial validation study of the MoCA recommended the same cutoff score as the MMSE (<26), however subsequent studies have indicated this cutoff may be too stringent and result in false positive indications of possible cognitive impairment (Wong et al., 2015;Carson et al., 2018;Ilardi et al., 2023).
To date, there has not been an extensive review and quantitative analysis of the MoCA's diagnostic accuracy for aMCI.Given that the MoCA has been translated into many different languages and dialects it is important to understand how consistent its diagnostic accuracy is across its various translations.The aims of this meta-analysis are to characterize the MoCA's diagnostic accuracy for aMCI and to characterize its relative effect size for mean differences between cognitively unimpaired (CU) older adults and those with aMCI using a large sample of published observational studies that cover a wide array of the languages that the MoCA has been translated in.

Inclusion criteria
Prior to conducting the literature searches, the following criteria for study selection and inclusion were established: (1) The data could not come from a treatment or intervention trial, (2) The study should report either raw means and standard deviations for MoCA performance in both the CU and aMCI groups OR the study should report should report area under the curve (AUC) values with standard error (SE) or 95% confidence intervals (CI), (3) The study should use either Petersen criteria (Petersen and Negash, 2008) to classify its aMCI subjects or DSM-V criteria for mild neurocognitive disorder (MND).Although in most circumstances using only one set of diagnostic criteria is preferred, we felt that including studies that used either the Petersen or DSM-V MND criteria would provide greater ecological validity for the study results since the MoCA is used primarily as a screening instrument in general practice settings where formal diagnostic criteria for cognitive impairment are not usually applied.PRISMA guidelines were followed for the analysis and a flow chart depicting study screening and selection is shown in Figure 1.

Literature search terms
Using the PubMed database, four different search terms were used.The first search term, "diagnostic accuracy and Montreal Cognitive Assessment" yielded 686 results from which 38 were screened and 15 were selected for inclusion.A second search using "mild cognitive impairment and Montreal Cognitive Assessment" yielded 1,884 results from which 60 were screened with 26 that were selected for inclusion.The third search using "area under the curve and Montreal Cognitive Assessment" yielded 146 results with 23 that were selected for screening from which five were included.A fourth search using the term "mild neurocognitive disorder and Montreal Cognitive Assessment" did not yield any additional studies beyond those already identified in the previous searches.All four searches were also carried out in the EMBASE database which yielded no additional articles.Nine additional articles were identified through reviews of references sections of the selected papers which brought the final total of included studies to 55 (Figure 1).The search approach taken for this study is consistent with "a multi-faceted approach that uses a series of searches" as described in the Cochrane Handbook of Systematic Reviews of Interventions (Lefebvre et al., 2023).

Data quality and extraction
From each of the included studies the following data were extracted: sample sizes for the CU and MCI groups, means and standard deviations of MoCA scores for the CU and MCI groups, AUC values with standard errors (SE).When 95% CIs were reported, SE was derived by taking the difference between the AUC estimate and the upper bound of the 95% CI and dividing by 3.92 (Higgins et al., 2022).The cutpoint associated with the AUC estimate, means and standard deviations for age and education levels (when education was reported in years), and the geographic region in which the study was conducted (Asia, Europe, North America) were also extracted from each study.The quality of each study was assessed using the National Heart, Lung, and Blood Institute (NHLBI) Study Quality Assessment of Case Control Studies1 which was used to grade each study as Good, Fair, or Poor.

Statistical analysis
The first analytic approach was a meta-analysis of AUC values derived from the receiver operator characteristic (ROC) analyses that differentiated aMCI from CU individuals.The second analytic approach included analyses of the standardized mean difference (SMD) (Hedge's g) and the raw mean difference (RMD) for MoCA scores between CU and aMCI.For both analytic approaches, results from random effects analyses were reported and the I2 statistic was used to quantify between-study heterogeneity which was classified as low, moderate, or high based on proposed guidelines (Higgins et al., 2003).Additional AUC and SMD analyses were carried out for subgroups based on geographic region (Asia, Europe, North America).The Egger's test was used to determine the presence of publication bias among the included studies.In addition, the median of the reported MoCA cutoff score was used to summarize the reported cutoff values for studies in the AUC analysis.Since the included studies came from a number of different geographic regions we anticipated a wide range of reported MoCA cutoff scores so using the median as a summary measure provides an overall estimate of the MoCA's cutoff that is relatively robust to the variability of reported cutoff values among the studies.All analyses were carried out using MedCalc Statistical Software version 20.109 (MedCalc Software Ltd., Ostend, Belgium, 2 2022).
The average age for CU groups was 71.06 ± 7.37 years with an average of 11.44 ± 3.27 years of education.For aMCI groups, the average age was 73.99 ± 7.65 years with an average of 9.89 ± 3.47 years of education.Mean MoCA scores for the CU groups was 24.98 ± 2.88 and 20.11 ± 3.76 for the aMCI groups.Among studies that reported optimal cutoff values (n = 44), the median was 24 (range = 17-27).Characteristics of each study included in the AUC meta-analysis are shown in Table 2.The overall AUC value was 0.84, 95% CI (0.81, 0.87), p < 0.001 with very high heterogeneity [I 2 = 90, 95% CI (87, 92%)] (Figure 2).The Egger's test indicated the presence of publication bias in the analysis (p = 0.002).The meta-analysis for differences in means demonstrated a large effect size [Hedge's g = 1.49, 95% CI (1.33, 1.64), p < 0.001; Table 3] with very high heterogeneity [I 2 = 93, 95% CI (91, 94%)] and an Egger's test that indicated the presence of publication bias (p = 0.003).The large effect size reported here equates to a 4.73 (95% CI: 4.20, 5.27) point difference on the MoCA between CU and aMCI groups (Figure 3).Characteristics of each study included in the SMD meta-analysis are shown in Table 2. Funnel plots depicting the publication bias in the AUC and SMD analyses are shown in Figure 4.

Discussion
This meta-analysis assessed the diagnostic accuracy and the mean difference of the MoCA when comparing aMCI older adults to those who are CU across several global regions.The overall AUC value of 0.84 indicates that the MoCA has good diagnostic accuracy for aMCI, however a very high degree of between-study heterogeneity was noted for this finding.The analysis of MoCA mean differences yielded a large effect size (Hedge's g = 1.49) and a high degree of between-study  heterogeneity was also noted for this analysis.English and Mandarin studies (n = 26) made up approximately half of the studies included in the meta-analysis and among the studies that assessed diagnostic accuracy a score of 24 was the most commonly-used cutoff for differentiating aMCI from CU individuals.However, it was noted that the range of reported cutoff values was 17 to 27 which suggests that optimal MoCA cutpoints may be population-and context-specific in order to avoid misclassification errors.A recent systematic review highlights this point by noting that cross-cultural differences necessitate the use of varying cutoff values as well as corrections for Forest plot for standardized mean difference of MoCA performance between CU and aMCI groups.Funnel plots for MoCA diagnostic accuracy (A) and standardized mean difference (B).
Others have also noted significant problems with misclassification on the MoCA when a single cutoff is used as higher rates of false positive indications of impairment were noted with increased age and decreased educational levels (Wong et al., 2015).Based on these previous reports the high levels of between-study heterogeneity in this meta-analysis may reflect the cultural, linguistic, and educational diversity among the included studies rather than any particular methodological weakness among them.These findings also emphasize the need to frame the MoCA's utilization in a screening rather than a diagnostic context.Here it is also important to consider the sensitivity and specificity of a cognitive screening measure and how this impacts the utilization of full neuropsychological evaluations.The high falsepositive rates of impairment on the MoCA using 26 as the cutoff could lead to many CU individuals being referred for unnecessary neuropsychological evaluations (Ilardi et al., 2023).In contrast, lowering the cutoff score for impairment also has the effect reducing the MoCA's sensitivity in correctly detecting aMCI which further underscores the notion that the MoCA's cutoff score can be adjusted for a given population in order to optimize its diagnostic accuracy.Additionally, adjustments to the cutoff score can be made when physical limitations (e.g., hearing loss) substantially impact MoCA performance (Utoomprurkporn et al., 2020).
A previous meta-analysis of nine studies investigating the MoCA's diagnostic accuracy showed that the optimal MoCA cutoff for detecting aMCI was 23 (Carson et al., 2018) and a recent systematic review found that the AUC value for the MoCA in differentiating aMCI from CU individuals ranged from 0.71 to 0.99 across 34 studies (Pinto et al., 2019b) putting the AUC value of this meta-analysis (0.84) near the midpoint of this range.The three global regions examined in this meta-analysis (North America, Asia, Europe) all had comparable AUC and effect size values despite each region having a high degree of between-study heterogeneity.
There are some limitations to this meta-analysis.Despite the very large number of studies included for both the AUC and SMD analyses, a high degree of between-study heterogeneity was noted for all analyses which decreases the level of confidence one may have in the findings that are reported.A number of different factors may account for the high heterogeneity such as study setting (clinic vs. communitybased), varying educational attainment of the populations among the different geographic regions, and cultural norms and values that may impact test performance.While the inconsistencies of the reported AUC and SMD values across studies warrant some degree of skepticism for the final results, there is also significant value in findings that are derived from such a large number of studies across different geographic regions and this aspect of the meta-analysis will likely appeal to clinicians who use the MoCA.
The findings of this meta-analysis provide further support for the use of the MoCA as an accurate cognitive screening tool for use in general practice settings.In line with other studies of the MoCA in aMCI and CU samples, a score of 24 appears to be the optimal cutoff to use for identifying cognitive impairment.

FIGURE 2
FIGURE 2Forest plot of MoCA diagnostic accuracy for amnestic mild cognitive impairment.

TABLE 1
Distribution of MoCA administration language among included studies.

TABLE 2
Characteristics of studies used in the diagnostic accuracy meta-analysis.

TABLE 3
Characteristics of studies used in the standardized mean difference meta-analysis.
Mean ± standard deviation; nr, not reported.*Standard deviation was not reported.

TABLE 4
Diagnostic accuracy and standardized mean difference analyses stratified by global region.