Over 40% of persons with multiple sclerosis (MS) are thought to experience cognitive impairment which adversely affects social participation, independence, and employment (1, 2). Cognitive impairment at diagnosis has been found to be associated with disability progression over time (3). Neuropsychological assessments objectively evaluate cognitive function, and are increasingly important in the care of persons with MS patients, as new rehabilitative strategies and pharmacologic therapies for cognitive impairments continue to emerge. Given that access to comprehensive neuropsychological assessments is often limited, several abbreviated test batteries have been recommended for use in persons with MS, including the Brief International Cognitive Assessment for Multiple Sclerosis (BICAMS) (4). Brief Repeatable Battery of Neuropsychological Tests (BRB-N) (5), and the Minimal Assessment of Cognitive Function in MS (MACFIMS) (5, 6). Interpretation of test results for both research and for clinical practice requires the use of normative data, although most available published normative data for these tests were developed in American populations. Application of American norms to Canadian populations is not recommended due to differences in performance between Canadian and American adults on measures of intellectual ability (7). Moreover, published norms were often established in samples that no longer reflect contemporary demographics; for example the proportion of individuals with higher levels of education was lower than in the present day population. Notably, Intelligence Quotient scores have risen over time (8), and use of outdated norms may lead to misclassification of cognitive status by underestimating the normal range of performance (9). In consideration of these issues, recommendations for international validation of the BICAMS were made to encourage its adoption (10).

Traditionally, normative data have been reported for discrete categories, such as age and/or education. More recently, continuous norms have been developed using multivariable regression equations that account for multiple demographic factors simultaneously. Regression-based norms for use in the Canadian population were recently developed for tests included in the MACFIMS battery (11), including the subset of tests included in BICAMS. Because these norms were derived from control populations recruited for other purposes, the number of participants available was fewer than the recommended 100 participants for some tests. In addition, while developed for use in Canada, the controls were drawn from only one region of Canada (i.e., province of Ontario), and the performance of these norms in an independently collected sample of healthy Canadian persons has not yet been assessed. Establishing the generalizability of norms is essential to determine if they may be appropriately applied in clinical and research settings more broadly than those from which the normative samples were drawn.

We sought to (i) test the performance of the previously published Canadian (Ontario) regression-based norms in independently collected samples of healthy controls from other Canadian regions; (ii) develop local regression-based norms for the tests included in the BICAMS; and (iii) examine differences in impairment classification rates in local healthy controls when applying BICAMS regression-based norms from different populations; and (iv) examine the ability of Canadian and non-Canadian norms to discriminate between local healthy and MS samples.

We conducted the primary analysis using MS and healthy control samples from Manitoba, Canada. Manitoba is a central Canadian province with a population of ~1.4 million people. We replicated our analyses in MS and healthy control samples from the eastern Canadian province of Nova Scotia (population ~1.0 million), which are described further in the replication section.

Throughout, we present the findings in Manitoba followed by the findings in the Nova Scotia replication sample. Of the 103 healthy participants from Manitoba, 96 were under age 65 years, and of 111 participants with MS, 93 were under age 65 years. The healthy participants were younger on average, but the age range of the healthy participants (18.2–64.4) was similar to that of the participants with MS (20.8–63.8) years. Most participants in each group were women, although the proportion who were women was higher in the MS group (Table 1). The average number of years of education was consistent with at least some post-secondary education in both groups although the healthy control group averaged 2.4 more years of education than the MS group. Race/ethnicity did not differ between the two groups, nor did estimated household income.

In the replication sample, most participants were also women, and the average number of years of education was consistent with at least some post-secondary education (Table 1).

In this cross-sectional study, we applied a set of previously developed regression-based norms from Ontario, Canada for tests comprising the BICAMS, to an independently collected healthy sample from Manitoba, Canada to assess their generalizability. We also replicated our findings in a second, smaller, normative sample from Nova Scotia, Canada. In healthy controls, the rates of impairment differed from standard population expectations, sometimes being higher than expected and sometimes being lower. The application of regression-based norms developed in other non-Canadian English-speaking populations also produced variable impairment rates that differed from expectations, as did the discrete norms from the test manuals. All of the norms differed in their ability to discriminate between MS and healthy populations from Manitoba, and between Manitobans with MS who had differing levels of disability or employment status. The local Manitoba norms generally had better discriminating ability in the Manitoba sample than other norms, but the CVLT-II and BVMT-R were still poor at discriminating between healthy participants and participants with MS. A prior report in a Belgian sample also found that the CVLT-II did not discriminate between persons with and without MS (33). Prior studies examining the sensitivity of neuropsychological tests suggest that the SDMT discriminates best between people with and without MS (34), and the SDMT is commonly found to be the test most associated with other clinically relevant factors (3). This high sensitivity of the SDMT to cognitive impairment in MS has been attributed to its assessment of commonly affected cognitive abilities including processing speed and working memory, as well as its requirements for efficient visual scanning and oculomotor functioning (27). Overall, our findings indicate that using regional norms to interpret all BICAMS tasks is likely to be most informative.

Spearman correlations between the different norms all exceeded 0.75 and most correlations exceeded 0.90. However, concordance coefficients were lower, indicating that while the norms rank ordered participants similarly, the absolute z-scores differed. Notably, in the Manitoba and Nova Scotia samples, concordance was lowest between the norms from Ireland and the other English language norms, which were developed in regions of Canada or the United States; potentially reflecting greater cultural differences between Ireland and North America than among North American regions for this verbal memory test. A prior study found that nationality influences performance on all three BICAMS tests, even after adjusting for age and years of education (35). That study highlighted the importance of considering both the language and culture of the individual being tested and called for additional studies across countries with common languages to address the potential influences of cultural factors. An approach by which BICAMS can be validated in other languages has been recommended (10) and a systematic review in 2018 reported on the performance of BICAMS as translated from English into 11 languages, following which performance was assessed (28). However, within countries, including Canada, where inhabitants may use one or more languages and/or are members of different cultural groups, there may be a need for particular effort to ensure appropriate norms are applied.

In principle, clinicians, and researchers may choose to use discrete norms that are commercially available for the cognitive tests they employ, locally validated norms, or regression-based norms from other populations. For example, regression-based norms derived from a Canadian sample have been employed in Sweden, albeit modified to exclude educational level (36). A large multi-center trial of exercise and cognitive rehabilitation will be applying Dutch norms at the Denmark site (37). Notably, even when we employed only norms developed in other regions of Canada, our local norms, and discrete norms from the manuals for each test that are used in clinical practice, we observed meaningful variations in impairment classification rates and in the ability to discriminate between and within groups. This reflects the differences in the absolute z-scores, as demonstrated by the lower concordance coefficients than correlation coefficients. These differences may reflect differences in the healthy populations enrolled, as well as differences in the approaches used to develop the norms. For example, Walker et al. used raw test scores in their regression models and did not incorporate a non-linear term for age (11), while Berrigan et al. used scaled scores and incorporated a non-linear term for age that reflected non-linear findings reported in large samples (22). Our findings suggest that methodological issues such as these constitute an important component of the wide variation in the frequency of cognitive impairment reported in the MS literature [reviewed in Chiaravalloti and DeLuca (38)]. Differences in the ability to discriminate between healthy and MS groups, and between groups of persons with MS at differing levels of neurologic disability and employment status, also highlight how the use of different norms affects the identification of factors influencing cognitive outcomes.

Within the Manitoba healthy sample, the contribution of demographic characteristics to cognitive performance also varied across the three cognitive tests evaluated, with the variance explained ranging from 15 to 21%, consistent with prior reports (26). The poorer performance seen on the SDMT and BVMT-R with increasing age is consistent with prior reports in healthy populations (39, 40). Sex was associated with performance of the CVLT-II, but not the SDMT or BVMT-R. One prior report suggested that the association of sex SDMT performance is only seen for the written version of this test, with women having better scores than men, whereas this is not the case of for the oral version used here and recommended for persons with MS (39). Education was not associated with cognitive performance, but most of our healthy sample was well-educated. Race predominantly contributed to performance on the SDMT in our sample although the association between race, ethnicity and performance of cognitive tests is well-recognized (40).

Raw scores on cognitive tests have been demonstrated to have higher sensitivity than demographically-corrected scores for discriminating between persons with and without cognitive impairment, but demographically-corrected scores have higher specificity (41). Several options exist for demographically correcting scores. Discrete norms are easy to develop but require continuous variables such as age to be categorized. This creates somewhat arbitrary and discontinuous changes in expected performance for individuals at the boundaries of those categories and relatively large sample sizes are required to develop precise norms with smaller categories that address this issue (15). Regression-based norms have become popular because they do not categorize continuous variables, and the improved efficiency of estimation allows for the use of substantially smaller sample sizes while providing more precise estimates. For the BICAMS, the international validation standards recommend that the minimum sample size is 65 healthy volunteers, provided that they are group matched on demographics to an MS sample (10). Samples of ≥150 persons or more are encouraged for generalizability. We used linear regression models to develop our norms as is common in the literature. This approach is affected by whether model assumptions are met, and model assumptions were met in this study. Nonetheless, skewness may interfere with norm accuracy (42), and outliers may exert a substantial influence on the norms that are developed, particularly in smaller samples. Linear regression examines the relationship between the conditional mean of the dependent variable to the independent variables of interest, and assumes that this adequately represents relationships across the entire distribution of the dependent variable. Moreover, traditional linear regression does not account for the fact that cognitive tests typically have a limited range of scores and therefore, we employed a truncated regression model to account for this issue.

Limitations of this study should be recognized. To ensure comparability with existing Canadian-Ontario regression-based norms, we did not include participants over age 65 years. However, after restricting our analyses to persons who were aged ≤65 years, we had 96 participants for developing local norms in Manitoba. While this exceeds the minimum 65 persons recommended in the BICAMS international standards for validation (10), it is slightly <100 recommended based on simulation studies (15). Like the healthy samples used to develop regression-based norms for BICAMS that we evaluated here (Supplementary Table e1), our healthy sample predominantly included women (n = 32 men). Most of our study population were white, thus further work is needed to develop norms that account for the racial/ethnic diversity in Canada and elsewhere. This is particularly important as recognition grows of the burden of MS in populations traditionally considered to be at a lower risk of MS such as indigenous Canadians and African Americans (43, 44). We did not capture acculturation which may also influence performance of norms (45). On average, the healthy control sample in Manitoba was younger than the MS sample, and more highly educated; differences in sex distribution were more modest as indicated by the standardized difference of <0.20. Norms should be applied cautiously in populations with different characteristics than those in whom they were developed due to limitations in generalizability, as illustrated by our findings. However, while the samples differed on average, the age and years of education distributions overlapped.

Regression-based norms have advantages over discrete norms. However, our findings emphasize the value of local norms when interpreting the findings of cognitive tests (46) and demonstrate the need to consider and assess the performance of regression-based norms developed in other populations when applying them to local populations, even when they are from the same country. This is important to avoid misclassifying individuals as to whether they are cognitively impaired or unimpaired. Our findings also strongly suggest that the development of regression-based norms should involve larger, more diverse samples to ensure broad generalizability. Specifically, greater representation is needed of men, individuals over age 65 years, and of varying racial, ethnic, and social backgrounds.

The datasets presented in this article are not readily available because some participants did not agree to data sharing; some data may be accessible to qualified investigators with the appropriate ethical approvals and data use agreements. Requests to access the datasets should be directed to Ruth Ann Marrie, rmarrie@hsc.mb.ca.

The studies involving human participants were reviewed and approved by the University of Manitoba Health Research Ethics Board and the Nova Scotia Health Authority Research Ethics Board. The patients/participants provided their written informed consent to participate in this study.

RM, JF, and RP conceived of the idea. RM, RP, CF, JK JB, LG, EM, JM, CB, and JF obtained study funding. RM and CW conducted the statistical analyses and drafted the manuscript. RM, RP, CW, CF, JK, JB, LG, EM, JM, CB, and JF revised the manuscript and approved of the final version. All authors contributed to the article and approved the submitted version.

RM receives research funding from: CIHR, Research Manitoba, Multiple Sclerosis Society of Canada, Multiple Sclerosis Scientific Foundation, Crohn's and Colitis Canada, National Multiple Sclerosis Society, CMSC, the Arthritis Society, and US Department of Defense. CW was supported by a Canadian Institutes of Health Research Frederick Banting and Charles Best Canada Graduate Scholarship Doctoral Award. RP receives research funding from the Workers Compensation Board of Manitoba. CF receives research funding from the Brain Canada Foundation, MS Society of Canada, Natural Sciences and Engineering Research Council of Canada, and Health Sciences Center Foundation. JK receives research funding from the MS Society of Canada, Crohn's and Colitis Canada, University of Manitoba, and Health Sciences Center Foundation. JB receives research funding from CIHR, Brain, and Behavior Research Foundation and the MS Society of Canada. LG receives research funding from: CIHR, the MS Society of Canada, Crohns and Colitis Canada, and the Health Sciences Center Foundation. EM received fellowship funding from NSERC and Alberta Innovates-Health Solutions and receives research funding from the MS Society of Canada. JM has conducted trials for Biogen Idec and Roche, and receives research funding from the MS Society of Canada. CB is supported in part by the Bingham Chair in Gastroenterology. He serves on Advisory Boards for Abbvie Canada, Janssen Canada, Takeda Canada, Pfizer Canada. He is a Consultant for Mylan Pharmaceuticals. He is receiving educational grants from Abbvie Canada, Pfizer Canada, Shire Canada, Takeda Canada, Janssen Canada. Speaker's panel for Abbvie Canada, Janssen Canada, Takeda Canada, and Medtronic Canada. He received research funding from Abbvie Canada. JF receives research funding from: CIHR, Multiple Sclerosis Society of Canada, Crohn's and Colitis Canada, the Nova Scotia Health Authority Research Fund; consultation and distribution royalties from MAPI Research Trust.