Inter-rater and intra-rater reliability of the Chinese version of the short orientation–memory–concentration test in people with stroke

Purpose: This study aimed to use three statistical methods to investigate the inter-rater and intra-rater reliability of the Chinese version of the Short Orientation-Memory-Concentration Test (C-SOMC) for people who have had a stroke.

Methods: Forty-four participants (31 men and 13 women) with a mean age of 59.05 ± 10.79 years who have had experienced a single episode of stroke were enrolled in this study. To determine the inter-rater reliability of the C-SOMC, the test was administered to each participant on the same day by two raters (A and B) who each had more than 1 year of work experience. To determine intra-rater reliability, rater B re-evaluated 36 of the 44 participants with the C-SOMC on the subsequent day. Intraclass correlation coefficients (ICCs), paired-samples t-tests, and Bland-Altman plots were used to analyze the inter-rater and intra-rater reliability.

Results: The evaluation of inter-rater reliability for the total score and item 1, 4, 5, 6, and 3/7 showed respective ICCs of 0.959, 0.918, 1.000, 0.942, 0.905 and 0.913, indicating excellent inter-rater reliability for the C-SOMC. Item 2 had an ICC of 0.796, indicating moderate to good inter-rater reliability. The evaluation of intra-rater reliability showed an ICC of 0.978 for the total score, and respective ICCs of 1.000, 1.000, 1.000, 0.968, 0.973 and 0.929 for the individual items, indicating excellent intra-rater reliability for the C-SOMC. The paired-samples t-test for the C-SOMC showed no statistically significant differences (P > 0.05) between ratings made by two different raters or by the same rater on separate occasions. The minimal detectable change value at the 95% confidence threshold (MDC₉₅) of the SOMC score was found to be 2.14. Bland-Altman plots showed a mean difference of 0.02 and 95% limits of agreement (95% LOA) ranging from −3.86 to 3.90 for the inter-rater measurements and a mean difference of 0.33 and 95% LOA of −2.71 to 3.37 for the intra-rater measurements.

Conclusions: The C-SOMC demonstrated excellent inter-rater and intra-rater reliability when administered to people who have had a stroke. The C-SOMC may be used to screen for cognitive impairment in people who have had a stroke.

Introduction

Stroke is the second leading cause of death and the third leading cause of disability worldwide (1). Given the growing global population and advancements in public health and medicine, the number of stroke survivors is increasing annually (2). In 2013, a nationally representative door-to-door survey in China found that the respective age-standardised prevalence and incidence rate of stroke were 1,114.8 per 100,000 people and 246.8 per 100,000 person-years (3). By 2017, stroke had become the leading cause of death, years of life lost, and disability-adjusted life-years in China (4). In addition, in China, the prevalence of stroke continuously rose from 2013 to 2019 (5). Stroke survivors often experience cognitive impairment, physical disability, fatigue, and psychological problems, all of which affect their daily living activities and place huge burdens on their families and society (6). Post-stroke cognitive impairment is the most prevalent clinical syndrome in stroke survivors (7), affecting approximately 20%–80% of this population (2, 8). It can lead to poor recovery of physical functions (9), great restriction in daily activities and social participation, and a lower quality of life (7). Post-stroke cognitive impairment is even associated with the recurrence of ischaemic stroke in high-risk patients (10). However, cognitive impairment is not as noticeable as physical disabilities and can be overlooked by patients and caregivers. Early diagnosis and intervention for cognitive impairment are therefore essential. Assessment for cognitive impairment can be improved by the routine use of validated screening instruments, and very short screening protocols are particularly useful due to their brevity and ease of administration (11).

The Mini-Mental State Examination (MMSE) is the most widely known and used screening tool for evaluating cognition among persons who have had a stroke (12). However, previous studies have reported that use of the MMSE among people with acute and subacute stroke has been limited (13), because of insensitive assessment of executive functioning, working memory, and visual perception. Moreover, many stroke survivors suffer from motor dysfunction of the upper limb, which can affect their ability to perform the necessary drawing task in the MMSE. Furthermore, the MMSE includes a larger number of items and requires a more extended period to administer. Therefore, researchers are searching for alternative screening tools to eliminate the impact of these unrelated factors on the result of cognitive assessment.

The Short Orientation-Memory-Concentration Test (SOMC) is a commonly used screening tool for assessing cognitive function, including the dimensions of orientation, concentration and memory (14, 15). Developed as a modification of the 26-item Information-Memory-Concentration Mental Status Test (16), the SOMC was streamlined to 6 items to enhance clinical utility and ease of administration. The SOMC demonstrates a high degree of correlation (r = 0.92) with the full orientation–memory–concentration test and exhibits nearly equivalent sensitivity (15). Additionally, the SOMC test scores are positively correlated with the immediate (Pearson's r = 0.68) and delayed (Pearson's r = 0.74) paragraph recall scores from the Rivermead Behavioural Memory Test (17), and shows excellent test-retest reliability (r = 0.99) in telephone screening of elderly patients (18). Our previous study (19) validated the Chinese version of SOMC for assessing cognitive impairment in stroke patients, demonstrating moderate to good correlations (r = 0.57–0.64) with the Chinese version of the MMSE.

The SOMC has been used in various populations such as patients with dementia or Alzheimer's disease, patients given an anesthesia, and healthy older adults (20–26), and previous studies have shown that the SOMC can distinguish between mild, moderate, and severe cognitive deficits (15, 17). The SOMC is brief and can be completed verbally at the patient's bedside without requiring them to write or draw (27), making it particularly friendly for patients with upper limb and hand movement disorders such as stroke survivors. Additionally, the SOMC has excellent test–retest correlation and concurrent validity when used in the telephone screening of elderly patients (18), which allows physicians to efficiently screen the cognitive status of homebound patients. Based on these characteristics, the SOMC is a promising tool for assessing the cognitive status of people who have had a stroke. However, to the best of our knowledge, no further studies investigated the other psychometric properties of SOMC in people with stroke.

Concise and efficient screening instruments such as the Short Orientation-Memory-Concentration (SOMC) test are essential for assessing cognitive function in large populations. The SOMC has already been translated into Dutch (28) and German (29). However, such instruments are currently lacking in China. To allow the use of the SOMC for assessing the cognitive function of Chinese individuals, there is a significant clinical need for a translated Chinese version of the SOMC (C-SOMC) and an investigation of the psychometric properties of this C-SOMC in Chinese population. We therefore translated the original SOMC into Chinese using a translation–back translation procedure and explored its concurrent validity in our previous study (19). Despite this, other important psychometric properties remain unexplored. To facilitate the use of the SOMC for assessing cognitive function in Chinese individuals, there is a significant clinical need for a comprehensive investigation of its psychometric properties in the Chinese population. In addition, to the best of our knowledge, the reliability of the SOMC has not yet been evaluated for use among people who have had a stroke. In this study, the inter-rater and intra-rater reliability of the C-SOMC during stroke population were explored using intra-class correlation coefficient (ICC), paired sample test and Bland-Altman plots.

Materials and methods

Study design

This study used a cross-sectional design. The participants' demographic information and major comorbidity data were collected from medical records. This study was approved by the Human Subjects Ethics Subcommittee of the First Affiliated Hospital of Sun Yat-sen University in China (Ethics Number: [2014]88). Informed written consent was obtained from all the participants.

Participants

The inclusion criteria for the participants were as follows: (1) had a first stroke with unilateral hemiparetic lesions confirmed by magnetic resonance imaging or computed tomography; (2) an interval of ≥5 days after the stroke; (3) aged of 18–80 years; (4) Glasgow coma scale score of 15; (5) no severe deficits in communication; (6) did not have delirium; and (7) able to give informed consent. Patients were excluded if they: (1) were unable to complete assessments due to medical instability; (2) were diagnosed with other neurological diseases that can affect cognitive function; and (3) had taken medication for mental illness.

Outcome measure

The Chinese version of the short orientation–memory–concentration test (C-SOMC)

The C-SOMC consists of 6 items and is used to assess cognitive function along the domains of orientation, concentration and memory. The C-SOMC test is presented in Appendices 1 and 2. Item 6 of the SOMC originally requires reciting the 12 months in reverse order. In Chinese culture, however, reciting months backwards is equivalent to counting backwards from 12 to 1, a task already covered by item 5 (counting backwards from 20 to 1). In C-SOMC, we therefore replaced the months with the twelve Chinese zodiac animals—rat, ox, tiger, rabbit, dragon, snake, horse, goat, monkey, rooster, dog and pig—which are universally familiar in China. Like the SOMC, the total score for the C-SOMC ranges between 0 and 28, with higher scores indicating better cognitive function (30). The scores are categorized as follows: 24–28 = normal cognition; 19–23 = questionable impairment, and ≤18 = dementia (31). The breakdown of scores for each item is as follows. For item 1 (“What year is it now?”), a correct answer is awarded 4 points, while an incorrect answer receives 0 points. For item 2 (“What month is it now?”), a correct answer is awarded 3 points, with 0 points for an incorrect answer. Item 4 (“About what time is it? within an hour”) is awarded 3 points for a correct answer and 0 points for an incorrect answer. Item 5 (“Count backwards 20 down to 1”) assesses completion accuracy. A score of 4 points is given for completion without errors, 2 points for completion with one error, and 0 points for completion with two or more errors. For item 6 (“State the twelve Chinese zodiac signs in reverse order”), the same criteria apply, with 4 points for completion without any error, 2 points for completion with one error, and 0 points for completion with two or more errors. Scores for item 3/7 (“Repeat the address given”) are deducted 2 points for each error on a first name, family name, number, road name, or city name, resulting in a score ranging from 0 to 10 points.

Procedure

Participants were recruited from the Department of Rehabilitation Medicine of the First Affiliated Hospital, SunYat-sen University between August 2015 and July 2019. To determine inter-rater reliability, two therapists (raters A and B) who were familiar with the C-SOMC administered it to all the participants. Rater A was a physiotherapist with more than 10 years of clinical experience in stroke rehabilitation, while rater B had 1 year of clinical experience in stroke rehabilitation. The Day 1 assessment was conducted by rater A and rater B, while the Day 2 (on the subsequent day of Day 1) assessment was conducted by rater B only. Rater B was blinded to the results of assessments performed by rater A. On item 3/7, rater A chose either address a or b for participants to recall, and rater B chose either address c or d for participants to recall. There were 8 participants who were either unable or unwilling to participate in a third round of testing within a 2-day period, which was used to minimise the possible effect of spontaneous recovery (32). To assess intra-rater reliability, the C-SOMC was readministered by rater B to the remaining participants in a third round of testing on the second day. Based on previous studies, the remaining sample of 36 participants was sufficient to determine the reliability of the C-SOMC (17, 18). For item 3/7, the third round of testing instructed to patients was chosen during either address c or d but different from that of the second round. The C-SOMC was administered in a quiet room to minimize confounding factors.

Statistical analysis

Sample size calculation

The sample size was calculated by an online calculator (33). As there was no previous study explore the reliability of C-SOMC, we assume the minimal acceptable reliability and the expected reliability as 0.75 and 0.90, with the significant level set at 0.05 and the power set as 80%. The minimal sample size was calculated to be 33 subjects. To draw a more robust conclusion, this study enrolled 44 in patients who have had a first episode of stroke.

Participants

Descriptive statistics were used to analyze the demographic and clinical characteristics of the participants in this study (n = 44).

Reliability

The inter-rater and intra-rater reliabilities were analysed using intraclass correlation coefficient, paired sample tests, and Bland–Altman plots.

(1) Intraclass Correlation Coefficient

To minimise the effects of spontaneous recovery and learning effect on the results, we used ICC Model 3 (Raters in the study are the specific raters of interest and fixed) to quantify the degrees of inter-rater and intra-rater reliability (34). To assess inter-rater reliability, ICC Model 3,2 (two-way mixed effects, absolute agreement, multiple raters) was used to analyze the two sets of C-SOMC data obtained though administration of the test by raters A and B at the same day. To assess intra-rater reliability, ICC Model 3,1 (two-way mixed effects, absolute agreement, single rater) was used to analyze the two sets of C-SOMC data obtained by administration of the test by rater B at two different times in 2 consecutive days. Based on a previous study, ICC values of <0.50, 0.50–0.75, 0.75–0.90, and >0.90 are considered to respectively indicate poor, moderate, good and excellent reliability (35). For the calculation of the 95% CI of ICC (35), please find it as below:

$95\text{\%}\mathrm{CI}=\mathrm{ICC}\pm t(df,0.025)\times \sqrt{\mathrm{Var}(\mathrm{ICC})*}$

where:

t(df, 0.025) is the t-statistic with degrees of freedom (df) and a two-tailed alpha of 0.025, Var(ICC) is the variance of the ICC, calculated as:

$\mathrm{Var}(\mathrm{ICC})={\displaystyle \frac{(1-\mathrm{ICC})2\cdot (1+(k-1)\cdot \mathrm{ICC})2}{k^2\times (n-1)}}$

where:

n is the number of subjects; k is the number of raters.

In this study, the 95% CI of ICC was calculated by the SPSS directly.

The minimal detectable change (MDC) is used to represent the real change in the test rather than the measurement error (36). It was calculated by the following formula: ${\mathrm{MDC}}_{95}=1.96\times$ $\mathrm{SDpool}\times \sqrt{2\times (1-\mathrm{ICC})}$, where the SDpool is the mean SD of the SOMC scores on the two time points and the ICC is the intra-rater reliability coefficient.

(2) Paired sample t-test

The paired sample t-test will be used to assess the difference in inter-rater (rater A vs. rater B) and intra-rater (rater B's Day 1 and rater B's Day 2) assessment of C-SOMC results. The t-statistic is calculated using the following formula:

$t={\displaystyle \frac{\overline{D}}{8_D/\sqrt{n^2}}}$

where:

8_D is the standard deviation of the differences.

$8_D=\sqrt{{\displaystyle \frac{\sum {(D-\overline{D})}^2}{n-1}}}$

(3) Bland–Altman plot

To provide a more detailed analysis of the differences between scores obtained by different raters or by the same rater at different times, a Bland–Altman plot is used to compare the mean differences and 95% limits of agreement (LOA) between the total C-SOMC scores. The 95% LOA is similar to the minimal detectable change (37) and sets a threshold to distinguish real biological change (such as the spontaneous recovery of impairment) from measurement noise (38). A narrow 95% LOA signifies that the measurement tool can detect more subtle biological changes occurring over time or with an intervention (37, 39). The mean difference is calculated as the average difference between 2 assessments of the same individuals. The upper and lower boundaries of the 95%LOA are obtained by calculating the mean values from the 2 assessment sessions and adding or subtracting 1.96 standard deviations (SD) (40). A smaller mean difference and 95%LOA indicates better agreement.

All statistical analyses were generated using SPSS version 20.0 (IBM, Inc., Armonk, NY, USA). All tests were two-tailed, and a significance level of p-value <0.05 was used to indicate statistical significance.

Results

Demographics

This study enrolled 44 participants who had a first episode of stroke (ischemic, n = 38; haemorrhagic, n = 6), of whom 31 were men and 13 were women. The mean age of the participants was 59.05 ± 10.79 years, with a range of 40–80 years. Details of the demographic and clinical characteristics of the participants are provided in Table 1.

Table 1

Table 1. Characteristics of the study participants (n = 44).

The C-SOMC total and item performance scores obtained by the different raters are summarised in Table 2. The data from the re-evaluation of 36 participants by rater B at separate times were pooled to assess the intra-rater reliability. The demographic information for these 36 participants is also provided in Table 1. The C-SOMC total and item performance scores for these 36 participants are shown in Table 3.

Table 2

Table 2. C-SOMC inter-rater reliability (n = 44).

Table 3

Table 3. C-SOMC intra-rater reliability (n = 36).

Inter-rater reliability

The inter-rater reliability of the C-SOMC was evaluated using data from all 44 participants. The ICC for the total score was 0.959 [95% confident interval (CI): 0.926–0.978], indicating excellent inter-rater reliability of the C-SOMC in individuals who have had a stroke. The ICC for item 2 scores was 0.796 (95% CI: 0.627–0.889), showing moderate to good interrater reliability. The ICC for the scores for items 1, 4, 5, 6, and 3/7 ranged from 0.905 to 1.000 (95% CI: 0.826–1.000), which suggested good to excellent inter-rater reliability for these items. The details of inter-rater reliability of the C-SOMC are summarised in Table 2.

Table 4 detailed the differences between the scores for total and individual items of the C-SOMC as administered by rater A (C-SOMC A) and rater B (C-SOMC B). The differences between the scores obtained by the two raters ranged from +6 to −4 (median 0.00, 95% percentiles +2 to +6), with a mean difference of 0.02 (SD 1.982; 95% CI: −0.59 to 0.61).

Table 4

Table 4. Difference between C-SOMC results administrated by rater A and rater B (n = 44).

Table 5 illustrated the paired sample t-test for the scores of the C-SOMC obtained by two raters demonstrating no statistically significant difference (P > 0.05).

Table 5

Table 5. The paired sample t-test of C-SOMC administered by two raters (n = 44) or by rater B on separate occasions (n = 36).

Figure 1 showed the analysis of inter-rater reliability. The mean difference between the scores obtained by the two raters was 0.02, which did not differ significantly from zero. The 95% LOA ranged from −3.86 to 3.90 and showed five outliers. The results showed a high level of concurrence between the C-SOMC scores obtained from different raters, corroborated by the mean difference and the 95% LOA.

Figure 1

Figure 1. Scatterplots of the differences between two raters. The dashed bold line represented the mean difference score between two raters. The dashed lines exhibited the limits of agreement (mean ± 1.96× the standard deviation of the different score).

Intra-rater reliability

The intra-rater reliability of the C-SOMC was assessed using data from 36 participants who were re-evaluated by rater B at different times. The ICC for the total score was 0.978 (95% CI: 0.956–0.988), indicating excellent intra-rater reliability of the C-SOMC for individuals who have had a stroke. The ICC for item 3/7 scores was 0.929 (95% CI: 0.865–0.965), indicating good to excellent intra-rater reliability. The ICCs for the scores of items 1, 2, 4, 5, and 6 ranged from 0.968 to 1.000 (95% CI: 0.938–1.000), indicating excellent intra-rater reliability for items 1, 2, 4, 5, and 6. The minimal detectable change value with a 95% confidence threshold (MDC₉₅) of SOMC score was found to be 2.14. The details of intra-rater reliability assessments of the C-SOMC are summarised in Table 3.

Table 6 showed the differences between the total and individual item scores obtained by rater B on separate occasions (C-SOMC B1 and C-SOMC B2). The differences between the scores obtained by rater B on administration of the C-SOMC at different times ranged from +4 to −4 (median 0.00, 95% percentiles +2 to +4), with a mean difference of 0.33 (SD 1.549; 95% CI: −0.17 to 0.78).

Table 6

Table 6. Difference between C-SOMC results administrated by rater B at different occasion (n = 36).

Table 5 illustrated the paired t-test for the comparing C-SOMC scores obtained by rater B on separate occasions demonstrating no statistically significant difference (P > 0.05).

Figure 2 showed the analysis of intra-rater reliability. The mean difference between the two evaluation scores obtained by rater B was 0.33, which did not differ significantly from zero. The 95%LOA ranged from −2.71 to 3.37 and showed two outliers. The findings indicated a high level of concurrence between the C-SOMC assessments conducted by the same rater at different time points, as corroborated by the mean difference and the 95% LOA.

Figure 2

Figure 2. Scatterplots of the differences between two measurements by the same rater at different time points. The dashed bold line represented the mean difference score between two raters. The dashed lines exhibited the limits of agreement (mean ± 1.96× the standard deviation of the different score).

Discussion

The aim of this study was to evaluate the reliability of the C-SOMC when used to assess individuals who have had a first episode of stroke. Our results indicated that the total C-SOMC scores and all individual items demonstrated good to excellent inter-rater and intra-rater reliability. There was no significant difference of C-SOMC score when assessed by different raters or by the same rater at different times. Furthermore, analyses with a Bland–Altman plot demonstrated that both the inter-rater and intra-rater evaluations showed small mean differences and 95%LOA ranges, indicating strong agreement. Therefore, the C-SOMC is a reliable test for screening cognitive impairment in individuals who have had a stroke.

The inter-rater reliability analysis conducted in our study demonstrated excellent reliability for the total score of the C-SOMC. A previous study by David et al. showed good inter-rater reliability (k = 0.63) of the SOMC in geriatric patients older than 75 years who presented to the emergency department (41). Age can be a significant factor contribute to the difference in the inter-rater reliability between the 2 studies. Compared with our study (mean age = 59.0 years old), older adults (e.g., >75 years) in David’ study may have lower cognitive reserve, making their cognitive performance more susceptible to fluctuations and measurement errors. Age-related changes in brain structure and function, including reduced processing speed, working memory capacity, and attention control, are well-documented in cognitive aging literature (42–44). Moreover, older adults may experience greater fatigue during testing, have more difficulty with test instructions, or show increased sensitivity to environmental distractions, all of which can contribute to measurement inconsistency across raters (45). In addition, geriatric patients in the emergency setting may have more variable cognitive states due to acute medical conditions or delirium (46). Our results also indicated excellent inter-rater reliability for items 1,4,5,6, and 3/7 and good inter-rater reliability for item 2. Although item 2 did not show excellent inter-rater reliability, a comparison of the scores obtained by rater A and rater B showed a difference in the assessment of only one participant. Additionally, the differences between the C-SOMC scores obtained for the same participant by rater A and rater B were mainly reflected in item 3/7. Twelve of the 44 subjects performed better on the item 3/7 when evaluated by rater A in the first assessment, which may be attributed to the participants being impatient or less nervous and anxious during the second round of testing on the same day. Seven of the 44 subjects performed better in item 3/7 when evaluated by rater B within one day, which could be attributed to a learning effect. Overall, our results demonstrated excellent inter-rater reliability for the total and individual item scores of the C-SOMC.

Our intra-rater reliability analysis indicated excellent reliability for both the total C-SOMC and individual item scores when the test was administered to the same participant by the same rater at different time points. Several studies have previously evaluated the intra-rater reliability of the SOMC. Davous et al. reported good intra-rater reliability of the SOMC when administered by the same rater during 18 patients with Alzheimer's disease at different evaluations done one month apart (47). Wade and Vergis investigated the intra-observer reliability of the SOMC in 38 people with a variety of neurological diseases and discovered that the SOMC was reasonably reliable when applied by one rater twice over an interval of one week (17). We administered the C-SOMC twice within a shorter time interval of one day to 36 patients with a first stroke. The higher intra-rater reliability shown in our study could be due to the fact that we included individuals with less severe cognitive impairment. Additionally, our results indicated excellent intra-rater reliability for all the individual items.

In this study, the MDC₉₅ was calculated to be 2.14. The mean difference of SOMC score between the two different time points was less than 2.14, which indicated that the inter-rater and intra-rater mean difference of SOMC was mainly contributed by the measurement error rather than the real change of cognitive function. Thus, the cognitive function among the subjects was stable in this study. The variances of cognitive function due to the spontaneous recovery may not make a significant impact on our result.

The differences between the C-SOMC scores for the same participant as evaluated by rater B at different time points, were mainly from the scores on item 3/7. Ten of the 36 subjects performed better on item 3/7 during the first assessment by rater B. All participants gained two more points on the second assessment, which could be due to the participants being impatient or less nervous and anxious during the third assessment done in the span of two days. Seven of the 36 subjects showed an improvement in item 3/7 during the second assessment by rater B, which could be attributed to a learning effect. In conclusion, our results demonstrated excellent intra-rater reliability for the C-SOMC and its individual items.

Our inter-rater and intra-rater analyses showed small mean differences and acceptable 95% LOA ranges, indicating good reliability of the C-SOMC. While most measurements fell within the 95% LOA, a small percentage of participants in the inter-rater (11.4%, 5 of 44) and intra-rater (5.6%, 2 of 36) analyses fell outside these limits. The Bland-Altman plots revealed a greater 95% LOA range in the inter-rater plot compared to the intra-rater plot, suggesting slightly lower inter-rater agreement than intra-rater agreement, which was an expected pattern in reliability studies. Nevertheless, both analyses demonstrated that the C-SOMC was a stable evaluation tool. The LOA in Bland-Altman analysis is conceptually similar to minimal detectable change (39), which helps distinguish true biological differences from measurement error (48). Our finding that 88.6%–94.4% of points fell within the 95% LOA indicated that the observed differences between measurements were primarily due to expected measurement error rather than actual changes in cognitive function (38). Thus, our results suggested that the C-SOMC is sufficiently sensitive to detect meaningful changes in cognitive function among stroke patients.

In conclusion, the C-SOMC demonstrated excellent inter-rater and intra-rater reliability when administered to individuals who have had a stroke. Most participants in this study showed good orientation and attention due to superior performance on items 1, 2, 4, and 5. Some had suboptimal scores on item 3/7, suggesting that they may have memory impairment. Many participants in this study had poor performance on item 6, and most participants reported that they have never tried to remember the order of the twelve Chinese zodiac animals and could not say it in order even before their stroke. The revision of item 6 may be required in the future version of the C-SOMC.

Clinical implication

The SOMC test is a valuable tool for assessing cognitive function, particularly in stroke populations, which comprising only 6 items (49). All items are easy to understand and administer, minimizing the risk of misunderstandings (50, 51). Its simplicity and brevity make it particularly suitable for stroke patients, who often face challenges with more complex cognitive assessments. For instance, stroke-related impairments such as hemiplegic paralysis, visual impairment, agraphia, and dyslexia can hinder the completion of common cognitive tests like the MMSE and MoCA, which require reading and writing tasks (27, 28, 52). The SOMC, however, can be completed verbally at the patient's bedside, making it accessible even in the acute phase of stroke.

Clinically, the SOMC has demonstrated significant predictive validity. Lucke et al. (53) reported that the SOMC was an independent predictor of adverse outcomes, including hospital length of stay, new institutionalization, and in-hospital mortality. Madison et al. (54) further highlighted the SOMC's importance by noting that its scores were the only acute clinical data significantly predicting communication, memory, and thinking abilities. This underscores the SOMC's potential as a prioritized screening measure for assessing acute neurologic status and cognitive dysfunction, guiding post-stroke rehabilitation recommendations.

This study has some limitations. First, this study only assessed the cognitive function level among the subjects with stroke, without evaluating other related factors such as stroke severity or emotional status. To achieve a comprehensive understanding of the factors affecting cognitive function, future cross-sectional studies should incorporate additional assessments, such as the National Institutes of Health Stroke Scale and Stroke Impact Scale, which cover a broader range of domains. Second, the study conducted three repeated measures of C-SOMC within 2 days potentially introducing a learning effect that should be considered when interpreting the results. Third, this study did not categorize the participants according to the cognitive impairment or chronicity of their stroke conditions. As a result, the findings and conclusions drawn from this study may only be relevant and applicable to individuals who possess a comparable level of cognitive function to those included in the study. Future research should consider incorporating a more diverse range of subjects with varying degrees of stroke severity and chronicity to enhance the generalizability and applicability of the results. At last, further research should explore the comprehensive psychometric characteristics of the instrument in patients with different degrees of cognitive impairment.

Conclusion

In conclusion, our preliminary findings suggest that the C-SOMC has excellent inter-rater and intra-rater reliability when administered to people who have had a stroke. Thus, the C-SOMC is a valuable tool for measuring cognitive impairment in people who have had a stroke.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving humans were approved by Medical Ethical committee of the First Affiliated Hospital of Sun Yat-sen University. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

J-LZ: Methodology, Formal analysis, Writing – original draft, Data curation, Writing – review & editing, Funding acquisition. P-MC: Formal analysis, Writing – review & editing, Methodology. TZ: Data curation, Writing – review & editing. HX: Formal analysis, Writing – review & editing. Y-SZ: Writing – review & editing. SN: Writing – review & editing. Y-RM: Writing – review & editing. D-FH: Investigation, Writing – review & editing, Funding acquisition, Supervision, Conceptualization, Methodology.

Funding

The author(s) declare that financial support was received for the research and/or publication of this article. This work was supported by the National Natural Science Foundation of China (No.82102649), the 5010 Planning Project of Sun Yat-Sen University of China (No.2014001).

Acknowledgements

We would like to acknowledge the individuals for their participating in the study. We also express our gratitude to our expert group who translated the original SOMC from English into Chinese and all the colleagues of the Department of Rehabilitation Medicine of the First Affiliated Hospital, Sun Yat-sen University, China.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Generative AI statement

The author(s) declare that no Generative AI was used in the creation of this manuscript.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fresc.2025.1614305/full#supplementary-material

References

1. Owolabi MO, Thrift AG, Mahal A, Ishida M, Martins S, Johnson WD, et al. Primary stroke prevention worldwide: translating evidence into action. Lancet Public Health. (2022) 7:e74–85. doi: 10.1016/S2468-2667(21)00230-9

PubMed Abstract | Crossref Full Text | Google Scholar

2. Sun JH, Tan L, Yu JT. Post-stroke cognitive impairment: epidemiology, mechanisms and management. Ann Transl Med. (2014) 2:80. doi: 10.3978/j.issn.2305-5839.2014.08.05

PubMed Abstract | Crossref Full Text | Google Scholar

3. Wang W, Jiang B, Sun H, Ru X, Sun D, Wang L, et al. Prevalence, incidence, and mortality of stroke in China: results from a nationwide population-based survey of 480 687 adults. Circulation. (2017) 135:759–71. doi: 10.1161/CIRCULATIONAHA.116.025250

PubMed Abstract | Crossref Full Text | Google Scholar

4. Wu S, Wu B, Liu M, Chen Z, Wang W, Anderson CS, et al. Stroke in China: advances and challenges in epidemiology, prevention, and management. Lancet Neurol. (2019) 18:394–405. doi: 10.1016/S1474-4422(18)30500-3

PubMed Abstract | Crossref Full Text | Google Scholar

5. Tu WJ, Hua Y, Yan F, Bian H, Yang Y, Lou M, et al. Prevalence of stroke in China, 2013–2019: a population-based study. Lancet Reg Health West Pac. (2022) 28:100550. doi: 10.1016/j.lanwpc.2022.100550

PubMed Abstract | Crossref Full Text | Google Scholar

6. Crichton SL, Bray BD, McKevitt C, Rudd AG, Wolfe CD. Patient outcomes up to 15 years after stroke: survival, disability, quality of life, cognition and mental health. J Neurol Neurosurg Psychiatry. (2016) 87:1091–8. doi: 10.1136/jnnp-2016-313361

PubMed Abstract | Crossref Full Text | Google Scholar

7. Ye M, Zheng Y, Xiong Z, Ye B, Zheng G. Baduanjin exercise ameliorates motor function in patients with post-stroke cognitive impairment: a randomized controlled trial. Complement Ther Clin Pract. (2022) 46:101506. doi: 10.1016/j.ctcp.2021.101506

PubMed Abstract | Crossref Full Text | Google Scholar

8. Zhu Y, Zhao Y, Lu Y, Fang C, Zhang Q, Zhang J, et al. The association between plasma soluble triggering receptor expressed on myeloid cells 2 and cognitive impairment after acute ischemic stroke. J Affect Disord. (2022) 299:287–93. doi: 10.1016/j.jad.2021.12.011

PubMed Abstract | Crossref Full Text | Google Scholar

9. Yuan M, Guo YS, Han Y, Gao ZK, Shen XY, Bi X. Effectiveness and mechanisms of enriched environment in post-stroke cognitive impairment. Behav Brain Res. (2021) 410:113357. doi: 10.1016/j.bbr.2021.113357

PubMed Abstract | Crossref Full Text | Google Scholar

10. Kwon HS, Lee D, Lee MH, Yu S, Lim JS, Yu KH, et al. Post-stroke cognitive impairment as an independent predictor of ischemic stroke recurrence: picasso sub-study. J Neurol. (2020) 267:688–93. doi: 10.1007/s00415-019-09630-4

PubMed Abstract | Crossref Full Text | Google Scholar

11. Borson S, Scanlan JM, Chen P, Ganguli M. The mini-cog as a screen for dementia: validation in a population-based sample. J Am Geriatr Soc. (2003) 51:1451–4. doi: 10.1046/j.1532-5415.2003.51465.x

PubMed Abstract | Crossref Full Text | Google Scholar

12. Rabadi MH, Rabadi FM, Edelstein L, Peterson M. Cognitively impaired stroke patients do benefit from admission to an acute rehabilitation unit. Arch Phys Med Rehabil. (2008) 89:441–8. doi: 10.1016/j.apmr.2007.11.014

PubMed Abstract | Crossref Full Text | Google Scholar

13. Toglia J, Fitzgerald KA, O’Dell MW, Mastrogiovanni AR, Lin CD. The mini-mental state examination and Montreal cognitive assessment in persons with mild subacute stroke: relationship to functional outcome. Arch Phys Med Rehabil. (2011) 92:792–8. doi: 10.1016/j.apmr.2010.12.034

PubMed Abstract | Crossref Full Text | Google Scholar

14. Ball LJ, Bisher GB, Birge SJ. A simple test of central processing speed: an extension of the short blessed test. J Am Geriatr Soc. (1999) 47:1359–63. doi: 10.1111/j.1532-5415.1999.tb07440.x

PubMed Abstract | Crossref Full Text | Google Scholar

15. Katzman R, Brown T, Fuld P, Peck A, Schechter R, Schimmel H. Validation of a short orientation-memory-concentration test of cognitive impairment. Am J Psychiatry. (1983) 140:734–9. doi: 10.1176/ajp.140.6.734

PubMed Abstract | Crossref Full Text | Google Scholar

16. Blessed G, Tomlinson BE, Roth M. The association between quantitative measures of dementia and of senile change in the cerebral grey matter of elderly subjects. Br J Psychiatry. (1968) 114:797–811. doi: 10.1192/bjp.114.512.797

PubMed Abstract | Crossref Full Text | Google Scholar

17. Wade DT, Vergis E. The short orientation-memory-concentration test: a study of its reliability and validity. Clin Rehabil. (1999) 13:164–70. doi: 10.1191/026921599673848768

PubMed Abstract | Crossref Full Text | Google Scholar

18. Dellasega CA, Lacko L, Singer H, Salerno F. Telephone screening of older adults using the orientation-memory-concentration test. Geriatr Nurs. (2001) 22:253–7. doi: 10.1067/mgn.2001.119472

PubMed Abstract | Crossref Full Text | Google Scholar

19. Zhao JL, Chen PM, Ng S, Mao YR, Huang DF. Translation and concurrent validity, sensitivity and specificity of Chinese version of short orientation memory concentration test in people with a first cerebral infarction. Front Hum Neurosci. (2023) 17:977078. doi: 10.3389/fnhum.2023.977078

PubMed Abstract | Crossref Full Text | Google Scholar

20. Fillenbaum GG, Landerman LR, Simonsick EM. Equivalence of two screens of cognitive functioning: the short portable mental status questionnaire and the orientation-memory-concentration test. J Am Geriatr Soc. (1998) 46:1512–8. doi: 10.1111/j.1532-5415.1998.tb01535.x

PubMed Abstract | Crossref Full Text | Google Scholar

21. Cohen-Mansfield J, Perach R. Sleep duration, nap habits, and mortality in older persons. Sleep. (2012) 35:1003–9. doi: 10.5665/sleep.1970

PubMed Abstract | Crossref Full Text | Google Scholar

22. Mavioglu H, Gedizlioglu M, Akyel S, Aslaner T, Eser E. The validity and reliability of the turkish version of Alzheimer’s disease assessment scale-cognitive subscale (adas-cog) in patients with mild and moderate Alzheimer’s disease and normal subjects. Int J Geriatr Psychiatry. (2006) 21:259–65. doi: 10.1002/gps.1457

PubMed Abstract | Crossref Full Text | Google Scholar

23. Bech RD, Lauritsen J, Ovesen O, Overgaard S. The verbal rating scale is reliable for assessment of postoperative pain in hip fracture patients. Pain Res Treat. (2015) 2015:676212. doi: 10.1155/2015/676212

PubMed Abstract | Crossref Full Text | Google Scholar

24. Vlaar AM, Wade DT. Verbal fluency assessment of patients with multiple sclerosis: test-retest and inter-observer reliability. Clin Rehabil. (2003) 17:756–64. doi: 10.1191/0269215503cr674oa

PubMed Abstract | Crossref Full Text | Google Scholar

25. Werner JG, Castellon-Larios K, Thongrong C, Knudsen BE, Lowery DS, Antor MA, et al. Desflurane allows for a faster emergence when compared to sevoflurane without affecting the baseline cognitive recovery time. Front Med (Lausanne). (2015) 2:75. doi: 10.3389/fmed.2015.00075

PubMed Abstract | Crossref Full Text | Google Scholar

26. Jayani RV, Magnuson AM, Sun CL, Ma H, Tew WP, Mohile SG, et al. Association between a cognitive screening test and severe chemotherapy toxicity in older adults with cancer. J Geriatr Oncol. (2020) 11:284–9. doi: 10.1016/j.jgo.2019.10.004

PubMed Abstract | Crossref Full Text | Google Scholar

27. Goring H, Baldwin R, Marriott A, Pratt H, Roberts C. Validation of short screening tests for depression and cognitive impairment in older medically ill inpatients. Int J Geriatr Psychiatry. (2004) 19:465–71. doi: 10.1002/gps.1115

PubMed Abstract | Crossref Full Text | Google Scholar

28. Tuijl JP, Scholte EM, de Craen AJ, van der Mast RC. Screening for cognitive impairment in older general hospital patients: comparison of the six-item cognitive impairment test with the mini-mental state examination. Int J Geriatr Psychiatry. (2012) 27:755–62. doi: 10.1002/gps.2776

PubMed Abstract | Crossref Full Text | Google Scholar

29. Hessler J, Bronner M, Etgen T, Ander KH, Forstl H, Poppert H, et al. Suitability of the 6cit as a screening test for dementia in primary care patients. Aging Ment Health. (2014) 18:515–20. doi: 10.1080/13607863.2013.856864

PubMed Abstract | Crossref Full Text | Google Scholar

30. Tieges Z, Stiobhairt A, Scott K, Suchorab K, Weir A, Parks S, et al. Development of a smartphone application for the objective detection of attentional deficits in delirium. Int Psychogeriatr. (2015) 27:1251–62. doi: 10.1017/S1041610215000186

PubMed Abstract | Crossref Full Text | Google Scholar

31. VT L-D. Neuropsychological screening tests in African Americans. J Natl Med Assoc. (2001) 93:323–8.11560287

PubMed Abstract | Google Scholar

32. Hsieh CL, Hsueh IP, Chiang FM, Lin PH. Inter-rater reliability and validity of the action research arm test in stroke patients. Age Ageing. (1998) 27:107–13. doi: 10.1093/ageing/27.2.107

PubMed Abstract | Crossref Full Text | Google Scholar

33. Walter SD, Eliasziw M, Donner A. Sample size and optimal designs for reliability studies. Stat Med. (1998) 17:101–10. doi: 10.1002/(sici)1097-0258(19980115)17:1%3C101::aid-sim727%3E3.0.co;2-e

PubMed Abstract | Crossref Full Text | Google Scholar

34. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. (1979) 86:420–8. doi: 10.1037/0033-2909.86.2.420

PubMed Abstract | Crossref Full Text | Google Scholar

35. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. (2016) 15:155–63. doi: 10.1016/j.jcm.2016.02.012

PubMed Abstract | Crossref Full Text | Google Scholar

36. Haley SM, Fragala-Pinkham MA. Interpreting change scores of tests and measures used in physical therapy. Phys Ther. (2006) 86:735–43. doi: 10.1093/ptj/86.5.735

PubMed Abstract | Crossref Full Text | Google Scholar

37. Hoonhorst MH, Kollen BJ, van den Berg PS, Emmelot CH, Kwakkel G. How reproducible are transcranial magnetic stimulation-induced meps in subacute stroke? J Clin Neurophysiol. (2014) 31:556–62. doi: 10.1097/WNP.0000000000000114

PubMed Abstract | Crossref Full Text | Google Scholar

38. Schambra HM, Ogden RT, Martinez-Hernandez IE, Lin X, Chang YB, Rahman A, et al. The reliability of repeated tms measures in older adults and in patients with subacute and chronic stroke. Front Cell Neurosci. (2015) 9:335. doi: 10.3389/fncel.2015.00335

PubMed Abstract | Crossref Full Text | Google Scholar

39. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. (1986) 1:307–10. doi: 10.1016/S0140-6736(86)90837-8

PubMed Abstract | Crossref Full Text | Google Scholar

40. Cleveland WS. Visualizing Data. Chicago, IL: Hobart Press (1993). p. 360.

Google Scholar

41. Barbic D, Kim B, Salehmohamed Q, Kemplin K, Carpenter CR, Barbic SP. Diagnostic accuracy of the Ottawa 3dy and short blessed test to detect cognitive dysfunction in geriatric patients presenting to the emergency department. Bmj Open. (2018) 8:e19652. doi: 10.1136/bmjopen-2017-019652

Crossref Full Text | Google Scholar

42. Salthouse TA. When does age-related cognitive decline begin? Neurobiol Aging. (2009) 30:507–14. doi: 10.1016/j.neurobiolaging.2008.09.023

PubMed Abstract | Crossref Full Text | Google Scholar

43. MacDonald SW, Hultsch DF, Dixon RA. Performance variability is related to change in cognition: evidence from the Victoria longitudinal study. Psychol Aging. (2003) 18:510–23. doi: 10.1037/0882-7974.18.3.510

PubMed Abstract | Crossref Full Text | Google Scholar

44. Stern Y. Cognitive reserve in ageing and Alzheimer’s disease. Lancet Neurol. (2012) 11:1006–12. doi: 10.1016/S1474-4422(12)70191-6

PubMed Abstract | Crossref Full Text | Google Scholar

45. Rabbitt P, Diggle P, Holland F, McInnes L. Practice and drop-out effects during a 17-year longitudinal study of cognitive aging. J Gerontol B Psychol Sci Soc Sci. (2004) 59:P84–97. doi: 10.1093/geronb/59.2.p84

PubMed Abstract | Crossref Full Text | Google Scholar

46. Hartigan I, O'Mahony D. The barthel index: comparing inter-rater reliability between nurses and doctors in an older adult rehabilitation unit. Appl Nurs Res. (2011) 24:e1–7. doi: 10.1016/j.apnr.2009.11.002

PubMed Abstract | Crossref Full Text | Google Scholar

47. Davous P, Lamour Y, Debrand E, Rondot P. A comparative evaluation of the short orientation memory concentration test of cognitive impairment. J Neurol Neurosurg Psychiatry. (1987) 50:1312–7. doi: 10.1136/jnnp.50.10.1312

PubMed Abstract | Crossref Full Text | Google Scholar

48. Beckerman H, Roebroeck ME, Lankhorst GJ, Becher JG, Bezemer PD, Verbeek AL. Smallest real difference, a link between reproducibility and responsiveness. Qual Life Res. (2001) 10:571–8. doi: 10.1023/a:1013138911638

PubMed Abstract | Crossref Full Text | Google Scholar

49. Brooke P, Bullock R. Validation of a 6 item cognitive impairment test with a view to primary care usage. Int J Geriatr Psychiatry. (1999) 14:936–40. doi: 10.1002/(SICI)1099-1166(199911)14:11%3C936::AID-GPS39%3E3.0.CO;2-1

PubMed Abstract | Crossref Full Text | Google Scholar

50. Yokomizo JE, Simon SS, Bottino CM. Cognitive screening for dementia in primary care: a systematic review. Int Psychogeriatr. (2014) 26:1783–804. doi: 10.1017/S1041610214001082

PubMed Abstract | Crossref Full Text | Google Scholar

51. Upadhyaya AK, Rajagopal M, Gale TM. The six item cognitive impairment test (6-cit) as a screening test for dementia: comparison with mini-mental state examination (mmse). Curr Aging Sci. (2010) 3:138–42. doi: 10.2174/1874609811003020138

PubMed Abstract | Crossref Full Text | Google Scholar

52. Abdel-Aziz K, Larner AJ. Six-item cognitive impairment test (6cit): pragmatic diagnostic accuracy study for dementia and mci. Int Psychogeriatr. (2015) 27:991–7. doi: 10.1017/S1041610214002932

PubMed Abstract | Crossref Full Text | Google Scholar

53. Lucke JA, van der Mast RC, de Gelder J, Heim N, de Groot B, Mooijaart SP, et al. The six-item cognitive impairment test is associated with adverse outcomes in acutely hospitalized older patients: a prospective cohort study. Dement Geriatr Cogn Dis Extra. (2018) 8:259–67. doi: 10.1159/000490240

PubMed Abstract | Crossref Full Text | Google Scholar

54. Bertolin M, Van Patten R, Greif T, Fucetola R. Predicting cognitive functioning, activities of daily living, and participation 6 months after mild to moderate stroke. Arch Clin Neuropsychol. (2018) 33:562–76. doi: 10.1093/arclin/acx096

PubMed Abstract | Crossref Full Text | Google Scholar