Estimating the Minimal Number of Repeated Examinations for Random Responsiveness With the Coma Recovery Scale—Revised as an Example

Objective: The aim of this study was to develop a general method to estimate the minimal number of repeated examinations needed to detect patients with random responsiveness, given a limited rate of missed diagnosis. Methods: Basic statistical theory was applied to develop the method. As an application, 100 patients with disorders of consciousness (DOC) were assessed with the Coma Recovery Scale–Revised (CRS-R). DOC patients were supposed to be examined for 13 times over 20 days, while anyone who was diagnosed as a minimally conscious state (MCS) in a round would no longer be examined in the subsequent rounds. To test the validation of this method, a series of the stochastic simulation was completed by computer software under all the conditions of possible combinations of three kinds of distributions for p, five values of p, and four sizes of the sample and repeated for 100 times. Results: A series of formula was developed to estimate the probability of a positive response to a single examination given by a patient and the minimal number of successive examinations needed based on the numbers of patients detected in the first i (i =1, 2,.) rounds of repeated examinations. As applied to the DOC patients assessed with the CRS-R, with a rate of missed diagnosis < 0.0001, the estimate of the minimal number of examinations was six in traumatic brain injury patients and five in non-traumatic brain injury patients. The outcome of the simulation showed that this method performed well under various conditions possibly occurring in practice. Interpretation: The method developed in this paper holds in theory and works well in application and stochastic simulation. It could be applied to any other kind of examinations for random responsiveness, not limited to CRS-R for detecting MCS; this should be validated in further research.


INTRODUCTION
In clinical practice, bedside evaluation of patients has always been regarded as the "gold standard" (Giacino et al., 2009). However, the fluctuation of behavioral responses of patients is very common, especially in the minimally conscious state (MCS) (Giacino et al., 2002). Patients with MCS are defined as those who exhibit inconsistent but repeatable purposeful behaviors (Giacino et al., 2002). That is to say, the fluctuations in the responsiveness of MCS patients are inherent (Majerus et al., 2005), and this is due to the patient's motor or language impairments and vigilance fluctuations (Seel et al., 2010). Both unresponsive wakefulness syndrome/vegetative state (UWS/VS) (Laureys et al., 2010) and MCS are referred to as "disorders of consciousness" (DOC), but clinicians often make mistakes when differentiating between these patients. Many studies have shown that the fluctuations of MCS often leads to missed diagnosis (the missed diagnosis rate may be higher than 35%) (Stender et al., 2014, van Erp et al., 2015, and that such diagnostic errors can affect patient outcomes and treatment decisions (Demertzi et al., 2011). Especially when DOC patients were tested only once, even when using the most sensitive scale of the Coma Recovery Scale-Revised (CRS-R) (Giacino et al., 2004), a single assessment resulted in a 25% missed diagnosis (Wang et al., 2020). Therefore, recent guidelines have emphasized the importance of repeated assessments to increase the accuracy of clinical diagnosis (Giacino et al., 2018;Kondziella et al., 2020). Wannez et al. assessed 123 chronic DOC patients using the CRS-R at least six times over a 10-day period, and their results suggest to perform at least five assessments in each patient with DOC within a short time interval. But this conclusion was statistically analyzed based on the best diagnosis of the six tests (Wannez et al., 2017). There was still a 10% missed diagnosis at the fifth assessment when compared with the best diagnosis from the seven tests in the subgroup (Wannez et al., 2017). So, it is questionable whether five assessments are sufficient. In fact, the number of repeated examinations needed depends on the probability of MCS patients giving a positive response at a single examination.
The aim of this study is to develop a general method to estimate the minimal number of repeated examinations needed to detect patients with random responsiveness, given a limited rate of missed diagnosis and to validate the method by computer stochastic simulation.
To avoid abstract mathematical derivation, we will use the language of detecting MCS from DOC to develop and explain the method. Considering that there are two types of DOC patients (including MCS and UWS/VS), we assume that there are n independent patients with MCS and N − n independent patients with UWS/VS in a group of N patients with DOC; and the probability of an MCS patient giving a positive response to a single CRS-R assessment is p in theory. We will develop a series of statistical formulas for the estimation of n and p, and, hence, the minimal number of successive examinations needed k min with a rate of missed diagnosis less than a given α, say α = 0.0001.

Bedside Examinations
The DOC Patients They were part of the patients treated in the rehabilitation department of Hangzhou Wujing Hospital. The inclusion criteria for this project were as follows: (1) over 18 years old, (2) injured for at least 28 days, and (3) no drugs affecting conscious expression (e.g., no neuromuscular blockers or sedatives within 72 h of enrollment). Comatose patients and patients who emerged from MCS were excluded.

Procedure of Examination
The standard operating procedure of the CRS-R is the same as the study of Wannez et al. (2017). The difference is the arrangement of appraisers and the frequency of appraisals. Examinations were performed independently by two appraisers who were well-trained and experienced in the use of the CRS-R, and recorded by a video. The results given by the two appraisers were compared right after each round; if they were not consistent, the final diagnosis was determined by discussion among the two appraisers and the third person incorporating with the recorded video. If the patient was first diagnosed as MCS in a round, then he or she would no longer be examined in the subsequent rounds; otherwise, the behavioral examination with the CRS-R was continued until the 13th round for a maximum of 20 days.
This study protocol was approved by the Ethics Committee of Hangzhou Normal University. Written informed consent was signed by the legal guardians of the patients enrolled in this study.

Development of Statistical Formulas
The Assumptions The assumptions of the formulas to be developed are the following: (1) The number of MCS patients n is big enough.
(2) Each of the MCS patients may independently give a positive response to a single examination with the same constant probability p at any time; and each of non-MCS patients may never give any positive response to the examination. (3) The examination is administered independently for several rounds according to unique standard criteria; if someone successively gives a negative response up to the (i -1)-th round of examination but gives a positive response at the i-th round, then the examination for this person is ended; otherwise, he or she keeps going. (4) The examination for the whole group is supposed to stop at the k min -th round with a small rate of missed diagnosis controlled by a given α . The key target is to determine the size of k min based on the observed numbers of new cases, giving positive response in the successive rounds, a 1 , a 2 , · · · , a k min , and the bi-product is the estimate of the probability p.

Data and Formulas
In Table 1, the second and third columns show the number of new cases of MCS, giving positive responses and the total number of MCS patients really assessed in each round of examinations. If the total number of MCS patients is big enough, based on the data of each round, any proportion in the fourth column can be an estimate of the probability of positive response to a single a 1 + a 2 + · · · + a i n + (n − a 1 ) + · · · + (n − a 1 − a 2 − · · · − a i−1 ) MCS, minimally conscious state. examination given by an MCS patient; and, in theory, based on the data of a series of examinations from the first up to the i-th round, the pooled estimate in the fifth column is better than the estimate based on the data of the i-th round only, i = 1, 2. . .
Since both of a 1 /n and any of the formulas in the 5-th column of Table 1 approximate to the same probability of positive response to a single examination given by a MCS patient, we have and a 1 n ≈ a 1 + · · · + a i n + (n − a 1 ) + · · · (n − a 1 − · · · − a i−1 ) Denote their solutions of n, respectively, bŷ These formulas have been summarized in Table 2.
The possible minimal number of rounds k min can be estimated right after any i-th round by finding a minimalk i,min such that

Validation by Stochastic Simulation
The "DOC Patients" Set up N binary variables Y j , j = 1, 2, · · · , n, n + 1, · · · N representing N "DOC patients, " Y j = 1, j = 1, 2, · · · , n representing n "MCS patients, " and Y j ≡ 0, j = n + 1, · · · N representing N-n "non-MCS patients." The Inherent Ability on Response The probabilities of j-th patient giving a positive response to a single examination p j , j = 1, 2, · · · , n representing his or her inherent ability on response: (1) Constant p j ≡ p to show that this method works well for a large sample (say n = 500, 1,000) and applicable to a small sample (say n = 100 or 50); (2) p j following a normal distribution N(p, σ 2 ), to show that this method still works well when there is some normality-like variation among patients (say σ = 0.1, 0.2, 0.3); the values of p j , j = 1, 2, · · · , n in each time of simulation were generated by statistical software; (3) p j following a uniform distribution U(p − δ, p + δ), to show that this method still works well when there is uniform-like variation among patients (say δ = 0.1, 0.2, 0.3); the values of p j , j = 1, 2, · · · , n in each time of simulation were generated by statistical software.
For non-MCS patients, let p j ≡ 0, j = n + 1, · · · N to ensure that they will never give any positive response to the "examination." "Examination" and "Responses" During i-th round of "examination, " generate a random number r ij in the interval of [0, 1] for the j-th patient; if p j ≥ r ij , letŶ ij = 1, indicating a "positive response" given by the j-th patient in the i-th round; otherwise,Ŷ ij = 0. After i-th round of "examination, " define a i = jŶ ij as the total number of MCS patients diagnosed in this round. From each time of simulation, we can obtain an estimated probabilityp, the minimal number of examinationsk min , the total numbers of diagnosis,n = k min i=1 a i , the rate of missed diagnosis (n −n)/n.

Repeated Simulation and the Rate of Missed Diagnosis
In order to see the robustness of this method, we have repeated the simulation for 100 times and calculated the means ofp,k min , (n −n)/n and corresponding 95%-confidence intervals.

RESULTS
To easily understand the procedure of this method, we would like to show the results on real data first, and then those on the simulation.

Outcome of Bedside Examinations
A total of 100 DOC patients, including 50 traumatic brain injury (TBI) patients and 50 non-traumatic brain injury (NTBI) patients were recruited and assessed with the CRS-R. The TBI patients consisted of 40 males and 10 females aged 18-77 years (standard deviation = 13.0); the time after the injury onset was from 1 to 24 months (SD = 4.4). The NTBI patients consisted of 34 males and 16 females aged 26-83 years (standard deviation = 14.9); the time after the injury onset was from 1 to 27.8 months (standard deviation = 4.8). There was no significant difference in age (the detailed information on patients is shown in Supplementary Material A). The above-developed method had been applied to the observed data with α= 0.0001 being assigned. Table 3 shows the diagnostic results of all the DOC patients after each examination.

For TBI Patients
After completing the first 2 rounds of examinations we obtained the numbers of MCSs giving positive response a 1 = 30 and a 2 = 3, using the formulas in the second row of (1 −p 2 ) 5 = 0.00001 < 0.0001 the examination should be kept on going, and might be ended at the 5-th round; and the total number of MCS patients in this group of DOCs might be around 34. After completing the 3rd round, we obtained a 3 = 3, and (1 −p 3 ) 5 = 0.00006 < 0.0001 the examination should be kept on going, and might be ended at the 5-th round; and the total number of MCS patients in this group of DOCs might be around 35.
After completing the 4-th round, we obtained a 4 = 2, and (1 −p 4 ) 6 = 0.00003 < 0.0001 the examination should be kept on going, and might be ended at the 6-th round, and the total number of MCS patients in this group of DOCs might be around 37.

Since
(1 −p 6 ) 6 = 0.00005 < 0.0001, the examination could be ended at this roundk min = 6. And up to this round, the total number of MCS patients had been detected in the group of DOCs waŝ n = a 1 + a 2 + a 3 + a 4 + a 5 + a 6 = 38 Following the guidance of our developed method, based on six rounds of examination, we could conclude with a missed diagnosis rate < 0.01% that there were 38 patients with MCS in total out of N = 50 DOCs in this group of patients with TBI and the probability of the patients with MCS giving positive response in a single examination was about 81%. This conclusion was supported by the real data collected from 13 rounds of successive examinations (Table 3), where the number of detected MCSs was a 1 = 30, a 2 = 3, a 3 = 3, a 4 = 2, a 5 =. . . = a 13 = 0, respectively.

For NTBI Patients
After completing the first two rounds of examination we obtained the numbers of MCSs giving positive response a 1 = 29 and a 2 = 3, using the formulas in the second row of (1 −p 2 ) 5 = 0.000012 < 0.0001 the examination should be kept on going, and might be ended at the 5-th round; and the total number of MCS patients in this group of DOCs might be 33. After completing the 3rd round, we obtained a 3 = 2, and (1 −p 3 ) 5 = 0.000039 < 0.0001 the examination should be kept on going, and might be ended at the 5-th round; and the total number of MCS patients in this group of DOCs might be 34. After completing the 4-th round, we obtained a 4 = 1, and   the examination could be ended at this roundk min = 5 and up to this round, the total number of MCS patients had been detected in the group of DOCs waŝ n = a 1 + a 2 + a 3 + a 4 + a 5 = 35 Following the guidance of our developed method, based on the five rounds of examination, we could conclude with a missed diagnosis rate < 0.01% that, out of N = 50 DOCs in this group of patients with non-TBI, there were 35 patients with MCS in total; and the probability of the patients with MCS giving positive response in a single examination was about 85%. This conclusion was also supported by the real data collected from the 13 rounds of successive examinations (Table 3), where the number of detected MCSs ws a 1 = 29, a 2 = 3, a 3 = 2, a 4 = 1, a 5 =. . . = a 13 = 0, respectively.

Outcomes of Simulation
The process of a single time of simulation included three conditions of probability p mentioned above; under each condition, p = 0.5, 0.6, 0.7, 0.8, 0.9 were simulated with sample size n = 50, 100, 500, 1, 000 respectively. The detailed outcome of the simulation has been packaged in a file of Supplementary Material B, which will be available upon request.
The key findings were summarized in Tables 4-6.
Estimation on p, k min and Missed Diagnosis Rate Table 4 gives the estimated probabilitiesp. One can see that the maximal error p −p is 0.003 for a bigger sample size (500, 1,000), and 0.015 for a smaller sample size (50, 100), which are not significant in practice. Table 5 gives the estimated minimal number of repeated examinations neededk min . One can see that the maximal error k min −k min is 0.5 for a bigger sample size (500, 1,000) and 0.7 for a smaller sample size (50, 100), which reminds us thatk min is a conservative estimate, and usingk min as the estimate might be more safe in practice. Table 6 gives the average rate of missed diagnosis. One can see that, since we controlled the rate of missed diagnosis by α = 0.0001, the real rates of missed diagnosis were mostly <0.0001 for bigger sample (500, 1,000), but, occasionally, it might be 0.0002 for a bigger sample and even 0.00640 for p = 0.5 and n = 50.

The Robustness of the Estimation
In Tables 4-6, through the comparison among p i ≡ p,p i˜N (p, 0.3 2 ) and p i˜U (p − 0.3, p + 0.3) we can see that the estimated p, k min and the missed diagnosis rate are fairly stable; the absolute errors p −p and k min −k min are not affected by whether p j being constant or varied as we simulated and are not affected by whether p j being normality-like varied or uniform-like varied as we simulated.

DISCUSSION
There is a consensus that the fluctuations in responsiveness are inherent to MCS patients (Giacino et al., 2002(Giacino et al., , 2009Majerus et al., 2005), so people should repeat the examination to obtain a reliable diagnosis while using the CRS-R. We assume that there is a probability p for a patient with MCS, giving a positive response to a single examination. In theory, the minimal number k min of repeated examinations needed for a homogeneous MCS population depends on the size of p; if p is bigger, k min is less, and, otherwise, k min should be bigger. We have developed a series of statistical formulas to estimate the probability p, the total number n of MCSs, and the necessary minimal number of k min of repeated examinations based on the numbers of new cases detected in successive rounds of examinations for a group of patients in DOC. These formulas had been applied to our clinical practice  along with the following procedure: After completing the i-th round of examinations (I = 2, 3. . . ), based on the numbers of new cases of MCS detected up to the i-th round ( a 1, a 2 ,. . . , a i ), to calculaten i andp i according to the formulas in the i-th row of Table 1 and given a very small probability α, if (1 −p i ) i ≥ α, then the examination should be continued; otherwise, ended. Meanwhile, to estimate the minimal number needed for repeated examinations at the i-th round by finding a minimal k i,min such that (1 −p i )k i,min < α , as a result given, α= 0.0001, 38 MCSs out of a group of 50 TBI patients were detected through the first six rounds of successive examinations without any more patients giving positive response in the following seven rounds, while 35 MCSs out of a group of 50 NTBI patients were detected in the first five rounds without any more positive response in the following eight rounds.
Based on our data, in case the patient's etiology was NTBI, the probability for a patient with MCS giving a positive response was about 85%, and the corresponding number of repeated rounds needed was 5. This number coincides with the number suggested in the article of Wannez et al. (2017). However, for TBI patients, the probability of the patients with MCS giving positive response in a single examination was about 81%, and the corresponding number of repeated rounds needed was 6. This coproduct might be important. As we all know, patients with different kinds of etiology should be managed in different ways. For example, the ethical and legal issues concerning treatment withdrawal can be discussed when the NTBI patients were injured after 3 months without recovery, while the TBI patients were injured after 12 months without recovery (Sazbon and Groswasser, 1991). This result also reminds us that the patients with TBI have a higher degree of behavioral instability, suggesting that we should carefully monitor the behavioral performance of such patients in clinical practice.
More than the application for the real data of bedside examination, this method had been validated by a series of stochastic simulations. Each simulation explored all the conditions of the combination of three kinds of distributions for p, five values of p, and four sizes of the sample and repeated for 100 times. The outcome of the simulation was fairly enlightening; the estimation on p was satisfied with very small absolute errors even when the sample size is small as n = 50; the estimation onk min was fine for a bigger sample, and using k min + 1 as the estimate might be safer for the small sample. Although we controlled for the rate of missed diagnosis by α = 0.0001, the real rate of missed diagnosis was not necessarily <0.0001, occasionally might be 0.0002 and even 0.00640 for p = 0.5 and n = 50. Therefore, in practice, one can still assign the limit as α = 0.0001 but should not clearly claim the rate of missed diagnosis being <0.0001, especially for a small sample size as n = 50. For safety, one may assign α = 0.00001 to control for the rate of missed diagnosis in case of the sample size being smaller. The simulation brought good news to us, indeed, that the performance of the estimations was not significantly different between the three kinds of distributions for p. It means that, as long as the group of patients comes from an identical population in the sense of statistics, reasonable variation is allowed. In practice, if one recruits a group of patients carefully, following a set of eligibility criteria and exclusion criteria, even though the probabilities of the patients were not being exactly the same, this method still works well, especially for a bigger sample.
In theory, the method developed in this paper could be applied to any other kind of repeated examinations for random responsiveness, not limited to CRS-R for detecting MCS. The fluctuation of the responses of DOC patients to stimuli is closely related to their inherent characteristics. This is not only shown in behavioral assessment but also in other detection and assessment methods (such as functional magnetic resonance imaging assessment technique) (Wang et al., 2015). As Wang et al. (2015) found, there was only a 52% overlap in brain region activation between the two-time functional magnetic resonance imaging tests in DOC patients. Therefore, this method might also be applicable to neuroimaging evaluation. Beyond these, we wish that this method can be further applied to capture some rare event, which is randomly occurring in nature or being detected in a single test. This paper has some limitations. For the convenience of the readers in the medical field, the abstract mathematical aspect of this method has been avoided, which is intended to be published elsewhere in a professional statistical journal. Since the probabilities of giving a positive response in the problems on MCS patients are possibly in the range of (0.6, 0.9), the stochastic simulation did not cover the range of p < 0.5. The application on the analysis of two groups of data is just to demonstrate the procedure; further studies with application to patients with different etiology backgrounds need to be explored.

DATA AVAILABILITY STATEMENT
The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author/s.

ETHICS STATEMENT
The studies involving human participants were reviewed and approved by the Ethics Committee of Hangzhou Normal University. The patients/participants provided their written informed consent to participate in this study. Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.