This study was designed according to the reported Delphi method with modification (17), a method for achieving consensus among a panel of experts. In total, 3 rounds of the Delphi survey were conducted during January 2018 and January 2019 via letter. The study process is shown in Figure 1.

This study was approved by the Shanghai Changhai Hospital Ethnics Committee (No: 2018-048).

A purposive sample of Chinese experts engaged in military medicine, sports and exercise medicine, rehabilitation medicine, kinesiology and exercise science, and medical laboratory science was selected for this Delphi study. The eligible experts had experience in their related profession more than 10 years, had a title of deputy equal to or higher than associate professor/associate chief physician/chief coach/associate researcher, and were available to complete all 3 rounds of surveys in Chinese before the required deadline.

All the experts included in this study were willing to participate and signed a written informed consent.

The related research of Delphi's expert consultation method (42) suggested that the preferable number of experts is 15–30: too few participants will limit the representation, and too many participants will result in low response and agreement rates and add to the complexity and cost. Therefore, the number of experts in this study is limited to this range.

According to the modified Delphi method (43), the consistency of participant opinions in this study was measured by the coefficient of concordance (Kendall's W) and coefficient of variation (CV). Kendall's W Test was conducted to calculate the value of Kendall's W, which is a number between 0 and 1, with higher values indicating better consistency of agreement (43). The CV of each aspect for each indicator was calculated as a standard deviation/mean of scores, with lower values indicating better consistency of agreement. “Consensus in” was achieved when Kendall's W of a round was between 0.2 and 0.5 and the CV of each aspect for an indicator was < 0.5 (44). If round 3 could not achieve “consensus in,” then more rounds would be conducted iteratively according to the process of round 3 (43), i.e., a round 4 or more should be conducted until whose Kendall's W is between 0.2 and 0.5 and CV < 0.5.

The indicators included in the recommendation set, namely, the recommended indicators, met the following criteria: the mean scores of aspects 1, 2, 3 and 5 are ≥ 3.0, and that of aspect 4 is ≥ 3.5. Additionally, we classified the recommend indicators into grade I (highly recommended) or grade II (recommended). The recommended indicators included in grade I met the following criteria: (1) the mean scores of aspects 1–5 are ≥ 3.5, and (2) the CV of aspect 5 is ≤ 0.25. The recommend indicators not included in grade I were classified into grade II.

The credibility of a Delphi study, which depends on the authority of the expert panel, is important to further research based on it. Thus, according to the Delphi method (43, 44), the quality of this study was assessed by the authority coefficient (Aa), which is the arithmetic mean value of the familiarity coefficient (As) and the coefficient of judgement basis (Ai). In this study, the value of As was calculated as the sum of familiarity scores of related disciplines (military medicine, sports and exercise medicine, rehabilitation medicine, kinesiology and exercise science, and medical laboratory science). Familiarity scores were reported by participants from 1.0 to 0 corresponding to the degree of familiarity as follows: very familiar (1.0 or 0.9), familiar (0.8 or 0.7), generally familiar (0.6, 0.5, or 0.4), unfamiliar (0.3 or 0.2) or very unfamiliar (0.1 or 0). The value of Ai was calculated as the sum of the judgement basis score, which was assigned based on theoretical analysis (0.30, 0.20, and 0.10), practical experience (0.50, 0.40, and 0.30), domestic and international references (0.10, 0.08, and 0.05) and intuitive judgement (0.10, 0.07, and 0.05) sequentially for large, medium, and small levels. The authority coefficient is positively correlated with the credibility. According to previous study, Aa > 0.7 indicates good credibility (43).

The levels of agreement between participants in the first, second, and third rounds were assessed with the intraclass correlation coefficient (ICC) as another evaluation of study reliability (45). The strength of reliability was defined as: very good (0.80 ≤ ICC < 1.00), good (0.60 ≤ ICC < 0.80), moderate (0.41 ≤ ICC < 0.60), fair (0.20 ≤ ICC < 0.40), and poor ( ≤ 0.20), with p < 0.05 (46).

Response data from each round were transferred into electronic data by one researcher and checked by another researcher, then saved as Excel files (Microsoft office 2016). Data analysis was conducted using SPSS 21.0 (SPSS, Chicago, IL).

Demographic characteristics of the Delphi panel are shown in Table 1. In total, 23 of the 40 invited experts agreed to participate in all the Delphi rounds. The mean age of the participants was 46.1 years (standard deviation [SD] = 5.4). The professional experience of the participants was 23.7 years on average (SD = 5.3). All 23 participants finished the Delphi survey in rounds 1, 2 and 3. The response rate of each round was 100.0%.

A total of 10 categories were put forward in this round. The recommendation proportions of categories varied from 34.8% to 100.0% (Table 2). Indicators of exercise capacity were recommended by all participants, followed by indicators of the cardiovascular system (91.3%), neuropsychological/psychological function (78.3%), and muscle/tissue damage (73.9%). The function of the respiratory, oxygen transport, neurological function, and energy metabolism/metabolite level were recommended by the majority to be important indicators of MIF degree (recommendation proportion > 60%, respectively). Although indicators showing endocrine and immune function were also recommended, the proportions were low.

According to the categories recommended by experts, potential indicators were searched according to the database search strategy in Methods (section 2.3.1. Delphi round 1). The initial search retrieved 26,527 articles. After removing 4,531 duplications, screen via title and abstract (15602 removed), and assessed for eligibility via full text (3,423 removed), finally, 2,971 qualified articles were selected out. Among these 2,971 articles, a total of 73 indicators were identified according to the recommended 10 categories (Figure 2). Indicators evaluating the function of the nervous system and the energy metabolism/metabolite level constituted the majority, with a total proportion of 50.7% (n = 19 and 18, respectively). The number of indicators for each category is shown in Table 2. The number of articles per indicator is shown in Supplementary material-Appendix 2.

All 73 indicators identified in round 1 were voted by the participants. The Kendall's W of this round was 0.174 (χ²: 287.835, P < 0.001). As shown in Table 3, the recommendation proportions varied from 30.4% to 100.0%. A total of 28 indicators had recommendation proportion ≥ 70% (Table 3), among which, the indicators Wingate test was recommended to be substituted by fatigue index and mean power, and muscle strength substituted by maximum voluntary contraction and twitch force (the experts' reasons were shown in Table 4). In addition, 14 new indicators were put forward by participants including 4 for oxygen transport system, 2 for neurological function, 3 for muscle/tissue oxidative damage and neuropsychological/psychological function respectively, and 1 for immune function and exercise capacity respectively (the indicators list and experts' reasons were shown in Table 5). Therefore, a total of 44 indicators were forwarded to round 3.

The Kendall's W of round 3 is 0.435 (χ²: 430.607, P < 0.001). The mean score and CV of each aspect of the individual indicators fluctuated between 2.00 and 4.83 (Figure 3A) and between 0.080 and 0.484, respectively (Figure 3B). The value of Kendall's W (> 0.2) and CV (all < 0.5) indicated this round achieved “consensus in.” As shown in Figure 3A, 28 indicators meet the inclusion criteria of the mean score for the recommendation set, among which 14 meet the criteria for grade I (highly recommended). As shown in Figure 3B, only 13 indicators meet the inclusion criteria of the CV for grade I (highly recommended), among which 2 indicators (serum myoglobin and delayed-onset muscle soreness) did not meet the mean score criterion. Thus, a total of 28 indicators were included, with 11 classified as grade I and 17 as grade II (Figure 3 and Table 6). As shown in Table 6, the categories covered cardiovascular system (n = 4), oxygen transport system (n = 5), energy metabolism/metabolite level (n = 6), muscle/tissue damage (n = 3), neurological function (n = 2), neuropsychological/psychological function (n = 3), endocrine function (n = 3), and exercise capacity (n = 2).

The mean familiarity coefficient and judgement coefficient were 0.64 and 0.83, respectively; thus, the mean authority coefficient was 0.733 (> 0.7). The authority of experts was good, and the results of the 3-round consultation were credible.

The ICC for the first, second, and third round were 0.821 (95% confidential interval [CI]: 0.607–0.947, p < 0.001), 0.787 (95% CI: 0.710–0.852, p < 0.001), and 0.932 (95% CI: 0.899–0.958, p < 0.001), which all meet the “very good (0.80 ≤ ICC < 1.00) or good (0.60 ≤ ICC < 0.80)” criteria of reliability, further illustrating the good reliability of this Delphi study.

The Delphi method is a way to solicit expert opinions to gain consensus by iterative stages of anonymous responses (66). The goal of this method is to reduce the range of responses to gain expert consensus quantitatively and qualitatively, which is often seen as more credible than conjecture or individual opinion (42).

A total of 23 experts participated all Delphi rounds, whose average professional experience year was 23.7 years. The mean authority coefficient of experts was 0.733, indicating good authority of participants. The majority of experts are male (19/23, 82.6%) and aged between 40 and 50 years. As previous comparison study exploring the potential confounding factor of Delphi consensus reported (67), no significant difference was observed between men and women experts when giving their opinion, but the authority of experts impacts their opinion significantly. In our study, to achieve a high authority of experts, we set the qualification criteria of experts as “has experience in their related profession more than 10 years, had a title of deputy equal to or higher than associate professor/associate chief physician/chief coach/associate researcher,” which results to the experts aged between 40 and 50 years.

In round 2, 14 new indicators were put forward by participants to evaluate MIF via oxygen transport function (4/14, 28.57%), muscle/tissue oxidative damage levels (3/14, 21.43%), neurological and psycho-behavioral functions (5/14, 35.71%), immune system function (1/14, 7.14%), and exercise capacity (1/14, 7.14%). The results showed that, on the one hand, in the previous literature research (Figure 2), the research group still had some shortcomings and failed to comprehensively identify potential indicators of MIF; on the other hand, the factors involved in MIF may be much larger than we surmised: both the physical and psychological status of soldiers should be considered when evaluating MIF. As previous study on the influence of the literature searches reported (68), the amount of information available (i.e., keywords, bibliographics, abstracts) and the cognitive characteristics of the searcher could both lead to different search outcomes. In our study, literature search aimed to identify potential indicators of MIF. The initial search retrieved 26,527 articles. When removing duplications (n = 4,531), there are still a huge number of articles should be screened for checking their eligibility. Therefore, careless omission or exclusion of literature could not be avoidable, which partially leads to the gap between literature search and experts' recommendation. While, for physical and psychological status, our research strategy did not pinpoint on keywords “physical” or “psychological,” which may also partially lead to leak detection. The above results acknowledged the value of extensive expert consultation and multidisciplinary knowledge exchange in the exploration of MIF indicators and comprehensive evaluation.

After 3 rounds of consultation, a total of 28 recommended indicators (grades I and II) were screened out. Most of these indicators were involved in evaluating MIF via cardiopulmonary and oxygen transport system function (9/28, 32.14%) and energy metabolism/metabolite level (6/28, 21.43%). These results were consistent with the current “wear-out doctrine” and “blockage doctrine” of the mechanism of MIF: that is, the consumption of a large amount of energy substances and accumulation of metabolites during physical exercise/training leads to a decline in the function capacity of tissues, muscles, and organs, ultimately resulting in fatigue (17). The function of the cardiopulmonary and oxygen transport systems is the basis of substance metabolism of energy supply during the exercise process, and the level of material energy metabolism can further regulate the function of the cardiopulmonary and oxygen transport systems (17). Therefore, these two systems have an important role in maintaining the movement of the human body, and thus, should be considered in the evaluation of MIF. Besides cardiopulmonary and oxygen transport system function, and energy metabolism/metabolite level, this Delphi consultation also recommended indicators about muscle/tissue damage, neurological function, neuropsychological/psychological function, endocrine function, and exercise capacity. These results were consistent with previous study about the relationship between exercise and fatigue: physical exercise affects the equilibrium of the internal environment of various physical systems which in turn create sensations of fatigue and exhaustion in the mind of the exercising subject (69). Muscle/tissue damage indicators like creatine kinase are related to the injury of muscle cell triggered by forced myotasis during exercise/training (26, 27), and the injured muscle cells will decrease the exercise capacity of a person, therefore, leading to fatigue. The change of neurological function is one manifestation of fatigue originating from central nervous system. Some invasive indicators such as motor-evoked potentials of transcranial magnetic stimulation (70) were reported before. However, invasive examination has difficulty in the application among soldiers. In our study, reaction time (71) and Stroop test (61) recommended after three-round consensus could avoid the defects of application, further acknowledged the value of extensive expert consultation among indicator selection. For indicators on endocrine function, testosterone (72) and cortisol (73) were reported to be related to the degree of EIF among male athletes. Although their application in female athletes remained unknown, for military populations, gender limitation (the majority are men) on the contrary adds to their feasibility in assessing MIF. Finally, though decreasing of exercise capacity is the most direct symptom of MIF, how to conduct exercise capacity test without adding to the degree of fatigue is an important issue. The grade I indicator recommended by our study is vertical jump height, a simple test that will almost not deteriorate exercise capacity (65), could be a good choice for MIF evaluation.

The results of response rate, quality assessment, Kendall's W and CV showed satisfied attendance of experts (100%), good credibility of this study (authority coefficient > 0.7), and consistency of experts' opinions (Kendall's W = 0.435, all CV < 0.5 in round 3), indicating this Delphi-consensus study made a reliable recommendation about the indicators that may be potentially useful in the comprehensive evaluation of MIF among soldiers. In China, this is the first MIF-evaluation study focusing on military personnel; thus, it may benefit further research on the management of MIF among soldiers.

This study initially clarified the value of comprehensive evaluation in the diagnosis and assessment of MIF. This study better followed the principle of Delphi consensus, and therefore, is a good reference for the management of MIF. But anyway, there do be some limitations. Firstly, this consensus study based on the previous literature of fatigue evaluation and the experience of expertise, therefore, the efficacy of recommended indicators in evaluation MIF needs further study to confirm. Secondly, this consensus study recommends to evaluate MIF comprehensively based on eight aspects of physical systems/capacity, but further comprehensive frame of these aspects and specific combination of indicators needs more sectional and cohort studies to support.

In China, this is the first MIF-evaluation study focusing on military personnel; thus, it could benefit further research on the management of MIF among soldiers. This consensus study indicates the necessary to evaluate MIF in a comprehensive way. Further researches of MIF will focus on constructing a comprehensive evaluation framework for MIF diagnosis and management.