Quality appraisal of clinical practice guidelines for motor neuron diseases or related disorders using the AGREE II instrument

Abstract

Objectives:

This study aimed to systematically assess the quality of CPGs for motor neuron diseases (MNDs) or related disorders and identify the gaps that limit evidence-based practice.

Methods:

Four scientific databases and six guideline repositories were searched for eligible CPGs. Three researchers assessed the eligible CPGs using the Appraisal of Guidelines Research and Evaluation II instrument. The distribution of the level of evidence and strength of recommendation of these CPGs were determined. The univariate regression analysis was used to explore the characteristic factors affecting the quality of CPGs.

Results:

Fifteen CPGs met the eligibility criteria: 10 were for MND and 5 were for spinal muscular atrophy. The mean overall rating score was 44.5%, and only 3 of 15 CPGs were of high quality. The domains that achieved low mean scores were applicability (24.4%), rigor of development (39.9%), and stakeholder involvement (40.3%). Most recommendations were based on low-quality evidence and had a weak strength. The CPGs that were updated, meant for adults, and evidence based, and used a CPG quality tool and a grading system were associated with higher scores in certain specific domains and overall rating.

Conclusion:

The overall quality of CPGs for MNDs or related disorders was poor and recommendations were largely based on low-quality evidence. Many areas still need improvement to develop high-quality CPGs, and the use of CPG quality tools should be emphasized. A great deal of research on MNDs or related disorders is still needed to fill the large evidence gap.

1. Introduction

Motor neuron diseases (MNDs) or related disorders, such as MND, spinal muscular atrophy (SMA), and post-polio progressive muscular atrophy (PPMA), are a group of disorders characterized by progressive weakness secondary to degeneration of the motor neurons (1). MND is caused by the loss of motor neurons leading to muscle atrophy, paralysis, and ultimately death within 3–5 years after the onset of the disease (2). Although the treatment options for MND are limited and patient care primarily focuses on controlling symptoms and optimizing functioning and quality of life (3), better multidisciplinary care and a better understanding of interventions may allow patients with MND to live longer life (4). The data from the Global Burden of Disease study show that the burden of MND is increasing (5).

SMA is an autosomal recessive disease caused by biallelic mutations in the survival motor neuron 1 gene, causing progressive muscle weakness and atrophy (6). It imposes a significant burden on patients, caregivers, and the healthcare system (7), but the financial burden could be significantly reduced by expanding newborn screening for SMA in combination with early treatment interventions for newborns with SMA (8). For example, newborn screening for patients with SMA is associated with earlier diagnosis and intervention, and motor milestones are often achieved at an earlier age, compared with clinically diagnosed patients (8). PPMA, also known as post-polio syndrome (PPS), is a chronic progressive disorder that may appear decades after the initial acute polio infection (9), affecting 20–40% of polio survivors and manifesting as neuromuscular complications (10). Although the incidence of paralytic poliomyelitis has declined significantly (11), PPS will remain a major health problem for many years (12). At present, the diagnosis and management of MNDs or related disorders remain a challenge for clinicians and vary widely in practice.

One of the foundations of improving healthcare work are clinical practice guidelines (CPGs) (13). CPGs are defined as “statements that include recommendations intended to optimize patient care that are informed by a systematic review of evidence and an assessment of benefits and harms of alternative care options” (14). CPGs help improve the quality of health care, such as informing clinicians about treatment decisions for patients and determining the appropriate standard of care, thus identifying gaps between evidence and practice (15).

The usefulness of CPGs depends primarily on quality, rigorous methodology, and transparent development processes (16). Better CPG quality appears to be associated with more positive treatment outcomes (17). Some common problems with CPGs include the sheer volume available, large amount of documentation that is difficult to assimilate or use, lack of clear supporting evidence, exclusion of key stakeholders, insufficient editorial independence, and poor applicability (18–20). Besides, determining the level of evidence on which the recommendations are based is important (21, 22). However, the systemic appraisal of CPGs for MNDs or related disorders remians unreported, and the distribution of the level of evidence on which these recommendations are based has not been described.

Various tools have been developed to assess the quality of CPGs (23). The Appraisal of Guidelines Research and Evaluation (AGREE) Instrument was first published in 2003 and updated in 2009 (AGREE II). It is designed to assess the methodological quality of CPGs and also provide methodological strategies for the development of new CPGs (24). The AGREE II instrument is currently the preferred tool for the quality assessment of CPGs in the world and can be used to assess CPGs of multiple diseases (15, 24–27).

Therefore, the present study aimed to assess the quality of CPGs for MNDs or related disorders using the AGREE II instrument, identify the distribution of the level of evidence and strength of recommendations of these CPGs, and also identify the potential influencing factors for the quality of CPG. The findings of this study would help identify the gaps that hinder evidence-based practice and highlight potential opportunities for improvement.

2. Materials and methods

2.1. Eligibility criteria

CPGs were included if they: (1) focused on the diagnosis and management of MNDs or related disorders, including MND, SMA, and PPMA (1); (2) were published between January 1, 2006, and September 4, 2022; and (3) were written in English. Consistent with previous studies (25, 26), both evidence-based and consensus-based CPGs were included. If the CPG was updated, the latest version was included.

CPGs were excluded if (1) full texts were unavailable; (2) they were editorials, comments, reviews, letters, and correspondence studies; (3) they were interpretation, translation, and adaptation of CPGs; and (4) they were duplicate publications.

2.2. Literature search

Scientific databases, including PubMed, Embase, Cumulative Index to Nursing and Allied Health Literature, and Physiotherapy Evidence Database PEDro, were systematically searched under the direction of a reference librarian. Besides, the following six online guideline repositories were also searched: National Institute for Health and Clinical Excellence (NICE), Scottish Intercollegiate Guidelines Network, Guidelines International Networks, Agency for Healthcare Research and Quality, National Health and Medical Research Council, and World Health Organization. Keywords and Medical Subject Headings related to CPGs and MNDs or related disorders were combined in the database search. The specific search strategy is displayed in Supplementary File 1. The search range was from January 1, 2006, to September 4, 2022.

2.3. Study selection and data extraction

Two researchers (Jia-yin Ou and Jun-Jun Liu) independently performed the study selection and data extraction. Any disagreements between the two were resolved through discussion or consultation with a third researcher (Jing Xu). For study selection, the search results were first exported into the EndNote X7 literature management software (Thomson Reuters Corporation, CA, United States), excluding duplicates. Two researchers reviewed the titles and abstracts of the studies for screening to exclude the explicitly irrelevant studies, and then read the full texts of the remaining studies to determine whether they were finally included. For each CPG included in the end, the accompanying technical and supporting documents were thoroughly searched to better inform our evaluation.

The data were extracted into a specially designed spreadsheet. The extracted variables included the year of publication, disease, development organization, first author (if applicable), country/region of origin, version, age range of target population (adult/children/all ages), development method, search dates covered, CPG quality tool used, CPG methodologist included, title, funding sources, and accompanying documents. The country/region of origin was described as “Europe” if the CPG was jointly developed by multiple European countries, and “international” if it was jointly developed by multiple countries from different continents. The development method was described as “evidence based” if the CPG performed a systematic search and evaluation of evidence and made recommendations based on the evaluation results during the CPG development process; otherwise, it was described as “consensus based.”

2.4. Quality assessment

Three researchers (Jia-Yin Ou, Jun-Jun Liu, and Jing Xu) assessed the quality of each CPG independently using the AGREE II tool under the guidance of a methodological expert (Liming Lu). The AGREE II tool included 23 key items organized in 6 domains followed by 2 global rating items (“Overall Assessment”). The six domains included scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability, and editorial independence. Each item was rated on a 7-point scale, with 1 indicating strongly disagree and 7 indicating strongly agree. Strongly disagree meant no information was relevant to an item, and strongly agree meant the quality of reporting was exceptional and the full criteria and considerations were met for an item. A score between 2 and 6 was assigned when the reporting did not meet the full criteria or considerations for an item. According to the calculation formula, the scores in each domain were calculated and the calculation formula was as follows: each domain score = (obtained score – minimum possible score)/ (maximum possible score – minimum possible score) × 100%. Consistent with a previous study (28), the score of a domain or overall rating ≤ 40% was considered as a low rating, >40 and ≤ 70% as a moderate rating, and > 70% as a high rating.

In the overall assessment, the first global rating item (“overall rating”) was scored on a 7-point scale and then calculated as a percentage, which was the same method used to calculate domain scores, as described in previous studies (15, 29). For the second global rating item, a CPG was rated as high quality when the score of three domains considered as the most important was ≥50% of the maximum possible score, consistent with previous studies (15, 30, 31). The three domains were as follows: stakeholder involvement (domain 2), rigor of development (domain 3), and editorial independence (domain 6).

Before the assessment, researchers received online training using the AGREE II Online Training Tool. Then, a meeting was held to discuss the specific assessment criteria of AGREE II, and researchers assessed some CPGs of different levels and discussed the results with each other. The formal assessment was performed only when the intraclass correlation coefficient (ICC) was >0.8. During the assessment process, researchers carefully read the document of each CPG and its accompanying documents or information on the Internet to make an accurate judgment. If the researchers’ score on an item differed significantly (more than 2 points), a consensus was reached through discussion.

2.5. Statistical analysis

The researchers’ AGREE score was entered into Microsoft Excel (Microsoft, WA, United States), and the standardized score of each domain and the overall score of each CPG were calculated. Continuous variables were expressed as mean ± standard deviation (SD) (normal distribution) or median (Q1–Q3) (skewed distribution), and categorical variables were expressed as frequencies and percentages. The independent-sample t test/nonparametric tests (Kruskal-Wallis test)/chi-square tests/Fisher exact tests were used to compare the differences between the two groups. As the overall scores of AGREE II domain, overall rating, and item of included CPGs had both normal and nonnormal distributions, consistent with the previous studies (27), the mean (SD) and median (Q1–Q3) of overall scores were both presented for the convenience of observation and comparison. The number of each level of evidence and the strength of recommendation of each CPG were evaluated. The univariate linear regression model and the logistic regression model were used to assess the associations between the characteristics and each AGREE II domain score and overall assessment of included CPGs. An ICC with a 95% confidence interval (CI) with a two-way random-effects model was used to detect the inter-rater agreement to ensure that researchers’ understandings of each item were basically the same, and ICCs for each domain and overall rating scores were calculated. Consistent with previous studies (25, 26) and according to Landis and Koch (32), the degree of agreement between 0.01 and 0.20 was considered minor, between 0.21 and 0.40 as fair, between 0.41 and 0.60 as moderate, between 0.61 and 0.80 as substantial, and between 0.81 and 1.00 as very good.

In the sensitivity analysis, a series of analyses were performed to test the robustness of our findings. First, other criteria were used for the overall assessment to identify whether the overall rating score and the number of high-quality CPGs differed from the initial results. For the first global rating item (“overall rating”), the score of each CPG was based on the average score of the six domains, consistent with previous studies (27, 33). A stricter standard was used for the second global rating item, consistent with previous studies (27, 34). A CPG was classified as of high quality if the score of domain 3 (rigor of development) was >70% and the scores of all other domains and overall rating were > 50%. Second, the univariate regression analysis, restricted to CPGs published after 2015 or evidence based or MND as the target disease, was performed separately. This helped assess the association between characteristics and each AGREE II domain score and overall assessment of included CPGs, determine whether these associations were consistent across different types of CPGs, and reduce confounders.

All analyses were performed using the following software packages: R (http://www.R-project.org, The R Foundation), EmpowerStats (http://www.empowerstats.com, X&Y Solutions, Inc., MA, United States), and SPSS 23.0 (IBM, IL, United States). A p value less than 0.05 (two-sided) indicated a statistically significant difference.

3. Results

3.1. Study selection

A total of 5,899 records were yielded initially, and 15 CPGs were finally included for assessment after screening by title, abstract, and full text (Figure 1). Two CPGs were both published in two parts but considered as one (6, 35–37).

Figure 1

3.2. Characteristics of CPGs

The characteristics of included CPGs are displayed in Table 1 and Supplementary Table S1. Supplementary Table S2 shows the results of descriptive statistics of these characteristics. Nine CPGs were published after 2015. Eight CPGs were developed by a medical society and five by an expert panel. Ten CPGs were developed for MND and five for SMA; however, no CPG existed for PPMA. Five CPGs were originally from individual European countries, five were from individual North American countries, and the remaining were international, Europe, and Brazil. Six CPGs were updated. Three CPGs were for adults, one for children, and five for all ages; the rest did not specify the age group. Eight CPGs were considered evidence based and used a grading system. Seven CPGs reported the search dates. Only three CPGs used the CPG quality tool, such as the AGREE II tool. One CPG reported the inclusion of methodologists in the development team. Five CPGs were funded, four were not, and the remainder did not report the funding status. Compared with SMA CPGs, MND CPGs were significantly more likely to state the search dates (70.0% vs. 0.0%, p = 0.026) (Supplementary Table S3).

3.3. Quality assessment of CPGs

Table 2 displays the overall mean (SD) and median (Q1–Q3) scores for each AGREE II domain, item, and overall rating of included CPGs. The mean overall rating score for all CPGs was 44.5% [SD = 24.7%, median = 39.0% (Q1–Q3, 25.0–58.5%)]. Overall, the highest domain score was for clarity of presentation (domain 4) [mean = 80.3% (SD, 14.8%), median = 85.0% (Q1–Q3, 67.0–92.0%)]. The next highest score was for scope and purpose (domain 1) [mean = 68.1% (SD, 23.8%), median = 67.0% (Q1–Q3, 52.0–88.5%)]. The lowest domain score was for applicability (domain 5) [mean = 24.4% (SD, 24.2%), median = 15.0% (Q1–Q3, 10.5–27.0%)], rigor of development (domain 3) [mean = 39.9% (SD, 29.5%), median = 28.0% (Q1–Q3, 14.0–57.5%)], and stakeholder involvement (domain 2) [mean = 40.3% (SD, 22.2%), median = 37.0% (Q1–Q3, 26.0–51.0%)]. The moderate domain score was for editorial independence (domain 6) [mean = 55.9% (SD, 34.4%), median = 47.0% (Q1–Q3, 32.0–94.0%)]. Compared with SMA CPGs, MND CPGs had a significantly lower domain 6 score [median = 80.5% (Q1–Q3, 47.8–96.2%) vs. median = 42.0% (Q1–Q3, 17.0–47.0%), p = 0.036] (Supplementary Table S4).

Item 1 [mean = 6.1 (SD, 1.6), median = 7.0 (Q1–Q3, 6.0–7.0)] and item 15 [mean = 6.1 (SD, 0.9), median = 6.0 (Q1–Q3, 6.0–7.0)] had the highest scores. Item 21 [mean = 2.1 (SD, 1.8), median = 1.0 (Q1–Q3, 1.0–2.0)], item 19 [mean = 2.1 (SD, 2.0), median = 1.0 (Q1–Q3, 1.0–2.0)], and item 13 [mean = 2.1 (SD, 2.1), median = 1.0 (Q1–Q3, 1.0–1.0)] had the lowest scores.

Table 3 displays the AGREE II domain scores and overall assessment of each CPG. Figure 2 displays the distribution of the degree of score across each domain and the overall rating. In the domain scope and purpose, seven CPGs received high ratings, which were scored >70%. Six CPGs received moderate ratings, which were scored >40% and ≤ 70%. The National Institute for Health and Clinical Excellence (NICE) CPG received the highest scores (100%), and the Canadian Thoracic Society (CTS), Brazilian Medical Association, and the 2020 Canada CPG also got high scores, which were all >90%. The distribution of the rating degree was similar in the stakeholder involvement and rigor of development domains, with eight CPGs receiving low ratings, which were ≤ 40%. The 2020 Canada CPG received the highest scores in these two domains (domain 2: 89%, domain 3: 91%). In addition, the NICE CPG also received high ratings in both domains (domain 2: 70%, domain 3: 76%). The CTS received the second highest score in domain 3 (90%). Regarding the domain clarity of the presentation, the results were satisfactory. More than half of the CPGs received high ratings, and none received low ratings. In the domain applicability, thirteen CPGs received low ratings and only two CPGs received high ratings, and these two CPGs were the CTS (85%) and NICE CPG (74%). In the editorial independence domain, five CPGs received high ratings (33.3%) and six received moderate ratings (40.0%). Many CPGs lacked funding information and statements of competing interests. In the overall rating, eight CPGs received low ratings, and only the CTS (89%), NICE (78%), and 2020 Canada CPG (83%) received high ratings.

Figure 2

Among the 15 CPGs, 3 were of high quality, which was consistent with the result of the overall rating. These three CPGs were all for MND, while the NICE and CTS CPGs were for adults, and the 2020 Canada CPG was for all ages. The CTS and 2020 Canada CPGs were from Canada, and the NICE CPG was from the UK. All three CPGs were evidence based and used the CPG quality tool.

The inter-rater reliability was very good for all domains and the overall rating (Table 4). Supplementary File 2 displays the individual scoring of the AGREE II tool for each CPG.

3.4. Level of evidence and strength of recommendation of CPGs

Among the 15 CPGs, 8 were evidence based. Table 5 displays the grading systems used and the distribution of the level of evidence and strength of recommendation among these CPGs. Two CPGs used the adapted Grading of Recommendations Assessment, Development and Evaluation (GRADE) system, one used the combined Oxford Centre for Evidence-based Medicine and GRADE system, one used the American Academy of Pediatrics criteria, two used the adapted American Academy of Neurology (AAN) criteria, one used the AAN criteria, and one did not report the name of the grading system. Four CPGs graded evidence by a body of evidence, whereas four CPGs graded only individual studies. One CPG did not grade the strength of recommendations, and two CPG did not report the strength of specific recommendations.

Although the CPGs used different grading systems, their grading criteria for the level of evidence and strength of recommendations were similar. Most of the evidence for which CPGs graded specific evidence (85.7%) were C, low, D, very low, EC, III, and IV level, indicating low-quality evidence. In the same way, most of the recommendations for which CPGs graded specific recommendations (88.2%) were graded as C, GCPP, U, 2, and weak strength, indicating weak recommendations.

3.5. Relationship between characteristics and AGREE II domain score and overall assessment of CPGs

Table 6 displays the results of the relationship between characteristics and AGREE II domain score and overall assessment of included CPGs. Compared with the MND CPGs, SMA CPGs had a lower domain 6 score (β = -37.9, 95% CI: −70.3 to −5.5). Compared with the CPGs of the first version, the updated CPGs had a higher domain 5 score (β = 24.9, 95% CI: 2.7–47.1), domain 6 score (β = 37.2, 95% CI: 6.3–68.1), and overall rating score (β = 26.4, 95% CI: 4.3–48.6). Compared with the CPGs for adults, CPGs for all ages had a lower domain 1 score (β = −36.8, 95% CI: −64.9 to −8.7), domain 3 score (β = −39.8, 95% CI: −76.5 to −3.1), and domain 5 score (β = −34.1, 95% CI: −64.7 to −3.6). CPGs that did not state the age range of the target population had a lower domain 1 score (β = −32.0, 95% CI: −59.3 to −4.7), domain 2 score (β = −30.7, 95% CI: −58.8 to −2.5), domain 3 score (β = −43.8, 95% CI: −79.3 to −8.3), domain 5 score (β = −36.7, 95% CI: −66.2 to −7.1), and overall rating score (β = -37.0, 95% CI: −68.1 to −5.9).

Compared with the consensus-based CPGs, evidence-based CPGs had a higher domain 1 score (β = 33.7, 95% CI: 16.7–50.7), domain 2 score (β = 26.2, 95% CI: 7.7–44.7), domain 3 score (β = 46.4, 95% CI: 28.3–64.4), and overall rating score (β = 38.6, 95% CI: 23.4–53.9). Compared with CPGs that stated the search dates, those that did not state the search dates had a lower domain 6 score (β = −42.3, 95% CI: −70.3 to −14.3). Compared with the CPGs using the CPG quality tool, those not using the CPG quality tool had a lower domain 1 score (β = −35.7, 95% CI: 60.1 to −11.2), domain 2 score (β = −42.2, 95% CI: −60.1 to −24.2), domain 3 score (β = −57.2, 95% CI: −80.2 to −34.3), domain 5 score (β = −50.3, 95%CI: −66.6 to −34.1), and overall rating score (β = −48.6, 95% CI: −67.3 to −29.8). Compared with the CPGs including a CPG methodologist, those not including a CPG methodologist had a lower domain 5 score (β = −43.0, 95% CI: −81.9 to −4.1), and those not stating had a lower domain 5 score (β = −63.3, 95% CI: −102.2 to −24.3). A comparison between CPGs using and not using the grading system showed the same results as that between consensus-based and evidence-based CPGs.

We also found that the year of publication, type of development organization, country/region of origin, and funding sources were not associated with the AGREE II domain score and overall assessment of included CPGs.

3.6. Sensitivity analysis

Supplementary Table S5 displays the overall assessment of included CPGs using other criteria. The mean overall rating score for all CPGs was 51.5% [SD = 19.9%, median = 51.0% (Q1–Q3, 36.5–61.0%)]. For the second global rating item, only the CTS CPG and the NICE CPG were of high quality under the stricter evaluation criteria. The 2020 Canada CPG was rated as low quality because its domain 5 score was less than 50%. Supplementary Tables S6–S8, respectively, display the relationship between characteristics and AGREE II domain score and overall assessment of included CPGs published after 2015, evidence-based CPGs, and CPGs for MND. The relationship between characteristics and each AGREE II domain score and overall assessment of different types of CPGs were generally consistent with the main analyses that the results in all included CPGs.

4. Discussion

This study was novel in assessing the quality of CPG for MNDs or related disorders using the AGREE II instrument and identifying the distribution of the level of evidence among these CPGs. The quality of CPGs for MNDs or related disorders varied widely. The overall quality of these CPGs was generally poor, and only three CPGs were rated as high quality. However, in a more stringent assessment standard, only two CPGs were of high quality. In comparison, the highest domain score was for clarity of presentation (domain 4) and the lowest domain score was for applicability (domain 5). Eight CPGs were considered evidence based, and most of the evidence (85.7%) on which these CPGs’ recommendations were based was low-quality evidence. Despite the improvement in the quality of CPGs in recent years, contemporary CPGs for MNDs or related disorders still lacked a consolidated evidence basis to provide recommendations for clinical practice. We also identified some factors that could affect the quality of CPGs, which could also serve as aspects for improvement.

Consistent with other studies (15, 30, 31), we used a less-stringent criterion in the overall assessment to judge whether a CPG was of high or low quality, but most CPGs were of poor quality. The development of CPGs requires significant resource consumption. Spending resources on low-quality CPGs with ineffective care recommendations is a waste and confusing for users (15). Coexisting problems are the uneven geographical distribution of CPGs and the duplication of CPGs. Most CPGs were from Europe and North America, only two CPGs were international, and one was from Brazil. Only Canada and the United Kingdom had high-quality CPGs, while other countries lacked them. All CPGs were for MND and SMA, of which SMA CPGs were all of low quality. No CPGs existed for PPS, although the prospect for the future is a continuous and ever-increasing demand for rehabilitation programs and management of patients with PPS (48, 49); currently around 18 million people are still affected by paralytic poliomyelitis (11). Dedicating resources to develop fewer, higher quality, and less-“redundant” CPGs can help reduce inefficient resource usage and user confusion (15, 51). Cooperation between countries and associations should be strengthened to reduce overlapping efforts and focus efforts and resources on developing high-quality CPGs and areas that need to be addressed.

Another problem is the inconsistent terminology used by CPG developers to describe the MND condition for which the CPGs are used. Also CPGs for MND include some CPGs for “amyotrophic lateral sclerosis (ALS)” and some for “MND.” Terminology can be confusing. MND is mainly divided into four categories: ALS, progressive bulbar palsy, progressive amyotrophy, and primary lateral sclerosis. When patients present with both upper and lower motor neuron signs, the disease is referred to as ALS, which is the most common form of MND (52). However, in the United Kingdom and Australia, MND is used as a general term for these disorders and also refers to the ALS subtype; however, in the United States, ALS is more commonly used as a general term and also denotes the ALS subtype (52). Consistent terminology is needed to define MND conditions, regardless of developer/professional group, to reduce CPG duplication and user confusion.

Many aspects of CPG development need to be improved. The domains of AGREE II that these aspects relate to are stakeholder involvement (domain 2), rigor of development (domain 3), applicability (domain 5), and editorial independence (domain 6). CPGs having problems in these domains were consistent with other studies (15, 19, 51). Among these, the problems of poor applicability and editorial independence of CPGs always existed, although the overall quality of CPGs in different health fields improved (51).

Stakeholder involvement reflects how well the CPG represents the views of its intended users, including patients. During the CPG development process, patients’ views, preferences, and expectations regarding care have become increasingly important (53). However, the overall score was low, with 53.3% of CPG scores ≤40% and only one CPG scoring greater than 70%. Most CPGs do not provide details about the involvement of patients or their representatives. Patients are important stakeholders and should be involved in the development process of CPGs, although this may introduce patient biases about costs, cultural background, and expectations (54).

Rigor of development is the most critical domain, as it significantly affects confidence in implementing CPGs (55), while nonsystematic development tends to lead to poor-quality CPGs (56). The strong heterogeneity found among the scores of included CPGs (Table 2) attested to the existence of significant gaps in the methodological development of the CPGs. Seven CPGs were not considered evidence based and did not use the grading system, or describe literature search and selection methods. The low score might be due to insufficient methodological consultation (55) or unfamiliarity with CPG development standards, poor reporting (19), or poor performance of the external peer review and update process (53).

The domain applicability considers factors that may affect guideline implementation, including identifying facilitators and barriers, providing tools for applying recommendations, identifying potentially relevant resources, and auditing standards (24). Thirteen CPGs received low ratings scoring ≤40%. The low applicability significantly hindered the implementation of CPG recommendations. CPG developers should conduct pilot tests to ensure feasibility before publication (57). In addition, all CPGS were just published as articles in journals. New ways to increase user adoption and usability should be considered by CPG developers (15), such as the use of smartphone applications (58, 59) or digital CPG platforms (60).

Editorial independence is also an important domain of CPG quality (61), and with only two statements required, it should be easy to score high (15). However, the overall score in this domain was not high [mean = 55.9% (SD, 34.4%)], with only five CPGs receiving high ratings, scoring >70%. Six CPGs did not state funding sources. Conflicts of interest are a common source of bias (62) and are often underestimated (63). CPG developers should clearly report conflicts of interest, including rigorous review processes and transparent review rules (64).

Overall, the NICE, CTS, and 2020 Canada CPGs (3, 41, 65) had the highest overall rating scores and were also rated as high quality. Based on this, these three CPGs should be favored by clinicians and policy makers, and are worthy of application in clinical practice.

Besides focusing on improving the transparency and methodological rigor of the CPG development process, CPGs should rely more on the growing body of evidence (26). However, nearly half of the CPGs for MNDs or related disorders were not considered evidence based, and most of the recommendations (85.7%) in the evidence-based CPGs were based on low-quality evidence, which were largely from observational studies or expert consensus. This finding constituted an obstacle to establishing CPGs for MND or related disorders, as recommendations were based on low-quality evidence, and it also showed a gap between clinical practice evidence and current medical research. Further research is needed on managing MND or related disorders, and more evidence-based recommendations would be extremely important to improve the standard of care for patients with MND or related disorders. Moreover, given that certain clinical questions may not addressed by high-quality research, it is expert consensus that can fill these knowledge gaps. Despite expert consensus may lack support from high-quality evidence, they can still provide valuable information. Consequently, CPGs developers should pay more attention to the rigor and standardization of consensus method and explore how to guide users to adopt expert consensus accurately.

Some characteristics and factors were found to be associated with the quality of CPGs. Specifically, among the CPGs for MND or related disorders, CPGs that were for MND, stated search dates, and included CPG methodologist were associated with higher scores in some specific domains, whereas CPGs that were updated, for adults, and evidence based, and used CPG quality tool and a grading system were associated with higher scores in both some specific domains and overall rating. In addition, the three CPGs that used the CPG quality tools were all rated as high quality, two of which used the AGREE II instrument. Therefore, the use of CPG quality tools, especially the AGREE II instrument, needs to be emphasized and improved during the development of CPGs. In addition, attention should also be paid to the use of the grading system, which has the greatest impact on the domain 3 score. The GRADE system is used to assess the level of evidence, while the AGREE II instrument is used to guide the CPG development process and set reporting standards (27), and they complement each other. Moreover, methodologists should be brought into the development team and should pay attention to CPG development details, such as providing the search date range of literature evidence.

This study had several limitations. Firstly, the search might have missed some CPGs, although the authors systematically searched major scientific databases and online guideline repositories. Secondly, only CPGs published in English were included, which might have excluded high-quality non-English CPGs, resulting in a lack of representation of CPGs from less-developed countries. Thirdly, the assessment of CPGs might reflect the researcher’s perspective, although our research team was multidisciplinary (including neurologists and other specialists, methodologists and other researchers), these limitations were unavoidable. Fourth, the AGREE II scoring system relied on the understandability and comprehensiveness of CPGs’ reporting and did not reflect the quality and strength of evidence (29). However, this study additionally analyzed the distribution of the level of evidence among included CPGs. Fifth, the AGREE II instrument did not provide a clear cut-off point to distinguish between high-quality and low-quality CPGs. To this end, based on some previous studies, we used a more widely used and less-stringent method to distinguish between high-quality and low-quality CPGs as the main analysis. Also, we used a more sensitive and stringent method to distinguish between high-quality and low-quality CPGs as a sensitivity analysis. Sixth, due to the small sample size, only univariate analysis was used to explore the relationship between CPG characteristics and AGREE II scores. However, we tried to reduce confounders by limiting the analysis to CPGs with certain characteristics, as a sensitivity analysis.

5. Conclusion

The quality of CPGs for MNDs or related disorders varied widely, and the overall quality was poor. No CPG existed for PPMA. Most recommendations were based on low-quality evidence. Many areas still need improvement, especially in the domains of stakeholder involvement, rigor of development, and applicability. CPGs for MNDs or related disorders should formulate recommendations with high-quality evidence and should be developed through rigorous methodology and transparency to minimize bias from external sources, and CPG quality tools should be used. In addition, a significant amount of studies on MNDs are still needed to fill the large evidence gap in the CPGs for these diseases.

Funding

This study was supported by the Youth Program of the National Natural Science Foundation of China (NO: 81904297), the Innovation Team and Talents Cultivation Program of National (No: ZYYCXTD-C-202004), the Special Project of “Lingnan Modernization of Traditional Chinese Medicine” in 2019 Guangdong Provincial R & D Program (NO: 2020B1111100008), the Key Laboratory Program of Universities in Guangdong Province (NO:2018KSYS006), and the Discipline Collaborative Innovation Team Program of Double First-class and High-level Universities for Guangzhou University of Chinese Medicine (NO: 2021XK01).

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

• 1.

  Lau D . Neuromuscular disease affecting the larynx. Adv Otorhinolaryngol. (2020) 85:144–57. doi: 10.1159/000456694

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 2.

  Liu B Li M Zhang L Chen Z Lu P . Motor neuron replacement therapy for amyotrophic lateral sclerosis. Neural Regen Res. (2022) 17:1633–9. doi: 10.4103/1673-5374.332123

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 3.

  Shoesmith C Abrahao A Benstead T Chum M Dupre N Izenberg A et al . Canadian best practice recommendations for the management of amyotrophic lateral sclerosis. Can Med Assoc J. (2020) 192:E1453–68. doi: 10.1503/cmaj.191721

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 4.

  Hardiman O al-Chalabi A Chio A Corr EM Logroscino G Robberecht W et al . Amyotrophic lateral sclerosis. Nat Rev Dis Primers. (2017) 3:17071. doi: 10.1038/nrdp.2017.71

  • CrossRef
  • Google Scholar

• 5.

  Logroscino G Urso D Tortelli R . The challenge of amyotrophic lateral sclerosis descriptive epidemiology: to estimate low incidence rates across complex phenotypes in different geographic areas. Curr Opin Neurol. (2022) 35:678–85. doi: 10.1097/WCO.0000000000001097

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 6.

  Mercuri E Finkel RS Muntoni F Wirth B Montes J Main M et al . Diagnosis and management of spinal muscular atrophy: part 1: recommendations for diagnosis, rehabilitation, orthopedic and nutritional care. Neuromuscul Disord. (2018) 28:103–15. doi: 10.1016/j.nmd.2017.11.005

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 7.

  Paracha N Hudson P Mitchell S Sutherland CS . Systematic literature review to assess the cost and resource use associated with spinal muscular atrophy management. PharmacoEconomics. (2022) 40:11–38. doi: 10.1007/s40273-021-01105-7

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 8.

  Servais L de Vivo D Kirschner J Mercuri E Muntoni F Tizzano E et al . SMA - treatment. Neuromuscul Disord. (2021) 31:S138. doi: 10.1016/j.nmd.2021.07.315

  • CrossRef
  • Google Scholar

• 9.

  Boyer FC Tiffreau V Rapin A Laffont I Percebois-Macadré L Supper C et al . Post-polio syndrome: pathophysiological hypotheses, diagnosis criteria, drug therapy. Ann Phys Rehabil Med. (2010) 53:34–41. doi: 10.1016/j.rehab.2009.12.003

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 10.

  Saraswathy R . The genetics of post-polio syndrome - much to be unravelled!Indian J Med Res. (2016) 144:9–10. doi: 10.4103/0971-5916.193277

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 11.

  World Health Organization . International classification of diseases 11th revision. (2022). Available at: https://icd.who.int/en

  • Google Scholar

• 12.

  de Lira C Santos D Viana RB Guimarães JM Oliveira J Sousa BS et al . Knowledge of healthcare professionals about poliomyelitis and postpoliomyelitis: a cross-sectional study. São Paulo Med J. (2021) 139:464–75. doi: 10.1590/1516-3180.2020.0617.16032021

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 13.

  Eccles MP Grimshaw JM Shekelle P Schünemann HJ Woolf S . Developing clinical practice guidelines: target audiences, identifying topics for guidelines, guideline group composition and functioning and conflicts of interest. Implement Sci. (2012) 7:60. doi: 10.1186/1748-5908-7-60

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 14.

  Trustworthy IOMU Guidelines CP . Clinical practice Guidelines we can trust. Washington (DC): National Academies Press (US) (2011).

  • Google Scholar

• 15.

  Lin I Wiles LK Waller R Goucke R Nagree Y Gibberd M et al . Poor overall quality of clinical practice guidelines for musculoskeletal pain: a systematic review. Br J Sport Med. (2018) 52:337–43. doi: 10.1136/bjsports-2017-098375

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 16.

  Birring SS Kavanagh J Lai K Chang AB . Adult and paediatric cough guidelines: ready for an overhaul?Pulm Pharmacol Ther. (2015) 35:137–44. doi: 10.1016/j.pupt.2015.01.007

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 17.

  Brouwers MC Kho ME Browman GP Burgers JS Cluzeau F Feder G et al . Development of the AGREE II, part 1: performance, usefulness and areas for improvement. Can Med Assoc J. (2010) 182:1045–52. doi: 10.1503/cmaj.091714

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 18.

  Baiardini I Braido F Bonini M Compalati E Canonica GW . Why do doctors and patients not follow guidelines?Curr Opin Allergy Clin Immunol. (2009) 9:228–33. doi: 10.1097/ACI.0b013e32832b4651

  • CrossRef
  • Google Scholar

• 19.

  Alonso-Coello P Irfan A Sola I Gich I Delgado-Noguera M Rigau D et al . The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. Qual Saf Health Care. (2010) 19:e58. doi: 10.1136/qshc.2010.042077

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 20.

  van Dijk MR Steyerberg EW Habbema JD . A decision-analytic approach to define poor prognosis patients: a case study for non-seminomatous germ cell cancer patients. Bmc Med Inform Decis Mak. (2008) 8:1. doi: 10.1186/1472-6947-8-1

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 21.

  Isaac A Saginur M Hartling L Robinson JL . Quality of reporting and evidence in American Academy of Pediatrics guidelines. Pediatrics. (2013) 131:732–8. doi: 10.1542/peds.2012-2027

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 22.

  Khan AR Khan S Zimmerman V Baddour LM Tleyjeh IM . Quality and strength of evidence of the Infectious Diseases Society of America clinical practice guidelines. Clin Infect Dis. (2010) 51:1147–56. doi: 10.1086/656735

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 23.

  Siering U Eikermann M Hausner E Hoffmann-Eßer W Neugebauer EA . Appraisal tools for clinical practice guidelines: a systematic review. PLoS One. (2013) 8:e82915. doi: 10.1371/journal.pone.0082915

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 24.

  Brouwers MC Kho ME Browman GP Burgers JS Cluzeau F Feder G et al . AGREE II: advancing guideline development, reporting and evaluation in health care. J Clin Epidemiol. (2010) 63:1308–11. doi: 10.1016/j.jclinepi.2010.07.001

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 25.

  Shen WQ Yao L Wang XQ Hu Y Bian ZX . Quality assessment of cancer cachexia clinical practice guidelines. Cancer Treat Rev. (2018) 70:9–15. doi: 10.1016/j.ctrv.2018.07.008

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 26.

  Jiang M Guan WJ Fang ZF Xie YQ Xie JX Chen H et al . A critical review of the quality of cough clinical practice Guidelines. Chest. (2016) 150:777–88. doi: 10.1016/j.chest.2016.04.028

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 27.

  O'Shaughnessy SM Lee JY Rong LQ Rahouma M Wright DN Demetres M et al . Quality of recent clinical practice guidelines in anaesthesia publications using the appraisal of Guidelines for research and evaluation II instrument. Br J Anaesth. (2022) 128:655–63. doi: 10.1016/j.bja.2021.11.037

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 28.

  Bakaloudi DR Chrysoula L Poulia KA Dounousi E Liakopoulos V Chourdakis M . AGREEing on nutritional Management of Patients with CKD-A quality appraisal of the available Guidelines. Nutrients. (2021) 13:624. doi: 10.3390/nu13020624

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 29.

  Andrade R Pereira R van Cingel R Staal JB Espregueira-Mendes J . How should clinicians rehabilitate patients after ACL reconstruction? A systematic review of clinical practice guidelines (CPGs) with a focus on quality appraisal (AGREE II). Br J Sport Med. (2020) 54:512–9. doi: 10.1136/bjsports-2018-100310

  • CrossRef
  • Google Scholar

• 30.

  Haran C van Driel M Mitchell BL Brodribb WE . Clinical guidelines for postpartum women and infants in primary care-a systematic review. BMC Pregnancy Childbirth. (2014) 14:51. doi: 10.1186/1471-2393-14-51

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 31.

  Bouwmeester W van Enst A van Tulder M . Quality of low back pain guidelines improved. Spine. (2009) 34:2562–7. doi: 10.1097/BRS.0b013e3181b4d50d

  • CrossRef
  • Google Scholar

• 32.

  Kramer MS Feinstein AR Clinical biostatistics LIV . The biostatistics of concordance. Clin Pharmacol Ther. (1981) 29:111–23. doi: 10.1038/clpt.1981.18

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 33.

  Luu NN Chorath KT May BR Bhuiyan N Moreira AG Rajasekaran K . Clinical practice guidelines in idiopathic facial paralysis: systematic review using the appraisal of guidelines for research and evaluation (AGREE II) instrument. J Neurol. (2021) 268:1847–56. doi: 10.1007/s00415-020-10345-0

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 34.

  Corp N Mansell G Stynes S Wynne-Jones G Morsø L Hill JC et al . Evidence-based treatment recommendations for neck and low back pain across Europe: a systematic review of guidelines. Eur J Pain. (2021) 25:275–95. doi: 10.1002/ejp.1679

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 35.

  Finkel RS Mercuri E Meyer OH Simonds AK Schroth MK Graham RJ et al . Diagnosis and management of spinal muscular atrophy: part 2: pulmonary and acute care; medications, supplements and immunizations; other organ systems; and ethics. Neuromuscul Disord. (2018) 28:197–207. doi: 10.1016/j.nmd.2017.11.004

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 36.

  Miller RG Jackson CE Kasarskis EJ England JD Forshew D Johnston W et al . Practice parameter update: the care of the patient with amyotrophic lateral sclerosis: multidisciplinary care, symptom management, and cognitive/behavioral impairment (an evidence-based review): report of the quality standards Subcommittee of the American Academy of neurology. Neurology. (2009) 73:1227–33. doi: 10.1212/WNL.0b013e3181bc01a4

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 37.

  Miller RG Jackson CE Kasarskis EJ England JD Forshew D Johnston W et al . Practice parameter update: the care of the patient with amyotrophic lateral sclerosis: drug, nutritional, and respiratory therapies (an evidence-based review): report of the quality standards Subcommittee of the American Academy of neurology. Neurology. (2009) 73:1218–26. doi: 10.1212/WNL.0b013e3181bc0141

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 38.

  Georges M Perez T Rabec C Jacquin L Finet-Monnier A Ramos C et al . Proposals from a French expert panel for respiratory care in ALS patients. Respir Med Res. (2022) 81:100901. doi: 10.1016/j.resmer.2022.100901

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 39.

  Corcia P Lumbroso S Cazeneuve C Mouzat K Camu W Vourc'H P . Pre-symptomatic diagnosis in ALS. Rev Neurol. (2020) 176:166–9. doi: 10.1016/j.neurol.2019.07.027

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 40.

  Pattee GL Plowman EK (Focht) Garand KL Costello J Brooks BR Berry JD et al . Provisional best practices guidelines for the evaluation of bulbar dysfunction in amyotrophic lateral sclerosis. Muscle Nerve. (2019) 59:531–6. doi: 10.1002/mus.26408

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 41.

  Motor neurone disease: assessment and management . London: National Institute for health and care excellence (NICE) (2019). Available at: https://www.nice.org.uk/guidance/ng

  • Google Scholar

• 42.

  Silvinato A Bernardo WM . Spinal muscular atrophy 5Q - treatment with nusinersen. Rev Assoc Med Bras. (2018) 64:484–91. doi: 10.1590/1806-9282.64.06.484

  • CrossRef
  • Google Scholar

• 43.

  Cuisset JM Estournet B . Recommendations for the diagnosis and management of typical childhood spinal muscular atrophy. Rev Neurol. (2012) 168:902–9. doi: 10.1016/j.neurol.2012.07.020

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 44.

  Pitarch Castellano I Cabrera-Serrano M Calvo Medina R Cattinari MG Espinosa García S Fernández-Ramos JA et al . Delphi consensus on recommendations for the treatment of spinal muscular atrophy in Spain (RET-AME consensus). Neurologia (Engl Ed). (2022) 37:216–28. doi: 10.1016/j.nrleng.2021.07.002

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 45.

  EFNS Task Force on Diagnosis and Management of Amyotrophic Lateral Sclerosis Andersen PM Abrahams S Borasio GD de Carvalho M Chio A et al . EFNS guidelines on the clinical management of amyotrophic lateral sclerosis (MALS)--revised report of an EFNS task force. Eur J Neurol. (2012) 19:360–75. doi: 10.1111/j.1468-1331.2011.03501.x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 46.

  Filippi M Agosta F Abrahams S Fazekas F Grosskreutz J Kalra S et al . EFNS guidelines on the use of neuroimaging in the management of motor neuron diseases. Eur J Neurol. (2010) 17:526–e20. doi: 10.1111/j.1468-1331.2010.02951.x

  • CrossRef
  • Google Scholar

• 47.

  ACOG committee opinion No . 432: spinal muscular atrophy. Obstet Gynecol. (2009) 113:1194–6. doi: 10.1097/AOG.0b013e3181a6d03a

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 48.

  Lo JK Robinson LR . Post-polio syndrome and the late effects of poliomyelitis: part 2. Treatment, management, and prognosis. Muscle Nerve. (2018) 58:760–9. doi: 10.1002/mus.26167

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 49.

  Lo JK Robinson LR . Postpolio syndrome and the late effects of poliomyelitis. Part 1. Pathogenesis, biomechanical considerations, diagnosis, and investigations. Muscle Nerve. (2018) 58:751–9. doi: 10.1002/mus.26168

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 50.

  de Carvalho M Dengler R Eisen A England JD Kaji R Kimura J et al . Electrodiagnostic criteria for diagnosis of ALS. Clin Neurophysiol. (2008) 119:497–503. doi: 10.1016/j.clinph.2007.09.143

  • CrossRef
  • Google Scholar

• 51.

  Armstrong JJ Goldfarb AM Instrum RS MacDermid JC . Improvement evident but still necessary in clinical practice guideline quality: a systematic review. J Clin Epidemiol. (2017) 81:13–21. doi: 10.1016/j.jclinepi.2016.08.005

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 52.

  Abdul WS Law ZK Ismail NA Azman AR Lai NM . Cell-based therapies for amyotrophic lateral sclerosis/motor neuron disease. Cochrane Database Syst Rev. (2016) 11:D11742. doi: 10.1002/14651858.CD011742.pub2

  • CrossRef
  • Google Scholar

• 53.

  Gallardo CR Rigau D Irfan A Ferrer A Caylà JA Bonfill X et al . Quality of tuberculosis guidelines: urgent need for improvement. Int J Tuberc Lung Dis. (2010) 14:1045–51.

  • Google Scholar

• 54.

  Hoydonckx Y Kumar P Flamer D Costanzi M Raja SN Peng P et al . Quality of chronic pain interventional treatment guidelines from pain societies: assessment with the AGREE II instrument. Eur J Pain. (2020) 24:704–21. doi: 10.1002/ejp.1524

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 55.

  Acuña-Izcaray A Sánchez-Angarita E Plaza V Rodrigo G de Oca MM Gich I et al . Quality assessment of asthma clinical practice guidelines: a systematic appraisal. Chest. (2013) 144:390–7. doi: 10.1378/chest.12-2005

  • CrossRef
  • Google Scholar

• 56.

  Burgers JS Cluzeau FA Hanna SE Hunt C Grol R . Characteristics of high-quality guidelines: evaluation of 86 clinical guidelines developed in ten European countries and Canada. Int J Technol Assess Health Care. (2003) 19:148–57. doi: 10.1017/s026646230300014x

  • CrossRef
  • Google Scholar

• 57.

  Jiang M Liao LY Liu XQ He WQ Guan WJ Chen H et al . Quality assessment of clinical practice Guidelines for respiratory diseases in China: a systematic appraisal. Chest. (2015) 148:759–66. doi: 10.1378/chest.14-3201

  • CrossRef
  • Google Scholar

• 58.

  Free C Phillips G Watson L Galli L Felix L Edwards P et al . The effectiveness of mobile-health technologies to improve health care service delivery processes: a systematic review and meta-analysis. PLoS Med. (2013) 10:e1001363. doi: 10.1371/journal.pmed.1001363

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 59.

  Ventola CL . Mobile devices and apps for health care professionals: uses and benefits. P T. (2014) 39:356–64. PMID:

  • Pubmed Abstract
  • Google Scholar

• 60.

  Siemieniuk RA Agoritsas T Macdonald H Guyatt GH Brandt L Vandvik PO . Introduction to BMJ rapid recommendations. BMJ. (2016) 354:i5191. doi: 10.1136/bmj.i5191

  • CrossRef
  • Google Scholar

• 61.

  Cosgrove L Bursztajn HJ Erlich DR Wheeler EE Shaughnessy AF . Conflicts of interest and the quality of recommendations in clinical guidelines. J Eval Clin Pract. (2013) 19:674–81. doi: 10.1111/jep.12016

  • CrossRef
  • Google Scholar

• 62.

  Detsky AS . Sources of bias for authors of clinical practice guidelines. Can Med Assoc J. (2006) 175:1033–5. doi: 10.1503/cmaj.061181

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 63.

  Neuman J Korenstein D Ross JS Keyhani S . Prevalence of financial conflicts of interest among panel members producing clinical practice guidelines in Canada and United States: cross sectional study. BMJ. (2011) 343:d5621. doi: 10.1136/bmj.d5621

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 64.

  Norris SL Holmer HK Ogden LA Burda BU . Conflict of interest in clinical practice guideline development: a systematic review. PLoS One. (2011) 6:e25153. doi: 10.1371/journal.pone.0025153

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 65.

  Rimmer KP Kaminska M Nonoyama M Giannouli E Maltais F Morrison DL et al . Home mechanical ventilation for patients with amyotrophic lateral sclerosis: a Canadian thoracic society clinical practice guideline. Can J Respir. (2019) 3:9–27. doi: 10.1080/24745332.2018.1559644

  • CrossRef
  • Google Scholar