Quality appraisal of clinical practice guidelines for the management of Dysphagia after acute stroke

Objectives Dysphagia is a common complication in stroke patients, widely affecting recovery and quality of life after stroke. The objective of this systematic review is to identify the gaps that between evidence and practice by critically assessing the quality of clinical practice guidelines (CPGs) for management of dysphagia in stroke. Methods We systematically searched academic databases and guideline repositories between January 1, 2014, and August 1, 2023. The Appraisal of Guidelines for Research and Evaluation (AGREE II) instrument was used by two authors to independently assess CPG quality. Results In a total of 14 CPGs included, we identified that three CPGs obtained a final evaluation of “high quality,” nine CPGs achieved “moderate quality” and two CPGs received “low quality.” The domain of “scope and purpose” achieved the highest mean score (91.1%) and the highest median (IQR) of 91.7% (86.1, 94.4%), while the domain of “applicability” received the lowest mean score (55.8%) and the lowest median (IQR) of 55.4% (43.2, 75.5%). Conclusion The CPG development group should pay more attention to improving the methodological quality according to the AGREE II instrument, especially in the domain of “applicability” and “stakeholder involvement;” and each item should be refined as much as possible.


Introduction
Globally, stroke remained the second-leading cause of death and the third-leading cause of death and disability combined in 2019 (1,2).Dysphagia is a common complication in stroke patients, widely affecting recovery and quality of life after stroke and increasing mortality risk through increased risk of dehydration, malnutrition and pneumonia (3).The incidence of dysphagia varies widely depending on the method of assessment, compared with clinical assessment (30-55%) and video rheology (64-78%), a lower incidence was detected using initial screening tools (37-43%) (4).However, managing dysphagia correctly and effectively can shorten hospital stays, reduce the risk of death, and decrease healthcare costs (5,6).Clinical practice guidelines (CPGs) are a type of declaration that include evidence-informed recommendations aimed at optimizing patient care that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options (4).To date, a number of CPGs have been developed and updated with the aim of ensuring optimal dysphagia management of stroke patients.CPGs would contribute to improving the quality of health care, for example, providing evidence for clinicians to make decisions about patient care and determining appropriate medical criteria, thereby identifying gaps between evidence and practice (7).Nevertheless, hospital personnel adherence to evidence-based stroke care is limited (8), translating evidence into clinical practice is challenging, and implementation of these CPGs in clinical practice remains suboptimal (9,10).
The quality of the CPGs has a direct impact on utilization (11,12), and the purpose of this study was to assess the quality of guidelines for managing poststroke dysphagia.Therefore, we used the Appraisal of Guidelines for Research and Evaluation II (AGREE II) instrument (13) to evaluate the quality of CPGs for dysphagia management after stroke, which may be helpful in identifying the potential factors that impact the quality of CPGs.The findings would illustrate the gaps between evidence-based guidelines and clinical practice and attempt to explore potential measures of improvement.

Search strategy
A comprehensive literature search was conducted by two authors to identify CPGs for the prevention, diagnosis, and treatment of dysphagia after acute stroke between January 1, 2014, and August 1, 2023.The following databases were searched: PubMed, Web of Science and EMBASE, Clinical Practice Guidelines, the National Institute for Health and Care Excellence, National Guideline Clearinghouse, World Health Organization, Scottish Intercollegiate Guideline Network, New Zealand Guidelines Group and BMJ Best Practice.Search strategies were tailored according to each database (The specific search strategy is displayed in Supplementary File 1).All results were imported into EndNote (Version.X9.2),where duplicates were removed.A third author resolved any disagreements.

Eligibility criteria
The inclusion criteria were as follows: (1) International and national CPGs published on the management of dysphagia after acute stroke; (2) Published or updated from January 1, 2014 to August 1, 2023; (3) Published in English; and (4) Guidelines focused on adult patients.The excluded criteria were as follows: (1) Guideline-related interpretation, application evaluation or brief versions, etc.; (2) Full text not available; and (3) Guidelines under development or withdrawal.

Data screening and extraction
The titles and abstracts of all search results were screened by two authors before checking the full text.In addition, two authors scanned the reference lists of the confirmed papers to identify more relevant CGPs.Then, they extracted the characteristics of the CPGs including year, developer, grading system, country/region, target population, and multidisciplinary team using a predesigned standardized data extraction form.

Quality assessment
The quality of the 14 CPGs was appraised by two authors trained using the AGREE II instrument, which is a reliable tool that is widely used to assess the quality of CPGs (13).AGREE II consists of 23 items organized into six domains and two overall assessment portions.Each item was scored from 1 to 7 (1 = strongly disagree, 7 = strongly agree).Prior to the formal assessment, we discussed the assessment criteria based on the AGREE II manual and training tools to maintain a consistent understanding of each item.After scoring, we organized the CPGs and randomly cross-checked 10% (14) to ensure consistency between authors, especially for items with wide variations in scoring.
The standardized scores for each domain were computed based on the achievement scores (13), as follows: The maximum possible score of domain = 7 (strongly agree) × number of items × number of evaluators; a minimum possible score of domain = 1 (strongly disagree) × the number of items x the number of evaluators.The standardized scores = (obtained score ─ minimum possible score)/ (maximum possible score ─ minimum possible score) × 100%.
The AGREE II manual does not offer any advice on how to explain the scores.In accordance with previous studies (15,16), if a CPG scored above 70% on six domains, it was classified as 'high quality'; if a CPG scored above 70% on three to five domains, it was classified as 'moderate quality'; and if a CPG scored less than 70% on ≥ two domains, it was classified as 'low quality' .

Statistical analysis
All data were analyzed using IBM SPSS Statistics Version 26.0 software and Microsoft Excel 2021.Mean, median and interquartile range (IQR) were computed for the domain scores.The intraclass correlation coefficient (ICC) was computed to measure the interrater agreement when performing a quality appraisal of the CPGs among the two appraisers to ensure the reliability of our conclusions.The level of ICC was classified according to commonly cited cutoffs: poor (< 0.50), fair (0.50-0.75), good (0.75-0.90) or excellent (0.90-1.00) (17).

Results
A total of 1,208 titles and abstracts were generated through database and manual searches.After deleting duplicates, 730 articles were filtered by title and abstract.A total of 36 full-text CPGs were screened for eligibility, and 14 CPGs were included in our systematic review.Figure 1 provides the PRISMA flow chart (18).Table 1 shows the general characteristics of the CPGs included in the analysis.Regarding geographical distribution, six of them are from Europe, the US and Canada all have two CPGs each, while Brazil, China, Turkey, Australia and New Zealand have only one.

Quality of CPGs according to the AGREE II domains
Table 2 reports the ICC score, overall quality and recommendation comments of all CPGs.In total, 14 CPGs were included, only three CPGs were found to be of high quality with all domains reaching a score higher than 70%, nine CPGs were graded as moderate quality and the remaining two were classified as low quality.The evaluation results of the two appraisers were reliably consistent, with ICCs (95% CI) ranging from 0.75 (0.48, 0.88) to 0.90 (0.82, 0.99).
The quality of CPGs evaluated by AGREE II varied widely, not only between guidelines, but also between domains within guidelines.Figure 2 shows the score distribution of the 6 domains among the 14 CPGs.Figure 3 shows the mean score of each domain for all CPGs sorted by quality classification.

Scope and purpose
The domain of "scope and purpose" obtained the highest mean (91.1%) and the highest median (IQR) score of 91.7% (86.1, 94.4%).Moreover, all guidelines achieved over 70% in this domain, but only two of them had a maximum score of 100% (22, 29).

Stakeholder involvement
The standardized scores in this domain ranged from 38.9 to 91.7%, with nine of 14 CPGs scoring above 70%.Most of the poor scores are due to the views and preferences of the target population (patients, public, etc.) have not been sought (22,32).

Rigor of development
Regarding the standardized scores in this domain, the mean was 70.5%, and the median (IQR) was 75.0%(62.2, 81.3%).GSN2021 (32) obtained the lowest scores (43.8%).Most CPGs lacked clarity in describing all stages of the methodological development or did not provide a procedure for updating the guidelines.

Clarity of presentation
In this domain, the mean score was 87.3%, and the median (IQR) was 90.3% (79.5, 97.2%).Six CPGs received above 90%.In contrast, GSN2021 (32) obtained the lowest score of 63.9%, which means that the guideline development group did not present recommendations clearly.

Editorial independence
In this domain, the mean was 90.2%, and the median (IQR) was 89.6% (83.3, 100%).Five CPGs (23,26,27,31,32) received full marks in this domain, with the exception of NICE2022 (22), which did not explicitly provide information on editorial independence and the competing interests of members of the CPG development group have not been recorded and addressed.

Discussion
The present study proposes a critical review that evaluates the quality of 14 CPGs developed to manage dysphagia in acute stroke using the AGREE II tool (13).Depending on our results, the quality of CPGs evaluated by AGREE II varied significantly, not only between guidelines, but also between domains within guidelines.RCP2016 (27), AHA/ASA2016 (28) and CSA2022 (31) were classified as high quality and thus were recommended based on the AGREE II tool.Among the domains, "scope and purpose" obtained the highest mean score of 91.1% and the highest median (IQR) score of 91.7% (86.1, 94.4%), while "applicability" yielded the lowest mean score of 56.8% and the lowest median (IQR) score of 55.4% (43.2, 75.5%).Based on the AGREE II reported items, the domain of "applicability" performed the worst, which is consistent with other quality assessment results of CPGs in different healthcare topics (33,34).Many CPGs failed to identify and describe the potential facilitators, barriers and advice or tools on how the recommendations can be put into practice.This may be one of the reasons why clinical implementation is not as effective as it could be (9,35).To address this issue, we find that implementation science approaches are feasible, and a quality improvement intervention that includes online educational videos, mobile health technology, simplified versions of the guidelines manual, audits and feedback, is recommended to improve the CPG adherence of medical staff and patients, user awareness and CPG uptake (36)(37)(38)(39).
Regarding the domain of "stakeholder involvement, " some CPGs did not clearly describe the guideline development group or the views and preferences of the target population (patients, public, etc.) were not been sought.During the development of CPGs, patients and a variety of stakeholders, such as clinicians of all types, insurance payers and funders, health policy decision makers, and experts should be involved in the development of CPGs to set priorities, ensure feasibility, and promote distribution and compliance (6,40,41).
Most of the CPGs lacked clarity in describing the crucial stages of the methodological development, especially in external review and procedure for updating, which is important for transparency and applicability (42).In addition, guidelines would benefit from a more prescriptive and standardized evidence-based approach to developing recommendations and avoiding the use of ambiguous recommendations.Two of the included CPGs [IMSWT2020 (20), CSA2022 (31)] used the AGREE II tool during the external review and development phase.Although IMSWT2020 (20) used the AGREE II instrument, high quality is still not achieved in the domain of applicability.Therefore, the AGREE II instrument should be considered in the process of planning, developing and publishing CPGs for guideline development groups (13).Our results were largely similar to the results of CPG quality appraisal in different clinical topics (43)(44)(45), indicating that the problems in CPG development have some commonality.The CPG development group should pay more attention to improving the methodological quality according to the AGREE II instrument, and each item should be refined as much as possible (16,42).
In addition to focusing on improving the transparency and methodological rigor of the guideline development process, the quality of guidelines is more dependent on high-quality evidence.However, most of the recommendations in the above guidelines are based on low to moderate quality evidence, and even some of them are not based on evidence.More high-quality evidence is needed for the management of post-stroke dysphagia, such as how to select instruments to evaluate swallowing with sensory tests (29), rational dietary programs (24), and effective therapies (31), which are extremely important for improving the quality of care for patients with post-stroke dysphagia.
Our study has several strengths.First, before the formal assessment, two assessors discussed the appraisal criteria according to the AGREE II manual and training tools to maintain the understanding of each item in line with each other.After scoring, the CPGs were collated with a randomized 10% cross-check (14) to ensure consistency between authors, especially for the items with significantly different scores.Furthermore, to the best of our knowledge, this is the first study that compares and evaluates the quality of CPGs in the nutritional management of stroke patients.
Due to language or publication restrictions, our review is limited to CPGs written in English, and excluding CPGs written in other languages may introduce bias.Furthermore, AGREE II does not provide an explicit cutoff to distinguish between high quality, moderate quality, and low quality CPGs.We defined them based on previous studies, but we are not exempt from misinterpretation that may derive from heterogeneity in the formulation and wording of recommendations.In addition, it is worth noting that in this study, only the critical appraisal of the quality development of the guidelines was performed, without any assessment of the quality of the guidelines' content.

FIGURE 1 PRISMA
FIGURE 1PRISMA flow chart of searching and selecting guidelines.

FIGURE 2
FIGURE 2Score distribution of the six domains among the 14 CPGs.

FIGURE 3
FIGURE 3Mean score of each domain for all CPGs sorted by quality classification.

TABLE 1
General characteristics of the CPGs included in the analysis., Brazilian Academy of Neurology; IMSWT, Integrative Medicine for Stroke working team; GPP, good practice points; GRADE, The Grading of Recommendations Assessment, Development and Evaluation; PMR, Turkish Society of Physical Medicine and Rehabilitation; WGO, World Gastroenterology Organization; CPGs, clinical practice guidelines; NICE, National Institute for Health and Care Excellence; ESO, European Stroke Organization; ESPEN, The European Society for Clinical Nutrition and Metabolism; NSF, National Stroke Foundation; RCP, Royal College of Physicians; AHA/ASA, American Heart Association/American Stroke Association; CSA, Canadian Stroke Association; AN, Australian and New Zealand; GSN, German Society of Neurology. BAN

TABLE 2
Appraisal of Guidelines for Research and Evaluation (AGREE) II version result for clinical practice guidelines.