Quality Assessment of Cancer Pain Clinical Practice Guidelines

Introduction Several clinical practice guidelines (CPGs) for cancer pain have been published; however, the quality of these guidelines has not been evaluated so far. The purpose of this study was to evaluate the quality of CPGs for cancer pain and identify gaps limiting knowledge. Methods We systematically searched seven databases and 12 websites from their inception to July 20, 2021, to include CPGs related to cancer pain. We used the validated Appraisal of Guidelines for Research and Evaluation Instrument II (AGREE II) and Reporting Items for Practice Guidelines in Healthcare (RIGHT) checklist to assess the methodology and reporting quality of eligible CPGs. The overall agreement among reviewers with the intraclass correlation coefficient (ICC) was calculated. The development methods of CPGs, strength of recommendations, and levels of evidence were determined. Results Eighteen CPGs published from 1996 to 2021 were included. The overall consistency of the reviewers in each domain was acceptable (ICC from 0.76 to 0.95). According to the AGREE II assessment, only four CPGs were determined to be recommended without modifications. For reporting quality, the average reporting rates for all seven domains of CPGs was 57.46%, with the highest domain in domain 3 (evidence, 68.89%) and the lowest domain in domain 5 (review and quality assurance, 33.3%). Conclusion The methodological quality of cancer pain CPGs fluctuated widely, and the complete reporting rate in some areas is very low. Researchers need to make greater efforts to provide high-quality guidelines in this field to clinical decision-making.


INTRODUCTION
The global burden of cancer has risen to 19.3 million new cases and 10.0 million deaths in 2020 (1). Pain is one of the most common symptoms in over 70% of cancer patients, but nearly 50% of these patients are not well controlled, which has a negative impact on patients' functional status and quality of life (2,3). Therefore, it is necessary to adequately control pain and improve the condition of cancer patients under the guidance of evidence-based guidelines.
Clinical practice guidelines (CPGs) are statements that included recommendations aimed at improving patient care based on a systematic review of evidence and an assessment of the benefits and harms of alternative care options (4). Policymakers and clinicians generally regard CPGs as an essential tool for selecting the most evidenced and costeffective treatments for practice. The purpose of the guidelines is to improve the quality of patient care by encouraging interventions with proven benefit and discouraging the use of ineffective or potentially harmful interventions; to reduce unnecessary variation in practice; to lessen disparities; and to influence public policy (5). A reliable guideline should follow strict methodological standards and provide complete and clear reporting (6,7). Inadequately reported research may affect their reliability and application and may cause misunderstandings to target users (8). The Appraisal of Guidelines for Research and Evaluation (AGREE) II tool, as an international guideline evaluation gold standard, is the most widely used now (9). The Reporting Items for Practice Guidelines in Healthcare (RIGHT) checklist also has been widely implemented as a standard for guideline reporting (10).
At present, many international organizations have issued guidelines on cancer pain, aiming at providing guidance for alleviating the pain of cancer patients. Among these, the World Health Organization (WHO) analgesic ladder is widely used (11). Moreover, NCCN updates the Adult Cancer Pain every year (12). Although multiple cancer pain guidelines have been issued, so far, there have been no studies to critically evaluate the quality of existing cancer pain guidelines. Therefore, we used the AGREE II and RIGHT checklist to evaluate the methodology and reporting quality in order to identify high-quality cancer pain CPGs and explore potential strategies for improvement.

Study Design
We conducted a comprehensive review of clinical guidelines using the AGREE II instrument and RIGHT statement. This study was conducted and reported based on Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA).

Eligibility Criteria
We included CPGs that met the following criteria: (1) focused on the assessment, diagnosis, management, or treatment of cancer pain; (2) the CPGs were accessible or available; (3) published in Chinese or English; (4) the CPGs were the latest version; and (5) there is no restriction on the type of cancer pain. The following types of CPGs were excluded: (1) translations, interpretations, summaries, protocols, and the draft of guidelines; (2) older guidelines if a new version was accessible; (3) recommendations not only for cancer pain; and (4) expert consensus statements.

Selection and Data Extraction
Two pairs of reviewers (QW and MS, XC and QY) independently screened the titles and abstracts of the identified records. Then, the full texts of potentially relevant articles were downloaded and their eligibility further reviewed. Any discrepancies were resolved by discussion or through adjudication by a third reviewer (LG or JT).
Data of interest were extracted independently by four authors (QW, MS, XC, and QY) using a standardized electronic form. Data extraction included the first author, country, publication year, sponsoring organization, version (original or updated), grading system, development methodology (evidence-based or not), and funding. Disagreements were solved by consultation with another reviewer (ZZ). We conducted two rounds of preextraction to improve the consistency of the results.

Quality Appraisal of Guidelines
The methodological quality of the eligible CPGs was assessed by the AGREE II instrument (9). It consists of 23 items divided into six domains: "scope and purpose," "stakeholder involvement," "rigor of development," "clarity of presentation," "applicability," and "editorial independence." The score for each item ranges from 1 (strongly disagree) to 7 (strongly agree).
The reporting quality of the eligible CPGs was assessed by the RIGHT checklist (10). It contains 35 items divided into seven domains: "Basic information," "Background," "Evidence," "Recommendations," "Review and quality assurance," "Funding and declaration and management of interests," and "Other information." The item was evaluated by "reported" (if the required information has been fully reported), "not reported" (if some required information was missing), or "not applicable" (if this item was not appropriate for assessing the specific CPGs).
Four reviewers (QW, MS, XC, and QY) independently assessed the quality of CPGs, and differences between the reviewers were resolved by consensus or consulting another reviewer (LG, JT, ZZ). Before a formal assessment, we conducted two rounds of pre-evaluation and discussed the appraisal criteria to ensure the consistency of results. Among the reviewers, MS and ZZ majored in nursing care, QW, XC, and QY participated in guideline development, and LG and JT are methodologists in guideline development. Therefore, most of the reviewers had rich experiences in guideline development or guideline evaluation, and some of them had experience of cancer patient care.

Statistical Analysis
Descriptive analysis was performed by calculating the value of each domain, total score, and standardized score by each reviewer based on AGREE II. The calculation formula of the domain score is as follows: (obtained scoreminimal possible score)/(maximal possible scoreminimal possible score) × 100%. The overall quality assessment score was based on the average score in all domains. The overall quality assessment of a guideline was appraised by four evaluators with a Likert scale from 1 to 7. The evaluator could give 6-7 points (if the score was more than 60% on all domains), 4-5 points (if the score was more than 30% on more than three domains but less than 60% on some domains), or 1-3 points (if the score was less than 30% on more than three domains). The guideline was used in practice to judge based on scores in six domains, and the AGREE-II manual recommends using "yes" (6-7 points), "yes with modifications" (4-5 points), or "no" (1-3 points). In addition, agreement among the four reviewers was determined by ICC, which was calculated by SPSS version 24.0 (IBM, Armonk, NY). The value of ICC ranged from 0 to 1. An ICC less than 0.10 was considered unreliable, that between 0.11 and 0.40 as mildly reliable, that between 0.41 and 0.60 as fairly reliable, that between 0.61 and 0.80 as moderately reliable, and that over 0.80 as indeed reliable (13).
We used Excel 2016 (Microsoft) to calculate the reporting rates for each item, domain, and overall, separately. In the calculation of the domain and the overall score, the proportion of 35 items is equal, and the items evaluated as "not applicable" are still included.

Search Results
Initially, we identified 3,954 records through database searching and 105 records through guideline websites. We excluded 3,734 studies after de-duplication and title screening and a further 202 after further screening the full text. Eighteen CPGs (14-31) proceeded to data extraction and synthesis. The selection process of the guideline is presented in Figure 1.

Methodological Quality of the Included Cancer Pain CPGs
The overall assessment and scores of CPGs assessed by the AGREE II tool are shown in Table 2 and Figure 2. The ICC was substantial reliability for four domains (ICC >0.80) and moderate reliability for two domains (ICC = 0.76). Four CPGs (20,26,27,31) were determined to be recommended, two CPGs (17, 21) not recommended, and the remaining 12 CPGs recommended with modifications. The highest mean score was 72% (range, 57% to 89%) on the "scope and purpose" domain, followed by 69% (range, 29% to 85%) on the "clarity and presentation" domain and 64% (range, 13% to 94%) on the "editorial independence" domain. The mean score on other three domains was less than 60%. The domains with low scores for all CPGs were "applicability," "stakeholder involvement," "rigor of development," and "editorial independence."

Scope and Purpose
All CPGs scored over 50% for this domain. Most CPGs (16,88.89%) clearly defined their objectives (item 1), health questions (item 2), and target populations (item 3). Only a limited number of CPGs (2, 11.11%) did not clearly define their health question and target populations.

Stakeholder Involvement
Half of the CPGs (9, 50%) scored over 50% for this domain. Over half of the CPGs (10, 55.56%) clearly introduced their development group (item 4). Only a small number of CPGs (5,

Applicability
Only a limited of CPGs (6, 33.33%) scored over 50% for this domain. Only a limited of CPGs (6, 33.33%) described facilitators and barriers in application (item 18). Half of the CPGs (9, 50%) provided tools and/or advice to help the recommendations be put into practice (item 19). Only a limited of CPGs (7, 38.89%) considered the resources that may be required to implement the recommendations (item 20). More than half of CPGs (11, 61.11%) provided auditing and/or monitoring criteria (item 21).

Basic Information
Overall, none of the included CPGs fully reported this domain. More than 50% CPGs reported the report type in the title (item1a, 17, 94%), the focus of the guideline (item 1c, 15, 83.33%), the summary of the recommendations (item 2, 11, 61.11%), and the corresponding developer or author (item 4, 13, 72.22%). Only three CPGs (item 1b, 16.67%) indicated the publication year, on the title, subtitle, cover page, or copyright page. Eight (item 3, 44.44%) CPGs provided a glossary or a list of acronyms or abbreviations. The descriptor of the report type included guideline(s) (n = 13), recommendation(s) (n = 2), management (n = 1), and various other terms (quick advice, manual, statement, toolkit, handbook, technical paper) (n = 1).

Evidence
Only three (16.67%) CPGs reported all the items in the evidence domain. One of the included CPGs was not an evidence-based guideline, so it cannot evaluate most items of this domain. In 17 other CPGs, 17 (100%) CPGs stated the key question (item 10a), 16 (94.12%) CPGs were identified and assessed based on systematic reviews (item 11a), 10 (58.82%) CPGs indicated how those reviews (item 11b), 15 (88.24%) CPGs introduced the method for evaluating the certainty of evidence (item 12), and 4 (2.53%) CPGs explained how the outcome (item 10b) was selected and sorted.

Recommendations
One of included CPGs was not an evidence-based guideline, so it cannot evaluate in this domain. Seventeen other CPGs (100%) introduced precise, clear, and operable recommendations (item 13a), 2 (11.76%) presented specific recommendations for important subgroups (item 13b), 13 (76.47%) reported both the strength of recommendations and the certainty of evidence (item 13c), 9 (52.94%) CPGs reported considering values and preferences (item 14a), 9 (52.94%) reported the cost and resource required to implement recommendation (item 14b), 5 (29.41%) described other factors (item 14c), and 13 (76.47%) reported the processes and methods used by the guideline work group for decision-making (item 15).

Review and Quality Assurance
A total of 9 (50%) CPGs mentioned information on the external review of the final draft guidelines (item 16), and 3 (16.67%) CPGs reported the quality assurance program (item 17).

Methodological and Reporting Quality by Publication Year and the Criteria for Rating Evidence
Overall, the AGREE II scores of the included CPGs showed a smooth trend in most domains according to publication year, which may be because most of the CPGs are evidence-based CPGs (Supplement Figure S1). Two CPGs which were published before 2000 had a low score in most domains, except the scope and purpose domain and the stakeholder involvement domain. The AGREE II score showed an obvious difference according to the criteria for rating evidence (Supplement Figure S2). The overall quality of CPGs using the GRADE approach was high. The overall quality of some guidelines that use other standards was moderate or even lower than those that do not use any standards. The RIGHT score showed an unclear trend according to publication year, which may be because the included CPGs are limited (Supplement Figure S3). The most significant increase was in the score of the "Evidence" domain, from 58.33% before 2018 to 90.00% after 2018. It experienced a slight improvement after the publication of RIGHT, but the overall quality of cancer pain is still moderate. The RIGHT score showed an obvious difference according to the criteria for rating evidence (Supplement Figure  S4).The significant increase was in the score of the "Recommendations" domain, from 31.43% to 62.64%. Although cancer pain CPG has improved in most domains of methodology and reporting, the overall quality was still moderate.

DISCUSSION
It is no doubt that improving the quality of the guidelines could play a large part in its advantages and promote the dissemination and implementation of the guideline. The guideline working group should follow the AGREE II to conduct a rigorous guideline development procedure and use RIGHT for detailed reporting. In recent years, the methodological quality and reporting quality of the cancer pain guidelines have been improved, but some aspects still need to be improved. We found that the methodological and reporting quality of cancer pain CPGs was highly heterogeneous among different domains even within the same guideline; the distribution of level of evidence and strength of recommendations also varied significantly among different categories of CPGs. According to our findings on the methodological quality of the guidelines, unlike lung cancer CPGs, which score poorly in the domain of applicability and editorial independence, cancer pain CPGs do not score well in the applicability and stakeholder involvement domains (34). Compared with the methodological quality of nasopharyngeal carcinoma and oral cancer CPGs, cancer pain CPGs scored less than 50% not only in the applicability domain but also in the stakeholder involvement domain (35,36). As with previous studies, this study also showed that all cancer-related CPGs scored low in the applicability domain, and there is a need to improve the quality of applicability and stakeholder involvement domains. According to the AGREE II result, two domains are in urgent need of improvement. Stakeholder involvement reflects whether CPGs better reflect the wishes of its target users, so it is necessary to include patient representatives in working groups or to conduct patient preference surveys. At present, the research on the patient preference survey is thriving, and it is feasible to increase patient preference in the guideline and recommend the optimal scheme according to patient preference. For the applicability domain, the guideline panels should conduct presurveys in advance to identify obstacles and facilitators in the implementation process and to provide appropriate responses and implementation criteria. Considering patient preferences and providing detailed implementation standards or manuals are more helpful for guidelines to be popularized in the field.
Based on our findings on the reporting quality of the guidelines, cancer pain CPGs only reported less than 50% in the review and quality assurance domain, which is different from the findings for lung cancer and gastric cancer CPGs with poor reporting rates for evidence, funding and declaration and management of interests, review and quality assurance, and other information (37,38). The clinical practice guidelines on prostate cancer reported more than 50% only in the background and basic information domain (39). We think the good reporting quality of CPGs must be more than 50% of reporting rate. According to the RIGHT result, improvements are needed in three domains. Review and quality assurance was an important part of ensuring the high quality of CPGs. The possible reason may be that the readers and the authors may consider the rich recommendations to take precedence over the review process in CPGs. Limitations in the layout of journals can also lead to poor reporting. However, we believe that reports on improved review details and quality assurance methods will enhance the transparency of the review process, thereby improving the overall quality of the guideline working group; they need to report this part in its manuscripts or appendices. The recommendation domain, funding and interest declaration domain, and other information domain also needed more detailed reports. There is still a lack of reports related to explaining recommendations and describing the decisionmaking process; these items can help readers understand the recommendations easily and improve the recommendation quality. Therefore, dissemination and implementation of the guideline and recommendations in detail is necessary to include in the guideline. The report of funding and declaration of interest items can ensure the independence and impartiality of the CPGs. The report of other information items can improve the dissemination and implementation of CPGs. As we all know, the formulation of guidelines requires a lot of financial support; a reasonable and public financial support can ensure the fairness and credibility of the guidelines. Guidelines supported by national or provincial projects are bound to be more credible than those that rely on private funding. In addition, the low score of these cancer pain CPGs may be related to the lack of reports caused by periodical restrictions.
Currently published cancer pain guidelines focus on comprehensive management (e.g., the evaluation of cancer pain, basic principles, and adverse reaction management) or treatment only. The treatment guidelines focus on pharmacological interventions, with less mention of non-pharmacological intervention (such as psychological interventions, physical and complementary treatment). Nevertheless, based on our preliminary findings, with the improvement of cancer pain treatment measures, effective, convenient, and safe nonpharmacological therapies will become an increasingly important factor in future guidelines. For example, anxiety, depression, fear, and other psychological problems that come with cancer pain should not be underestimated. Several studies have reported that cancer patients pay attention to the psychological aspects associated with treatment (40)(41)(42). However, the analysis of the recommendations was not the purpose of this study. We expected that future studies could further analyze the recommendations of the guidelines based on multiple aspects (e.g., pharmacological, nonpharmacological treatments), in order to understand the recommendations of cancer pain in different areas and different interventions.

Strengths and Limitations
To the best of our knowledge, this is the first study to evaluate quality systematically for cancer pain guidelines. We strictly control the quality in the process of evaluation, carried out a comprehensive and systematic literature search, and obtained the highly unanimous opinion of the four evaluators which improved the credibility of our conclusion. Nonetheless, this study has some limitations. Due to the limitation of language skills, we only evaluated CPGs written in Chinese and English, which may affect the completeness of the current results. Moreover, evaluating the methodology and reporting quality of CPGs using the AGREE II and RIGHT checklists is a subjective process. Although we independently assessed and used ICC scores to detect consistency, bias was still possible. Meanwhile, due to limitations of the journal layout, the report of the guideline may indeed have some important information that leads to lower quality of the report. Furthermore, further analysis of the recommendations in these guidelines is also of interest. Despite these shortcomings, the consistency of comprehensive search and evaluation ensures the credibility of our research results.

CONCLUSION
This study found that more than half of the CPGs were of high quality that could be recommended. However, the reporting rate of existing cancer pain CPG is low in some domains, and its methodological quality is also inadequate. Researchers need to make greater efforts to provide high-quality guidelines for clinical decision-making.

AUTHOR CONTRIBUTIONS
ZZ: conceptualization, methodology, data curation, writingoriginal draft preparation; XC: conceptualization, methodology, data curation, writing-original draft preparation; QW: methodology, data curation, writing-original draft preparation; QY: data curation, writing-original draft preparation; MS: data curation, writing-original draft preparation; LG: writingreviewing and editing, project administration; JT: supervision, project administration. All authors contributed to the article and approved the submitted version.