We systematically searched five databases, including PubMed, DynaMed, UTD (UpToDate), CNKI (China National Knowledge Infrastructure), and CBM (China Biology Medicine). Additionally, WHO (World Health Organization), GIN (Guidelines International Network), NICE (National Institute for Health and Care Excellence), SIGN (Scottish Intercollegiate Guidelines Network), ECRI (Emergency Care Research Institute), and Medlive were searched. Our comprehensive search was limited from inception to 14 March 2022. The search strategy consisted of keywords: (“rheumatoid arthritis”) AND (“guidance” OR “guideline” OR “recommendation”). The detailed search strategy was shown in Supplementary Table S1.

Inclusion criteria: 1) guidelines for the diagnosis, treatment, and management of RA; 2) the latest and available version of the guidelines. Exclusion criteria: 1) guidelines published in languages other than Chinese or English; 2) an interpretation version of the guidelines; 3) the protocol of guidelines.

Two researchers independently screened the titles and abstracts to exclude irrelevant literature. The literature, which could not be determined by screening the titles and abstracts, needed to be identified by reviewing the full text. Data extraction was conducted using pre-designed tables in Microsoft Excel 2016. The extracted items included the title, publication year, publication language, development institution/organization, type of guideline, the adaption of other guidelines or not, the method for grading the quality of evidence and strength of recommendations, and counts of recommendations. Data extraction was carried out by two researchers independently. If there were disagreements between literature selection and data extraction, a third researcher was consulted to resolve the disagreements by reaching a consensus.

Firstly, recommendations with unspecified strength or guidelines without using a method for grading the quality of evidence and strength of recommendations were excluded from viewing the changes in the direction and strength of recommendations. Secondly, leave-one-out sensitivity analyses were used to assess the robustness of consistency after excluding one guideline at a time, looking at changes in the direction and strength of each PIC in turn. Sensitivity analysis was carried out through these two steps to check whether the scope of included guidelines in the consistency analysis and the strength of the recommendation has a clear impact on the direction and strength of the recommendation and the fluctuation range of the consistency rate.

We initially identified 2,207 records, 54 duplicates removed by Endnote software, 1,721 records were excluded after the full-text screening, and 406 records were excluded by reviewing the full text. Ultimately, 26 guidelines met the criteria (see Figure 1). (Lau et al., 2018; Smolen et al., 2020; Ho et al., 2019; Fraenkel et al., 2021; Cardiel et al., 2018; NICE, 2020; Peter et al., 2021; Fang et al., 2020; Alhajeri et al., 2019; Daien et al., 2019; Parisi et al., 2019; Leon et al., 2018; Ataman et al., 2018; Chinese Rheumatology, 2018; Henrique da Mota et al., 2012; Hurkmans et al., 2011; Forestier et al., 2009; Gossec et al., 2006; Jiang et al., 2018; Huang et al., 2011; Jiang, 2019-10; Guidelines for diagnosis and treatment of rheumatoid arthritis, 2010; Luo et al., 2020; Management of Rheumatoid Arthritis, 2019; Management of early rheumatoid arthritis, 2011; Clinical guideline for the diagnosis and management of early rheumatoid arthritis, 2009) Twenty-six guidelines were included in publication years ranging from 2006 to 2021. Eight CPGs were developed in China, three in France, two in the Netherlands, and one in each of the United States, Mexico, the United Kingdom, Europe, Asia Pacific, Italy, Malaysia, Brazil, Scotland, Australia, Spain, Kuwait, and Turkey. The types of included guidelines consisted of diagnosis, treatment, and management. The specific types of therapy were pharmacotherapy, physiotherapy, TCM therapy, psychotherapy, and rehabilitation therapy. The methods for grading the quality of evidence and the strength of recommendations were different. Seven CPGs used the Grading of Recommendations, Assessment, Development, and Evaluations (GRADE), two used the adapted GRADE, three used the Oxford Centre for Evidence-Based Medicine (OCEBM) criteria, one used the Scottish Intercollegiate Guidelines Network (SIGN), one used the Evidence-Based Recommendation Development (EBRO), one used the National Health and Medical Research Council (NHMRC), and four adopted other unnamed standards or tools. The method for grading the quality of evidence and strength of recommendations was not reported in one CPG, and six CPGs not used (see Supplementary Table S2).

We selected 14 guidelines for consistency analysis from the included guidelines, with the number of recommendations ranging from 10 to 47 (see Supplementary Table S2). The publish year of 11 (78.57%, 11/14) guidelines ranged from 2018 to 2021. Other three guidelines (RACGP, SIGN, and BSR) were published in 2009, 2011, and 2012 respectively. Sixty-four recommendations of the 14 included guidelines were extracted and analyzed. After PIC coding, 18 PICs were obtained (see Table 1), and the details of the direction and strength of the recommendations were shown in Supplementary Table S3. There were 17 (17/18, 94%) PICs with consistent direction, and the only PIC (1/18, 6%) with an inconsistent direction was HCQ monotherapy. There were 7 (7/18, 38%) PICs with consistent direction and strength, including csDMARDs monotherapy, leflunomide (LEF) monotherapy or salazosulfapyridine (SSZ) monotherapy when methotrexate (MTX) was contraindicated, csDMARDs in combination with bDMARDs, and dosage reduction of csDMARDs and bDMARDs (see Figure 2). Among the 18 PICs, the direction of the recommendations was highly consistent, but poor consistency was found in the strength of the recommendations.

The inconsistency rate remained high after excluding recommendations with unspecified strength (see Table 2; Supplementary Table S4). The leave-one-out sensitivity analysis revealed varying degrees of variation in the consistency rate (see Supplementary Tables S5–S18). The number of coded PICs varied from 13 to 18. The variation in the direction of recommendation consistency rate ranged from -1% to 6%; the variation in the rate for both direction and strength consistency ranged from -15.8% to 14.1% (see Table 2).

Here we took the CPGs which had the most significant rate changes in the leave-one-out sensitivity analysis as examples to describe. After excluding recommendations with unspecified strength, the direction consistency rate increased to 100% after NICE or HKSR guideline was excluded (see Supplementary Tables S8, S11) and decreased to 92.3% when SIR guideline was excluded (see Supplementary Table S12). After excluding APLAR guideline, the number of recommendations with consistent direction and strength increased to nine, and the consistency rate increased to 53.0% (see Supplementary Table S10). In contrast, after excluding SIR guideline, the number of recommendations with consistent direction and strength remained only three, and the consistency rate decreased to 23.1% (see Supplementary Table S12).

The recommendations among different CPGs create difficulties in clinical practice and decision-making, specifically when faced with inconsistent directions or various conditions related to recommendations (Mackey and Liang, 2011). In our study, HCQ was recommended in different directions. NICE guideline indicated that HCQ should be considered as a first-line treatment (NICE, 2020), but HKSR stated not to use HCQ because of its slow onset and the availability of more effective csDMARDs (Ho et al., 2019). Recommendations on the HCQ of these two guidelines were formed merely on primary studies, not systematic reviews. Based on a systematic review, HCQ in monotherapy had similar (or even lower) efficacy compared to other csDMARDs. However, the combination of HCQ and other DMARDs would increase the clinical efficacy (Rempenault et al., 2020). Therefore, guideline developers should clearly describe the evidence on which the recommendations are based, the factors considered, and the process and methods used to form the recommendations. Therefore, clinicians might face a dilemma when treating RA with pharmacotherapy. Because following the recommendations in one guideline may violate the recommendations in other guidelines. Clinicians often give special attention to the strength of evidence, but we found that the strength of recommendation in the same PIC was inconsistent. Thus, it is essential to specify the circumstances in which the pharmacotherapy of RA should be used. These issues have complicated the implementation and application of guidelines and resulted in the disconnection between evidence and clinical practice. The inconsistency in CPGs indicated that the clinical questions were still unspecific and unclear, which needed to be supported by more studies. Therefore, we advocated paying more attention to controversial recommendations and updating the guidelines in time to improve the consistency.

There are many underlying reasons for the poor consistency among recommendations. First, the time span of the guideline development may cause heterogeneity because the emergence of new evidence may overturn old concepts or perspectives (Kow et al., 2021; Zhang et al., 2021). Therefore, the differences among the guidelines could be observed according to the publication year. Second, the lack of high-quality evidence may lead to formulating recommendations based on expert opinions rather than evidence. Further, the selection and interpretation of evidence may be inconsistent among guidelines. Third, different countries and regions consider local conditions, factors, values, preferences, and types and degrees of stakeholder participation differently. Fourth, different countries and regions may have quite different populations and population composition, and drug policies, which might impact the development of clinical recommendations. RA imposes a substantial economic burden and the expenditures for RA pharmacotherapy increase with severity (Santos-Moreno et al., 2021). Drug policy, clinical recommendations, and consensus conferences of different countries or regions should be considered judiciously during specific applications of innovative and expensive pharmacology. Fifth, different methods for grading the quality of evidence and strength of recommendations may form inconsistent recommendations based on the same evidence (Jolliffe et al., 2018). These reasons might lead to high-inconsistent recommendations. Further research should determine the reasons for the inconsistency.

The strengths of this study were as follows. First, to our knowledge, this is the first study that used the PIC framework to analyze the consistency of RA pharmacotherapy recommendations among CPGs. Second, we used a robust scientific PIC framework to encode the recommendations and analyzed the consistency among the recommendations in detail. Third, we focused on commonly-used and essential pharmacotherapy in the clinic. We also visualized the overall consistency among these recommendations, providing concise and detailed information for clinicians and researchers who may benefit from these findings while noting the existing gaps.

This study had some limitations as well. There may be some bias as the methods for grading the quality of evidence and strength of recommendations were different. The strength of recommendations was divided into strong and weak, and the highest degree of strength was coded as strong. The strength less than the highest degree was coded as weak according to the strength given in the guidelines. Local circumstances and drug policies vary among countries and should also be noticed, which may make recommendations different and create bias.

Developers need to refer to internationally recognized methods and standards for guideline development. First, developers should write a protocol and register before the guideline development work is carried out (Ge et al., 2018). Second, forming recommendations should adopt the currently recognized and more applicable methods for grading the quality of evidence and strength of recommendations, such as GRADE (Group, 2012; Dijkers, 2013). Further, the recommendations should be based on the highest quality of evidence, and their usage conditions should be clarified (Brignardello-Petersen et al., 2021). Accordingly, CPGs could be better implemented to provide evidence-based support for clinical practice and decision-making. However, the communications between CPG developers may be confused or hindered if the grading systems of evidence are different (Brozek et al., 2009; Sun et al., 2019). Third, high-quality evidence for RA should be produced via original research and systematic reviews in the future to develop RA guidelines and support the recommendations. Many factors need to be considered and balanced during the development of guidelines, so it is hard to maintain a high degree of consistency in the strength of the recommendations. Forth, systematic reviews of guidelines are suggested to conduct, indicating research gaps. Future studies on the consistency of recommendations among CPGs should pay attention to the following points. First, with the CPGs updated quickly, the publish time period of CPGs need to be considered. Second, whether CPGs are national or just developed by the single institution (or organization) and should be checked. It is inappropriate to analyze these CPGs together. Third, the availability of new drugs in the nations should be scrutinized. Because the local circumstances and drug policies are various. With the continuous revising and updating of the CPGs, we believe that high-quality guidelines could be developed in the future to form clear, definite, and consistent recommendations for the pharmacotherapy of RA.