Quality evaluation of metabolic and bariatric surgical guidelines

Objective To evaluate the quality of surgical guidelines on bariatric/metabolic surgery. Methods Four independent reviewers used the AGREE II (The Appraisal of Guidelines for Research and Evaluation II) tool to assess the methodological quality of the included guidelines and conducted a comparative analysis of the main recommendations for surgical methods of these guidelines. Results Nine surgical guidelines were included in this study. Five articles with AGREE II scores over 60% are worthy of clinical recommendation. The field of rigor of development was relatively low, with an average score of 50.82%. Among 15 key recommendations and the corresponding best evidence in the guidelines, only 4 key recommendations were grade A recommendations. Conclusions The quality of metabolic and bariatric guidelines is uneven, and there is much room for improvement.

simultaneously (4, 5). Obesity and overweight can lead to shortened life expectancy and lower quality of life (6).
Although some new drugs (7) and diet and exercise programs (8) have been developed to counteract the continuous increase in the obesity rate, the above measures are still not ideal for controlling obesity. Bariatric surgery has been proven to be an effective method for the control of morbid obesity and metabolic syndrome (3) and has also received attention from international academic groups and experts on bariatric/metabolic surgery. Currently, metabolic and weight loss surgical guidelines have been developed (9)(10)(11)(12)(13)(14)(15)(16)(17). However, the methodological quality of these guidelines and the heterogeneity of the main recommendations have caused great confusion to the users of these guidelines. Therefore, the purpose of this study was to analyse the methodological quality of the guidelines for bariatric/metabolic surgery and the differences in recommendations between these guidelines and to provide the best evidence to help guideline users in choosing an appropriate guideline and to inform guideline developers when they update them.

Research design
In this study, the AGREE II tool was adopted to conduct methodological evaluation of clinical guidelines for metabolism and weight loss, and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) principles were followed (18).

Retrieval strategy
In this study, metabolism and weight loss surgery-related guidelines were retrieved from PubMed, Ovid, Springer, Web of Science, CNKI, VIP database, Wanfang Database and other databases. This study examined relevant guidelines published from January 1, 2014, to January 1, 2021, that included supporting evidence for the main recommendations included in the guidelines and the impact of time span on evidence updates. This study also searched Google and Baidu Academics to obtain more guidelines. The language was limited to English and Chinese. Keywords included the Chinese and English keywords "bariatric surgery", "metabolic surgery" and "guidelines". Additionally, the reference lists of the included guidelines were manually searched.

Selection principle of guidelines 2.3.1 Inclusion criteria
(1) complete guideline text (2); the guidelines include information about metabolism and bariatric surgery (3); if the guidelines are updated, only the latest version will be included (4); guidelines published in English or Chinese.

Exclusion criteria
(1) duplicate guidelines, translated versions of the guidelines, secondary or multiple publications, and brief abstracts (2); the translated versions of the guidelines may have lost the information of the original versions, which may have affected the accuracy of the evaluation of this study (3); if multiple guidelines were issued by the same organisation, older versions of the guidelines were excluded.
According to the above inclusion and exclusion criteria, two reviewers (Xiao Wang and Yu-Lu Du) independently evaluated the obtained literature to determine whether to include or exclude the literature. Disagreements were resolved through negotiation until consensus or by consulting a third expert reviewer (Ya-Qi Zheng).

Quality assessment of the guidelines 2.4.1 AGREE II tools
This research used the latest version of the AGREE II tool (https://www.agreetrust.org/resource-centre/) to assess each bariatric/metabolic guideline that met the standards of this study. This tool consists of 6 domains and 23 items. (Table S1).
In this study, each bariatric/metabolic guideline was graded according to the AGREE II user manual by four independent reviewers (Zi-Han Qin, Xin Yang, Li-Ya An and Ting Yang). The reviewers were trained on the use of the AGREE II tool through a rigorous online training course on the AGREE website. Reviewers are guided and supervised by experts (Da-Li Sun, Yue-Ying Lin, Li-Ya An and Ting Yang) who have published a number of articles through the use of AGREEII. The team includes experienced specialists in bariatric/metabolic surgery (Da-Li Sun, Yue-Ying Lin, Li-Ya An and Ting Yang). The user manual defines each item and helps the user determine the guideline score for the item. Items are rated on a scale of 1 (completely inconsistent with the item) to 7 (completely consistent with the item). Domain scores are calculated by adding up the project scores for each domain for each reviewer, then normalizing them to the percentage of the highest score (19).
For each area of the AGREE II tool, "points earned" is calculated as the sum of all points scored by the grader for all items contained in that area. The "proportional domain score" is calculated as a standardized score using the following formula: (score obtained − lowest possible score)/(maximum possible score − lowest possible score). The maximum score for each area is obtained by multiplying the number of items in that area by the number of raters, multiplying by 7 (which corresponds to "strong agreement"). The minimum score is obtained by multiplying the number of items in the field by the number of raters and multiplying by 1 (which corresponds to "strong disagreement").

Guidelines for extracting and regrading key recommendations and best evidence
In this study, the relatively high AGREE II scoring guidelines were used to extract and analyse important recommendations related to metabolism and bariatric surgery, and a database search was conducted to further obtain the highest level of evidence supporting these recommendations. This study reclassified recommendations and evidence using the Oxford Centre for Evidence-based Medicine (OCEBM) grading system. (Table S2).

Statistical analysis
We used a descriptive statistical analysis method to calculate the standardized scores for each guideline, which were expressed as a percentage, and we also listed the median scores and the range of each domain. We adopted a two-way ANOVA to calculate the intra-class correlation coefficients (ICCs) to examine the agreement among the scores from the four reviewers. Consistency among raters was determined by ICCs and 95% confidence intervals (CIs). ICC is equal to the individual variation divided by the total variation, with a value between 0 and 1 for the 23 items identified in AGREE II. If the ICC is between 0.01 and 0.20, the degree of agreement is considered to be slight; if the ICC is between 0.21 and 0.40, the consistency is considered fair. If the ICC is between 0.41 and 0.60, the consistency is moderate. If the ICC is between 0.61-0.80, the degree of agreement is considered to be high; if the ICC is between 0.81-1.00, it is considered perfect. P < 0.05 indicates statistical significance. Statistical analysis was performed using IBM SPSS Version 19.0 (SPSS Inc., Chicago, IL, USA). (Table S3).

Guideline features
A total of 264 records were obtained through database retrieval and other retrieval methods, which were evaluated by reading titles, abstracts and full texts. Finally, 9 guidelines for bariatric/metabolic surgery were included (Figure 1), 8 of which are original (9-14, 16, 17) and 1 of which was updated in 2020 (15). All guidelines were developed by local or national medical associations. Seven of the guidelines are international or from > 1 country (9)(10)(11)(12)(13)(14)16), one is from the EU (15), and one is from India (17). One of the nine guidelines is for children and adolescents (10), and the other eight are for adults (9,(11)(12)(13)(14)(15)(16)(17). The basic features of the included guidelines are listed in Table 1.

Quality evaluation of the guidelines
The AGREE II standardized area scores for each metabolic and bariatric/metabolic surgery recommendation guideline and their overall recommendations are shown in Table 2. Domain scope and purpose and domain clarity and presentation methods had the highest median scores of 84.09% and 76.70%, respectively (range 77.78% to 91.67% and 62.50% to 91.67%, respectively). The median score for domain stakeholder involvement was 60.03% (range 47.22% to 86.11%). Only one guideline (11%) scored less than 50% (13). Editorial independence of the domain had the highest score range (0 to 100%). Two guidelines scored zero (9,12), and three guidelines scored less than 50% (9,10,12). The median scores for applicability and rigor of development were 45.95% and 50.82% (ranging from 27.08% to 78.13% and 20.83% to 77.08%, respectively). According to the overall score of the guidelines, the five included guidelines scored well in all areas (10-12, 14, 15) and were classified as strongly recommended for clinical practice. Four guidelines were recommended for revisions (9,13,16,17).
In this study, four evaluators participated in the evaluation of the guidelines, and the ICC value range of the metabolic and bariatric surgery guidelines was > 0.8 using AGREE II, indicating a high degree of internal item score consistency among reviewers.

Key recommendations in the guidelines and the best available evidence
To further analyse the reasons for the heterogeneity of recommendations for metabolism and bariatric surgery among different guidelines, this study referred to guidelines (15) with relatively high scores and relatively clear recommendation items, extracted the main recommendations in the guidelines, and sorted out the highest evidence supporting these main recommendations by searching the database. Additionally, recommendations (9-11, 14, 16) not included in the high quality guideline (15) are also sorted (Table 3). It mainly includes surgical indications, surgical methods, and preoperative and postoperative recommendations (Table 3).

Discussion
Compared to other disease-specific guidelines, the development of guidelines for metabolism and bariatric surgery can be a complex Flowchart of study selection. issue, as metabolism and bariatric surgery is a multidisciplinary, global issue. In this study, even within the same guidelines, the quality of metabolic and bariatric surgery guidelines was highly heterogeneous among different fields, and there were significant differences in the distribution of evidence level and recommendation strength between different categories of guidelines.
Analysing the included guidelines, the study identified several areas where improvement was needed in the development of the guidelines. In the development of guidelines, patient perceptions, expectations and preferences for medical care have become increasingly important. Stakeholder participation can well reflect the views of prospective users and patients. The implementation of the guidelines also requires multidisciplinary medical expertise. However, the guidelines included in this study did not provide details about the involvement of patients and their representatives.
Rigour of development is closely related to credibility in the implementation of the guidelines. This field assessment is used to locate and synthesize evidence and to develop and update recommendations (35). Unsystematic methods of retrieving evidence tend to lead to low-quality guidelines (11,16,17). The lack of clear evidence selection criteria and their strengths and limitations (9,16) also lead to the low quality of the guidelines. Other causes include vague connection between recommendations and evidence, lack of external evaluation, failure to provide guideline update steps (9,11,16), and failure to consider side effects and risks when forming recommendations (17).
All nine guidelines included in the study scored > 60% for clarity and presentation. This shows that the recommendations in the guidelines are clear and easy to identify.
The overall score for the applicability of the guidelines was low in the included guidelines. This indicates that the hindrance and promotion factors in the application of the guidelines have not been fully understood in the formulation of the guidelines (16). No recommendations or tools are provided in the application to ensure its feasibility (9,17). Additionally, the neglect of relevant resources (9,11,12,(14)(15)(16)(17)) that may be needed in the application of recommendations and the lack of monitoring and auditing standards (9,11,13,14,17) are also important reasons for the low score of the guidelines in the application field.
In the area of editorial independence of the guideline. The influence of sponsors' views on the guideline and the conflicts of interest between members of the organisations involved in the development of the guideline are rarely mentioned (9,12). Therefore, conflicts of interest among members should be clearly recorded and publicized in the formulation of the guidelines to improve their independence.
There are significant differences in recommendations in the included guidelines. Therefore, this study further analysed the consistency and controversy between current recommendations and corresponding evidence of metabolic and bariatric surgery guidelines with reference to key recommendations from the guidelines for metabolic and bariatric surgery with relatively high scores.  (20) For this recommendation, only 2 guidelines (15, 17) are relatively consistent in this recommendation, and the other 7 guidelines (9-14, 16) do not specify this recommendation. However, there is currently a lack high-quality empirical evidence, and the best evidence derives from a randomized controlled trial of 57 patients. The main conclusion is that surgery is very effective in the short term for patients with T2DM and obesity (20).

(2)
Bariatric/metabolic surgery should be considered for patients with BMI≥ 30 kg/m 2 and type 2 diabetes with poor control despite optimal medical therapy (recommendation strength: A; evidence level: 1a) (21) For this recommendation, four guidelines (14-17) are relatively consistent in this regard, while the other five guidelines (9-13) do not explicitly state this recommendation. The best evidence to date is a systematic review of 11 randomized controlled trials, with the main conclusion that bariatric/metabolic surgery is more effective than various medical/lifestyle interventions in reducing body weight, controlling blood glucose, alleviating T2DM, and improving other cardiovascular disease risk factors (21).

(3)
Bariatric/metabolic surgery should be considered for patients with BMI≥ 30 kg/m 2 and obesity-related comorbidities that cannot lose enough weight through nonsurgical treatment (recommendation strength: B; evidence level: 2b) (22) For this recommendation, two guidelines (12,14) are relatively consistent in this regard, while the other seven guidelines (9-11, 13, 15-17) do not explicitly state this recommendation. There is a lack of high-quality evidence from a large sample. The best available evidence is a randomized controlled trial involving 80 patients who shows that surgical treatment was statistically significant compared with nonsurgical treatment in terms of weight loss, solving metabolic syndrome and improving quality of life for adults with mild to moderate obesity (body mass index, 30 kg/m 2 to 35 kg/ m 2 ) (22). Three guidelines [9, 10 and 15] agree on this recommendation. The other six guidelines (11-14, 16, 17) do not specify this recommendation. There is still a lack of high-quality research evidence with a large sample, and the best evidence at present is a cohort study of 71 patients, which shows that AGB surgery is inferior to SG surgery in weight loss (23).

(2) RYGB should be preferred over AGB (recommendation strength: B; evidence level: 2b) (24)
Two guidelines (10,15) agree on this recommendation. The other seven guidelines (9, 11-14, 16, 17) do not explicitly state this recommendation. Currently, there is a lack of large-scale, highquality randomized controlled studies, and the best evidence is based on a cohort study of 1295 patients in whom RYGB has a lower incidence of long-term complications than AGB (24).

(3) OAGB may offer greater short-term weight loss than SG (recommendation strength: B; evidence level: 2b) (25)
For this recommendation, two guidelines (15,16) have agreed on this recommendation. The other seven guidelines (9-14, 17) do not specify this recommendation. Currently, there is a lack of highquality research evidence, and the best evidence is a cohort study involving 123 patients. The main conclusion shows that the two surgery methods have excellent weight loss and maintenance effects in the short and medium term, and the results of T2D and HTN after OAGB are better (25).

(4)
OAGB may offer greater short-term weight loss than RYGB (recommendation strength: A; evidence level: 1b) (26) For this recommendation, two guidelines (15,16) have agreed on this recommendation. The other seven guidelines (9-14, 17) do not specify this recommendation. The best evidence to date is a 253-patient randomized controlled trial that shows that the OAGB group has better short-term weight loss (26).

(5) RYGB is an acceptable revisional bariatric surgery option after AGB (recommendation strength: C; evidence level: 4) (27)
Two guidelines (10, 11) agree on this recommendation. There is currently a lack of related high-quality RCT studies, and the best evidence thus far comes from a case series analysis of 58 patients, which found that RYGB is a safe operation with good weight loss within 5 years. It can be regarded as a good revision operation after failure of AGB (27). 4.2.6 (6) BPD/DS and SADIs are acceptable revisional bariatric surgery options after SG (recommendation strength: C; evidence level: 4) (28, 29) The best evidence to date comes from a case-analysis study of 96 patients, which found that BPD/DS is a safe and effective option after initial SG failure, especially in patients with severe obesity before SG. SADI-S results in a more significant reduction in overall weight than RYGB after failure of SG (28, 29). 4.2.7 (7) RYGB and BPD/DS are acceptable surgical options for patients with GERD after SG surgery, and BPD/DS is better than RYGB (recommendation strength: C; evidence level: 4) (28, 30) Two guidelines (9, 13) agree on this recommendation, and the best evidence thus far comes from a case study of 10 patients. The main conclusion is that RYGB is an effective treatment for BE and reflux after SG, and RYGB alleviates BE and reflux in most cases (30).   3 The key recommendations and the best evidence to support the recommendations at present.

The key recommendations
The best evidence to support the recommendations at present Four guidelines (9)(10)(11)15) have consistent opinions on this article. The other five guidelines (12-14, 16, 17) do not specify this recommendation. At present, the best evidence comes from a randomized controlled study involving 120 obese patients (31), and the results suggest that proper nutritional assessment and preoperative preparation of a balanced energy diet for morbidly obese patients can reduce the risk of surgery and improve efficacy.

(2)
Psychological evaluation can be considered before bariatric/metabolic surgery (recommendation strength: B; evidence level: 2b) (32) Two guidelines (11,15) recommend psychological assessment. The other seven guidelines (9, 10, 12-14, 16, 17) do not explicitly state this recommendation. The best evidence to date comes from a cohort study of 2,458 patients, which showed that measures such as preoperative psychological evaluation can improve the prevalence of alcohol use disorder after bariatric/metabolic surgery (32).

(2)
Postoperative behavioural advice should be provided to patients undergoing bariatric/ metabolic surgery (recommendation strength: A; evidence level: 1b) (34) For this proposal, two guidelines (10, 15) are consistent. The guidelines recommend providing motivation intervention for postoperative patients through a randomized controlled trial of 144 obese patients. The result indicates that participants accept behavioural intervention based on the scores on the Beck depression rating scale being significantly lower than those among standard treatment participants (34).

(3) Pregnancy after bariatric/metabolic
surgery should be delayed during the weight loss phase (recommendation strength: D; evidence level: 5) (15) Three guidelines (10,15,17) are consistent in this recommendation. At present, the best evidence comes from  expert opinions (15), with a low level of evidence and a lack of highquality randomized controlled studies. In summary, suggestions for improving the quality of metabolic and bariatric surgery guidelines are as follows: (1) Developers should set different groups (including patients and the public, etc.) clearly and fully consider the views and wishes of target groups when formulating clinical guidelines. (2) In the formulation, evidence standards should be clearly described, the link between recommendations and evidence should be shown clearly, and update steps should be provided. (3) Guideline developers should be familiar with guideline development standards, such as the AGREE II tool. (4) The guideline shall be externally reviewed by experts before publication. (5) Most of the key recommendations for metabolic and bariatric surgery are not supported by high-quality research evidence. It is recommended that international academic groups on metabolic and bariatric surgery organize and carry out multicentre high-quality research to provide high-quality evidence for the key recommendations on metabolic and bariatric surgery.
This study has some advantages and limitations. The advantages of this study are as follows: (1) This study collated and analysed the key recommendations and relevant evidence in the recent guidelines for metabolic and bariatric surgery. This study identified issues of recommendations and evidence related to metabolic and bariatric surgery and suggests improvements that may help guideline makers and users identify gaps in practice and provide a reference for guideline users to select more reliable guidelines. (2) Most of the developers of the guidelines included in this study come from international organisations and from different backgrounds, including clinical experts and methodologists, who have rich experience in developing clinical guidelines, which improves the reliability of the results of this study.
The limitations of this study are as follows: (1) In this study, only the guidelines written in English were evaluated, and the guidelines published in other languages may be missed in this study, resulting in inadequate representation of some less developed countries. (2) AGREE II is a methodological tool that does not evaluate the content and clinical significance of guidelines and focuses mainly on the formulation of guidelines. Thus, even if the guidelines are based on low-quality evidence, but the methodology is developed in compliance with AGREE II standards, the guidelines may still be scored highly by the AGREE II tool. (3) We decided to include all populations without setting age limits, but only one remaining article deals with the population of children and adolescents. This may be due to the lack of literatures and guidelines.

Conclusion
The quality of metabolic and bariatric surgical guidelines varies visibly. High-quality guidelines require multidisciplinary collaboration. Using the AGREE II tool, this study found significant room for improvement in the guidelines for metabolic and bariatric surgery, especially in terms of rigor, stakeholders, adaptability, and independence of guideline development. Effectively addressing these issues has vital implications for developing high-quality recommendations for metabolic and bariatric surgery guidelines.