Mitigating Disputes Originated by Multiple Discordant Systematic Reviews and Meta-Analyses: A Survey of Methodologists and Clinicians

Abstract

Background:

Overlapping systematic reviews (SRs) are increasingly frequent in the medical literature. They can easily generate discordant evidence, as estimates of effect sizes and their interpretation might differ from one source to another.

Objective:

To analyze how methodologists and clinicians make a decision when faced with discordant evidence formalized in structured tables.

Methods:

We conducted a 16-item survey exploring how methodologists and clinicians would react when presented with multiple Summary of Findings (SoF) tables (generated using the GRADE tool) derived from 4 overlapping and discordant SRs and meta-analyses on thrombolytic therapy for intermediate-risk pulmonary embolism. SoF tables reported 4 different magnitudes of effects and overall certainty. Participants were asked to provide their recommendations regarding the intervention and the reasons behind their conclusion.

Results:

Of the 80 invitees, 41 (51%) participated. The majority described themselves as “somewhat familiar” or experts with SoF tables. The majority recommended the therapy (pharmacological systemic thrombolysis), grading the recommendation as weak positive. Certainty of evidence and benefit-risk balance were the two criteria that prevailed in generating the recommendation. When faced with overlapping meta-analyses, the preferred approach was to use only high-quality SRs and exclude redundant SRs. Several participants suggested integrating the SoF tables with additional information, such as a more comprehensive evaluation of the risk of bias of systematic reviews (71%), heterogeneity/inconsistency (68%) and studies included within each SR (62%).

Conclusion:

When faced with multiple controversial SR results, the type and completeness of reported information in SoF tables affect experts' ability to make recommendations. Developers of the SoF table should consider collating key information from overlapping and potentially discordant reviews.

Introduction

In 1992 evidence-based medicine (EBM) was pronounced as a new approach for teaching and practicing clinical medicine (Evidence-Based Medicine Working Group, 1992). In the past 25 years, the EBM movement succeeded in promoting rigorous and explicit methods to identify, select and appraise relevant research studies. Systematic reviews (SRs), meta-analyses (MAs), clinical practice guidelines, health technology assessment and, more recently, overviews of systematic reviews (OSRs) are evidence synthesis designs that gained increasing popularity. SRs and MAs play a pivotal role, as they often inform other evidence synthesis designs. Guidelines, health technology assessment and OSRs are built systematically from the lower levels of evidence and provide substantially more useful information for guiding clinical decision-making (Alper and Haynes, 2016). For instance, many systematically derived recommendations rely on SRs that were either previously published or created de novo by guideline developers (Cook et al., 1997).

The number of SRs and MAs is increasing. For example, Hoffmann and colleagues observed a more than 20-fold increase in the number of SRs indexed on PubMed between 1990 and 2019 (Hoffmann et al., 2021). It has been suggested that the increasing number of redundant, unnecessary and misleading evidence syntheses, described by the claim “too much evidence,” is one of the reasons for a potential crisis of the EBM movement (Greenhalgh et al., 2014).

OSRs, as the next-generation overarching study design, have been proposed as a solution for the limited utility of multiple SRs (Ioannidis, 2017). OSRs are studies that allow synthesizing a large amount of evidence from multiple SRs and identifying gaps, weaknesses and biases that affect certain research fields (Puljak, 2017). However, one of the problems that OSRs can detect is the existence of discordant SRs, i.e., reviews that cover the same topic but reach different results and different conclusions (Riva et al., 2018; Augustin et al., 2020).

Summary of Findings (SoF) tables have been designed to summarize the key results of an SR or MA and evaluate the authors' confidence in the estimates of effect in evidence syntheses (Guyatt et al., 2011). However, it is unknown which information methodologists and clinicians prioritize when faced with discordant evidence from multiple SRs or MAs, and how their decision-making algorithm could be used to improve SoFs.

This study aimed to analyze how methodologists and clinicians decide when faced with discordant evidence and how the SoF table should be constructed in the event of overlapping MAs.

Methods

Ethics

The study protocol was approved by the Ethics Committee of the University of Split School of Medicine (Approval no. 2181-198-03-04-17-0064).

Study Design

This was a cross-sectional study.

Participants

This study was conducted during the year 2017 among clinicians considered opinion-leaders in the field of pulmonary embolism and methodologists skilled in EBM. We were targeting researchers with a high number of articles (i.e., more than 10) published on topics related to this survey (i.e., pulmonary embolism, systematic, reviews, SoF tables) and a high number of citations (i.e., more than 1,000), to ensure that these individuals are recognized experts in the field. Participants were recruited by using the snowballing method.

Discordant Evidence Scored by the Participants

We invited participants to complete a scenario-based survey. The participants were asked to identify the source of discordance between contrasting evidence on pharmacological systemic thrombolytic therapy for intermediate-risk pulmonary embolism and suggest how to incorporate them in developing recommendations. They were also asked to provide suggestions for an accurate and transparent reporting of legitimate controversies in the final guideline.

For this study, we prepared four SoF tables (Supplementary File 1) from our OSR about thrombolytic therapy for intermediate-risk pulmonary embolism using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) tool (Riva et al., 2018). We chose four SRs, purposively creating a sample of real SRs with different salient characteristics. Namely, we included the SR with the highest AMSTAR (A MeaSurement Tool to Assess systematic Reviews) score (Hao et al., 2015), the SR with the lowest AMSTAR score (Gao et al., 2015), the SR with the highest number of included trials (Chen and Ren, 2014) and the SR with the highest number of citations (Chatterjee et al., 2014). Four authors produced these SoF tables (LM, EP, NR, LP) in a way that each SoF was created independently by two authors, and then the results were compared and discrepancies discussed until they were resolved. Authors resolved discrepancies via discussion. Respondents did not receive information about the SRs from which the SoF tables were extracted.

Survey

We created a 16-item survey (available in Supplementary File 2) divided into three sections for this study. In the first section, there were questions about characteristics of the participants, including their sex, age, place of residence, personal description of their role and familiarity with an SoF table. In the second section, the participants were asked to make a decision based on the four presented SoFs, explaining how they reached their decision when faced with discordant evidence and how they would judge the quality of presented evidence. In the third section, there were general questions about decision-making in the event of discordant evidence from overlapping MAs and how SoF tables should be designed in the event of overlapping MAs.

Dissemination of the Survey

Prospective participants were invited by e-mail to participate in the study, which was available as an online survey at the SurveyMonkey platform (SurveyMonkey Inc., Palo Alto, CA, USA). The survey was set up as completely anonymous; it did not collect participants' internet protocol (IP) addresses. Participants received a maximum of 3 reminders before the survey was closed.

Statistics

Descriptive statistics with frequencies and percentages were used based on the built-in statistical features of SurveyMonkey. Characteristics of clinicians vs. methodologists were compared using the Chi Square or the Fisher's exact tests, as appropriate. The software STATA/BE version 17 (StataCorp LP, College Station, TX) was used for statistical analysis with two-tailed p < 0.05 considered statistically significant.

Results

Of the 80 invited participants, 41 (51%) accepted the invitation and participated in the study. A similar number of women (N = 20; 49%) and men participated. Most of the participants were aged 41 to 50. Most had a residence in Europe (29, 71%), and most of them were either methodologists (N = 22; 54%) or clinicians (N = 15; 37%). The majority of participants described themselves as ‘somewhat familiar' with the SoF tables (N = 15; 37%) or experts involved in GRADE methods and who can teach others about creating the SoF tables (N = 12; 29%) (Table 1).

Based on the evidence/SoF tables presented, most participants proposed that thrombolytic therapy (i.e., infusion into a vein of a medicine that will break up or dissolve blood clots, in addition to an anticoagulant medicine) “is suggested” or “might be suggested” in patients with pulmonary embolism at intermediate-risk of death at 30 days. They also mostly agreed that the grade for this recommendation was weak positive. Most of them (N = 27; 79%) based their recommendation on two or more SoF tables. They identified the most important outcome (i.e., mortality) and then considered that the quality of evidence for this outcome was marked in the preferred SoF tables as “high.” They reported that the certainty of the evidence and the benefit-risk balance were two criteria that most often guided their choice of recommendation (Table 2).

When faced with overlapping MAs, which may reach different conclusions, the participants suggested using only high-quality SRs and excluding redundant reviews as the most effective strategy in supporting guideline development (N = 25; 76% of the participants gave this option the highest priority mark) (Figure 1).

Figure 1

In case of overlapping MAs, in addition to the SoFs, additional information that the participants would like to have were: 1. the risk of bias in each review (measured with AMSTAR or a tool for assessing the risk of bias in systematic reviews—ROBIS) (N = 24; 71%), 2. the heterogeneity / consistency among the results of included studies/SRs (N = 23; 68%), and 3. the included studies within each SR (N = 21; 62%) (Figure 2). In addition, 41% of the participants indicated that they do not think that the actual form of the SoF table supports users in capturing differences across overlapping SRs (Figure 3). However, a minority did not consider overlapping SRs a problem deserving a dedicated approach in GRADE. Some participants indicated that it is not possible to improve the SoF to present evidence from overlapping reviews, mainly because clinical decisions should be based on one SR, which is the most up-to-date, or the one with the highest quality, accompanied by a clear justification for this choice (Table 3).

Figure 2

Figure 3

An open-ended question about ideas to improve the SoF in case of overlapping SRs yielded 19 responses (Table 3). Participants mostly provided suggestions for improving the SoF table by having the option to present the results of the overlapping SRs. Most of them suggested that various information should be added to the SoF, which may help in discerning reasons for discrepancies.

When we compared responses between clinicians and methodologists, we found that methodologists were more often experts or very familiar with the SoF. There were no significant differences in the type of recommendation these two groups provided based on the evidence presented and their grade of recommendation (Table 4). There was no difference in the frequency of the chosen criteria related to the SoF table, which guided the decision making. Methodologists more commonly indicated that they would need detailed information on the included studies within each systematic review to make a decision (Table 4).

We published raw data from this study on the Open Science Framework; the raw data table is available on the following link: https://osf.io/qsbyh/.

Discussion

We presented different SoF tables from overlapping SRs to methodologists and clinicians, who were somewhat familiar or experts with GRADE methodology and the development of SoF tables. Even though they addressed the same clinical question, i.e., systemic thrombolysis for intermediate-risk pulmonary embolism, the SoF tables reported results that were contradictory to some extent. Nevertheless, based on the quality of the SRs, participants were able to identify a recommendation-making process, giving preference to high-quality reviews. When faced with discordant evidence, they were more likely to formulate a weak recommendation. Additional information relevant to participants for supporting the clinical decision-making process were risk of bias, heterogeneity, and information about included studies in discordant SRs.

In 1997, the question of interpreting discordant results from similar SRs was first addressed (Jadad et al., 1997). It has been documented that escalation in the number of SRs increases the likelihood of finding conflicting evidence (Lucenteforte et al., 2015; Riva et al., 2018). These disputes between different SRs on the same topic may lead to the dissemination of inconsistent recommendations, slowing the transfer of research evidence into practice.

OSRs can easily identify cases of overlapping and discordant reviews. However, a recent analysis by Lunny and colleagues of 50 randomly sampled overviews showed that only a minority of them reported methods for handling overlapping reviews and discordant results and/or conclusions (Lunny et al., 2020). Another challenge for information users is also to appraise whether some of the discordant SRs meet the criteria of a SRas one or more may not be systematic in part or at all (Dettori and Norvell, 2020). Several typologies of literature reviews have been proposed (Par et al., 2015; Munn et al., 2018), contributing to different definitions of SR over time and across fields. The lack of a consensual definition of an SR and agreement on core methodological elements (e.g., identification of all relevant records, selection of eligible studies, assessment of the risk of bias, qualitative synthesis of the included studies, and meta-analyses when possible) can challenge an explicit and reproducible selection of SRs by researchers (Krnic Martinic et al., 2019).

This survey offers preliminary insights on concepts and practices among reviewers and health professionals who read SRs, or use them in the context of recommendations development when facing multiple sources of evidence. Based on our findings, we suggest that the structure of SoF tables should be complemented with a dedicated module of information to present users with details about overlapping and potentially discordant SRs. The purpose of such a tool would be to inform the reporting of overlapping evidence syntheses rather than provide a method to identify or classify discordant evidence. It could, however, be also used by reviewers, guideline developers and journal editors to plan for a comprehensive appraisal of evidence syntheses in situations where it is important to consider overlapping sources and conflicting results.

Using a new type of SoF table for OSRs, as a tool that would help collate the essential information about overlapping and potentially discordant reviews is likely to support clinical decision-making and more transparent guideline recommendations. As Bobbio and colleagues wrote in 1991 in a seminal paper in the Lancet about optimal reporting of trial results, “the key question in most therapeutic controversies is not whether the proposed treatment is effective or not, but how data should be presented in order to allow physicians and patients to decide for themselves” (Bobbio et al., 1994). Overall, a broader appreciation of the need for reporting findings from multiple sources, particularly when they reach high-quality thresholds, would be a welcome evolution in evidence synthesis science.

Our study has several potential limitations. There is no consensus definition of an opinion leader, which might have hampered our selection of experts that assessed the SoF tables. We used snowball sampling for the recruitment of participants. Such sampling is frequently used as a form of non-random sampling where a high external validity is not sought as prominent feature of a study (Parker et al., 2019). The initial sample relies on personal contacts. Thus, with this sampling strategy, we cannot generalize our results. In our study, we compared the responses of clinicians and methodologists. As the numbers of participants in each group were small, any difference should be interpreted as hypothesis-generating. Our study targeted a single medical question, namely thrombolytic therapy for intermediate-risk pulmonary embolism, again limiting external validity. Familiarity with the topic could have influenced the responses of some participants.

In conclusion, when faced with multiple controversial SR results, the type and completeness of reported information in SoF tables could affect experts' ability to make recommendations. Therefore, developers of the SoF table should consider collating the critical information from overlapping and potentially discordant reviews.

Funding

This work was supported by the Italian Ministry of Health (Giovani Ricercatori GR-2011-02348048) and by Regione Lombardia (R.L. d.g.r. n. IX/4662, 9/01/2013).

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

• 1

  AlperB. S.HaynesR. B. (2016). EBHC pyramid 5.0 for accessing preappraised evidence and guidance. Evid. Based Med. 21, 123–125. 10.1136/ebmed-2016-110447

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 2

  AugustinG.BoricM.BarcotO.PuljakL. (2020). Discordant outcomes of laparoscopic versus open appendectomy for suspected appendicitis during pregnancy in published meta-analyses: an overview of systematic reviews. Surg. Endosc. 34, 4245–4256. 10.1007/s00464-020-07674-6

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 3

  BobbioM.DemichelisB.GiustettoG. (1994). Completeness of reporting trial results: effect on physicians' willingness to prescribe. Lancet343, 1209–1211. 10.1016/S0140-6736(94)92407-4

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 4

  ChatterjeeS.ChakrabortyA.WeinbergI.KadakiaM.WilenskyR. L.SardarP.et al. (2014). Thrombolysis for pulmonary embolism and risk of all-cause mortality, major bleeding, and intracranial hemorrhage: a meta-analysis. JAMA311, 2414–2421. 10.1001/jama.2014.5990

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 5

  ChenH.RenC. (2014). Thrombolysis versus anticoagulation for the initial treatment of moderate pulmonary embolism: a meta-analysis of randomized controlled trials. Respir. Care59, 1880–1887. 10.4187/respcare.03197

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 6

  CookD. J.GreengoldN. L.EllrodtA. G.WeingartenS. R. (1997). The relation between systematic reviews and practice guidelines. Ann. Intern. Med. 127, 210–216. 10.7326/0003-4819-127-3-199708010-00006

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 7

  DettoriJ. R.NorvellD. C. (2020). Discordant systematic reviews: which to believe?Global Spine J. 10, 237–239. 10.1177/2192568219899945

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 8

  Evidence-Based Medicine Working Group (1992). Evidence-based medicine. A new approach to teaching the practice of medicine. JAMA268, 2420–2425. 10.1001/jama.268.17.2420

  • CrossRef
  • Google Scholar

• 9

  GaoG. Y.YangP.LiuM.DingM.LiuG. H.TongY. L.et al. (2015). Thrombolysis for acute intermediate-risk pulmonary embolism: a meta-analysis. Thromb. Res. 136, 932–937. 10.1016/j.thromres.2015.09.012

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 10

  GreenhalghT.HowickJ.MaskreyN. (2014). Evidence based medicine: a movement in crisis?BMJ348, g3725. 10.1136/bmj.g3725

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 11

  GuyattG.OxmanA. D.AklE. A.KunzR.VistG.BrozekJ.et al. (2011). GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J. Clin. Epidemiol. 64, 383–394. 10.1016/j.jclinepi.2010.04.026

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 12

  HaoQ.DongB. R.YueJ.WuT.LiuG. J. (2015). Thrombolytic therapy for pulmonary embolism. Cochrane Database Syst Rev. 12, CD004437. 10.1002/14651858.CD004437.pub4

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 13

  HoffmannF.AllersK.RombeyT.HelbachJ.HoffmannA.MathesT.et al. (2021). Nearly 80 systematic reviews were published each day: observational study on trends in epidemiology and reporting over the years 2000-2019. J. Clin. Epidemiol. 138, 1–11. 10.1016/j.jclinepi.2021.05.022

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 14

  IoannidisJ (2017). Next-generation systematic reviews: prospective meta-analysis, individual-level data, networks and umbrella reviews. Br. J. Sports Med. 51, 1456–1458. 10.1136/bjsports-2017-097621

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 15

  JadadA. R.CookD. J.BrowmanG. P. (1997). A guide to interpreting discordant systematic reviews. CMAJ156, 1411–1416.

  • Pubmed Abstract
  • Google Scholar

• 16

  Krnic MartinicM.PieperD.GlattA.PuljakL. (2019). Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Med. Res. Methodol. 19, 203. 10.1186/s12874-019-0855-0

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 17

  LucenteforteE.MojaL.PecoraroV.ContiA. A.ContiA.CrudeliE.et al. (2015). Discordances originated by multiple meta-analyses on interventions for myocardial infarction: a systematic review. J. Clin. Epidemiol. 68, 246–256. 10.1016/j.jclinepi.2014.11.004

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 18

  LunnyC.BrennanS. E.ReidJ.McDonaldS.McKenzieJ. E. (2020). Overviews of reviews incompletely report methods for handling overlapping, discordant, and problematic data. J. Clin. Epidemiol. 118, 69–85. 10.1016/j.jclinepi.2019.09.025

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 19

  MunnZ.SternC.AromatarisE.LockwoodC.JordanZ. (2018). What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC Med. Res. Methodol. 18, 5. 10.1186/s12874-017-0468-4

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 20

  Par,éGTrudelM.-C.JaanaM.KitsiouS. (2015). Synthesizing information systems knowledge: A typology of literature reviews. Information Manage.52, 183–199. 10.1016/j.im.2014.08.008

  • CrossRef
  • Google Scholar

• 21

  ParkerC.ScottS.GeddesA. (2019). “Snowball sampling,” in SAGE Research Methods Foundations, eds P. Atkinson, S. Delamont, A. Cernat, J. W. Sakshaug, and R. A. Williams (London: Sage Publications). 10.4135/9781526421036831710

  • CrossRef
  • Google Scholar

• 22

  PuljakL (2017). If there is only one author or only one database was searched, a study should not be called a systematic review. J. Clin. Epidemiol. 91, 4–5. 10.1016/j.jclinepi.2017.08.002

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 23

  RivaN.PuljakL.MojaL.AgenoW.SchunemannH.MagriniN.et al. (2018). Multiple overlapping systematic reviews facilitate the origin of disputes: the case of thrombolytic therapy for pulmonary embolism. J. Clin. Epidemiol. 97, 1–13. 10.1016/j.jclinepi.2017.11.012

  • Pubmed Abstract
  • CrossRef
  • Google Scholar