Potential limitations in systematic review studies assessing the effect of the main intervention for treatment/therapy of COVID-19 patients: An overview

Background Although several studies have assessed the safety, efficacy, and effectiveness of interventions in treating the COVID-19, many of them have limitations that can have an immense impact on their results. This study aims to assess the potential limitations in systematic reviews (SRs) that evaluate the effect of interventions on the treatment of the COVID-19. Methods PubMed, Scopus, and Web of Sciences (WOS) databases were searched from inception to January 1, 2022. All systematic reviews investigated the effectiveness, efficacy, safety, and outcome of the main intervention (Favipiravir, Remdesivir, Hydroxychloroquine, Ivermectin, Lopinavir/Ritonavir, or Tocilizumab) for the treatment of COVID-19 patients and reported the potential limitations of the included studies. We assessed the quality of the included studies using the Quality Assessment Tool (QAT) for review articles. We conducted a content analysis and prepared a narrative summary of the limitations. Results Forty-six studies were included in this review. Ninety one percent of the included studies scored as strong quality and the remaining (9%) as moderate quality. Only 29.7% of the included systematic reviews have a registered protocol. 26% of the included studies mentioned a funding statement. The main limitations of the included studies were categorized in 10 domains: sample size, heterogeneity, follow-up, treatment, including studies, design, definitions, synthesis, quality, and search. Conclusion Various limitations have been reported in all the included studies. Indeed, the existence of limitations in studies can affect their results, therefore, identifying these limitations can help researchers design better studies. As a result, stronger studies with more reliable results will be reported and disseminated. Further research on COVID-19 SRs is essential to improve research quality and also, efficiency among scientists across the world.

a result, stronger studies with more reliable results will be reported and disseminated. Further research on COVID-19 SRs is essential to improve research quality and also, efficiency among scientists across the world. KEYWORDS COVID-19, systematic review, limitations, intervention, treatment

Background
The COVID-19 pandemic began in early 2020 with major health consequences (1). According to live data from Worldometer website, the total number of coronavirus cases and the number of deaths so far is 595,494,252 and 6,455,301, respectively (Tue, 16 Aug 2022). Numerous studies have assessed the effects of the different interventions on the treatment of the COVID-19 patients (2)(3)(4)(5)(6). These studies differ in many ways, including the type of treatment, follow-up time, study design, patient type, and disease severity, each of which can have a positive or negative effect on the results of these studies (7).
As the global community eagerly awaits credible scientific solutions for this pandemic, researchers and scientists are under much pressure to identify effective therapeutic and preventive strategies for COVID-19. Also, there are many unknowns, and the massive demand for evidence on the treatment of a novel disease such as COVID-19 may be unintentionally affecting studies' design and conduct. Furthermore, it may inadvertently affect the peer-review and publication process, leading to significant methodology gaps and overall lower quality evidence on COVID-19. These gaps lead to less-informative studies, loss of precious time, and valuable resources (8).
With the growth of evidence in this area (9), there is a need for studies that report the results of these individual studies in general. Systematic reviews objectively summarize large amounts of information, identifying gaps in medical research, and identifying beneficial or harmful interventions which will be useful for clinicians, researchers, and even for public and policymakers. The value of a systematic review depends on what was done, what was found, and the clarity of reporting (10). The results of a systematic review are influenced by the quality of the primary studies included. Methodologically, poor studies tend to exaggerate the overall estimate of treatment effect and may lead to incorrect inferences (11).
While a need to disseminate information to the medical community and general public was paramount, concerns have been raised regarding the scientific rigor, quality, and limitations in published reports which may potentially effect on the systematic reviews and meta-analysis results (1). In this study, we aim to identify the potential limitations in systematic reviews that evaluated the effect of interventions on the treatment of the COVID-19 which can help to improve and make the result of studies more accurate in the future.

Methodology Protocol and registration
We conducted this overview based on Smith et al. guideline for conducting a systematic review of systematic reviews of healthcare interventions (12). We also followed the PRISMA guideline for reporting the methods and results of this study (13).

Eligibility criteria
All systematic reviews with available full text and in EN languages investigated the effectiveness, efficacy, safety, and outcome of the main intervention (Favipiravir, Remdesivir, Hydroxychloroquine, Ivermectin, Lopinavir/Ritonavir, or Tocilizumab) for treatment of COVID-19 patients and reported the potential limitation of the study were included.
We exclude articles that are full-text not available or used other treatment options than mentioned drugs. For example, acupuncture or traditional medicine, or supplement therapy. Preprint and without peer review articles also was excluded.

Information sources and search strategy
We searched PubMed, Scopus, and Web of Sciences (WOS) databases from inception to January 1, 2022, for the keywords COVID-19, "SARS-CoV-2, " "novel coronavirus, " "systematic review, " OR limitation in the title, abstract, or main text of the published article. There was no limitation regarding time or language. We also conducted a manual search in Google Scholar for potential missing articles. In addition to database searches, we screened reference lists of included studies after screening records were retrieved via databases and also contacted the corresponding authors of the included studies. The full search strategy for all databases is presented in Supplementary

Selection process
After the search was completed, all retrieved records were imported in EndNote, version X7, and duplicate removed. Two independent reviewers (HA, MM) screened the records based on the title, abstract, and full text. For increasing the agreement between reviewers we piloted a set of 30 studies before the screening. Discrepancies at this stage were resolved by consensus with a third reviewer (MA-Z).

Data collection process and data items
Two independent reviewers (HA, MM) extracted the data. We designed a data extraction table for this study, which was piloted by two reviewers (5 studies). we extracted the following data: first author name, corresponding authors name and email, Publication year, number of authors, study design, number of included studies in each included systematic review, investigated drug, country, language limitation, time of the search, number of the investigated outcome, sample size, limitations, funding statement, mean age, gender (%), protocol and registration information. Discrepancies at this stage were resolved by consensus with a third reviewer (MA-Z).

Quality appraisal
Two reviewers (HA and MM) independently assessed the quality of the included studies. We assessed the quality of the included studies using the Quality Assessment Tool (QAT) for review articles developed by healthevidence.org, which was piloted by two reviewers (5 studies) including ten quality criteria. A final review quality rating for each review is assigned: strong (8-10/10), moderate (5-7/10), or weak (1-4/10). Any discrepancies were resolved upon consultation with a third reviewer (MA-Z).

Synthesis of results
For data synthesis, we prepared a table summarizing systematic review information. We also used graphs for presenting some information. We then conducted a content analysis and prepared a narrative summary of the limitations. Two authors (HA, MM) read and reread the results reported in published articles to extract limitations. The coding frame and final categories were developed by 3 authors (HA, MM, and MA-Z) using these data.

Study selection
A total of 773 records were retrieved from the database search. After removing duplicates, 525 records were screened by title, abstract, and full-text based on eligibility criteria, of which 46 studies were included in the final review (14-58, 59). Twenty-seven studies were excluded after Full-text screening. The reasons for exclusion were as follows: Protocol (5 records), Preprint (6 records), Full-text not available after contacting the corresponding author (2 records), Not reporting limitation (5 records), and not investigating our target intervention (9 records). The PRISMA flow diagram for the complete study selection process is presented in Figure 1.

Study characteristics
The number of authors of the included systematic reviews varied between 3 and 58 people. Most studies were from Asia (46%), America (31%), and Europe (17%). Also, by country, most studies were reported from the United States and India (Figure 2). 80.4% of the included systematic review conducted a meta-analysis. The number of included studies in the included systematic reviews varied between 2 and 136. Only 29.7% of the included systematic reviews have a registered protocol. Also, 26% of the included systematic reviews mentioned a funding statement. More details about the characteristics of included systematic reviews are presented in Table 1. The most studied drug in the included studies was Remdesivir (17.37%) (Figure 3).

Quality appraisal
The overall mean quality score of the included studies was 9.5. Overall, 91% of the included studies were scored as strong quality and the remaining (9%) as moderate quality. The overall scores ranged between 7 and 10. About 74% of the included studies had a score of 10, 11% had a score of 9, 6% had a score of 8, and the remaining (9%) had a score of 7 (Figure 4) (for more details about items on the QAT checklist see Supplementary Table 2).

Results of synthesis Potential limitations of the included studies
Various studies have listed different limitations for the studies, some related to how the systematic review was conducted and some related to the studies included in these Frontiers in Medicine 03 frontiersin.org systematic reviews. The main limitations of the included studies were categorized in 10 domains: Heterogeneity (4 sub-categories), sample size (2 sub-categories), follow-up (2 sub-categories), treatment (7 sub-categories), included studies (4 sub-categories), design (10sub-categories), definitions (3 sub-categories), synthesis (4 sub-categories), quality (2 sub-categories), and search (4 sub-categories). The highest frequencies reported in the included studies related to the heterogeneity in sample population, small sample size, and database searches. Heterogeneity in studies has been reported for a variety of reasons, including differences in the sample population regarding age, gender, ethnicity, and racial groups; different level of disease severity in the included patients; different control group; and difference in the investigated outcome. Treatmentrelated limitations were mostly related to differences regarding the administration of drug, dose, duration of treatment, and different standard protocols and guidelines. Also, there are differences related to discontinuation, combination therapy, and supportive care which obscure the effect of the main treatment.
The studies had several design shortcomings. Many studies have suffered from a lack of randomization, placebo, blinding, and comparator arm. Selection bias, and publication bias, confounding bias were also reported in the studies. Also, different strategies regarding search were another limitation. Different databases, using pre-print and un-published data, limitations on language, and missing some eligible studies were the important limitation in this regard.
More details about the potential limitations of the included systematic reviews are presented in Table 2.

Discussion
With the spread of the COVID-19 pandemic and has many consequences, the need arose to conduct studies and disseminate their findings (1).
Systematic reviews are a valuable resource in academia and practice. Well-done systematic reviews, which include but are not limited to meta-analyses, offer an efficient way to evaluate a large amount of information for decision-makers in areas of research, policy, and patient care. Systematic reviews can help us know what we know about a topic, and what is not yet known, often to a greater extent than the findings of a single study (60)(61)(62)(63)(64). Systemic review studies on the safety and efficacy of COVID-19 have grown in numbers. Regarding the growing number of studies and rapid publication time, there are concerns about accuracy, quality, and limitations. Richard et al. performed a systematic review to evaluate the methodological quality of currently available COVID-19 studies compared  Distribution of the included studies by countries.
to historical controls. This research showed that COVID-19 clinical studies have a shorter time to publication and have lower methodological quality scores than control studies in the same journal (1). We tried to identify the potential limitations of COVID-19 systematic reviews which can improve and make the result of studies more accurate. The current review examined 46 systematic reviews and all of them were conducted on COVID-19 patients. These studies differ in many aspects, including the type of treatment, followup time, study design, patient type, and disease severity. Most of them (80.4%) have conducted meta-analyses. Overall, 91% of the included studies were scored as strong quality, and the rest of them were moderate. The number of studies in the included systematic reviews ranged from 2 to 136.
In this study, we classified the reported limitations into 10 categories and 42 sub-categories. Heterogeneity, sample size, follow-up, treatment, including studies, design, definitions, synthesis, quality, and search are identified as the main limitation of included studies. These limitations were attributed to the included systematic reviews or due to primary studies in these systematic reviews. Among all the limitations, sample population, sample size, and database search were found to be the most-mentioned limitations with frequencies of 22, 21, and 18 in the studies, respectively. It seems that a greater number of limitations could be due to primary studies in the systematic reviews including heterogeneity, small sample size, short follow-up time, and low quality of included studies. Limitations in systematic review studies result from selection of studies, choice of relevant outcomes, methods of analysis, interpretation of heterogeneity, and generalization, application of results, and proper search (65).
Heterogeneity contains four subcategories including differences in the sample population, differences in disease severity of patients, different control group, and different measured outcomes. Differences in the sample population in terms of age, gender, race, and comorbidities in the participants are the most reported limitation. Heterogeneity across the studies may affect the study results (65). For instance, pooling the data of the original articles would be highly difficult due to heterogeneity in the study design and reported outcomes (25), and heterogeneity in disease severity could affect the treatment output.
The small sample size is the second most frequently reported limitation in 21 studies. The number of participants in the included studies was small which could decrease the power of the studies, furthermore comparing the interventions regarding the efficacy would not be incontrovertible. Therefore, the findings need to be interpreted with caution. Distribution of the drugs in the included studies. Database search is another important item that belongs to the search category and is reported in 18 studies. It may potentially limit access to eligible trials for inclusion and miss some data.
Treatment-related limitations are mostly associated with differences regarding the administration of drugs, dose, duration of treatment, and different standard protocols and guidelines. The lack of uniform guidelines for administering additional treatments and providing supportive care for COVID-19 patients in clinical trials may lead to inaccurate and unreliable outcomes. These limitations can generate confounding bias (36). Also, there are other items belonging to this category such as differences related to discontinuation, combination therapy, and supportive care which obscure the effect of the main treatment.
In addition to the above-mentioned limitations, different follow-up times, low quality of the included studies, prepublish and unpublished studies, different comparator arms, and heterogeneity in control groups are the other highly

Sub-category Frequency
Heterogeneity •Heterogeneity in sample population 22 • Heterogeneity in disease severity 9 •Heterogeneity in control groups 15 •Heterogeneity in outcome 13 Sample size •Small sample size 21 •Different inclusion criteria 12 Follow up •Short follow up time 14 •Different follow up time 17 Treatment •Different standard protocol and guideline 7 •Different administration of drug 7 •Different duration of treatment 11 •Different dose of treatment 8 •Treatment discontinuation 6 •Combination therapy 6 •Supportive care 5 Included studies •Different type of included studies 7 •Low number of included studies 14 •Different level of quality of the included studies 9 •Short duration of studies 7 Design •Different design of the included studies 6 •Randomization 9 •Placebo 12 •Blinding 8 •Single-arm 7 •Lack of comparator arm 7 •Different comparator arm 15 •Selection bias 8 •Publication bias 6 •Confounding bias 4 Definition •Different definition of disease severity 8 •Different definition of outcomes 9 •Different definition of ordinal scales 3

Synthesis
•Different meta-analysis approach 5 •Sub-group analyses 5 reported items. The lack of a comparison/control group can limit the validity of the meta-analysis. As mentioned, although systematic reviews are considered the gold standard of evidence for clinical decision-making, one should keep in mind that meta-analyses should neither be a replacement for well-designed large-scale randomized studies nor a justification for conducting small underpowered studies (65). As other studies reported, the quality of the methodology and reporting of present COVID-19 SR is far from optimal. In addition, Differences in disease definition and heterogeneity in studies are important factors influencing the results of these studies. Following existing guidelines and proper study design can be one of the factors reducing the limitations of these studies (66,67). Taken together, poor designs and various limitations of the studies render them ineffective in gaging the full extent of its safety and efficacy and thus warrant further research into the use of interventions in COVID-19 patient treatment. Our study further highlighted the importance of conducting quality studies so that the results can be trusted with more certainty.

Implications for future research
Our results can be used as a guide for designing and reporting the future studies in this field. Undoubtedly, awareness of the limitations of articles in this field can reduce bias and on the other hand increase the power of studies. Considering these issues helps researchers to report studies in a more integrated way, which can also help readers to better understand the results of studies and prevent the repetition of errors and mistakes or limitations reported in previous studies. It is recommended that researchers interested in research related to COVID-19, as well as those interested in investigating the effectiveness of treatments for this disease, must consider the points mentioned in this study when designing, implementing, and reporting their studies. In addition, respected researchers can design similar studies for other fields related to this disease and report their results. Designing such studies can greatly contribute to evidence-based decision making.

Strengths and limitations
Although this study is an overview, and the quality appraisal is optional, but the quality of the articles has been evaluated in it, which is one of the strong points of the study. Also, we conducted this overview based on Smith et al. guideline for conducting a systematic review of systematic reviews and report the results of this study based on PRISMA guideline. All the steps of this study were done by two independent reviewers, which reduces errors and increases the power of the study. There are many potential limitations to this overview. First, a literature search was conducted in the three major electronic databases, Scopus, Pubmed, and WOS, but no other databases were searched, as was the "gray" literature. Therefore, additional relevant studies might have been missed. Second, we included all systematic reviews with available full text and in EN languages investigated the effectiveness, efficacy, safety, and outcome of the main intervention (Favipiravir, Remdesivir, Hydroxychloroquine, Ivermectin, Lopinavir/Ritonavir, or Tocilizumab) for treatment of COVID-19 patients. However, there are other interventions for the treatment of this disease, which can be investigated in other studies, but due to the small number of them, they were not included in this study and only the main interventions were used. Third, we excluded articles published in preprint databases due to lack of peer review.

Data availability statement
The original contributions presented in this study are included in the article/Supplementary material, further inquiries can be directed to the corresponding author.