Methodological Quality and Risk of Bias Assessment of Cardiovascular Disease Research: Analysis of Randomized Controlled Trials Published in 2017

Background All randomized-controlled trials (RCTs) are required to follow high methodological standards. In this study, we aimed to assess the methodological quality of published cardiovascular clinical research trials in a representative sample of RCTs published in 2017. Methods Cochrane Central Register of Controlled Trials was used to identify cardiovascular clinical research trials with adult participants published in 2017. Overall, 250 (10%) RCTs were randomly selected from a total of 2,419 studies. Data on general trial characteristics were extracted and the risk of bias (RoB) was determined. Results Overall, 86% of RCTs have reported at least one statistically significant result, with the primary outcome significant in 69%, treatment favored in 55%, and adverse events reported in 68%. Less than one-third (29%) of trials were overall low RoB, while the other two-thirds were rated unclear (40%) or with high RoB (31%). Sequence generation, allocation concealment, and selective reporting were the domains most often rated with high RoB. Drug trials were more likely to have low RoB than non-drug trials. Significant differences were found in RoB for the allocation concealment and blinding of participants and personnel between industry-funded and non-industry-funded trials, with industry-funded trials more often rated at low RoB. Conclusion Almost two-thirds of RCTs in the field of cardiovascular disease (CVD) research, were at high or unclear RoB, indicating a need for more rigorous trial planning and conduct. Prospective trial registration is a factor predicting a lower risk of bias.


INTRODUCTION
Randomized controlled trials (RCTs) are widely recognized as the most optimal methodology for causal inference, where humans are prospectively included and randomly allocated to groups to evaluate the efficacy and safety of an intervention (1). The strength of the RCTs comes from the randomization procedure, which ensures that all participants have the same chance of being assigned to each of the study groups (1) and guarantees that the characteristics of the participant are similar through the different groups at the baseline (2). However, the extent to which we can draw final conclusions based on RCTs strongly depends on how rigorous study methodology is; methodological inaccuracies during trial planning and conduct will subsequently reduce the reliability of results and their usability in medical practice (3). Biased results can finally lead to the underestimation or overestimation of the true intervention effect (3).
In cardiovascular disease (CVD) research, RCTs have been widely used to provide reliable knowledge on the best treatment strategies, such as therapies which are able to improve patient's symptoms, correct disease markers, and improve clinical outcomes (4,5). However, the risk of bias (RoB) has not been assessed in these studies.
The risk of bias (RoB) reflects the degree to which the results of a trial should be believed (6,7). To reduce the possibility of RoB in RCT's, the Cochrane Collaboration introduced a tool designed to appraise RoB (8,9), involving six domains related to the internal validity of a trial: sequence generation, allocation concealment, blinding, incomplete outcome data, selective outcome reporting, and "other" potential threats to validity (6,7). The risk of bias assessment enables the assessment of flaws in the trial design, conduct, and analysis that may affect study results (10).
In this study, we aimed to describe the reliability of evidence of cardiovascular diseases from a representative sample of cardiovascular RCTs published in 2017. Specific objectives were to examine: (1) the reliability of published cardiovascular trials using the RoB tool; (2) specific trial characteristics which increase the likelihood of unclear/high RoB; and (3) any potential differences in methodological issues between studies funded by the industry or the academy.

Sample Selection
We used the Cochrane CENTRAL Register of Controlled Trials to search for RCTs published in 2017 using subject headings and keywords related to adults (aged ≥ 18 years) and CVDs (such as, atherosclerosis, arrhythmia, cardiomyopathy, heart failure, hypertension, ischemic heart disease, heart attack, angina, sudden death, cardiac arrest, hypercholesterolemia, high blood pressure, CVD, ejection fraction, echocardiography, pericarditis, coronary artery disease, angioplasty, and angiography). The search and the screening of identified studies for eligibility were conducted by the first author (OB). Our search yielded a total of 2,556 studies (Supplementary File 1). Following deduplication, 2,419 studies underwent further analysis. Cochrane CENTRAL was the priority search source as it is the most comprehensive resource available of RCTs, containing publications from MEDLINE and EMBASE, as well as hand-search results, and gray literature (6,11). Results of the search were randomly ordered in Excel, by the following method: after exporting the search result as an Excel file from Cochrane CENTRAL, we assigned a random number between 0 and 1 to each record using Excel's random number generator, then reordered them from the smallest to the highest number (12). As a next step, we screened studies consecutively for eligibility, and the first 250 (∼10%) RCTs matching our prespecified inclusion criteria were selected (Supplementary File 2). Trials were eligible for inclusion if they were published in the year 2017, were written in English, the described results of an RCT in the field of cardiovascular medicine, and included participants aged ≥ 18 years. Decision on the inclusion of a study was made after a careful consideration of the methodology in the full text.

Data Extraction
For data extraction, we used a data extraction sheet already tested and described in a previous study (11), data extracting guide available here: https://doi.org/10.1016/j.jpeds.2017.09.014 (12). The following data were extracted: journal type (e.g., specialty cardiovascular, or general medical), the publication details and characteristics of the published trials (such as study design, intervention, trial conduct, study sample, sample size, presence of a data monitoring committee, research outcomes, and conclusions). Further, we collected information about trial registration. Data extraction was completed by two reviewers (OB, OF): the first reviewer extracted the data and then, the second reviewer double-checked the sample. Conflicts were resolved through discussion and by reaching a consensus. Trial registration and protocol availability were investigated by retrieving information from the publications and via additional Internet searches (in Google and Google scholar). For the internet searches, we used the trial register number, the investigators' names, and keywords describing the intervention or the condition.

Assessment of Methodological Quality and Reporting
We used the Cochrane RoB assessment tool (13) to evaluate the methodological quality of included RCTs. This tool assesses seven domains: (1) random sequence generation (whether the method used to generate the allocation sequence is described in sufficient detail to allow an assessment of whether it should produce comparable groups); (2) allocation concealment (whether the method used to conceal the allocation sequence is described in sufficient detail to determine whether intervention allocations could have been foreseen before or during enrollment); (3) the blinding of participants and personnel; (4) the blinding of outcome assessors; (5) incomplete outcome data (whether attrition and exclusions were reported, whether missing data were balanced across groups or were related to the outcomes); (6) selective reporting (whether pre-specified outcomes were reported in a pre-specified way); and (7) other bias.
We used the Cochrane RoB tool to assess RoB for the primary outcome. When the primary outcome was not clearly defined, we presumed it was the outcome either (1) described under aims/objectives of the study, (2) the outcome used to determine the sample size, or (3) the first outcome reported in the publication (the first applicable was used). One researcher performed a RoB assessment, while a second researcher was assigned to ensure the correctness of the assessments for each study.
Following Cochrane procedures (7), we classified each domain as low, unclear, or high risk. Then, the overall RoB was determined as follows: low when all domains were assessed as low RoB; unclear when at least 1 domain was assessed as unclear and no domains were assessed as high RoB; and high if any domain was assessed as high RoB (7).

Statistical Analysis
Statistical analyses were conducted by the statistical software R version 4.1.2 (14). The data were analyzed descriptively, using means and standard deviations (SDs) or medians and ranges for continuous variables and proportions for categorical variables. A multivariable logistic regression analysis was conducted to investigate the association between pre-specified study characteristics and the odds of high/unclear RoB. The value of p < 0.05 was considered as a significant result.

Study Design and Reporting Characteristics of the Study Sample
Out of the 2,419 studies identified via search, we included the first 250 randomly selected trials, which met our search inclusion criteria as shown in Figure 1. The publication and trial  The funding source was specified in 91.6% of the included trials: most of the trials were funded by an academic grant or a research institute (37.6%), while industrial and pharmaceuticals funding were reported in 23.2% of the trials.
When analyzing the main results of trials, we observed that at least one statistically significant result was reported in 86.0% of the studies; in these studies, the primary outcome was reported to be statistically significant in 69.2% of the cases. The treatment was favored in 55.6% and control in 3.6%. At least one adverse event was reported in 68% of the trials. A data monitoring committee was reported in 42% and sample size calculation in 60.4%. A total of 83.6% of the studies were registered in one of the clinical trials registries out of the 77.5% were registered in clinicaltrials.gov. Table 2 shows the RoB assessment results. Overall, 29.2% of the studies were deemed as low RoB, while the remaining studies were at either unclear (39.6%) or high risk (31.2%). We rated the domains sequence generation, allocation concealment, and selective reporting to be the domains most often at high RoB (13.2, 9.6, and 10.4%, respectively).

Risk of Bias Assessment
We investigated whether the RoB was associated with the following variables: type of the intervention (drug vs. non-drug); single or multiple study centers; sample size; the presence of a Data Monitoring Committee; statistical significance of the primary outcome and trial registration ( Table 3). Of these

Risk of Bias According to a Funding Source
When funding source was added as an additional independent variable to the multivariable regression model, funding did not seem to influence the likelihood of overall low RoB (industry funding: OR 0.76, 95% CI 0.40-1.45).
In the sub-group of industry-funded trials, multicenter trial and Data Monitoring Committee were factors that increased the likelihood of overall low RoB. None of the investigated factors influenced the overall RoB within the sub-group of trials with non-industry funding ( Table 3).  Compared with non-industry funded studies more industry funded studies were rated as low RoB (84.9 vs. 63.9%) and less were rated as unclear (12.3 vs. 26.9%) or high RoB (2.8 vs. 9.2%) for allocation concealment (p < 0.001) ( Table 4). More industry funded studies were rated low (66.0 vs. 42.9%) and fewer were rated as unclear (33.0 vs. 51.3%) or high risk (0.94 vs. 5.9%) for the blinding of participants and personnel (p < 0.001).

Summary of Main Findings
To our knowledge, this is the first research evaluating the RoB and its association with specific trial characteristics in a randomly selected sample of recently published clinical trials in adult cardiovascular disease. Included trials were mainly parallel RCTs investigating the efficacy of an intervention, with a very diverse trial scope and published in a variety of cardiovascular and general medical journals.
Of the 250 studies included, more than 85% have reported at least one statistically significant result, with the primary outcome significant in 69%. Treatment was favored in 55% of the studies, and adverse events were reported in 68%.
Less than one-third of our samples were overall low RoB, while the other two-thirds were unclear or high RoB. Sequence generation, allocation concealment, and selective reporting were the RoB domains most frequently rated at high risk. Trial registration influenced overall RoB to the greatest extent. Drug trials were more likely to be at low RoB than non-drug trials, and multicenter trials were more likely at a low risk RoB than single-center trials.
In the subgroup of industry-funded trials, multicenter trial and Data Monitoring Committee were factors that increased the likelihood of overall low RoB, while none of the investigated factors influenced the overall RoB within the subgroup of nonindustry-funded trials. Significant differences were found in the RoB for the domains allocation concealment and the blinding of participants and personnel between industry-funded and nonindustry-funded trials, with industry-funded trials more often rated at low risk.

Strengths and Weaknesses of the Study
We aimed to select a sample of studies representative for all randomized controlled trials published in 2017; we did not exclude published trials based on the country, type of journal, type of participants, or type of intervention. Included trials were randomly selected from all eligible trials. We used the most well-recognized tool for methodological assessment and the results indicated several areas of methodological weaknesses. To increase the reliability of findings, both data extraction and RoB assessment were conducted by two independent researchers.
This study was not pre-registered. Our study is limited by the included trials published in the English language only; conclusions cannot be generalized to trials published in other languages. We could not identify register entries or trial protocols for a subsample of trials, and it was difficult to properly evaluate the selective outcome reporting in these trials. We have not contacted the authors to get additional information about their trials, therefore the assessments are solely based on published information. Additionally, our study focused on the internal validity of trials, but have not assessed factors that may impact the external validity.

Discussion of Findings Considering Other Studies
Of the 250 analyzed trials, more than two-thirds (70.8%) were at high or unclear RoB, which is consistent with previous study results (15,16). The RoB domains sequence generation, allocation concealment, and selective reporting were rated most often to be at high risk in our study.
Trial registration had the most beneficial effect on RoB in our study. In previous studies, random sequence generation, allocation concealment, and selective reporting were shown to differ between registered and unregistered gynecology and fertility trials (17). Registration was shown to be a factor influencing all RoB domains except selective outcome reporting in a sample of pediatric research trials (6). Trial registration is a factor that has the potential to facilitate higher methodological quality (18).
We found that drug trials were more likely at a low RoB than non-drug trials. This might be in connection with the strict regulations surrounding drug trials. Differences in protocol quality (19) registration and publication tendencies were described already in previous publications (20,21), between trials with regulated and non-regulated interventions.
In our study, multi-center trials have demonstrated lower RoB compared with single-center trials. This is consistent with the conclusions of Landoni et al., who underlined bias issues characteristic for single-center trials (e.g., local effect bias, selection and performance bias, detection and reporting bias, analysis and attrition bias, concomitant therapy bias, low fragility index, and publication bias). Caution is advised for the results of single-center RCTs (22).
The source of funding may have an important impact on trial planning, conduct, and reporting (23)(24)(25). In our study, we investigated the role of funding sources on RoB by comparing trials both funded or not by the industry. In our study, industryfunded studies were rated more often low for the allocation concealment and the blinding of participants and personnel, as non-industry-funded studies. Our findigs on the higher probability of industry-funded trials to have adequate blinding are supported by the results of a similar study conducted in pediatric RCTs (6,25). The blinding of participants and personnel may not always be feasible, however, studies should attempt blinding wherever possible.

Implications for Practice and Future Research
Our results underline the need of further improvement in the process of planning and performing a clinical trial in the field of CVD research. Trial registration was associated with a larger likelihood of low RoB, therefore mandatory trial registration should be endorsed and enforced by ethic committees, funders, and journal editors. Further favorable trial features associated with the lower RoB were multicenter trials, larger trials with more than 500 participants, and trials with a Data monitoring Committee. Trials not funded by the industry were more often at a high RoB for the allocation concealment and the blinding of participants and personnel, indicating, that studies without industry involvement need to pay greater attention to following certain methodological recommendations. The same is applicable for cardiovascular research trials investigating the effects of a non-drug intervention. In the present study, we focused on the methodological quality of CVD research trials but did not assess reporting issues in detail. This may need further assessed in the future.

CONCLUSION
Almost two-thirds of RCTs published in 2017 in the field of CVD research were at high or unclear RoB. This indicates a need for more rigorous trial planning and conduct. Prospective trial registration is a factor that predicts the lower RoB.

DATA AVAILABILITY STATEMENT
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

AUTHOR CONTRIBUTIONS
SL conceptualized the research, supervised the research, and reviewed and edited the manuscript. OB and OF performed the research. OB led data extraction and assessment, responsible for data curation, and wrote the first draft of the manuscript. DN conducted statistical analyses. All authors critically revised and approved the final version of the manuscript.

ACKNOWLEDGMENTS
We would like to thank Kinga Amália Sándor-Bajusz for reviewing our manuscript for language accuracy.