Core Outcome Set for Clinical Trials of COVID-19 Based on Traditional Chinese and Western Medicine

Abstract

Background:

Development of a core outcome set (COS) for clinical trials for COVID-19 is urgent because of the pandemic wreaking havoc worldwide and the heterogeneity of outcomes in clinical trials.

Methods:

A preliminary list of outcomes was developed after a systematic review of protocols of clinical trials for COVID-19. Then, two rounds of the Delphi survey were conducted. Stakeholders were traditional Chinese medicine (TCM) experts, Western medicine (WM) experts, nurses, and the public. Patients with confirmed COVID-19 were also invited to participate in a questionnaire written in understandable language. Then different stakeholders participated in a consensus meeting by video conference to vote.

Results:

Ninety-seven eligible study protocols were identified from 160 clinical trials. Seventy-six outcomes were identified from TCM clinical trials and 126 outcomes were identified from WM clinical trials. Finally, 145 outcomes were included in the first round of the Delphi survey. Then, a COS for clinical trials of TCM and WM was developed. The COS included clinical outcomes (recovery/improvement/progression/death), etiology (SARS-CoV-2 nucleic-acid tests, viral load), inflammatory factor (C-reactive protein), vital signs (temperature, respiration), blood and lymphatic-system parameters (lymphocytes, virus antibody), respiratory outcomes (pulmonary imaging, blood oxygen saturation, PaO2/FiO2 ratio, arterial blood gas analysis, mechanical ventilation, oxygen intake, pneumonia severity index), clinical efficacy (prevalence of preventing patients with mild-to-moderate disease progressing to severe disease), and symptoms (clinical symptom score). Outcomes were recommended according to different types of disease. Outcome measurement instruments/definitions were also recommended.

Conclusion:

Though there are some limitations for the research, such as insufficient patients and the public involvement, and the unbalanced stakeholders' region, the COS for COVID-19 may improve consistency of outcome reporting in clinical trials. It also should be updated with research progression.

Introduction

Since December 2019, a novel coronavirus causing pneumonia has been spreading around the world. It was temporarily named “2019 novel coronavirus” on 2 January, 2020, but the World Health Organization (WHO) officially named it “coronavirus disease 2019” (COVID-19) on 11 February 2020. The coronavirus that causes COVID-19 was termed “severe acute respiratory syndrome coronavirus 2” (SARS-CoV-2) on 11 February 2020 by the WHO.

As of 10:00 CET on 2 April 2020, 896,450 cases of COVID-19 had been reported to the WHO, and 45,526 of these cases have died (WHO, 2020). COVID-19 is now a global threat, so its outbreak was declared to be a pandemic on 11 March 2020 by the WHO. There are significant knowledge gaps in the epidemiology, transmission dynamics, investigation tools, and management of COVID-19 (Khot and Nadkar, 2020). A specific drug or vaccine has not been approved to treat it. Hence, COVID-19 management is a major challenge for clinicians and researchers worldwide.

The first clinical trial of COVID-19 was registered on 23 January 2020 (Huang, 2020). Since then, an increasing number of clinical trials of COVID-19 have been registered using regimens based on traditional Chinese medicine (TCM) and Western medicine (WM). As of March 18, 2020, 585 protocols were searched from all the databases of International Committee of Medical Journal Editors (ICMJE)-accepted platforms of clinical-trial registries.

Previously, we found several problems regarding the protocols of clinical trials of COVID-19 [e.g., unclear study objectives, heterogeneity of outcome choices, and small sample size (Zhang et al., 2020)] that may reduce the value of clinical trials. In the meantime, clinicians' understanding of COVID-19 characteristics has been changing because they are treating many more patients than before. The diagnosis and management plan of COVID-19 also keeps changing. We believe that certain inappropriate outcomes may be chosen by researchers. To improve the consistency of outcomes and include more clinical trials in systematic reviews, development of a “core outcome set” (COS) for COVID-19 is crucial.

A COS is an agreed standardized set of outcomes that should be measured and reported, as a minimum, in all clinical trials in specific areas of health or healthcare (COMET). When researchers report outcomes in a COS, they can also report other outcomes.

This COS was based on: (i) a population with confirmed COVID-19 cases of “mild”, “ordinary”, “severe”, or “critical” types; (ii) interventions that include TCM and WM; and (iii) the COS being applied in randomized controlled trials (RCTs) and observational studies.

Methods

Registry

This COS has been registered on the Core Outcome Measures in Effectiveness Trials (COMET) database (Shang and Qiu, 2020). This research was conducted and reported following COS-STAndards for Development (COS-STAD) (Kirkham et al., 2017) and COS-STAndards for Reporting (COS-STAR) (Kirkham et al., 2016).

Participants

Steering Group

A steering group was formed by a TCM expert, WM expert, methodologist, nurse, and statistician. They conducted the research protocol, made decisions if there was confusion, and attended the consensus meeting to facilitate COS development.

Stakeholders in the Delphi Survey

The stakeholders in the Delphi survey included TCM experts (clinicians and researchers), WM experts (clinicians and researchers), nurses, patients, and the public.

COVID-19 is a new infectious disease that is spreading rapidly. In China, many clinicians have been trained to face emergencies, irrespective of whether they are on the “frontline” of the “battle” against COVID-19. More than 40,000 clinicians and nurses from other areas of China moved to Hubei Province to support the local medical system. Not all of these clinicians and nurses were trained in respiratory medicine or critical care. To obtain perspectives on a larger scale, we used “snowball” sampling to extend the sample size. We invited members from the Clinical Research Information Association of China and the Information Association for Traditional Chinese Medicine and Pharmacy to participate in the Delphi survey. We asked them to send the questionnaire to their colleagues.

We believe that the perspectives of patients and the public are important. Hence, we sent the questionnaire via social media (WeChat, Tencent) to invite the public to participate.

To obtain patients' perspectives, frontline clinicians of Dongzhimen Hospital, Beijing University of Chinese Medicine (Beijing, China) invited and helped patients who consented to complete the questionnaire.

Stakeholders in the Consensus Meeting

The stakeholders in the consensus meeting were TCM clinicians, WM clinician, nurse, methodologist, evidence-based medicine researcher, and staff from the Chinese Clinical Trials Registry.

Information Sources

All the databases of ICMJE-accepted platforms of clinical-trial registries (ICMJE) were considered. Search terms for Chinese Clinical Trial Registry (ChiCTR) were: “COVID-19,” “2019-novel Corona Virus (2019-nCoV),” “Novel Coronavirus Pneumonia (NCP),” “Severe Acute Respiratory Infection (SARI),” and “Severe Acute Respiratory Syndrome - Corona Virus- 2 (SARS-CoV-2).” Search terms for the Netherlands National Trial Register were “nCoV,” “Coronavirus,” “SARS,” “SARI,” “NCP,” and “COVID.” Search terms for other databases were “2019-nCoV OR Novel Coronavirus OR New Coronavirus OR SARS-CoV-2 OR SARI OR NCP OR Novel Coronavirus Pneumonia OR COVID-19 OR Wuhan pneumonia.”

The search was conducted on 14 February 2020. The details of inclusion criteria, exclusion criteria, study identification, data extraction, and rejected/combined outcomes are described in the systematic review of protocols of clinical trials of COVID-19 (Qiu et al., 2020).

Consensus Process

Two rounds of the Delphi survey for professionals and the public, as well as one round of the Delphi survey for patients, were conducted. After the Delphi survey had been completed, a consensus meeting was conducted to determine the final COS.

Delphi Survey

The questionnaire for professionals and the public was sent by smartphone. It included individual outcomes in different outcome domains and scoring. At the end of the questionnaire of the first round of Delphi survey, there were two open-ended questions: (i) which outcomes do you think are important but were not included in the questionnaire? (ii) what is your opinion of this questionnaire?

The questionnaire for patients was sent by smartphone, too. It included outcomes/outcome domains that were understood readily by patients. Patients were asked to vote on which outcomes/outcome domains were important to them. There was one open-ended question: which outcomes do you think are important but were not included in the questionnaire?

Outcome Scoring

The questionnaire for professionals and the public employed a nine-point scoring system, which has been used in previous COS studies (Qiu et al., 2018; Qiu et al., 2019). A score of: “1–3” denoted that the outcome was not important for inclusion in the COS; “4–6” meant the outcome was important but not critical for inclusion in the COS; “7–9” denoted that the outcome was critical for inclusion in the COS. An outcome scored as ≥7 by ≤50% of participants for all stakeholders was removed from the next consensus process. The outcomes recommended by participants were added in the second round of the Delphi survey after discussion by the steering group.

Consensus Definitions

For the Delphi survey administered to professionals and the public, the consensus definitions were: (i) consensus in: ≥70% of participants in all stakeholders scored the outcome as 7–9, and <15% of participants in all stakeholders scored the outcome as 1–3; (ii) consensus out: ≤50% of participants in TCM experts and WM experts scored the outcome as 7–9; (iii). no consensus: anything else.

The voice of patients should be considered. Hence, for the patients' survey, the consensus definition was outcomes that were voted by >50% of patients.

For the consensus meeting, the consensus definitions were: (i) consensus in: outcomes that were voted by ≥70% of participants; (ii). consensus out: outcomes that were voted by <70% of participants.

Consensus Meeting

The consensus meeting was held by teleconference. The contents of the consensus meeting covered: (i) the reporting background and methods of the research; (ii) reporting the results of the Delphi survey of professionals and the public, and the results of the patients' questionnaire; (iii) discussing the candidate outcomes and their instruments/definitions; and (iv) voting on the outcomes and reaching a consensus.

Ethics and Consent

The entire project is part of a clinical trial of COVID-19, which was approved by the Ethics Committee of Dongzhimen Hospital (DZMEC-KY-2020-09). Because of the special circumstances of the COVID-19 pandemic, participants who completed the questionnaire were assumed to have provided consent for their data to be used.

Results

A total of 160 protocols from 19 platforms of clinical-trial registries were searched. After reading the titles and study details, 63 non-relevant or ineligible study protocols were excluded. Finally, 97 eligible study protocols were included from ChiCTR and ClinicalTrials.gov. Thirty-four clinical trials were for TCM therapy and 63 clinical trials were for WM therapy. All clinical trials will be conducted in China. These clinical trials comprised 75 RCTs (53 for WM and 22 for TCM) and 22 non-RCTs (10 for WM and 12 for TCM). For 34 protocols of TCM clinical trials, there were 76 individual outcomes from 16 outcome domains after the merging and grouping of outcomes. For 63 protocols of WM clinical trials, there were 126 individual outcomes from 17 outcome domains after merging and grouping. The list of outcomes can be obtained from the systematic review (Qiu et al., 2020). There were >40 duplicated outcomes between the TCM and WM protocols for clinical trials.

After removing duplicated outcomes, we developed the questionnaire for first round of the Delphi survey. After review by the steering group, 145 outcomes were included in the questionnaire.

Round 1 of the Delphi Survey

We had incentive measures to improve the response of the Delphi survey (randomized rewards after completing and submitting the questionnaire). The time planned for round 1 of the Delphi survey was from 4 March to 12 March 2020. As of March 9, 2020, 176 participants had completed the questionnaire. After review, 51 questionnaires were found to be invalid. On March 8 and 9, 2020, ≤5 questionnaire/day were completed, and almost all of them were invalid. Most of the invalid questionnaires had been completed by the public within 5 min (it was not possible for people who were unfamiliar with COVID-19 to complete the questionnaire) or who had chosen the same score for all outcomes. After discussion with the steering group, we decided to stop the Delphi survey.

Finally, 125 valid questionnaires were evaluated. The characteristics of participants in the round 1 of the Delphi survey are shown in Table 1. The number of outcomes that achieved consensus and no consensus in different stakeholders are shown in Table 2. The list of outcomes is shown in Supplement 1.

Only 15 (15/125, 12%) participants were in Hubei Province. However, the Internet Protocol (IP) address that the electronic questionnaire obtained showed that 30 (30/125, 24%) participants were in Hubei Province. Only one person (1/125, 0.8%) was from outside of China (Canada). The regions of participants are shown in Figure 1.

Figure 1

More than 20 participants provided outcomes or significant proposals for round 1 of the Delphi survey. After discussion with the steering group, six of them were added to round 2 of the Delphi survey. There are some “consensus out” outcomes in different stakeholders, but no “consensus out” outcomes by all of the stakeholders were noted, so we did not delete any outcomes in the round 2 of Delphi survey.

Round 2 of the Delphi Survey

According to the significant proposals from participants in round 1 of the Delphi survey, the steering group decided to add more personal information. To reduce the risk of invalid questionnaires, participants would receive a random reward if the completed questionnaire was considered to be valid. Participants were also asked if they agreed to be mentioned in the “acknowledgements” section when the research was published. Computed tomography and magnetic resonance imaging of the hip were grouped as “hip imaging”. There were 150 individual outcomes in round 2 of the Delphi survey.

The feedback from participants in round 1 of the Delphi survey showed that scoring for some outcomes was difficult. Hence, in round 2 of the Delphi survey, participants had the opportunity to choose “unclear” for any outcome that was difficult to determine. The median score of each outcome from each stakeholder group was shown in round 2 of the Delphi survey. The steering group wanted more people to participate in the Delphi survey. Hence, the questionnaire was sent to potential participants (irrespective of whether they completed round 1 of the Delphi survey) and they were asked to invite colleagues who might be interested in this research. Round 2 of the Delphi survey was conducted from 11 to 13 March 2020.

A total of 110 questionnaires were completed, but seven of them were invalid, so 103 valid questionnaires were assessed. The characteristics of participants in round 2 of the Delphi survey are shown in Table 3.

The IP address that the electronic questionnaire obtained showed that 28 (28/103, 27.2%) participants were in Hubei Province. The regions of participants are shown in Figure 2. The number of outcomes that achieved consensus and no consensus in different stakeholders are shown in Table 4. The list of outcomes is shown in Supplement 2.

Figure 2

After the results of round 2 of the Delphi survey had been reviewed by the steering group, outcomes that achieved “consensus out” by TCM experts and WM experts were excluded. Outcomes that achieved “consensus in” from stakeholders were grouped and presented according to the classification of disease and interventions. They were presented to consensus-meeting participants with “no consensus outcomes” before the consensus meeting was held.

Patients' Survey

Results suggested that nurses and the public may find it difficult to score outcomes because they may misunderstand the terminology. Hence, we developed a simple questionnaire with understandable language for patients. There were 43 outcomes/outcome domains in the questionnaire. The list of outcomes is in Supplement 3. Patients were recruited by frontline clinicians in our team on 12 and 13 March 2020. Finally, 10 cured patients agreed to participate in the survey. They were asked to choose which outcomes were important to them. The characteristics of patients are shown in Table 5. More than 50% of patients care about outcomes of pulmonary imaging, lung function, respiratory symptoms such as cough and dyspnea, fever, SARS-CoV-2 nucleic acid tests, recovery, and mental state.

Consensus Meeting

The consensus meeting was held on 18 March 2020 and was a video conference. Six frontline clinicians (one from a WM hospital and five from TCM hospitals) as well as one frontline nurse, one methodologist, and one researcher who participated in the design of clinical trials of COVID-19 were invited to attend the consensus meeting. The participants were from Shanghai (one), Beijing (five), Tianjin (two), and Guizhou (one) and all were voting participants. All clinicians and nurses had worked in Hubei Province after the COVID-19 outbreak. Two additional participants (one coordinator and one staff member from the Chinese Clinical Trial Registry) attended the meeting but did not participate in the discussion or voting.

After reporting the results of the Delphi survey and patients' survey, participants discussed some outcomes they believed should/should not be measured in clinical trials. After discussion, voting participants were invited to vote on which outcomes should be included in the COS of COVID-19. The outcomes voted by ≥70% of participants were included in the COS. The voting results are shown in Supplement 4. The COS of COVID-19 is shown in Table 6.

Discussion

This COS was conducted rapidly and rigorously to report on an emergency in a specific environment. It can be used for any type of disease, intervention, and design. There is a specific outcome for TCM clinical trials: clinical symptom score. Researchers can measure the clinical symptom score according to different TCM syndromes. For some individuals, there are several measurements because there is no evidence to show which measurement is the best one. We hope researchers of clinical trials can use this COS to reduce heterogeneity in outcome reporting. Furthermore, our COS may help decision-makers to approve new agents for COVID-19 if researchers report important outcomes. However, researchers can report other outcomes according to the purpose of their research.

Our study had four main limitations. First, due to the highly infectious nature of SARS-CoV-2, patients and the public did not participate in the design or development of the preliminary list of outcomes, but they participated in the process of Delphi survey. Second, the preliminary list of outcomes was developed from protocols of clinical trials when there were knowledge gaps in the prevalence, therapy, prognosis, and clinical characteristics of COVID-19. With the research progression, new important outcomes may be reported. Hence, the COS must be updated in the future. Third, the number of patients was small and all of them were from Hubei Province, so their perspectives may not reflect those of other regions in China or overseas. Fourth, almost all stakeholders were from China. Though one participant in round 1 of the Delphi survey was from Canada, his/her opinion reflected a Chinese perspective because the questionnaire was written in Chinese.

Funding

This work was supported by the National High-level Personnel of Special Support Program (W02020052) and the COVID-19 Project, Dongzhimen Hospital, Beijing University of Chinese Medicine (No.2020-dzmyy-lczx-yj001).

References

• 1

  COMET. Available from: http://www.comet-initiative.org/. Date last accessed: February 14, 2020.

• 2

  HuangC. (2020). A randomized, open-label, blank-controlled trial for the efficacy and safety of lopinavir-ritonavir and interferon-alpha 2b in hospitalization patients with novel coronavirus pneumonia (COVID-19), Chinese Clinical Trial Registry. Available from: http://www.chictr.org.cn/showproj.aspx?proj=48684. Date last accessed: February 14, 2020.

  • Google Scholar

• 3

  ICMJE. Which trials registries are acceptable to the ICMJE?, International Committee of Medical Journal Editors Available from: http://www.icmje.org/about-icmje/faqs/clinical-trials-registration/. Date last accessed: February 14, 2020.

  • Google Scholar

• 4

  KhotW. Y.NadkarM. Y. (2020). The 2019 Novel Coronavirus Outbreak - A Global Threat. J. Assoc. Phys. India68 (3), 67–71.

  • Google Scholar

• 5

  KirkhamJ. J.GorstS.AltmanD. G.BlazebyJ. M.ClarkeM.DevaneD.et al. (2016). Core outcome set-standards for reporting: The COS-STAR Statement. PloS Med.13 (10), e1002148. doi: 10.1371/journal.pmed.1002148

  • CrossRef
  • Google Scholar

• 6

  KirkhamJ. J.DavisK.AltmanD. G.BlazebyJ. M.ClarkeM.TunisS.et al. (2017). Core outcome set-standards for development: the COS-STAD recommendations. PloS Med.14 (11), e1002447. doi: 10.1371/journal.pmed.1002447

  • CrossRef
  • Google Scholar

• 7

  QiuR.LiM.ZhangX.ChenS.LiC.ShangH. (2018). Development of a core outcome set (COS) and selecting outcome measurement instruments (OMIs) for non-valvular atrial fibrillation in traditional Chinese medicine clinical trials: study protocol. Trials.19 (1), 541. doi: 10.1186/s13063-018-2904-0

  • CrossRef
  • Google Scholar

• 8

  QiuR.ZhongC.HanS.HeT.HuangY.GuanM.et al. (2019). Development of a core outcome set for myocardial infarction in clinical trials of traditional Chinese medicine: a study protocol. BMJ Open9 (12), e032256. doi: 10.1136/bmjopen-2019-032256

  • CrossRef
  • Google Scholar

• 9

  QiuR.WeiX.ZhaoM.ZhongC.ZhaoC.HuJ.et al. (2020). Outcome reporting from protocols of clinical trials of Coronavirus Disease 2019 (COVID-19): a review. medRxiv. https://medrxiv.org/cgi/content/short/2020.03.04.20031401v1. Date last accessed: February 14, 2020.

  • Google Scholar

• 10

  ShangH.QiuR. (2020). Core Outcome Set for Traditional Chinese and Western Medicine Clinical Trials of COVID-19, The COMET Database. Available from: http://www.comet-initiative.org/Studies/Details/1507. Date last accessed: February 14, 2020.

  • Google Scholar

• 11

  WHO. (2020). Coronavirus disease 2019 (COVID-19) Situation Report – 73, World Health Organization. Available from: https://www.who.int/docs/default-source/coronaviruse/situation-reports/20200402-sitrep-73-covid-19.pdf?sfvrsn=5ae25bc7_2. Date last accessed: April 3, 2020.

  • Google Scholar

• 12

  ZhangX.ZhaoC.SunY.WeiX.GuanM.ZhaoM.et al. (2020). [Promoting the Establishment of a Collaboration and Sharing Mechanism for Clinical Trials: perspectives from the Outbreak of COVID-19]. J. Tradit. Chin. Med.61(8), 650–654. doi: 10.13288/j.11-2166/r.2020.08.002

  • CrossRef
  • Google Scholar