Concordance Study Between IBM Watson for Oncology and Real Clinical Practice for Cervical Cancer Patients in China: A Retrospective Analysis

Watson for Oncology (WFO) is a artificial intelligence clinical decision-support system with evidence-based treatment options for oncologists. WFO has been gradually used in China, but limited reports on whether WFO is suitable for Chinese patients. This study aims to investigate the concordance of treatment options between WFO and real clinical practice for Cervical cancer patients retrospectively. We retrospectively enrolled 300 cases of cervical cancer patients. WFO provides treatment options for 246 supported cases. Real clinical practice were defined as concordant if treatment options were designated “recommended” or “for consideration” by WFO. Concordance of treatment option between WFO and real clinical practice was analyzed statistically. The treatment concordance between WFO and real clinical practice occurred in 72.8% (179/246) of cervical cancer cases. Logistic regression analysis showed that rural registration residences, advanced age, poor ECOG performance status, stages II-IV disease have a remarkable impact on consistency. The main reasons attributed to the 27.2% (67/246) of the discordant cases were the substitution of nedaplatin for cisplatin, reimbursement plan of bevacizumab, surgical preference, and absence of neoadjuvant/adjuvant chemotherapy and PD-1/PD-L1 antibodies recommendations. WFO recommendations were in 72.8% of concordant with real clinical practice for cervical cancer patients in China. However, several localization and individual factors limit its wider application. So, WFO could be an essential tool but it cannot currently replace oncologists. To be rapidly and fully apply to cervical cancer patients in China, accelerate localization and improvement were needed for WFO.

Introduction

Artificial intelligence (AI) is the frontier and dominating terrain of Information Technology which able to simulate human mental status and cognitive function (Jiang et al., 2017). With the development of AI and medical diagnosis technology, clinical decision-support systems (CDSS) with intelligent diagnostic function has become one of the important issues of science for medical information (Meyer et al., 2018). Watson for Oncology (IBM) is a representative AI CDSS that developed by IBM Co.Ltd in United States. WFO can provide a reasonable individualized treatment plan for cancar patients by obtaining valuable information from medical records. WFO first officially landed in China in 2016, until now, more than 80 hospitals use WFO as an important medical diagnostic tool for individualized treatment of tumor (IBM, 2017). WFO can provide counseling services for almost all cancer patients. However, whether WFO was fit for Chinese cancer patients, especially cervical cancer patients.

Cervical cancer is common in the female genital tract malignant tumors, and the incidence of which is second only to that of breast cancer among women worldwide, making it the second-most serious cancer threatening the health and lives of women (Jassim et al., 2018). Compared to breast cancer, cervical cancer is more common in developing countries due to poor health status, and it is the most common in China (Gu X. Y. et al., 2018). And rural and remote areas are also a prevalent regions for cervical cancer in China. But the current problem of the medical service is that the main hospitals hold too many premium resources, but in the meantime, the primary health agencies are excessively lack of resources (Bao et al., 2018). Cervical cancer patients in rural and remote areas can not reach the effective treatment recommendation, especially at centers where cancer expert resources are limited. So, WFO is of great significance for Chinese patients with cervical cancer, especially patients in rural and remote areas with limited medical resources.

Therefore, we conducted a retrospective and observational study on cervical cancer at The Second Xiangya Hospital Cancer Center to explore consistency between WFO and clinical treatment recommendations supported by an expert panel of cancer specialists for Cervical cancer patients.

Materials and Methods

Study Population

This retrospective study was reviewed and approved by the Medical Ethics Committee of The Second Xiangya Hospital of Central south university (approval number was 2017-S104). We retrospectively and randomly selected 300 cases of cervical cancer patients from 05/2016 to 08/2018. All patients with cervical cancer confirmed by pathology at The Second Xiangya Hospital Cancer Center. Untreated Patients and recurrent tumors, rare histology that not yet trained to offer treatment options by WFO system were excluded. A total of 18% (54/300) cases excluded from our study and 82% (246/300) cases were included in our study. The detailed patient selection process is shown in Figure 1.

FIGURE 1

Figure 1. CONSORT diagram. WFO, Watson for Oncology.

Watson for Oncology

Watson for Oncology (IBM Corporation, United States, version 18.1R) used in our study were provided by Baheal Intelligent Technology Co., Ltd¹. The clinicopathologic data of supported cases were extracted from medical records and entered into the WFO system. Treatment options recommended by WFO were presented in three categories: Blue represents “Recommended” with a strong evidence supported, Orange represents “For consideration” with a potentially suitable evidence-based alternative considered by oncologists based on their clinical judgment, and Red represents that is “Not recommended” that a treatment with contraindications or strong evidence against its use.

Real Clinical Practice for Cervical Cancer

The Second Xiangya Hospital Cancer Center one of the biggest and best oncology departments in the Hunan Province of China. Gynecological Oncology Center is the most important part of The Second Xiangya Hospital Cancer Center and mainly serves cervical cancer, ovarian cancer, endometrial cancer, and other gynecological malignant tumors. Gynecological Oncology Center has a multidisciplinary team (MDT) composed of oncologists, gynecologists, radiologists, pathologists, and nutritionists, et al. MDT forms and implements a comprehensive regimen based on NCCN guidelines and the patient’s specific conditions. This comprehensive regimen was considered to be a real clinical practice for cervical cancer.

Data Acquisition and Concordance Judgment

The available clinicopathologic data of 246 patients included a registered residence, age, performance status, pathological type, differentiation degree, FIGO stage, lymphatic and distant metastasis, HPV status, and detailed clinical treatment plan were collected from Second Xiangya Hospital Cancer Center clinical electronic medical records and inputted into WFO system by 2 oncologists manually. Treatment options generated by WFO and recorded through two trained oncologists. It should be noted that in the data analysis process, real clinical practice were categorized as concordant if treatment options were designated “recommended” or “for consideration” by WFO. And if the real clinical practice was not recommended by WFO or if WFO did not provide the same treatment options, the recommendations were considered as discordant. The discordant cases were reevaluated by two senior oncologists provided their reasons for choosing the real treatment options. The specific study design and procedures and are shown in Figure 2.

FIGURE 2

Figure 2. Flow diagram of the study design. WFO, Watson for Oncology.

Statistical Analysis

SPSS20.0 statistics software (SPSS, United States) and Microsoft Excel (2012) were employed to undergo statistical analysis. Descriptive statistics of 246 patients were calculated and presented as means ± standard (x ± s) or median. Differences between the clinicopathological characteristics of the groups were analyzed by Pearson’s χ² test. Correlation between real clinical practice and WFO recommendations were assessed by the chi-square test. A logistic regression model was estimated with odds ratios (OR) and 95% confidence intervals (CIs). Thep values were designated as ^∗P < 0.05.

Results

Clinicopathological Characteristics of Supported Cases

Of the 300 accrued cervical cancer patients, 246 patients were eligible for WFO analysis. Overall, 82% (246/300) of our enrolled cases were supported by WFO. Clinicopathological characteristics of 246 supported cases are detailed in Table 1. Among the 246 supported cases in our study, median age was 53 years (range, 35–78 years), and rural registration patients, stage II/II disease, squamous cell carcinoma, middle/poorly differentiated accounted for 66.2% (165/246), 77.7% (101 + 90/246), 89.0% (219/246), and 80.6% (90 + 108/246), respectively.

TABLE 1

Table 1. Clinicopathological characteristics of cervical cancer patients (N = 246).

Concordance Between WFO and Real Clinical Practice

After reevaluated by two senior oncologists of discordant cases, there was no change to the primary concordance. Overall treatment concordance between WFO and real clinical practice occurred in 72.8% (179/246) of cervical cancer cases, among the concordant cases, treatment options that designated “Recommended” or “For consideration” by WFO accounted for 41.5% (102/246) and 31.3% (77/246), respectively. Also, there were 27.2% (67/246) of case cannot consistent with real clinical practice, among the discordant cases, treatment options that not recommended by WFO or did not provided by WFO accounted for 4.8% (12/246) and 22.4% (55/246), respectively. Dates are shown in Table 2.

TABLE 2

Table 2. Concordance between WFO and real clinical practice (N = 246).

Subgroup Analyses

Subgroup analyses of treatment concordance with clinicopathological characteristics were also carried out. The result showed that urban registration patients [96.3% (78/84)], low age group (≤45 years and 45–65 years groups) [86.2% (25/29), 81.2% (134/165), respectively], good ECOG performance status (0–1 and 2 points groups) [77.9% (145/186), 66.1% (31/47), respectively], and stage II/III disease [80.2% (87/101), 86.7% (78/90), respectively] exhibiting higher concordance than rural registration patients [61.2% (101/165)], advanced age group (≥60 years) [38.5% (20/52)], poor ECOG performance status (≥3 points) [23.1% (3/13)], and stage I/IV disease[41.4% (12/29), 7.96% (2/26), respectively]. While, there were no obvious difference among lymphatic and distant metastasis disease, pathological types, differentiation degrees. Dates shown in Figures 3–5.

FIGURE 3

Figure 3. Treatment concordance between WFO and real clinical practice, divided by age, registered residence, and ECOG performance status. WFO, Watson for Oncology.

FIGURE 4

Figure 4. Treatment concordance between WFO and real clinical practice, divided by FIGO stage, lymphatic and distant metastasis. WFO, Watson for Oncology.

FIGURE 5

Figure 5. Treatment concordance between WFO and real clinical practice, divided by pathological types and differentiation degrees.

Logistic Regression Analysis

The logistic regression analysis showed that, compared with patients ≤45 years of age, concordance declined significantly in patients ≥65 years of age and older[0.08 (0.03–0.28), P = 0.032]. And Concordance was particularly low for patients with rural registration[0.64 (0.427–0.946), P = 0.025],compare with urban patients. Poor ECOG performance status (≥3 points) patients exhibiting lower concordance than good ECOG performance status patients[0.29 (0.083–1.058), P = 0.048]. Odds ratios of concordance varied by stage, showed that compared with stage I disease, stages II-III disease were significantly more likely to be concordant ([2.08 (1.002–4.325), P = 0.046],[2.09 (1.001–4.381), P = 0.047], respectively), whereas, concordance declined remarkably in stages IV disease [0.19 (0.038–0.91), P = 0.025]. While, lymphatic and distant metastasis disease, pathological types, differentiation degrees were not found to affect concordance. Dates are shown in Table 3.

TABLE 3

Table 3. Logistic regression model of concordance between Watson for Oncology and real clinical practice (N = 246).

Analysis of Reasons for Discordant Cases

There were four critical factors attributed to 27.2% (67/246) of the discordant cases. Firstly, Cisplatin is the main chemotherapy drug recommended by WFO, but in our study, of 46.4% (31/67) cases select nedaplatin due to cannot tolerate gastrointestinal reactions of cisplatin. Next, bevacizumab as a routine option recommended by WFO for stage IV stage, but bevacizumab is not in medical reimbursement plan for cervical cancer in China, of 26.9% (18/67) patients reject bevacizumab therapy for the financial burden. Thirdly, for stage Ib2 and IIb disease, only concomitant radiochemotherapy was recommended by WFO, in our study, of 19.4% (13/67) patients prefer surgical therapy instead of concomitant radiochemotherapy. Moreover, neoadjuvant/adjuvant chemotherapy and programmed death-1 and ligand antibodies (PD-1/PD-L1 antibodies) drugs recommendations are not included in the WFO system, in our study, there were 9.1% (6/67),2.8% (2/67) patients chose neoadjuvant/adjuvant chemotherapy and pembrolizumab therapy. Dates are shown in Table 4.

TABLE 4

Table 4. Analysis of reasons for discordant cases (N = 67).

Discussion

From 2013, concordance studies between WFO and physicians have been performed in various countries and cancer types. A double-blind study showed that 93% concordance rate for 638 breast cancer patients (Kaur and Singh Mann, 2018; Somashekhar et al., 2018). A retrospective study from India for 1000 consecutive cases showed 80% concordance between multidisciplinary team (MDT) (Baek et al., 2017). A observational study from Korea showed a 73% concordance rate for colon cancer and a 49% concordance rate for gastric cancer (Somashekhar et al., 2016; Suwanvecho et al., 2017). And, a comparative Study from Korea indicated that WFO without the gene expression assay has limited clinical utility (Kim et al., 2018). It appears that the concordance results varies by countries and cancer types (Zhou et al., 2018). For China, a huge population and regional differences created a different therapeutic experiences and considerations for cancer patients, as well as large differences with Western countries. Also, a retrospective study (Liu et al., 2018) reported by our center revealed that treatment concordance between WFO and MDT occurred in 65.8% (98/149) of lung cancer. Another retrospective study (Zhou et al., 2019) from China showed that Ovarian cancer, lung cancer and breast cancer obtained a high concordance, the concordance of gastric cancer was very low, Incidence and pharmaceuticals may be the major cause of discordance. However, limited reports on whether WFO is suitable for Chinese cervical cancer patients, Zhou et al. reported 14 cervical cancer patients in this study, but the sample size is too small.

Our retrospective study provides the first evidence that accelerates localization and improvement were needed for WFO before comprehensive application in cervical cancer patients in China. Although treatment options generated by WFO were mostly concordant with real clinical practice, there are still unresolved issues. Firstly, as mentioned in the manual (Gu X. et al., 2018), some clinical settings are not yet supported by WFO system. In our study, of 73.7% (28/38) unsupported cases were recurrent tumors patients. But compare with our center, grass-roots hospitals have a greater proportion of patients with recurrent tumors. So, the cases that cannot be supported by WFO system are very large for cervical cancer patients in China. Secondly, localization factors such as physical of patients, medical reimbursement plan, economic condition, and patient preferences of China were different from western countries, and they ultimately affect the inconsistency. In our study, of 46.4% (31/67) cases select nedaplatin due to cannot tolerate gastrointestinal reactions of cisplatin, of 26.9% (18/67) patients reject bevacizumab therapy for financial burden. of 19.4% (13/67) patients prefer surgical therapy instead of concomitant radiochemotherapy. Moreover, registered residence, age, performance status, FIGO stage have a remarkable impact on consistency. Urban registration patients, low age group, good performance status, and stage II/III disease exhibiting higher concordance than rural registration patients, advanced age group, poor performance status, and stage I/IV disease. These personal factors make WFO unable to achieve individualized treatment and affect the consistency significantly in China. Finally, neoadjuvant/adjuvant chemotherapy (Sun et al., 2018). Chemotherapeutic drugs Goffin et al. (2010) such as gemcitabine, docetaxel, mitomycin, irinotecan, pemetrexed, vinorelbine, and PD-1/PD-L1 antibodies (Kim et al., 2017) drugs recommendations that performed in real clinical practice are not included in the WFO system.

Compared with previous research, our study provides the first evidence that WFO is not suitable for Chinese cervical cancer patients currently, and the sample size of this study was the largest among all cervical cancer studies performed. Also, we not only reported the consistency between WFO and real clinical practice, but also analyzed several influence elements and offered certainly advises for the improvement of WFO to better suit Chinese patients. But, our study contains some limitations. Firstly, this was a retrospective and observational study with control groups lacked, several unmeasured elements may influence the outcome. Secondly, treatment preferences among different experts also affect consistency. Thirdly, the distribution of clinicopathological characteristics among patients is imbalanced, for example, fewer patients were stage IV diseases may lead to a large disagreement for Stage IV tumors. Finally, molecular parameters, such as mutations, gene expression or protein localization can affect the treatment decision. But, in China, unlike lung cancer and breast cancer, gene detection were lacked for cervical cancer. Although there are some targeted drugs that may be effective for cervical cancer, such as PARP inhibitors (for BRCA1 or BRCA2 mutations patients), EGFR tyrosine kinase inhibitors (for EGFR mutations patients), gene detection is still not widely used in China. So, in our study, Because of the lack of gene detection datas, we cannot observe the effect of molecular parameters on treatment decisions.

For WFO, WFO could be an essential tool for clinicians, provides good references and literature for medical students, or even give some treatment advice to non-specialist (Malin, 2013; Werner et al., 2016). However, we believe that human physicians will not be replaced by AI in the foreseeable future, WFO still has a long way to go to replace oncologists. Medicine is not just a science, but also a social and psychological subject. Any tool and guidelines can only be used as a doctor’s reference, localization factors and individual elements should considered for different patients, especially for cancer patients with large heterogeneous (Kemin et al., 2017). Therefore, WFO must be significantly improved to adapt the real clinical practice in different countries. Patient’s physical and mental state, economic situation, complications, patient’s treatment preference and medical reimbursement plan in different countries should be taken into account and not just provide advice based on existing knowledge. For China, a unique medical database with Chinese characteristics should be created by WFO to adapt and serve Chinese cancer patients.

Conclusion

In conclusion, WFO recommendations were in 72.8% of concordant with real clinical practice for cervical cancer patients in China. However, several localization and individual factors limit its wider application. So, WFO cannot replace oncologists for cervical cancer patients in China currently. WFO could be an effective decision-support tool in cancer therapy for Chinese physicians, it also helps to standardize the treatment of cervical cancer. To be rapidly and fully apply to cervical cancer patients in China, accelerate localization and improvement were needed for WFO.

Data Availability Statement

The datasets generated for this study are available on request to the corresponding author.

Ethics Statement

This retrospective study was reviewed and approved by the Medical Ethics Committee of The Second Xiangya Hospital of Central South University (approval number was 2017-S104).

Author Contributions

CH was responsible for overall planning for research. FZ was responsible for data collection and statistical analysis. YT and CL were involved in data analysis. JM was participated in the preparation of manuscript.

Funding

This work was supported by the Fundamental Research Funds for the Central Universities of Central South University (2019zzts356).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We highly appreciate the kindly help and support from Tianle Li of Qingdao Baheal Corporation and Irene Dankwa-Mullan of IBM Watson Health for their support on the description and use of WFO. Personally, for my parents and wife and daughter, I must say I really appreciate all your love and support up to my work. Thank you for all you did to us, really thanks, I love you-all.

Footnotes

1. ^ https://www.bsmartd.com

References

Baek, J. H., Ahn, S. M., Urman, A., Ahn, H. K., Won, P. K., Lee, W.-S., et al. (2017). Use of a cognitive computing system for treatment of colon and gastric cancer in Korea. J. Clin. Oncol. 35(Suppl. 15):e18204A.

Google Scholar

Bao, H., Zhang, L., Wang, L., Zhang, M., Zhao, Z., Fang, L., et al. (2018). Significant variations in the cervical cancer screening rate in China by individual-level and geographical measures of socioeconomic status: a multilevel model analysis of a nationally representative survey dataset. Cancer Med. 7, 2089–2100. doi: 10.1002/cam4.1321

PubMed Abstract | CrossRef Full Text | Google Scholar

Goffin, J., Lacchetti, C., Ellis, P. M., Ung Md, Y. C., and Evans Md, W. K. (2010). First-line systemic chemotherapy in the treatment of advanced non-small cell lung cancer: a systematic review. J. Thorac. Oncol. 5, 260–274.

Google Scholar

Gu, X., Zheng, R., Xia, C., Zeng, H., Zhang, S., Zou, X., et al. (2018). Interactions between life expectancy and the incidence and mortality rates of cancer in China: a population-based cluster analysis. Cancer Commun. 38:44. doi: 10.1186/s40880-018-0308-x

PubMed Abstract | CrossRef Full Text | Google Scholar

Gu, X. Y., Zheng, R. S., and Sun, K. X. (2018). [Incidence and mortality of cervical cancer in China, 2014]. Zhonghua Zhong Liu Za Zhi 40, 241–246. doi: 10.3760/cma.j.issn.0253-3766.2018.04.001

PubMed Abstract | CrossRef Full Text | Google Scholar

IBM (2017). Available at: https://www.ibm.com/us-en/marketplace/ibm-watson-for-oncology.

Google Scholar

Jassim, G., Obeid, A., and Nasheet, H. A. A. (2018). Knowledge, attitudes, and practices regarding cervical cancer and screening among women visiting primary health care Centres in Bahrain. BMC Public Health 18:128. doi: 10.1186/s12889-018-5023-7

PubMed Abstract | CrossRef Full Text | Google Scholar

Jiang, F., Jiang, Y., Zhi, H., Dong, Y., Li, H., Ma, S., et al. (2017). Artificial intelligence in healthcare: past, present and future. Stroke Vasc. Neurol. 2:230. doi: 10.1136/svn-2017-000101

PubMed Abstract | CrossRef Full Text | Google Scholar

Kaur, J., and Singh Mann, K. D. (2018). AI Based HealthCare Platform for Real Time, Predictive and Prescriptive Analytics using Reactive Programming (Singapore: Springer), 138–149.

Google Scholar

Kemin, L. I., Yin, R., Wang, D., and Li, Q. (2017). Human papillomavirus subtypes distribution among 2309 cervical cancer patients in West China. Oncotarget 8, 28502–28509. doi: 10.18632/oncotarget.16093

PubMed Abstract | CrossRef Full Text | Google Scholar

Kim, M., Kim, H., Suh, D. H., Kim, K., Kim, H., Kim, Y. B., et al. (2017). Identifying rational candidates for immunotherapy targeting PD-1/PD-L1 in cervical cancer. Anticancer Res. 37, 5087–5094.

PubMed Abstract | Google Scholar

Kim, Y. Y., Oh, S. J., Chun, Y. S., Lee, W. K., and Park, H. K. (2018). Gene expression assay and Watson for Oncology for optimization of treatment in ER-positive, HER2-negative breast cancer. PLoS One 13:e0200100. doi: 10.1371/journal.pone.0200100

PubMed Abstract | CrossRef Full Text | Google Scholar

Liu, C., Liu, X., Wu, F., Xie, M., Feng, Y., Hu, C., et al. (2018). Using artificial intelligence (Watson for Oncology) for treatment recommendations amongst chinese patients with lung cancer: feasibility study. J. Med. Internet Res. 20:e11087. doi: 10.2196/11087

PubMed Abstract | CrossRef Full Text | Google Scholar

Malin, J. L. (2013). Envisioning Watson as a rapid-learning system for oncology. J. Oncol. Pract. 9, 155–157. doi: 10.1200/jop.2013.001021

PubMed Abstract | CrossRef Full Text | Google Scholar

Meyer, M., Donsa, K., Truskaller, T., Frohner, M., Pohn, B., Felfernig, A., et al. (2018). Development of a protocol for automated glucose measurement transmission used in clinical decision support systems based on the continua design guidelines. Stud. Health Technol. Inform. 248, 132–139.

Google Scholar

Somashekhar, S., Kumar, S. P. R., Kumar, A., Patil, P., and Rauthan, A. (2016). Validation study to assess performance of IBM cognitive computing system Watson for oncology with Manipal multidisciplinary tumour board for 1000 consecutive cases: an Indian experience. Ann. Oncol. 27(Suppl._9):2568.

Google Scholar

Somashekhar, S. P., Sepúlveda, M. J., Puglielli, S., Norden, A. D., Shortliffe, E. H., Rohit Kumar, C., et al. (2018). Watson for Oncology and breast cancer treatment recommendations: agreement with an expert multidisciplinary tumor board. Ann. Oncol. 29, 418–423. doi: 10.1093/annonc/mdx781

PubMed Abstract | CrossRef Full Text | Google Scholar

Sun, H., Huang, K., Tang, F., Li, X., Wang, X., Long, S., et al. (2018). Adjuvant chemotherapy after surgery can improve clinical outcomes for patients with IB2-IIB cervical cancer with neoadjuvant chemotherapy followed by radical surgery. Sci. Rep. 8:6443.

Google Scholar

Suwanvecho, S., Suwanrusme, H., Sangtian, M., Norden, A. D., Urman, A., Hicks, A., et al. (2017). Concordance assessment of a cognitive computing system in Thailand. J. Clin. Oncol. 35:6589. doi: 10.1200/jco.2017.35.15_suppl.6589

CrossRef Full Text | Google Scholar

Werner, N. E., Gurses, A. P., Leff, B., and Arbaje, A. I. (2016). Improving care transitions across healthcare settings through a human factors approach. J. Healthc. Qual. 38, 328–343. doi: 10.1097/jhq.0000000000000025

PubMed Abstract | CrossRef Full Text | Google Scholar

Zhou, N., Zhang, C. T., Lv, H. Y., Hao, C. X., Li, T. J., Zhu, J. J., et al. (2018). Concordance study between IBM Watson for oncology and clinical practice for patients with cancer in China. Oncologist 23, 1–8. doi: 10.1634/theoncologist.2018-0255

PubMed Abstract | CrossRef Full Text | Google Scholar

Zhou, N., Zhang, C. T., Lv, H. Y., Hao, C. X., Li, T. J., Zhu, J. J., et al. (2019). Concordance study between IBM Watson for oncology and clinical practice for patients with cancer in China. Oncologist 24, 812–819.

PubMed Abstract | Google Scholar