Bin Zhu1†
Jianlei Zhao2†
Mingnan Cao1
Wanliang Du3Liuqing Yang4Mingliang Su4
Yue Tian1
Mingfen Wu1
Tingxi Wu1Manxia Wang2*
Xingquan Zhao3*
Zhigang Zhao1*- 1Department of Pharmacy, Beijing Tiantan Hospital, Capital Medical University, Beijing, China
- 2Department of Neurology, The Second Hospital of Lanzhou University, Lanzhou, China
- 3Department of Neurology, Beijing Tiantan Hospital, Capital Medical University, Beijing, China
- 4DHC Mediway Technology Co. Ltd., Beijing, China
Background: Thrombolysis with r-tPA is recommended for patients after acute ischemic stroke (AIS) within 4.5 h of symptom onset. However, only a few patients benefit from this therapeutic regimen. Thus, we aimed to develop an interpretable machine learning (ML)–based model to predict the thrombolysis effect of r-tPA at the super-early stage.
Methods: A total of 353 patients with AIS were divided into training and test data sets. We then used six ML algorithms and a recursive feature elimination (RFE) method to explore the relationship among the clinical variables along with the NIH stroke scale score 1 h after thrombolysis treatment. Shapley additive explanations and local interpretable model–agnostic explanation algorithms were applied to interpret the ML models and determine the importance of the selected features.
Results: Altogether, 353 patients with an average age of 63.0 (56.0–71.0) years were enrolled in the study. Of these patients, 156 showed a favorable thrombolysis effect and 197 showed an unfavorable effect. A total of 14 variables were enrolled in the modeling, and 6 ML algorithms were used to predict the thrombolysis effect. After RFE screening, seven variables under the gradient boosting decision tree (GBDT) model (area under the curve = 0.81, specificity = 0.61, sensitivity = 0.9, and F1 score = 0.79) demonstrated the best performance. Of the seven variables, activated partial thromboplastin clotting time (time), B-type natriuretic peptide, and fibrin degradation products were the three most important clinical characteristics that might influence r-tPA efficiency.
Conclusion: This study demonstrated that the GBDT model with the seven variables could better predict the early thrombolysis effect of r-tPA.
Introduction
Acute ischemic stroke (AIS) is a disturbance in cerebral blood flow and has been the leading cause of serious disability and death worldwide (Khandelwal et al., 2016; Kamel et al., 2020). As an acute emergency, AIS is estimated to lead to a loss of 1.9 million neurons per minute, causing irreversible brain injury around the infarction area (Saver, 2006). Timely treatment to remove the obstruction and restore the blood flow has been shown to improve the outcomes in AIS patients (Herpich and Rincon, 2020).
Intravenous thrombolysis with r-tPA is widely recommended to be beneficial for patients with ischemic stroke within 4.5 h of symptom onset (Dong et al., 2017; Phipps and Cronin, 2020). Nevertheless, not all patients can benefit from this therapeutic regimen. Only approximately 40% of the treated patients show improvement, whereas other patients still experience a low recanalization rate (Orban-Kalmandi et al., 2021). Thus, predicting the intravenous thrombolytic effect at an early stage is important for both the clinician and patient.
Machine learning (ML) algorithms are a crucial branch of artificial intelligence and can learn from complex data by using computational techniques to find potential characteristics to make predictions (Heo et al., 2019; Rajkomar et al., 2019; Sirsat et al., 2020). Compared to conventional statistics, ML includes multiple unique algorithms, such as the random forest classifier, support vector machine, and extreme gradient boosting (XGBoost), which can allow computers to create a model by iterative learning (Deo, 2015; Wang et al., 2020). As a multidisciplinary approach, ML algorithms are gaining popularity in addressing complex problems of healthcare decision-making, and some studies have demonstrated that ML tools provide better accuracy and discrimination (Kamal et al., 2018; Fan et al., 2021). Kalscheur et al. (2018) reported the use of ML to create a model that can help doctors discriminate clinical outcomes in those accepting cardiac resynchronization therapy and improve the shared decision-making with patients. In addition, Lee et al. (2020) also found that ML algorithms using magnetic resonance imaging features were much more sensitive than human readings in identifying patients after a stroke within the time window for acute thrombolysis.
Although ML algorithms have been widely used in the prediction of stroke outcomes, few studies have reported the predictive value of the thrombolysis effect at 1 h after r-tPA infusion. In this study, we aimed to develop a model using the ML algorithm to predict the potential effect of r-tPA at admission in patients with AIS.
Methods
Study Design and Patients
We retrospectively conducted an analysis using a cohort of AIS patients who received intravenous thrombolysis with r-tPA between January 2019 and October 2020 at the Beijing Tiantan Hospital. The inclusion and exclusion criteria were previously reported by Zhu et al. (2021). Briefly, patients were recruited if they met the following criteria: diagnosis with AIS through magnetic resonance imaging or computed tomography and admittance to the emergency department within 4.5 h of symptom onset. Patients with a history of the following conditions were excluded from the study: having received a mechanical thrombectomy and having a large vessel occlusion. The study was approved by the ethics committee of the Beijing Tiantan Hospital. The requirement for written informed consent was waived because of the retrospective nature of the study. Confidential patient information was deleted from the entire data set prior to the analysis.
Laboratory values before r-tPA use were recorded. Other medical records, including demographic data and the NIH stroke scale (NIHSS) score at admission and 1 h after r-tPA infusion, were carefully extracted.
Machine Learning Algorithms
Missing values were replaced before the development of the ML-based predictive models. In our analysis, the median value and model were employed for continuous variables and dispersive eigenvalues, respectively. Six ML algorithms, including logistic regression (LR), random forest (RF), XGBoost, adaptive boosting (AdaBoost), gradient boosting decision tree (GBDT), and light gradient boosting machine (LGBM) were applied to determine the best performing model. In addition, the recursive feature elimination method was used to calculate the contribution of each of the variables. The most optimal variables were further validated using the GBDT method.
Model Validation
All patients were randomly divided into training and testing sets at a ratio of 8:2. Once the models were derived, the performances of the different models were validated using the receiver–operating characteristic (ROC) curve as the evaluation metric. The area under the curve (AUC) was calculated to evaluate the performance of the ML algorithm between the training and testing sets. The F1 score was calculated to evaluate the performance of each model. Finally, the optimal ML algorithm was selected.
Model Interpretation
The Shapley additive explanations (SHAP) value, which was developed from the cooperative game theory, was used to interpret the predictions made by the models. By marginally calculating the contributions of the variables, the SHAP method can better explain the importance of each variable in all factor sequences. The local interpretable model–agnostic explanation (LIME) was also used to explain the predictions. The rationale by which a model predicts a single sample using a local linear approximation of the model behavior can be better trusted.
Statistical Analysis
According to the NIHSS score at admission and 1 h after r-tPA infusion, all patients were divided into two groups, i.e., those having a favorable effect and those with an unfavorable r-tPA effect. Those with a 1-h NIHSS score of ≤1 point after thrombolysis or a 1-h NIHSS score that decreased by at least four points below the score at admission were considered favorable. By contrast, other effects were considered unfavorable (Guo et al., 2020).
Data are presented as medians with interquartile ranges (IQRs) (for non–normally distributed variables), means with standard deviations, or percentages (%). A Fisher exact test or a χ2 test was conducted for binary variables, and a Student t-test or Mann–Whitney U test was used for the continuous variables. The six ML algorithms were developed and validated using Python software. Statistical significance was set at p ≤ 0.05. Statistical analyses were conducted using Python software (version 3.8).
Results
AIS patients were divided into favorable and unfavorable effects according to the NIHSS score at admission and 1 h after r-tPA. The baseline characteristics of the patients have been reported elsewhere. Briefly, 353 patients with an average age of 63.0 (56.0–71.0) years were enrolled in the study. Of them 156 patients with an average age of 63.0 (55.0–70.25) years had a favorable effect and 197 patients with an average age of 63.0 (56.0–71.0) years had an unfavorable effect. At admission, the average NIHSS score for the patients with a favorable effect was 4.0 (2.0, 9.0) and for those with an unfavorable effect was 5.0 (3.0, 8.5). About 1 h after receiving rt-PA, the NIHSS scores were 1.0 (0.0, 2.0) and 4.0 (2.0, 8.0) for the favorable and unfavorable effect groups, respectively (Table 1).
TABLE 1. Demographic and laboratory data of the AIS patients stratified according to the NIHSS score.
Model Performance
A total of 14 variables including activated partial thromboplastin clotting time [APTT], monocytes percent (MO), B-type natriuretic peptide (BNP), red cell distribution width (RDW-CV), fibrin degradation products (FDP), glucose (Glu), granulocyte ratio (GR), cardiac troponin I, eosinophil ratio (EO, %), creatine kinase-MB (CK-MB), myoglobin (Myo), D-dimer (D-D), and neutrophil–lymphocyte ratio (NLR) were enrolled in our modeling (Supplementary Material). The AIS patients were randomly stratified (8:2) into the training set for developing the models and the testing set for evaluating the model performance. The clinical characteristics of the two data sets are presented in Table 2.
TABLE 2. Patient characteristics divided by training data set and testing data set.
Six ML algorithms including LR, RF, XGBoost, AdaBoost, GBDT, and LGBM were used to find the model with the best predictive performance. The results showed that the GBDT model demonstrated the best performance (AUC = 0.82, specificity = 0.68, sensitivity = 0.87, and F1 score = 0.80). In addition, the AUC, sensitivity, specificity, F1 score, positive predict (PPV), negative predict (NPV), positive likelihood ratio (PLR), and NLR values of the other five models in the training and testing data sets are shown in Table 3 and Figure 1.
TABLE 3. Summary of prediction results of six ML algorithms based on the training data set and testing data set.
FIGURE 1. ROC curves of six ML algorithms based on variables in the testing data set.
To further optimize the GBDT model, all 14 samples were filtered individually using the F method. Finally, seven variables were employed for the optimal GBDT algorithm (AUC = 0.81, specificity = 0.61, sensitivity = 0.9, and F1 score = 0.79) (Table 4; Figure 2A,B).
TABLE 4. Results of LIME with GBDT model under seven most important variables. Four patients were random selected to interpret of sample prediction results using true negative, true positive, false negative, and false positive.
FIGURE 2. Using recursive feature elimination method to screen the optimal variables. (A) Seven variables were employed for the optimal GBDT algorithm. (B) ROC curves of the GBDT model based on selected variables.
Interpretation and Evaluation of Machine Learning Model
The SHAP method was also used to interpret the predictions achieved by the GBDT model. Based on the SHAP results, the feature ranking interpretation showed that APTT, BNP, and FDP were the three most important features. Overall, the characteristics of Myo, RDW-CV, FDP, BNP, Glu, and GR correlated positively with the outcomes, indicating increased risk; in addition, the effect of APTT on the patient outcome is nonlinear and fluctuates (Figure 3).
FIGURE 3. Seven most important variables and their impact on the GBDT model output by SHAP analysis. (A) Summary of SHAP analysis on the data set. One dot represents a case in the data set, and the color of a dot indicates the value of the feature. Blue indicates the lowest range and red the highest range. (B) Ranking of the seven variables' importance indicated by SHAP analysis.
Moreover, the LIME algorithm was applied to explain the influence of different variables of the GBDT model on the prediction results. Four cases (true positive, true negative, false positive, and false negative) in the testing data set were randomly selected to interpret the visualized prediction results (Figure 4).
FIGURE 4. Results of LIME with GBDT applied to four random selected patients.
Discussion
Extensive efforts have been made to stratify the long-term outcomes of AIS after r-tPA (Di Lorenzo et al., 2021; Goh et al., 2021). However, early intervention for AIS is particularly important as a disease with a high mortality and disability rate. Thus, this study was designed to evaluate the risk factors which may influence the thrombolytic effect of r-tPA in the early stage of AIS by using ML algorithm. Our study using GBDT model demonstrated that Myo, RDW-CV, FDP, BNP, Glu, and GR are risk factors and should be considered before thrombolysis is applied in clinical scenarios. As an important tool in artificial intelligence, ML has recently received increasing attention and is widely utilized in medical data processing. One of the advantages of ML is the development of predictive models for prognostic outcomes among various confounding factors. Ou et al. (2020) reported a positive result in aneurysm rupture risk assessment using the ML method when compared with a conventional statistical model. Sung et al. (2020) found that ML algorithms that integrate clinical and neuroimaging information can better predict early neurological deterioration in patients after an acute minor stroke. Although these studies showed a good performance through the use of ML, they were not aimed at predicting the super-earlier therapeutic response of r-tPA. To the best of our knowledge, this is the first study to use ML algorithm to predict the 1-h clinical outcome of AIS patients after r-tPA thrombolysis.
Making an accurate prediction under a clinical scenario is usually difficult owing to the existence of too many potential heterogeneous factors and the inherent noise of data encountered in clinical care. Thus, it is preferable to use a simple model if it is sufficiently accurate for a particular application. In our study, we used both 14 and seven variables during the modeling. Both showed good prediction efficiency; however, because the use of seven variables is more feasible under real clinical scenarios, we applied the seven variables to the GBDT model.
Although ML algorithms enable a computer to process complex calculations between variables and outcomes to achieve a more relevant prediction, some aspects of ML algorithms, such as their “black box” characteristic, have limited their predictive value (Petch et al., 2021). Owing to the black box characteristic, ML algorithms are considered to lack a transparent interpretation of the learning. Thus, an ML algorithm model should be used along with the experiences of the physician, rather than as a clinical judgment tool.
To overcome the weaknesses of ML, the SHAP method, which employs a game theory–based approach, was used to interpret the predictions made by the best model. As a unified framework that improves the interpretability and maintains the predictability of complex models, the Shapley value can provide insight for determining the relationships among numerous variables. By using the SHAP method, we identified the characteristics of the important variables that contribute to a prediction, including APTT (time), Myo, and BNP.
The present study has several limitations that should not be ignored. First, this was a single-center retrospective study with a small sample size, which may influence the model efficiency. Further validation in a multicenter setting is required. Second, only the NIHSS score at 1 h was employed in the analysis to evaluate the thrombolytic effect, and long-term follow-up results should be added in the future study. Third, further validation and assessment of the model is required. Because AIS is a complex disease which may be influenced by various factors (such as disease history and home therapies), the analysis of this study was mainly based on the NIHSS score, which may not be enough to accurately predict the thrombolysis effect.
Conclusion
In summary, this study demonstrated the feasibility of applying ML algorithm in predicting the curative effect 1 h after thrombolysis. However, further studies with a larger cohort are needed to validate the model accuracy in the future, and the predictive value of the model needs to be further examined in clinical practice.
Data Availability Statement
The original contributions presented in the study are included in the article/Supplementary Material; further inquiries can be directed to the corresponding authors.
Ethics Statement
The studies involving human participants were reviewed and approved by the Ethics Committee of Beijing Tiantan Hospital. The Ethics Committee waived the requirement of written informed consent for participation.
Author Contributions
BZ wrote the manuscript. MC, WD, MW, YT, and TW collected the data. JZ, LY, and MS assisted with data statistics and interpretation. MW, XZ, and ZZ conceived the presented idea. All authors have read and approved the final version of the manuscript.
Funding
This study was supported by the Clinical Toxicology Project of Chinese Society of Toxicology (number: CST2020CT109) and Beijing Municipal Administration of Hospitals (number: ZYLX201827).
Conflict of Interest
Authors LY and MS were employed by the company DHC Mediway Technology Co. Ltd.
The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s Note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors, and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Acknowledgments
The authors would like to thank Kejia Liu and Yansheng Li for their suggestions in conducting the data analysis.
Supplementary Material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fphar.2021.759782/full#supplementary-material
References
Deo, R. C. (2015). Machine Learning in Medicine. Circulation 132 (20), 1920–1930. doi:10.1161/CIRCULATIONAHA.115.001593
Di Lorenzo, R., Saqqur, M., Buletko, A. B., Handshoe, L. S., Mulpur, B., Hardman, J., et al. (2021). IV tPA Given in the golden Hour for Emergent Large Vessel Occlusion Stroke Improves Recanalization Rates and Clinical Outcomes. J. Neurol. Sci. 428, 117580. doi:10.1016/j.jns.2021.117580
Dong, Q., Dong, Y., Liu, L., Xu, A., Zhang, Y., Zheng, H., et al. (2017). The Chinese Stroke Association Scientific Statement: Intravenous Thrombolysis in Acute Ischaemic Stroke. Stroke Vasc. Neurol. 2 (3), 147–159. doi:10.1136/svn-2017-000074
Fan, Y., Li, Y., Bao, X., Zhu, H., Lu, L., Yao, Y., et al. (2021). Development of Machine Learning Models for Predicting Postoperative Delayed Remission in Patients with Cushing's Disease. J. Clin. Endocrinol. Metab. 106 (1), e217–e231. doi:10.1210/clinem/dgaa698
Goh, S., Tan, N. H. W., Tan, C. H., Leow, A. S. T., Sia, C.-H., Ho, A. F. W., et al. (2021). Evaluating the Safety and Efficacy of Intravenous Thrombolysis for Acute Ischemic Stroke Patients with a History of Intracerebral Hemorrhage: a Systematic Review and Meta-Analysis. J. Thromb. Thrombolysis. doi:10.1007/s11239-021-02531-2
Guo, Y., Zhang, S., Li, M., Sun, B., Shang, X., Li, S., et al. (2020). Leukoaraiosis and Earlier Neurological Outcome after Mechanical Thrombectomy in Acute Ischemic Stroke. J. Neuroradiol 47 (6), 428–432. doi:10.1016/j.neurad.2019.10.005
Heo, J., Yoon, J. G., Park, H., Kim, Y. D., Nam, H. S., and Heo, J. H. (2019). Machine Learning-Based Model for Prediction of Outcomes in Acute Stroke. Stroke 50 (5), 1263–1265. doi:10.1161/STROKEAHA.118.024293
Herpich, F., and Rincon, F. (2020). Management of Acute Ischemic Stroke. Crit. Care Med. 48 (11), 1654–1663. doi:10.1097/CCM.0000000000004597
Kalscheur, M. M., Kipp, R. T., Tattersall, M. C., Mei, C., Buhr, K. A., Demets, D. L., et al. (2018). Machine Learning Algorithm Predicts Cardiac Resynchronization Therapy Outcomes: Lessons from the COMPANION Trial. Circ. Arrhythm Electrophysiol. 11 (1), e005499. doi:10.1161/CIRCEP.117.005499
Kamal, H., Lopez, V., and Sheth, S. A. (2018). Machine Learning in Acute Ischemic Stroke Neuroimaging. Front. Neurol. 9, 945. doi:10.3389/fneur.2018.00945
Kamel, H., Navi, B. B., Parikh, N. S., Merkler, A. E., Okin, P. M., Devereux, R. B., et al. (2020). Machine Learning Prediction of Stroke Mechanism in Embolic Strokes of Undetermined Source. Stroke 51 (9), e203–e210. doi:10.1161/STROKEAHA.120.029305
Khandelwal, P., Yavagal, D. R., and Sacco, R. L. (2016). Acute Ischemic Stroke Intervention. J. Am. Coll. Cardiol. 67 (22), 2631–2644. doi:10.1016/j.jacc.2016.03.555
Lee, H., Lee, E. J., Ham, S., Lee, H. B., Lee, J. S., Kwon, S. U., et al. (2020). Machine Learning Approach to Identify Stroke within 4.5 Hours. Stroke 51 (3), 860–866. doi:10.1161/STROKEAHA.119.027611
Orbán-Kálmándi, R., Szegedi, I., Sarkady, F., Fekete, I., Fekete, K., Vasas, N., et al. (2021). A Modified In Vitro Clot Lysis Assay Predicts Outcomes and Safety in Acute Ischemic Stroke Patients Undergoing Intravenous Thrombolysis. Sci. Rep. 11 (1), 12713. doi:10.1038/s41598-021-92041-1
Ou, C., Liu, J., Qian, Y., Chong, W., Zhang, X., Liu, W., et al. (2020). Rupture Risk Assessment for Cerebral Aneurysm Using Interpretable Machine Learning on Multidimensional Data. Front. Neurol. 11, 570181. doi:10.3389/fneur.2020.570181
Petch, J., Di, S., and Nelson, W. (2021). Opening the Black Box: the Promise and Limitations of Explainable Machine Learning in Cardiology. Can. J. Cardiol. doi:10.1016/j.cjca.2021.09.004
Phipps, M. S., and Cronin, C. A. (2020). Management of Acute Ischemic Stroke. BMJ 368, l6983. doi:10.1136/bmj.l6983
Rajkomar, A., Dean, J., and Kohane, I. (2019). Machine Learning in Medicine. N. Engl. J. Med. 380 (14), 1347–1358. doi:10.1056/NEJMra1814259
Saver, J. L. (2006). Time Is Brain-Qquantified. Stroke 37 (1), 263–266. doi:10.1161/01.STR.0000196957.55928.ab
Sirsat, M. S., Fermé, E., and Câmara, J. (2020). Machine Learning for Brain Stroke: A Review. J. Stroke Cerebrovasc. Dis. 29 (10), 105162. doi:10.1016/j.jstrokecerebrovasdis.2020.105162
Sung, S. M., Kang, Y. J., Cho, H. J., Kim, N. R., Lee, S. M., Choi, B. K., et al. (2020). Prediction of Early Neurological Deterioration in Acute Minor Ischemic Stroke by Machine Learning Algorithms. Clin. Neurol. Neurosurg. 195, 105892. doi:10.1016/j.clineuro.2020.105892
Wang, W., Kiik, M., Peek, N., Curcin, V., Marshall, I. J., Rudd, A. G., et al. (2020). A Systematic Review of Machine Learning Models for Predicting Outcomes of Stroke with Structured Data. PLoS One 15 (6), e0234722. doi:10.1371/journal.pone.0234722
Keywords: thrombolysis, acute ischemic stroke, machine learning algorithms, r-tPA, models
Citation: Zhu B, Zhao J, Cao M, Du W, Yang L, Su M, Tian Y, Wu M, Wu T, Wang M, Zhao X and Zhao Z (2022) Predicting 1-Hour Thrombolysis Effect of r-tPA in Patients With Acute Ischemic Stroke Using Machine Learning Algorithm. Front. Pharmacol. 12:759782. doi: 10.3389/fphar.2021.759782
Received: 17 August 2021; Accepted: 29 November 2021;
Published: 03 January 2022.
Edited by:
Jianxin Chen, Beijing University of Chinese Medicine, ChinaReviewed by:
Hang Chang, Lawrence Berkeley National Laboratory, United StatesQian Hua, Beijing University of Chinese Medicine, China
Copyright © 2022 Zhu, Zhao, Cao, Du, Yang, Su, Tian, Wu, Wu, Wang, Zhao and Zhao. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Manxia Wang, d214MzIyQGFsaXl1bi5jb20=; Xingquan Zhao, enhxQHZpcC4xNjMuY29t; Zhigang Zhao, MTAyMnp6Z0BzaW5hLmNvbQ==
†These authors have contributed equally to this work