Edited by: Fabien Scalzo, University of California, Los Angeles, United States
Reviewed by: Ivana Galinovic, Centrum für Schlaganfallforschung Berlin, Germany; Henry Ma, Monash University, Australia
This article was submitted to Stroke, a section of the journal Frontiers in Neurology
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
In developed countries, the second leading cause of death is stroke, which has the ischemic stroke as the most common type. The preferred diagnosis procedure involves the acquisition of multi-modal Magnetic Resonance Imaging. Besides detecting and locating the stroke lesion, Magnetic Resonance Imaging captures blood flow dynamics that guides the physician in evaluating the risks and benefits of the reperfusion procedure. However, the decision process is an intricate task due to the variability of lesion size, shape, and location, as well as the complexity of the underlying cerebral hemodynamic process. Therefore, an automatic method that predicts the stroke lesion outcome, at a 3-month follow-up, would provide an important support to the physicians' decision process. In this work, we propose an automatic deep learning-based method for stroke lesion outcome prediction. Our main contribution resides in the combination of multi-modal Magnetic Resonance Imaging maps with non-imaging clinical meta-data: the thrombolysis in cerebral infarction scale, which categorizes the success of recanalization, achieved through mechanical thrombectomy. In our proposal, this clinical information is considered at two levels. First, at a population level by embedding the clinical information in a custom loss function used during training of our deep learning architecture. Second, at a patient-level through an extra input channel of the neural network used at testing time for a given patient case. By merging imaging with non-imaging clinical information, we aim to obtain a model aware of the principal and collateral blood flow dynamics for cases where there is no perfusion beyond the point of occlusion and for cases where the perfusion is complete after the occlusion point.
Stroke ranks second as leading cause of death worldwide (
Neuroimaging plays an essential role in the diagnosis and treatment of stroke, where Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) are the preferred imaging modalities. However, MRI provides a better detection and assessment of potentially salvageable tissue, due to its multi-spectral property (
Predicting stroke lesion outcome (i.e., at 3-month follow-up), and the potential efficacy of the treatment according to the nature of the lesion, has a great potential to guide the decision making of physicians. An automatic stroke tissue outcome prediction method would help the physician in such time-critical decision-making process (
Several methods have been proposed for stroke lesion segmentation (
Scalzo et al. (
Most recent methods are based on deep learning. Choi et al.(
In previous approaches, the clinical information related to the success of reperfusion (TICI scale) has either been used within multivariate linear regression models (
In this paper, we propose an automatic method for stroke lesion outcome prediction, whose main contributions are:
1. The combination of imaging and non-imaging clinical data in an end-to-end deep learning architecture. 2. The development of a customized loss function to incorporate clinical information during the learning phase. Therefore, learning relationships between imaging and non-imaging information at a population level. 3. The inclusion of clinical information during the prediction phase at a patient-specific level, allowing us to perform predictions of different outcome scenarios in clinical environment.
The following sections are organized as follows: section 2 describes the proposed method. Section 3 details the database used and evaluation methods. Section 4 presents the results and its discussion. Finally, section 5 summarizes up the main aspects of the proposal.
Stroke lesion outcome prediction consists of characterizing follow-up changes in location and extension of lesions over time from multi-sequence MRI and clinical information. In our proposal, to perform tissue outcome prediction, the method assigns to each voxel of the MRI volume one out of two classes, healthy tissue or stroke lesion. The following subsections describe the main steps of our proposal.
Our proposal uses diffusion and perfusion maps, adding up to six MRI parametric maps: diffusion ADC map, and perfusion relative Cerebral Blood Flow (rCBF), relative Cerebral Blood Volume (rCBV), Mean Time to Transit (MTT), Time-to-Peak (TTP), and Tmax maps. Figures
MRI parametric maps of a stroke patient with TICI score 0, and the respective manual segmentation. Only one class is defined, describing simultaneously the infarct core and the penumbra regions.
MRI parametric maps of a stroke patient with TICI score 3, and the respective manual segmentation.
ISLES 2017 dataset provides MRI acquisitions from different centers (
Deep learning encompasses a variety of representation learning techniques capable of automatically learning hierarchical and complex features from the data. This property grants various levels of abstraction, translating to higher discriminative features, when comparing to hand-crafted features. In imaging processing, the most common techniques of deep learning are the Convolutional Neural Networks (CNNs) (
CNNs have recently achieved remarkable success in well-known computer vision challenges (
Gated RNNs, which achieved success in the biomedical imaging field (
Our proposal is inspired by the fully convolutional U-net architecture (
Overview of the proposed architecture. Blue feature maps result from 2D-dimensional convolutions. The green feature maps represent the 2D-dimensional GRU layer. The first dimension corresponds to the number of feature maps. The dashed line consists of a cropping step to connect the U-Net with the GRU layer. The prediction is provided by the last layer, corresponding to a SoftMax activation.
Besides MRI imaging data, non-imaging clinical information is also gathered during the acute phase of stroke, such as the Time Since Stroke (TSS), Time to Treatment (TTT), modified Ranking Scale (mRS) score, and TICI score. TSS and TTP are time measures that mark the time-points when the stroke incident was diagnosed and when clinical intervention was performed, respectively. The mRS score characterizes the degree of disability 90 days after a stroke incidence. However, the most relevant factor is the TICI score (
Incorporating clinical information at a population-level is achieved through a custom loss function, which drives the learning process to solutions conditioned to the clinical TICI score. Due to the presence or absence of perfusion beyond the location of the occlusion, stroke lesion extension can present changes between the TSS and the follow-up acquisitions. For cases with no perfusion, it is expected that the lesion grows between the two exams, while cases with existent perfusion should present a shrinkage of the lesion volume. In our proposal, we aim to model such lesion dynamics when predicting the lesion progression from the MRI parametric maps at the first exam to a future time. To do so, the training procedure is performed based on the MRI sequences from the first exam and the manual segmentation of the lesion at the follow-up acquisition. When the lesion shrinks, our system must learn that although the lesion presents a larger extension in the MRI sequences, it should produce a smaller segmentation, and when the lesion grows, it should learn to predict a larger segmentation, although the information provided by the MRI sequences indicates it is smaller. We may model this dynamic by interpreting the growth as oversegmentation, and the shrinkage as undersegmentation in relation to the information supported by the MRI sequences in the present time. We may interpret the oversegmentation as an increase in false positives (FP) and the shrinkage as an increase in false negatives (FN), since these are not supported by the information in the MRI sequences, acquired at the first medical exam. Such dynamic in our proposal is modeled by the
The Precision score, defined as
The sum is performed for the
The inclusion of the TICI score at a patient level is achieved by an extra channel before the final layer of the architecture (see Figure
As post-processing step, we performed simple morphological filtering. Stroke lesions vary significantly in size. The post-processing should take this variation into account to avoid the complete removal of stroke lesions; therefore, a threshold to remove only connected components with less than 25 voxels was defined using cross-validation.
We evaluated our proposal on the ISLES 2017 training and testing datasets, where the online platform also includes an automated evaluation of prediction results submitted to the system. In this work, we compared the performance of our proposal with and without using clinical meta-data.
ISLES 2017 dataset comprises a total of 75 ischemic stroke patients divided into two groups: training (
TICI distribution for ISLES 2017 training and testing datasets.
Training | 6 (14%) | 3 (7%) | 3 (7%) | 11 (26%) | 20 (46%) |
Testing | 3 (9%) | 2 (6%) | 4 (13%) | 6 (19%) | 17 (53%) |
The performance of each method was evaluated using five metrics: Dice Similarity Score (DSC), Precision, Recall, Hausdorff Distance and Average Symmetric Surface Distance (ASSD). DSC measures the similarity between two volumes and is defined by
The validation set comprised seven cases, while the testing set of 36 cases from ISLES 2017 training set. To assess the added value of our contributions, we perform a 7-fold-cross-validation scheme within the training set. We compare our proposal with a baseline architecture, which does not encompass any clinical meta-data. In addition, we changed the loss function to the soft dice (
For each subject, around 500 patches of size 88 × 88 were extracted, using a uniform random sampling scheme. The network was trained with ADAM optimizer (
When considering cases with low TICI score, predicting the maximal extent of tissue loss eases the clinical decision-making process, therefore decreasing the chances of tissue death by hypo-perfusion. In such circumstances, with the inclusion of the TICI score we aim to drive the model to predict the worst-case scenario of stroke lesion outcome. Conversely, in a case with a high TICI score we would prefer a prediction where the recovered hypo-perfused tissue due to reperfusion is achieved with success, holding on the same principles as before. It is worth mentioning that such relationship is further affected by several other clinical and patient-specific pathophysiological aspects, such as collateral flood, onset time of the stroke, etc.
Giving the available number of cases per TICI in ISLES 2017 dataset, we merged TICI scores, increasing the number of cases per score. Therefore, at a population level, β in Equation (4) encodes the TICI score as follows:
In this way, for
In this section, we first evaluate the main contribution of our proposal in the training set. Using cross-validation we compare the performance of the baseline method without non-imaging clinical information against our proposal. Afterwards, we present the results obtained in ISLES 2017 testing dataset, performing a comparison against state-of-the-art methods.
Due to the large diversity of appearance, size and shape, the tissue outcome prediction presents as a challenging task (
Results obtained through cross-validation in ISLES 2017 training dataset for the baseline method and our proposal. Each metric contains the average ± standard deviation.
Baseline | 0.34 ± 0.22 | 35.09 ± 17.27 | 6.08 ± 5.27 | 0.37 ± 0.29 | 0.54 ± 0.26 |
Proposal | 0.35 ± 0.22 | 31.38 ± 15.81 | 5.55 ± 5.00 | 0.41 ± 0.30 | 0.47 ± 0.24 |
When comparing with the baseline, our proposal is capable of achieving higher DSC and lower Hausdorff Distance, showing the added value of incorporating the TICI score into the neural network. Considering the precision and recall metrics, our proposal achieved higher precision but lower recall. This suggests a higher capability to perform stroke lesion outcome prediction, by depicting gradual changes in the hypo-perfused tissue. We hypothesize that making the model aware to intrinsic biological phenomena of lesion growth or shrinkage (TICI dependent) lead to more precise predictions, which is sustained by the lower values of distance metrics and higher DSC score.
However, in clinical practice the TICI score is only obtained after recanalization. Being so, predicting the stroke lesion at a 90 day follow up, during the sub-acute phase, needs to consider different reperfusion scenarios. In our proposal, we grant such property at patient-level domain. By adding an extra input channel that contains the TICI score, we aim to obtain tissue outcome predictions with successful and unsuccessful reperfusions. When accessing both case scenarios, during the decision-making process, our method could provide to clinicians additional information on the salvaged tissue if mechanical thrombectomy was performed. In Figures
Example case of stroke lesion outcome prediction, with and without non-imaging clinical information in a patient with unsuccessful reperfusion. For sake of description we present the ADC and Tmax maps and the GT. In the presence of clinical information, we show the two possible outcomes: unsuccessful (TICI = 0) and successful reperfusion (TICI = 3), respectively.
Example case of stroke lesion outcome prediction, with and without non-imaging clinical information in a patient with successful reperfusion. We also present the ADC and Tmax maps and the GT. In the presence of clinical information, we show the two possible outcomes: successful (TICI = 3) and unsuccessful reperfusion (TICI = 0), respectively.
For each case, we present the tissue outcome predictions with and without non-imaging clinical information. In the absence of the TICI score, the tissue outcome prediction performs worse than our proposal, for both cases. Our proposal is capable of employing the TICI score to yield better predictions, which are corroborated by higher Dice scores, but also provides a result that physiologically is more plausible. Observing the stroke lesion outcome predictions of our proposal against the baseline, it is noticeable the presence of physiologically infeasible isolated regions in the latter. Additionally, we also tested if our method was capable of predicting different lesion outcomes by changing the TICI score. When changing the TICI score, we obtained different lesion outcomes for each patient. Furthermore, such scenarios agreed with the expected outcome describe for each TICI score (e.g., by changing from a TICI score of 3 to 0 it was observed a larger lesion outcome volume). From the latter study, we show that our proposal gained awareness to scenarios of no-perfusion and complete perfusion. Such capability could provide the clinicians useful insight on the benefits and risks associated to the mechanical thrombectomy. Moreover, it can also be used to forecast recovery, which is important for patient treatment and the complete standard care associated to patient recovery. To corroborate our qualitative analysis, Table
Results obtained by our proposal on two patient cases with different TICI scores, alongside the obtained result after changing the original TICI score to its opposite (marked with a *).
24 | 21,310 | 0 | 0.48 | 0.87 | 0.33 | 8170 |
3* | 0.44 | 0.90 | 0.29 | 6840 | ||
42 | 288 | 3 | 0.43 | 0.59 | 0.33 | 163 |
0* | 0.24 | 0.17 | 0.39 | 651 |
On Table
In Table
Results of ISLES 2017 testing dataset, alongside our baseline method and proposal. Each metric contains the average ± standard deviation.
Challenge | Mok et al. |
0.32 ± 0.23 | 40.74 ± 27.23 | 8.97 ± 9.52 | 0.34 ± 0.27 | 0.39 ± 0.27 |
Kwon et al. |
0.31 ± 0.23 | 45.26 ± 21.04 | 7.91 ± 7.31 | 0.36 ± 0.27 | 0.45 ± 0.30 | |
Bertels et al. |
0.30 ± 0.21 | 33.85 ± 16.82 | 6.81 ± 7.18 | 0.34 ± 0.26 | 0.51 ± 0.32 | |
Monteiro et al. |
0.30 ± 0.22 | 46.60 ± 17.50 | 6.31 ± 4.05 | 0.34 ± 0.27 | 0.51 ± 0.30 | |
Lucas et al. |
0.29 ± 0.21 | 33.85 ± 16.82 | 6.81 ± 7.18 | 0.34 ± 0.26 | 0.51 ± 0.32 | |
Choi et al. |
0.28 ± 0.22 | 43.89 ± 20.70 | 8.88 ± 8.19 | 0.36 ± 0.31 | 0.41 ± 0.31 | |
Robben et al. |
0.27 ± 0.22 | 37.84 ± 17.75 | 6.72 ± 4.10 | 0.44 ± 0.32 | 0.39 ± 0.31 | |
Pisov et al. |
0.27 ± 0.20 | 49.24 ± 32.15 | 9.49 ± 10.56 | 0.31 ± 0.27 | 0.39 ± 029 | |
Niu et al. |
0.26 ± 0.20 | 48.88 ± 11.20 | 6.26 ± 3.02 | 0.28 ± 0.25 | 0.56 ± 0.26 | |
Sedlar et al. |
0.20 ± 0.19 | 58.30 ± 20.02 | 11.19 ± 9.10 | 0.23 ± 0.24 | 0.40 ± 0.29 | |
Rivera et al. |
0.19 ± 0.16 | 63.58 ± 18.58 | 11.13 ± 7.89 | 0.27 ± 0.25 | 0.21 ± 0.17 | |
Islam et al. |
0.19 ± 0.18 | 64.15 ± 28.51 | 14.17 ± 15.80 | 0.29 ± 0.28 | 0.25 ± 0.25 | |
Chengwei et al. |
0.18 ± 0.17 | 65.95 ± 25.94 | 9.22 ± 6.99 | 0.37 ± 0.30 | 0.21 ± 0.23 | |
Yoon et al. |
0.17 ± 0.16 | 45.23 ± 19.14 | 12.43 ± 11.01 | 0.23 ± 0.27 | 0.36 ± 0.32 | |
Baseline | 0.24 ± 0.20 | 53.29 ± 26.95 | 10.59 ± 4.98 | 0.27 ± 0.27 | 0.50 ± 0.35 | |
Proposal | 0.29 ± 0.22 | 47.17 ± 22.13 | 7.20 ± 4.14 | 0.26 ± 0.23 | 0.61 ± 0.28 |
Incorporating clinical information through the proposed custom loss function and the extra TICI channel resulted in a higher performance, in comparison to the baseline. Our proposal was able extract information from non-imaging data and to drive its training and testing phases toward better predictions. Therefore, the simultaneous incorporation of the reperfusion status, as an additional feature and in the loss function, improved performance of the classifier. In addition, we show the higher generalization capability of our proposal, since the performance metrics or our proposal for both datasets present less variation.
Although a previous work (
When comparing to the state-of-the-art methods, our proposal can reach competitive results, being placed among top scoring methods. With single model method, our proposal yields results within the top five methods, alongside ensemble approaches [e.g., Choi et al. (
Hausdorff Distance vs. Dice score from methods of ISLES 2017 in the testing database. Note that closer to the horizontal axis and further away from the origin is better (i.e., high Dice and low Hausdorff). Ensemble methods are marked with a purple plus.
From Figure
Prediction of stroke lesion outcome has the potential to assist interventionists when assessing the risks and benefits associated to mechanical thrombectomy. Therefore, having such tool can provide useful information during the clinical decision process.
In this work, we propose a novel deep learning architecture that beyond previously proposed architectures incorporates clinical information in a principled way. To do so, our proposal integrates clinical information at two different levels of the architecture. The first level considers the population domain-knowledge, achieved through the development of a custom loss function, to depict relationships between the TICI score and the tissue outcome prediction. The second level considers the patient-specific domain, where the TICI is encoded into an input channel of the architecture. From the latter level, we showed that our proposal was able to characterize different outcome scenarios of successful and unsuccessful reperfusion. Such methodology presents itself as a ground-breaking tool with potential to access the risks and benefits associated to the mechanical thrombectomy. The evaluation of our proposal was conducted on the publicly available ISLES 2017 dataset. We observe that the proposed method has benefited from the combination of imaging and non-imaging information. In addition, when comparing to the state-of-the-art methods, we observed that a single architecture with fewer parameters, such as ours, yields competitive performance metrics similar to more elaborate and/or ensemble methods.
However, there is still room from improvement since none of the current state-of-the-art methods, provides the robustness and accuracy needed for clinical practice, and are currently bellow the inter-rater performance of expert radiologists (DSC=0.58) (
The study utilizes anonymized data from the Bernese stroke registry, a prospectively collected database approved by the Kantonale Ethikkomission Bern. All patients were treated for an acute ischemic stroke at the University Hospital of Berne between 2005 and 2013. The study was performed according to the ethical guidelines of the Canton of Bern (Swiss Humanforschungsgesetz) with approval of our institutional review board (Kantonale Ethikkomission Bern). Some cases were supplied by the University Medical Center Schleswig-Holstein in Lübeck, Germany. They were acquired in diagnostic routine with varying resolutions, views, and imaging artifact load. A smaller group of cases were scanned at the Department of Neuroradiology at the Klinikum rechts der Isar in Munich, Germany. Both centers are equipped with 3T Phillips systems. The local ethics committee approved their release under Az.14-256A. Full data anonymization was ensured by removing all patient information from the files and the facial bone structure from the images.
AP is the main author of the research presented in the manuscript, being supervised by RM during an internship at Bern. CS, VA, RW and RM gave thoughtful insights during this research.
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.