- 1Stroke Medicine, Royal Berkshire NHS Foundation Trust, Reading, United Kingdom
- 2Stroke Medicine, Oxford University Hospitals NHS Trust, Oxford, United Kingdom
- 3Brainomix Limited, Oxford, United Kingdom
- 4Radcliffe Department of Medicine, University of Oxford, Oxford, United Kingdom
Introduction: In a drip-and-ship model for endovascular thrombectomy (EVT), early identification of large vessel occlusion (LVO) and timely referral to a comprehensive center (CSC) are crucial when patients are admitted to an acute stroke center (ASC). Several artificial intelligence (AI) decision-aid tools are increasingly being used to facilitate the rapid identification of LVO. This retrospective cohort study aimed to evaluate the impact of deploying e-Stroke AI decision support software in the hyperacute stroke pathway on process metrics and patient outcomes at an ASC in the United Kingdom.
Methods: Except for the deployment of e-Stroke on 01 March 2020, there were no significant changes made to the stroke pathway at the ASC. The data were obtained from a prospective stroke registry between 01 January 2019 and 31 March 2021. The outcomes were compared between the 14 months before and 12 months after the deployment of AI (pre-e-Stroke cohort vs. post-e-Stroke cohort) on 01 March 2020. Time window analyses were performed using Welch’s t-test. Cochran–Mantel–Haenszel test was used to compare changes in disability at 3 months assessed by modified Rankin Score (mRS) ordinal shift analysis, and Fisher’s exact test was used for dichotomised mRS analysis.
Results: In the pre-e-Stroke cohort, 19 of 22 patients referred received EVT. In the post-e-Stroke cohort, 21 of the 25 patients referred were treated. The mean door-in-door-out (DIDO) and door-to-referral times in pre-e-Stroke vs. post-e-Stroke cohorts were 141 vs. 79 min (difference 62 min, 95% CI 96.9–26.8 min, p < 0.001) and 71 vs. 44 min (difference 27 min, 95% CI 47.4–5.4 min, p = 0.01), respectively. The adjusted odds ratio (age and NIHSS) for mRS ordinal shift analysis at 3 months was 3.14 (95% CI 0.99–10.51, p = 0.06) and the dichotomized mRS 0–2 at 3 months was 16% vs. 48% (p = 0.04) in the pre- vs. post-e-Stroke cohorts, respectively.
Conclusion: In this single-center study in the United Kingdom, the DIDO time significantly decreased since the introduction of e-Stroke decision support software into an ASC hyperacute stroke pathway. The reduction in door-in to referral time indicates faster image interpretation and referral for EVT. There was an indication of an increased proportion of patients regaining independent function after EVT. However, this should be interpreted with caution given the small sample size. Larger, prospective studies and further systematic real-world evaluation are needed to demonstrate the widespread generalisability of these findings.
1 Introduction
Endovascular thrombectomy (EVT) is an established treatment for ischaemic stroke patients with large vessel occlusion (LVO) and has been shown in numerous studies to improve patient outcomes (1). Although selected patients may benefit from EVT up to 24 h after the onset of stroke, the benefit of treatment diminishes with increasing time between the onset of symptoms and treatment (2).
UK services to provide EVT are built around a ‘drip and ship’ model where those patients presenting at acute stroke centers (ASCs) within a geographical region, who are eligible for EVT, are transferred urgently to a comprehensive stroke center (CSC) after intravenous thrombolytic therapy, if indicated. The drip-and-ship model inevitably introduces a delay in EVT time to treatment for patients presenting to an ASC compared to those who present directly to a CSC.
Early identification of LVO and timely treatment are critical to achieving optimal clinical outcomes, particularly when patients require transfer from an ASC to a CSC. In a drip-and-ship model, for transferred patients to achieve the most benefit, the door-in-door-out (DIDO) time at an ASC should be less than 60 min (3, 4). One factor that has been identified as preventing this is a lack of expertise to quickly diagnose an LVO at the ASC, especially at times when expert staff are less likely to be available. Timely and accurate interpretation of imaging has been identified as an element of the patient pathway where an artificial intelligence (AI) decision support tool may be used to improve decision-making and expedite care (5–7).
e-Stroke (Brainomix, United Kingdom) is a software medical device for processing CT scans for stroke patients. It uses AI to analyse stroke CT brain image data to provide automated detection of specific clinically validated imaging features (8, 9), including identification and quantification of early ischaemic change and hemorrhage, and identification of large vessel occlusion (10). The version used in this study comprised two separate image assessment modules of e-Stroke: e-ASPECTS, and e-CTA for non-contrast CT (NCCT) and CT angiography (CTA), respectively (Figure 1).
Figure 1. (A) e-ASPECTS. The heat map indicates early CT changes in NCCT and (B) e-CTA. The red circle indicates LVO in the CTA.
In this study, we aimed to evaluate the impact of the introduction of e-Stroke into the hyperacute stroke pathway at a representative ASC in the United Kingdom, assessing changes over time in relevant time metrics and clinical endpoints. We used STrengthening the Reporting of OBservational Studies in Epidemiology (STROBE) criteria to report its findings (11).
2 Materials and methods
A retrospective cohort study design was used to analyse the impact of e-Stroke before and after it was deployed into the stroke pathway at an ASC in the UK on 1 March 2020. All patients who were referred and underwent EVT were eligible and included in the analysis. The pre-e-Stroke cohort included patients referred for and underwent EVT from 01 January 2019 to 29 February 2020, and the post-e-Stroke cohort included patients from 01 March 2020 to 31 March 2021. We measured the following pre-specified endpoints: door-in-door-out time (DIDO), broken down into door-in-to-referral time (DTR), referral to acceptance time (RTA), acceptance-to-door-out time, and mRS at 3 months and dichotomised mRS 0–2 and mRS 5–6 between groups.
The data on age, sex, admission National Institute of Health Stroke Scale (NIHSS), and comorbidity were extracted from the Stroke Sentinel National Audit Program (SSNAP) and the local stroke registry. As this was a service evaluation at a single-center where routinely collected anonymised data were utilised, a separate ethical approval was not sought. All patients who were referred for EVT from the center were included in the registry, and the data on time-stamps and mRS at 3 months were collected prospectively. No major changes to the stroke pathway were made during the study period at this center other than the implementation of e-Stroke and remote decision-making facilitated by e-Stroke during the first wave of COVID-19.
2.1 Stroke pathway
The study ASC serves a population of approximately 600,000. Patients presenting with acute ischaemic stroke who are eligible for EVT are referred to one of the two regional CSCs, depending on the EVT service hours in the CSCs. Before the patient arrives at the emergency department (ED), the ambulance crew pre-alerts the ED team at the ASC for any suspected acute stroke patients, which triggers an acute stroke call to the local stroke team. The stroke team meets the patient on arrival at the ED, undertakes a clinical review, and then the patient is transferred to the CT unit for, NCCT and CTA.
The stroke team is led by a stroke specialist clinician during daytime hours or an emergency department physician after hours and at weekends, with stroke specialists available remotely for advice as required. All stroke specialists are consultant physicians with specialist training in stroke medicine or neurology. The team also includes a stroke specialist nurse at all hours. A trainee doctor and/or physician associate also support the team.
NCCT images are reviewed by a stroke specialist or an ED physician. However, not all stroke specialists or ED physicians have expertise in reviewing CTA images, and a radiologist review is often sought to confirm findings. Prior to e-Stroke implementation, the specialists had to use a laptop computer to access CT images remotely, and the images were transferred to CSCs via the picture archiving and communication system (PACS) or image exporting portal (IEP). Potentially EVT-eligible patients are referred to the on-call stroke specialist at the CSC and accepted after reviewing the images and eligibility. Subsequently, an ambulance transfer is arranged from the ASC to the CSC. In the meantime, eligible patients are given intravenous thrombolysis (Figure 2).
2.2 Introduction of e-Stroke
e-Stroke was implemented in March 2020. Acute stroke images (NCCT and CTA) were directly sent to the e-Stroke server from the CT scanner in parallel to being sent to PACS (Figure 2). e-Stroke result outputs were available on the PACS image viewer as well as on a mobile app, typically in less than 2 min from scan acquisition. In addition, one of the CSCs (CSC1) was linked via an integrated cloud-based image worklist to enable immediate review of e-Stroke images. The other CSC (CSC2) was able to view the e-Stroke images through image-sharing functionality via the e-Stroke app. The software was implemented as part of a proactive quality improvement initiative.
Mandatory face-to-face training was provided to all the clinical users before they were given access to the e-Stroke software. Additional training was provided to radiographers. Brainomix also supported a virtual training platform for user onboarding.
The pre-hospital alert system, the composition of the acute stroke team, and other elements of the pre-existing protocol for the stroke pathway were not changed during the study period, except for stroke specialists making treatment decisions remotely for stroke patients admitted to the ED with suspected symptoms of COVID-19 during the first wave of the pandemic. The remote communication was facilitated by instant WhatsApp imaging between the stroke nurse and stroke consultant group. The legal assurance by the Information Commissioners’ Office to the NHS England for adopting digital technologies for information sharing during COVID-19 supported this change (12).
2.3 Statistical analysis
Door-in-door-out, door-in to referral, referral to acceptance, and acceptance to door-out time comparisons before and after the introduction of AI-based decision support were performed using Welch’s t-test. To assess clinical outcomes, both shift and dichotomised analyses were used for the modified Rankin Scale (mRS) at 3 months, in keeping with standard practice. The distribution of mRS scores was compared using the Cochran–Mantel–Haenszel test. Proportional odds model regression was used for mRS shift odds ratios, confidence intervals, and adjustment for age and NIHSS. Dichotomised mRS 0–2, mRS 5–6, and baseline characteristics were compared between groups with Fisher’s exact test. All analyses were performed in R version 3.6.3 (13) and figures were drawn using ggplot2.
3 Results
During the 14-month period from 01 January 2019 to 29 February 2020, 846 patients were admitted to the ASC with acute stroke and 22 were referred to a CSC for EVT (pre-e-Stroke cohort). Of which, 19 patients received EVT. During the 12-month period following e-Stroke deployment (1 March 2020 to 31 March 2021), 785 patients were admitted with acute stroke and 25 were referred for EVT (post-e-Stroke cohort). Of which, 21 patients underwent EVT. The EVT referral rate in the pre- and post-e-Stroke cohorts was 2.6% vs. 3.2%, respectively (p = 0.55).
In the post-e-Stroke cohort, 13 of 25 (52%) patients were transferred to CSC1, which was linked via a cloud-based e-Stroke image sharing network. In the pre-e-Stroke cohort, 10 of 22 (45%) patients were transferred to CSC1. This difference was not statistically significant (p = 0.77).
Baseline patient characteristics are shown in Table 1. In the post-e-Stroke cohort, patients were more frequently older and had slightly less severe stroke severity. Although there were numerically more patients with vertebrobasilar occlusion in the pre-e-Stroke cohort, this difference is not statistically significant.
The mean treatment decision time differences between pre- and post-e-Stroke cohorts are as follows (Figure 3): door-in to referral time reduced from 71 min to 44 min (difference 27 min, 95% CI 47.4–5.4 min, p = 0.01), referral to acceptance time reduced from 29 min to 17 min (difference 12 min, 95% CI 24.9–0.6 min, p = 0.06), acceptance to door-out time reduced from 38 to 25 min (difference 13 min, 95% CI 26.4–0.19, p = 0.053), and door-in door-out (DIDO) time reduced from 141 min to 79 min (difference 62 min, 95% CI 96.9–26.8 min, p < 0.001).
Figure 3. Time differences in the pre- and post-e-Stroke cohorts; (A) door-in to referral, (B) referral to acceptance, (C) acceptance to door out, and (D) door-in-door-out.
The mRS shift analysis of the patients who received EVT pre- and post-e-Stroke implementation, (19 and 21 patients, respectively) was not statistically significant, with only a trend towards improved outcomes following AI implementation. The adjusted (age and NIHSS) odds ratio for mRS ordinal shift analysis at 3 months was 3.14 (95% CI 0.99–10.51, p = 0.06) (Figure 4). Dichotomised mRS 0–2 at 3 months was achieved by 16% vs. 48% of patients for pre- and post-e-Stroke cohorts, respectively (p = 0.046); there were no cases with mRS 2. There was no significant difference in dichotomised mRS 5–6 at 3 months (53% before vs. 29% following implementation; p = 0.20) or in mortality (32% before vs. 19% following implementation, p = 0.47).
Figure 4. Three-month mRS shift analysis of the patients who received EVT pre- and post-e-Stroke implementation, 19 and 21 patients, respectively.
4 Discussion
Overall, in this exploratory study, we found an improvement in important time and patient flow metrics, particularly door-in-door-out time, after the implementation of the e-Stroke software. The ordinal shift analysis of mRS at 3 months did not reach significance. However, the dichotomised mRS analysis showed a trend towards improved clinical outcomes. The implementation of e-Stroke did not have a significant impact on EVT referral rates at the study ASC.
4.1 Time metrics
The results show a significant reduction in the mean DIDO time of 62 min, from 141 min to 79 min after the introduction of e-Stroke (95% CI 96.9–26.8 min, p < 0.001). As described above, no other significant changes occurred within the stroke pathway at the ASC during this time. Furthermore, the proportion of patients referred to each CSC was almost equal. Therefore, the reduction in DIDO time can be attributed to the improved efficiency of the existing patient assessment and referral pathway. The breakdown of the time segments that comprise the DIDO time shows that the impact was primarily driven by a reduction in door-to-referral time, likely reflecting improved patient identification at the ASC facilitated by e-Stroke.
The ambulance waiting time for secondary transfers can also affect the DIDO time. Our data show that the mean acceptance to door-out time reduced from 38 to 25 min (95% CI 26.4–0.19, p = 0.053). This trend may be explained by enabling faster secondary transfers when ambulance crews had not left the ASC by the time of the decision, providing availability for quicker secondary transfers. However, this information was not collected in the study.
The study period covers the pre- and post-COVID-19 pandemic. According to the Health Foundation Trust’s data for the United Kingdom, during the first wave of the COVID-19 pandemic, ambulance waiting time, patients waiting for more than 4 hours for an ambulance and staff sickness were on the rise (14). Further, McConachie et al. (15) have reported a 27% reduction in the number of EVTs performed in the UK and process delays due to the need for patient testing and isolating during the first wave of COVID-19. Therefore, the COVID-19 pandemic is likely to have had a negative impact on the processes around acute stroke care. Nevertheless, our study demonstrates the improved efficacy of the pathway with the introduction of e-Stroke.
The study ASC is one of the high-performing stroke centers in the United Kingdom for delivering hyperacute stroke care. The center consistently achieved an A rating (the rating ranges from A – the highest to E – the lowest) in the national audit data (SSNAP), with an IVT rate of more than 20% (national average 11%) and a median door-to-needle time of 30 min (national average 55 min) (16). The significant reduction in DIDO time in our study shows that even in high-performing centers, significant improvements can be attained in the EVT referral pathway with the help of e-Stroke.
4.2 Clinical outcomes
Dichotomised mRS 0–2 at 3 months, showing the proportion of patients achieving functional independence, increased 3-fold in the post-e-Stroke cohort (16% vs. 48%, p = 0.046). There was also a trend for improvement in the mRS shift analysis, but this did not reach statistical significance. It is important to consider that in a complex multi-step stroke pathway, it is difficult to directly attribute improved outcomes to a single intervention. However, in a recent study, Flores et al. (17) have shown that in transferred patients undergoing EVT, DIDO has a significant impact on clinical outcomes. Therefore, the significant reduction in DIDO time in our study could be attributed to the trend observed in the mRS analysis.
This study is a real-world evaluation of the introduction of AI decision-aid software into a stroke pathway and its impact on clinically meaningful endpoints. Another study that evaluated the impact of e-Stroke in a stroke pathway demonstrated an increase in the rate and timing of thrombolysis but did not describe the impact on the broader patient pathway metrics (7). Our study reports patient outcomes linked to the implementation of AI software into a National Health Service (NHS) stroke pathway as well as the process metrics.
In the transformation of any complex clinical pathway, human factors play a key role in the adoption of the change. However, the perception, usage, or impact of e-Stroke at an individual clinician level was outside the scope of this study. Furthermore, an assumption was made that the e-Stroke was used similarly in every case, but it is unlikely to have happened in real life. All stroke scans were automatically analysed by e-Stroke software. Although it was an individual clinician’s decision whether the outputs were used, the cloud-linked images of AI output had been used during referral to CSC. Therefore, one could assume the e-Stroke would have been used in most cases.
Our study is limited by a relatively small sample size owing to the volume of transferred cases at our center, and will require validation in larger datasets. This study focused on the pathway in an ASC and did not consider procedural success, pathway efficiency at CSCs, and access to rehabilitation that contributes to patient outcomes. Despite the limitations of a single-center, retrospective design, we believe this study adds important evidence to help understand the potential benefits of AI software in an acute stroke pathway.
5 Conclusion
In this single-center study in the United Kingdom, the DIDO time significantly decreased since the introduction of e-Stroke decision support software into an ASC hyperacute stroke pathway. The reduction in door-in to referral time indicates faster image interpretation and referral for EVT. There was an indication of an increased proportion of patients regaining independent function after EVT. However, this should be interpreted with caution, given the small sample size. Larger, prospective studies and further systematic real-world evaluations are needed to demonstrate the widespread generalisability of these findings.
Data availability statement
The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.
Ethics statement
Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. Written informed consent from the patients/participants or patients/participants' legal guardian/next of kin was not required to participate in this study in accordance with the national legislation and the institutional requirements.
Author contributions
KN: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Validation, Writing – original draft, Writing – review & editing. AN: Data curation, Formal analysis, Methodology, Validation, Writing – review & editing. JB: Writing – review & editing. GF: Writing – review & editing. ZW: Writing – review & editing. DM: Data curation, Writing – review & editing. GH: Writing – review & editing.
Funding
The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.
Conflict of interest
JB, ZW, and GH were employed by Brainomix Limited. KN received honoraria for lectures from Brainomix Limited.
The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
1. Goyal, M, Menon, BK, Van Zwam, WH, Dippel, DWJ, Mitchell, PJ, Demchuk, AM, et al. Endovascular thrombectomy after large-vessel ischaemic stroke: a meta-analysis of individual patient data from five randomised trials. Lancet. (2016) 387:1723–31. doi: 10.1016/S0140-6736(16)00163-X
2. Saver, JL, Goyal, M, Van Der Lugt, A, Menon, BK, Majoie, CBLM, Dippel, DW, et al. Time to treatment with endovascular thrombectomy and outcomes from ischemic stroke: Ameta-analysis. JAMA. (2016) 316:1279. doi: 10.1001/jama.2016.13647
3. Pérez De La Ossa, N, Abilleira, S, Jovin, TG, García-Tornel, Á, Jimenez, X, Urra, X, et al. Effect of direct transportation to Thrombectomy-capable center vs local stroke center on neurological outcomes in patients with suspected large-vessel occlusion stroke in nonurban areas: the RACECAT randomized clinical trial. JAMA. (2022) 327:1782–94. doi: 10.1001/jama.2022.4404
4. Holodinsky, JK, Williamson, TS, Demchuk, AM, Zhao, H, Zhu, L, Francis, MJ, et al. Modeling stroke patient transport for all patients with suspected large-vessel occlusion. JAMA Neurol. (2018) 75:1477–86. doi: 10.1001/jamaneurol.2018.2424
5. Dusenbury, W, Mathiesen, C, Whaley, M, Adeoye, O, Leslie-Mazwi, T, Williams, S, et al. Ideal foundational requirements for stroke program development and growth: a scientific statement from the American Heart Association. Stroke. (2023) 54:e175–87. doi: 10.1161/STR.0000000000000424
6. Campbell, BCV. Optimal imaging at the primary stroke center. Stroke. (2020) 51:1932–40. doi: 10.1161/STROKEAHA.119.026734
7. Gunda, B, Neuhaus, A, Sipos, I, Stang, R, Böjti, PP, Takács, T, et al. Improved stroke Care in a Primary Stroke Centre Using AI-decision support. Cerebrovasc Dis Extra. (2022) 12:28–32. doi: 10.1159/000522423
8. Mallon, DH, Taylor, EJR, Vittay, OI, Sheeka, A, Doig, D, and Lobotesis, K. Comparison of automated ASPECTS, large vessel occlusion detection and CTP analysis provided by Brainomix and RapidAI in patients with suspected ischaemic stroke. J Stroke Cerebrovasc Dis. (2022) 31:106702. doi: 10.1016/j.jstrokecerebrovasdis.2022.106702
9. Brinjikji, W, Abbasi, M, Arnold, C, Benson, JC, Braksick, SA, Campeau, N, et al. E-ASPECTS software improves interobserver agreement and accuracy of interpretation of aspects score. Interv Neuroradiol. (2021) 27:781–7. doi: 10.1177/15910199211011861
10. Seker, F, Pfaff, JAR, Mokli, Y, Berberich, A, Namias, R, Gerry, S, et al. Diagnostic accuracy of automated occlusion detection in CT angiography using e-CTA. Int J Stroke. (2022) 17:77–82. doi: 10.1177/1747493021992592
11. von Elm, E, Altman, DG, Egger, M, Pocock, SJ, Gøtzsche, PC, and Vandenbroucke, JP. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Int J Surg. (2014) 12:1495–9. doi: 10.1016/j.ijsu.2014.07.013
12. Nagaratnam, K, Harston, G, Floßmann, E, Canavan, C, Geraldes, RC, and Edwards, C. Innovative use of artificial intelligence and digital communication in acute stroke pathway in response to COVID-19. Future Healthc J. (2020) 7:169–73. doi: 10.7861/fhj.2020-0034
13. Team RC (2023). R Core team 2023 R: A language and environment for statistical computing. R foundation for statistical computing. Available at: https://www.R-project.org/
14. The Health Foundation. Why have ambulance waiting times been getting worse? - the Health Foundation. The Health Foundation. Available at: https://www.health.org.uk/publications/long-reads/why-have-ambulance-waiting-times-been-getting-worse?gclid=Cj0KCQiA7OqrBhD9ARIsAK3UXh1V-2AcYViLQtUri8OdYL0OCMMVYiQa9yNbfKzK9edWQIliHejVBIQaAssNEALw_wcB
15. McConachie, D, McConachie, N, White, P, Crossley, R, and Izzath, W. Mechanical thrombectomy for acute ischaemic stroke during the COVID-19 pandemic: changes to UK practice and lessons learned. Clin Radiol. (2020) 75:795.e7–795.e13. doi: 10.1016/j.crad.2020.07.001
16. SSNAP. Home. Available at: https://www.strokeaudit.org/
Keywords: stroke, artificial intelligence, endovascular thrombectomy, clinical outcome analysis, door-in-door-out time
Citation: Nagaratnam K, Neuhaus A, Briggs JH, Ford GA, Woodhead ZVJ, Maharjan D and Harston G (2024) Artificial intelligence-based decision support software to improve the efficacy of acute stroke pathway in the NHS: an observational study. Front. Neurol. 14:1329643. doi: 10.3389/fneur.2023.1329643
Edited by:
Lucio D'Anna, Imperial College London, United KingdomReviewed by:
Raffaele Ornello, University of L'Aquila, ItalyVivek Yedavalli, Johns Hopkins Medicine, United States
Abid Malik, Imperial College London, United Kingdom
Copyright © 2024 Nagaratnam, Neuhaus, Briggs, Ford, Woodhead, Maharjan and Harston. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Kiruba Nagaratnam, kiruba.nagaratnam@nhs.net