Clinical assessment of an AI tool for measuring biventricular parameters on cardiac MR

Introduction

Cardiac magnetic resonance (CMR) allows detailed, non-invasive assessment of the heart and its function. CMR can yield a variety of quantitative metrics - such as tissue characterization (including scar assessment) as well as quantitative assessment (chamber volumes, function and myocardial wall thickness - that have diagnostic and prognostic value in the acute and outpatient settings for several cardiopulmonary diseases, including ischemic evaluation, cardiomyopathy, myocarditis, and pulmonary hypertension (1–5).

The demand for CMR is increasing, and this is reflected in the growing inclusion of CMR in European Society of Cardiology guidelines, including those for coronary artery disease and heart failure (6, 7). There has been a rapid increase in the number of clinical CMR studies and demand is increasing 15%/year (8). In the UK, 114,967 scans are performed yearly as part of routine clinical practice (8). Locally, our tertiary UK centre has seen a continuous increase in the number of CMR studies being performed, with over 2,000 scans now performed in 2023 compared to just under 400 in 2008. Currently, each scan is evaluated manually by a consultant cardiac radiologist or other expert CMR reporter. In order to obtain cardiac volumetric and functional measurements, contours must be drawn manually around each of the cardiac chambers, a process that is time-consuming, repetitive and prone to inter-observer error.

The growth in demand for CMR is occurring in an era of worsening reporter shortages (9). The implications are reporting backlogs and delayed scan reporting, resulting in delays to patient diagnosis and management. The number of CMR consultants is estimated to be 5.4 per million/population in England, while the prevalence of cardiac disease is rising each year (8–10). The average waiting time for outpatient referrals was 41 days across England in 2018, which is considerably higher since the COVID-19 pandemic. According to the Royal College of Radiologists, it would take an additional 406 consultants employed overnight to be able to clear a 6 week backlog of scans in one month (8, 11, 12). Artificial intelligence (AI) solutions are being proposed to help alleviate the mismatch between demand and reporting capacity, with the NHS now actively funding AI solutions in this area (13).

Recent years have seen the emergence of AI tools that aim to automate repetitive and labour-intensive aspects of CMR evaluation. AI tools offer potential benefits including improved efficiency and reproducibility of contouring and segmentation (14, 15). While these offer an attractive solution to the rising demand for CMR and have demonstrated excellent results in non-clinical testing, the clinical deployment of these tools and their performance in “real world” situations remains poorly understood (16). This study aimed to assess the utility of a locally developed AI tool automated measurement of CMR volumes.

Methods

CMR study selection

CMR studies performed for any indication at our tertiary centre between January and October 2022 were eligible for inclusion prospectively if both manual and automated CMR measurements were available.

Manual evaluation

Manual segmentation was performed for each included CMR study by two CMR radiographers (RT and TLD). Manual segmentation is performed at a later date to the study using the Siemens Syngo.via that does not contain a deep learning segmentation algorithm, these segmentations were drawn by the radiographers (Figure 1). Short-axis stack images were used, with epicardial and endocardial contours of the left ventricle and endocardial contours of the right ventricle for end systolic and end diastolic phase. Trabeculations were included in the blood pool and the outflow tract was included for both ventricles. The following measurements were obtained for each included CMR study: end-diastolic volume (EDV), end-systolic volume (ESV), systolic volume (SV), myocardial mass and ejection fraction (EF).

Figure 1

Figure 1. (A,B) automatic segmentation (A) and manual segmentation (B) of left ventricular volume showing good agreement. (C,D) automatic segmentation (C), manual segmentation (D) of left ventricular volume showing moderate disagreement.

Evaluation by AI tool

Following the acquisition of each included CMR study, the images were automatically evaluated by the AI tool. The development and testing of the tool is described in detail elsewhere (17). In short, the tool is based on a convolutional neural network comprising a UNET-like architecture with 16 convolutional layers. The tool was trained using a total of 611 CMR studies from 539 patients with various cardiac abnormalities from Sheffield Teaching Hospitals NHS Foundation Trust (Sheffield, UK) and Semmelweis University (Budapest, Hungary). Testing was performed using CMR studies from a total of 4047 patients from the same centres and an external dataset from 32 centres across the UK. For each included CMR study, the AI tool automatically performed segmentation of both ventricles and provided automated measurements of EDV, ESV, SV, myocardial mass and EF across the full cardiac cycle.

Two consultant radiologists (AJS and KK) with 12 and 15 years specialist cardiac imaging experience visually assessed the automatic contours prospectively and rated them on the following disagreement scale; none, minor, moderate, and major. Those rated as minor were considered to have volume differences that were not clinically relevant, whereas the moderate and major groups represented errors that may affect the volume results.

Data collection and analyses

The clinical indication and final CMR diagnosis were reviewed for included studies retrospectively. These were obtained from radiological reports and information provided by clinicians for the scan. All of the scans were reported by cardiothoracic radiology consultants working in the centre. Additionally, the time taken for manual segmentation was measured prospectively on a subset of cases (Figure 2). The time for manual segmentation includes the entire process, from loading images, selecting relevant slices, the segmentation process itself and finalising contours to produce the measurements.

Figure 2

Figure 2. Study design and proportion of studies for each analysis

Statistical analysis and graph production were performed using RStudio (2022.07.1 running R 4.2.1.) and Prism (version 9.4.1; San Diego, CA, USA). Continuous data were compared using the paired T test, and categorical data compared using the chi-squared test, with a significance threshold of p < 0.05. Agreement between automated and manual measurements was assessed using the intraclass correlation coefficient (ICC), with values interpreted according to established thresholds: <0.6 poor, 0.6–0.8 good, >0.8 excellent (18). Bias between the measurements was also assessed using Bland-Altman analysis; these results are presented in accordance with published guidelines (19). The following measurements were compared for both right and left ventricles: EDV, ESV, SV, EF and mass.

Results

Included cases

462 CMR studies from 462 consecutive patients were included in the study (62.0% male, median age 57 years). Most patients were white British (68.0%); however, ethnicity information was not available for 19% of patients.

The indications for undergoing CMR included: known structural disease (29.5%) such as dilated cardiomyopathy, hypertrophic cardiomyopathy, arrhythmogenic right ventricular cardiomyopathy (ARVC), hypertrophy or outflow obstruction; arrhythmia (25.0%), such as ventricular fibrillation, ventricular tachycardia, atrial fibrillation or ectopics; symptomatic patients (14.0%) with chest pain, shortness of breath or syncope; evaluation of cardiac function (11.0%); surveillance imaging (7.0%) for conditions such as Tetralogy of Fallot or aortic aneurysm; heart failure (5.0%), myocarditis (4.0%), valve disease (2.0%), vascular disease (2.0%), and COVID (0.5%). The CMR diagnoses included: ventricular dysfunction (24.5%); followed by no CMR alterations (21.0%); dilatation (ventricles, atria or vessels) (18.0%); left ventricular hypertrophy (12.5%); myocardial infarction (11.0%); fibrosis (3.0%); pericarditis (3.0%); valve disease (3.0%); pulmonary hypertension (2.0%); thickening (1.0%), thrombus (1.0%) (Figure 3).

Figure 3

Figure 3. A Sankey diagram outlining the flow between indication (left) and CMR diagnoses (right).

Comparison between automated and manual measurements

No significant difference was identified between the distribution of manual and automated ventricular measurements (Table 1). There was excellent agreement between the automated and manual measurements, with ICC values exceeding 0.85 in all cases (Table 2) and minimal bias on Bland-Altman analysis (Table 3, Figure 4).

Table 1

Table 1. Mean and standard deviation (SD) for left and right ventricular end-diastolic volume (EDV), end-systolic volume (ESV), systolic volume (SV) in millilitres, ejection fraction (EF) and mass in grams.

Table 2

Table 2. Intraclass correlation coefficient (ICC) for left and right ventricular end-diastolic volume (EDV), end-systolic volume (ESV), systolic volume (SV) in millilitres, ejection fraction (EF) and mass in grams.

Table 3

Table 3. Bias and standard deviation (SD) with 95% confidence interval (CI) for left and right ventricular end-diastolic volume (EDV), end-systolic volume (ESV), systolic volume (SV) in millilitres, ejection fraction (EF) and mass in grams.

Figure 4

Figure 4. Bland-Altmann plot for left ventricular ejection fraction (A) and right ventricular ejection fraction (B) demonstrating a small amount of bias between automatic and manual measurements.

Radiologist assessment of automated contours

The automated contours for 251 of the cases were assessed visually by a consultant radiologist prospectively. The radiologist agreed with the contours in 91.0% of cases, including 60.0% with no disagreement and 31.0% with minor disagreement (Figure 5). Moderate or major disagreement was reported in 8.0% and 0.5%, of cases respectively. Additionally, the model failed to produce any segmentation in a single case (0.4%). Disagreement was seen most commonly at the cardiac apex (29.0% of cases with disagreement), or at both the apex and base (14.0%).

Figure 5

Figure 5. Level of agreement based on visual assessment of automatic contours. None and minor—no clinically relevant difference; moderate and major—clinically significant difference; failure—unable to produce any contouring.

Time required for automated and manual segmentation

The length of time taken to create manual segmentation of the end-systolic and end-diastolic phase was available in 100 of CMR studies. The mean length of time across all scans was 29.1 min ± 13.9 (SD). Biventricular segmentation takes considerably longer than the left ventricle alone (Table 4), whilst a handful of cases also required flow segmentation.

Table 4

Table 4. The length of time taken for manual segmentation in minutes for a sample of 100 cases. Biventricular segmentation takes considerably longer than left ventricular segmentation alone.

In contrast, the automatic method took 30 s for Siemens and 40 s for GE scanners to produce a biventricular segmentation for the full cardiac cycle. Data transfer, software initiation and graph production added an extra 45 s taking the total to a maximum of 1 min 25 s.

Discussion

AI tools for automated segmentation of cardiac structures on CMR have the potential to improve clinical workflows. Here, we assessed the performance of our previously reported AI tool in a single centre consecutive cohort of 462 patients undergoing CMR for a variety of indications. For each CMR study, biventricular measurements of EDV, ESV, SV, EF and myocardial mass were obtained using both automated segmentation by the AI tool and traditional manual segmentation. We showed excellent agreement between the automated and manual measurements for all metrics, infrequent disagreement of cardiac radiologists with the automated contours, and considerably faster segmentation using the automated method. The results demonstrate non-inferiority of the AI tool to expert CMR segmentation in a heterogeneous real-world clinical cohort and highlight the potential of automated segmentation to improve the efficiency of CMR reporting.

Agreement between the measurements derived from automated and manual segmentation was excellent for all metrics in both ventricles. The high ICC values, minimal bias on Bland-Altman analysis and lack of statistically significant differences in measurements is indicative of robust results. Left ventricular measurements showed levels of agreement consistent with those observed previously (20). However, the right ventricle showed a slightly lower level of agreement, which has been observed in other studies and is thought to be due to significant variability in shape and intricate movement (21–23).

Furthermore, visual assessment of the automatic contours found infrequent radiologist disagreement, which is unsurprising given the close agreement in the measurements derived from automated and manual segmentation. Qualitative visual assessment confirmed the automatic contours to be accurate and reliable without needing further adjustments in 91% of cases. Our findings are highly similar to Bai et al. who have an agreement of 84.8 to 91.6% between automated and manual contours for mid-ventricular region (22). They also found the apex and base of the heart to be regions with the most disagreement likely due to the more complex anatomy and therefore more challenging contouring (22). Likewise, our cardiac radiologists frequently noted disagreements in the cardiac apex and base regions. Notably, our AI tool failed to perform segmentation in only a single case.

A key strength of this study was the evaluation of the AI tool's performance in a consecutive and unselected clinical cohort, with the included CMR studies encompassing a broad range of indications and radiological findings across multiple scanner systems. The evaluation of AI tools in their intended populations and settings ensures generalisability and is essential for their translation to routine clinical practice. The importance of generalisability is increasingly recognised and a number of multi-centre and multi-vendor studies have attempted to address this issue in the field of CMR (24, 25). Our findings add to the growing body of evidence that automatic CMR measurements derived from automated segmentation are both accurate and reliable compared to the existing standard of manual segmentation by CMR experts (21, 26).

Additionally, we have demonstrated a significant reduction in the time taken to perform segmentation: the automated method using the AI tool was able to perform segmentation in under 90 s, compared to around thirty minutes for a manual approach. The range of times for manual segmentation was due to the experience of the operators, the complexity of the cases and the quality of images. Cases with shorter time are when the operator has been working efficiently by loading cases simultaneously and the images were of high quality. It is important to note that whilst the manual segmentation is only performed at the end-systolic and end-diastolic phase, the automatic segmentation is throughout the whole cardiac cycle producing more accurate results (Table 5). This emphasises the potential for AI tools to improve the speed and accuracy of CMR interpretation and reporting in an era in which the demands for CMR continue to rise.

Table 5

Table 5. Comparison of advantages and disadvantages of manual and automatic segmentation for cardiac segmentation.

There is also the wider issue of AI trustworthiness which is prevalent within cardiovascular imaging. Our model has some mitigating features as highlighted by Szabo et al. (27) such as the inclusion of multicentre data in the initial training and creating results that are explainable. This study also adds to this by demonstrating that it performs consistently in a real-world patient population without any exhibition of bias.

Our study is not without its limitations. Right ventricular segmentation is not routinely performed manually due to the additional time required - consequently, only left ventricular measurements were available for analysis in some of the included CMR studies. The AI-derived contours were assessed visually and qualitatively, with no direct quantitative comparison using a similarity score. Manual segmentations were also only performed by two experts at our centre, and it is unknown how these would vary if performed by other experts from different centres. Another limitation is the lack of comparison with commercially available algorithms provided by lenders such as Siemens and Medis. Future studies could build on our findings by examining the algorithm in a multicentre setting where more experts provide manual measurements allowing for the assessment of interobserver variability between them. Additionally, the algorithm could be compared to commercially available solutions in terms of both performance and time-saving benefits.

Conclusion

We evaluated an AI tool for the automated segmentation of both ventricles on CMR in a consecutive single-centre real-world clinical cohort. Excellent agreement was demonstrated between measurements derived from automated and manual segmentation using multiple statistical methods. Additionally, cardiac radiologists agreed with the AI-derived ventricular contours in the majority of cases. As expected, automated segmentation was performed considerably faster than the existing standard of manual segmentation. The findings suggest that the AI tool can undertake ventricular segmentation accurately and reliably, and has the potential to improve the efficiency of CMR reporting.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving humans were approved by Research ethics committee approval (17/YH/0142). The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

MaS: Data curation, Formal Analysis, Investigation, Methodology, Visualization, Writing – original draft, Writing – review & editing. AM: Methodology, Writing – review & editing. SS: Data curation, Writing – review & editing. ZA: Data curation, Visualization, Writing – review & editing. KK: Resources, Supervision, Writing – review & editing. RT: Data curation, Writing – review & editing. TL: Data curation, Writing – review & editing. DC: Writing – review & editing. KD: Methodology, Writing – review & editing. MiS: Investigation, Methodology, Writing – review & editing. PM: Data curation, Resources, Writing – review & editing. RV: Data curation, Resources, Software, Supervision, Writing – review & editing. SA: Data curation, Resources, Supervision, Writing – review & editing. AS: Conceptualization, Resources, Supervision, Writing – review & editing.

Funding

The author(s) declare financial support was received for the research, authorship, and/or publication of this article.

Study supported by the NIHR (grant no. AI_AWARD01706). AS supported in part by the Wellcome Trust (grant no. 205188/Z/16/Z). KD 222930/Z/21/Z and 4ward North 203914/Z/16.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The author(s) declared that they were an editorial board member of Frontiers, at the time of submission. This had no impact on the peer review process and the final decision.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Kanagala P, Cheng ASH, Singh A, McAdam J, Marsh A-M, Arnold JR, et al. Diagnostic and prognostic utility of cardiovascular magnetic resonance imaging in heart failure with preserved ejection fraction - implications for clinical trials. J Cardiovasc Magn Reson. (2018) 20(1):4. doi: 10.1186/s12968-017-0424-9

PubMed Abstract | Crossref Full Text | Google Scholar

2. Stukalova OV, Gupalo EM, Chumachenko PV, Samko AN, Butorova EA, Shakhnovich RM, et al. The value of cardiovascular magnetic resonance in myocarditis with different clinical presentation. Ter Arkh. (2019) 91:28–36. doi: 10.26442/00403660.2019.04.000078

PubMed Abstract | Crossref Full Text | Google Scholar

3. Lobeek M, Badings E, Lenssen M, Uijlings R, Koster K, van ‘t Riet E, et al. Diagnostic value of the electrocardiogram in the assessment of prior myocardial infarction. Neth Heart J. (2021) 29:142–50. doi: 10.1007/s12471-020-01515-w

PubMed Abstract | Crossref Full Text | Google Scholar

4. Aquaro GD, Ghebru Habtemicael Y, Camastra G, Monti L, Dellegrottaglie S, Moro C, et al. Prognostic value of repeating cardiac magnetic resonance in patients with acute myocarditis. J Am Coll Cardiol. (2019) 74:2439–48. doi: 10.1016/j.jacc.2019.08.1061

PubMed Abstract | Crossref Full Text | Google Scholar

5. Swift AJ, Capener D, Johns C, Hamilton N, Rothman A, Elliot C, et al. Magnetic resonance imaging in the prognostic evaluation of patients with pulmonary arterial hypertension. Am J Respir Crit Care Med. (2017) 196:228–39. doi: 10.1164/rccm.201611-2365OC

PubMed Abstract | Crossref Full Text | Google Scholar

6. Knuuti J, Wijns W, Saraste A, Capodanno D, Barbato E, Funck-Brentano C, et al. 2019 ESC guidelines for the diagnosis and management of chronic coronary syndromes. Eur Heart J. (2020) 41:407–77. doi: 10.1093/eurheartj/ehz425

PubMed Abstract | Crossref Full Text | Google Scholar

7. Ponikowski P, Voors AA, Anker SD, Bueno H, Cleland JGF, Coats AJS, et al. 2016 ESC guidelines for the diagnosis and treatment of acute and chronic heart failure: the task force for the diagnosis and treatment of acute and chronic heart failure of the European society of cardiology (ESC)Developed with the special contribution of the heart failure association (HFA) of the ESC. Eur Heart J. (2016) 37:2129–200. doi: 10.1093/eurheartj/ehw128

PubMed Abstract | Crossref Full Text | Google Scholar

8. Keenan NG, Captur G, McCann GP, Berry C, Myerson SG, Fairbairn T, et al. Regional variation in cardiovascular magnetic resonance service delivery across the UK. Heart. (2021) 107:1974–9. doi: 10.1136/heartjnl-2020-318667

PubMed Abstract | Crossref Full Text | Google Scholar

9. of Radiologists RC. Clinical radiology UK workforce census 2023 report. The Royal College of Radiologists London; (2023).

10. Facts and Figures. In: British Heart Foundation (cited June 7, 2023). Available online at: https://www.bhf.org.uk/what-we-do/news-from-the-bhf/contact-the-press-office/facts-and-figures.

11. State of the Wait (cited June 8, 2023). Available online at: https://www.rcr.ac.uk/press-and-policy/state-wait.

12. Griffin S. COVID-19: waiting times in England reach record highs. Br Med J. (2020) 370:m3557. doi: 10.1136/bmj.m3557

Crossref Full Text | Google Scholar

13. Health Education England Publishes Roadmap into Use of AI in the NHS. In: Health Education England. (cited June 8, 2023). Available online at: https://www.hee.nhs.uk/news-blogs-events/news/health-education-england-publishes-roadmap-use-ai-nhs (accessed February 10, 2022).

14. Küstner T, Munoz C, Psenicny A, Bustin A, Fuin N, Qi H, et al. Deep-learning based super-resolution for 3D isotropic coronary MR angiography in less than a minute. Magn Reson Med. (2021) 86:2837–52. doi: 10.1002/mrm.28911

Crossref Full Text | Google Scholar

15. Fahmy AS, El-Rewaidy H, Nezafat M, Nakamori S, Nezafat R. Automated analysis of cardiovascular magnetic resonance myocardial native T1 mapping images using fully convolutional neural networks. J Cardiovasc Magn Reson. (2019) 21(1):7. doi: 10.1186/s12968-018-0516-1

PubMed Abstract | Crossref Full Text | Google Scholar

16. Fotaki A, Puyol-Antón E, Chiribiri A, Botnar R, Pushparajah K, Prieto C. Artificial intelligence in cardiac MRI: is clinical adoption forthcoming? Front Cardiovasc Med. (2021) 8:818765. doi: 10.3389/fcvm.2021.818765

PubMed Abstract | Crossref Full Text | Google Scholar

17. Alabed S, Alandejani F, Dwivedi K, Karunasaagarar K, Sharkey M, Garg P, et al. Validation of artificial intelligence cardiac MRI measurements: relationship to heart catheterization and mortality prediction. Radiology. (2022) 305:68–79. doi: 10.1148/radiol.212929

PubMed Abstract | Crossref Full Text | Google Scholar

18. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. (2016) 15:155–63. doi: 10.1016/j.jcm.2016.02.012

PubMed Abstract | Crossref Full Text | Google Scholar

19. Gerke O. Reporting standards for a bland-altman agreement analysis: a review of methodological reviews. Diagnostics (Basel). (2020) 10(5):334. doi: 10.3390/diagnostics10050334

PubMed Abstract | Crossref Full Text | Google Scholar

20. Böttcher B, Beller E, Busse A, Cantré D, Yücel S, Öner A, et al. Fully automated quantification of left ventricular volumes and function in cardiac MRI: clinical evaluation of a deep learning-based algorithm. Int J Cardiovasc Imaging. (2020) 36:2239–47. doi: 10.1007/s10554-020-01935-0

Crossref Full Text | Google Scholar

21. Budai A, Suhai FI, Csorba K, Toth A, Szabo L, Vago H, et al. Fully automatic segmentation of right and left ventricle on short-axis cardiac MRI images. Comput Med Imaging Graph. (2020) 85:101786. doi: 10.1016/j.compmedimag.2020.101786

PubMed Abstract | Crossref Full Text | Google Scholar

22. Bai W, Sinclair M, Tarroni G, Oktay O, Rajchl M, Vaillant G, et al. Automated cardiovascular magnetic resonance image analysis with fully convolutional networks. J Cardiovasc Magn Reson. (2018) 20:65. doi: 10.1186/s12968-018-0471-x

PubMed Abstract | Crossref Full Text | Google Scholar

23. Yilmaz P, Wallecan K, Kristanto W, Aben J-P, Moelker A. Evaluation of a semi-automatic right ventricle segmentation method on short-axis MR images. J Digit Imaging. (2018) 31:670–9. doi: 10.1007/s10278-018-0061-3

PubMed Abstract | Crossref Full Text | Google Scholar

24. Tao Q, Yan W, Wang Y, Paiman EHM, Shamonin DP, Garg P, et al. Deep learning-based method for fully automatic quantification of left ventricle function from cine MR images: a multivendor, multicenter study. Radiology. (2019) 290:81–8. doi: 10.1148/radiol.2018180513

PubMed Abstract | Crossref Full Text | Google Scholar

25. Chen C, Bai W, Davies RH, Bhuva AN, Manisty CH, Augusto JB, et al. Improving the generalizability of convolutional neural network-based segmentation on CMR images. Front Cardiovasc Med. (2020) 7:105. doi: 10.3389/fcvm.2020.00105

PubMed Abstract | Crossref Full Text | Google Scholar

26. Sander J, de Vos BD, Išgum I. Automatic segmentation with detection of local segmentation failures in cardiac MRI. Sci Rep. (2020) 10:21769. doi: 10.1038/s41598-020-77733-4

PubMed Abstract | Crossref Full Text | Google Scholar

27. Szabo L, Raisi-Estabragh Z, Salih A, McCracken C, Ruiz Pujadas E, Gkontra P, et al. Clinician’s guide to trustworthy and responsible artificial intelligence in cardiovascular imaging. Front. Cardiovasc. Med. (2022) 9:1016032. doi: 10.3389/fcvm.2022.1016032

PubMed Abstract | Crossref Full Text | Google Scholar