Does artificial intelligence enhance physician interpretation of optical coherence tomography: insights from eye tracking

Background and objectives The adoption of optical coherence tomography (OCT) in percutaneous coronary intervention (PCI) is limited by need for real-time image interpretation expertise. Artificial intelligence (AI)-assisted Ultreon™ 2.0 software could address this barrier. We used eye tracking to understand how these software changes impact viewing efficiency and accuracy. Methods Eighteen interventional cardiologists and fellows at McMaster University, Canada, were included in the study and categorized as experienced or inexperienced based on lifetime OCT use. They were tasked with reviewing OCT images from both Ultreon™ 2.0 and AptiVue™ software platforms while their eye movements were recorded. Key metrics, such as time to first fixation on the area of interest, total task time, dwell time (time spent on the area of interest as a proportion of total task time), and interpretation accuracy, were evaluated using a mixed multivariate model. Results Physicians exhibited improved viewing efficiency with Ultreon™ 2.0, characterized by reduced time to first fixation (Ultreon™ 0.9 s vs. AptiVue™ 1.6 s, p = 0.007), reduced total task time (Ultreon™ 10.2 s vs. AptiVue™ 12.6 s, p = 0.006), and increased dwell time in the area of interest (Ultreon™ 58% vs. AptiVue™ 41%, p < 0.001). These effects were similar for experienced and inexperienced physicians. Accuracy of OCT image interpretation was preserved in both groups, with experienced physicians outperforming inexperienced physicians. Discussion Our study demonstrated that AI-enabled Ultreon™ 2.0 software can streamline the image interpretation process and improve viewing efficiency for both inexperienced and experienced physicians. Enhanced viewing efficiency implies reduced cognitive load potentially reducing the barriers for OCT adoption in PCI decision-making.


Introduction
To employ intracoronary imaging findings in procedural decisions, physicians require the ability to perform real-time image interpretation with speed and accuracy (1).The lack of such proficiency among many interventional cardiologists limits the adoption of optical coherence tomography (OCT) despite its clinical utility (2).The use of artificial intelligence (AI) to automate substantial portions of the image interpretation task may improve physician experience with imaging (3)(4)(5)(6).The image review software Ultreon TM 2.0 improves on AptiVue TM by using AI to detect calcium and external elastic lamina (EEL), reducing the need for physician identification of these structures.In addition, a simplified interface is used that toggles different representations of the vessel lumen rather than simultaneously showing multiple representations, thereby reducing the cognitive load on the physician (7,8).The goal of these changes is to streamline the interpretation process for less experienced physicians.
Eye-tracking studies are a way to gain insight into the ease with which physicians extract information from these two software platforms.Eye tracking has been used to investigate decisionmaking in other fields such as psychology, neuroscience, marketing, human-computer interaction, and medicine (9-18).It provides detailed information on where and how long participants focus on different elements of a visual scene, whether their searching is strategic and whether their gaze coincides with the location of the pertinent information of the visual scene (17,19).
In this study, an eye-tracking technology was used to understand how visual changes in the OCT software platform from AptiVue TM to Ultreon TM 2.0 impact the viewing efficiency and accuracy of physicians using OCT for percutaneous coronary intervention (PCI) decision-making.We compared experienced and inexperienced physicians to explore the impact differed among who were physicians who experienced vs. inexperienced with the AptiVue TM platform.

Methods
We conducted a descriptive study of eye-tracking patterns of interventional cardiologists reviewing OCT console images from both Ultreon TM 2.0 and AptiVue TM software platforms.

Participants
Interventional cardiologists and fellows in training were recruited from a tertiary center in Canada.Operators were considered inexperienced if they self-reported performing less than 50 OCT studies as primary operator with the AptiVue TM platform.Written informed consent was obtained from all participants.

Imaging materials and interpretation tasks
Six static images were selected for each of three OCT imaging tasks, namely, (1) calcium identification, (2) stent sizing, and (3) stent assessment.Two versions of each static image were prepared, one using Ultreon TM 2.0 and the other using AptiVue TM .Two counterbalanced sets of 18 images were constructed such that each set contained six images of each task, half on Ultreon TM 2.0 and half on AptiVue TM , but only one version of each image is required in each set so that a participant would only see each image once.Example images of morphology assessment, stent sizing, and stent review are shown in Figure 1 contrasting the visual interface of Ultreon TM 2.0 and AptiVue TM .
For calcium identification, participants were asked to interpret if the calcium arc was more or less than 180°based on the static image.For stent sizing, participants were required to give the correct stent diameter and length.Stent diameter was based on rounding down from the distal EEL diameter (if available) or rounding up from the distal lumen diameter.Stent length was based on the measurement identified by both software platforms.For stent assessment, participants needed to identify whether any OCT-identified concerns included a composite of (1) significant edge dissection (defined as a 60°arc and more than 3 mm length), ( 2) malapposition (defined as more than 300 μm from stent struts to the lumen), and (3) underexpansion (defined as less than 90% of the tapered reference) (20).If one or more features were not identified, the answer was deemed incorrect.

Training
All participants were oriented to both platforms.All inexperienced operators received formal training in the Ultreon TM 2.0 software.This training consisted of a standardized instruction tutorial provided by Abbott Vascular, structured in two separate videos on pre-and post-PCI image review following the MLD MAX algorithm (21).

Image review and eye tracking
Participants were randomized to review one of the two sets of 18 static images.Images were equally distributed between each of the three different tasks and each of the two console types.Participants provided an interpretation while their eye movements were tracked.Eye movements were recorded using an SMI RED 250 Hz eye tracker (SensoMotoric Instruments, Germany), a noninvasive tool which uses a small infrared camera mounted at the bottom of a laptop screen to follow the pupils.Areas of interest (AOI) specific to each task and software system were graphically identified within the static images and verified by two experts (GC and MS).Within each task, the most relevant area of interest (in special cases multiple areas of interest) was determined as the area of interest containing all necessary information to correctly complete the task as shown in Figure 2.

Outcomes
The primary study outcomes were viewing efficiency and interpretation accuracy.Viewing efficiency was assessed as follows: (a) time to first fixation (time taken for the participant to look at the target area of interest for the prescribed task), (b) total task time (time taken to complete the prescribed task), and (c) dwell time in target defined as the percentage time in target compared with total task time (s).Interpretation accuracy was defined as the proportion of correct responses provided by the participant.Heat maps of eye movements were created to qualitatively compare experienced vs. inexperienced operator eye movements.

Data analysis
For each participant and task, viewing efficiency and accuracy metrics were averaged and subjected to a mixed multivariate model.Platform type (AptiVue TM vs. Ultreon TM 2.0) and task type (identification of calcium, stent sizing, stent assessment) were treated as within-subject variables.Operator experience (inexperienced vs. experienced) was treated as a between-subject variable.Statistical analysis was performed using SPSS version 26 (IBM, Redmond).
Heat maps were reviewed by three OCT imaging experts (GC, TS, MS), to identify key themes in the viewing patterns using an inductive approach with constant iterative comparisons across platforms, experience levels, and task.

Results
Eighteen physicians participated in the study.Of these, 10 (56%) were practicing interventional cardiologists, and eight (44%) were fellow physicians [two (11%) were women].Twelve (67%) self-reported performing less than 50 OCTs as first operator and were considered inexperienced.Tables 1 and 2 summarize the results by experience level and task type.

Interpretation accuracy
Overall interpretation accuracy of the OCT images was 73% (95% CI 0.67-0.78).In a mixed multivariate analysis, the software platform did not affect interpretation accuracy (Ultreon TM 74%, 95% CI 67%-81%, vs. AptiVue TM 71%, 95% CI 64%-78%, Table 3).However, experienced participants outperformed inexperienced participants (87%, 95% CI 79%-95%, vs. 58%, 95% CI 52%-64%, p < 0.001).Interpretation accuracy also varied across task type (calcium identification 82%, 95% CI 73%-90%; stent sizing 56%, 95% CI 47%-64%; stent assessment 80%, 95% CI 72%-88%, p < 0.001).Interactions between the software platform and experience level or task type were not significant.Comparison of AptiVue TM and Ultreon TM 2.0 interfaces.In the AptiVue TM software, panel A provides a cross-sectional image with the possibility of utilizing measurement tools inside the curtain menu (*) (length, area, thickness, angle) and a visual estimation of the mean diameter (green text).In Ultreon TM 2.0, all information is at a glimpse in panel A1.AI-plaque morphology recognizes the calcium burden at a given threshold (orange arc), mean EEL diameter (dashed line -E), lumen diameter (L), and mean lumen area (LA) (green-highlighted boxes).B and C are fused together in Ultreon TM 2.0.This provides several information (highlighted green box): in orange the calcium-plaque burden above the preselected threshold, lesion length, EEL diameter, lumen diameter and LA at the proximal and distal references, and MLA.(A) Morphology assessment.In comparison with AptiVue TM , Ultreon TM 2.0 presents AIplaque recognition (A1) with orange band recognizing calcium.This is also presented in orange bands in the longitudinal/lumen profile (B) where the calcium is detected (*) based on the preselected threshold (in this case ≥180°).(B) Stent sizing.In comparison with AptiVue TM , Ultreon TM 2.0 presents AI recognition of EEL (dashed line -E) which is displayed in the cross-sectional area (A1) and the lumen/longitudinal section (B).In addition, L and LA are displayed at the cross-sectional area (A1) and at the proximal and distal references together with the selected length (B).MLA is also shown (B).(C) Stent assessment.In comparison with AptiVue TM , Ultreon TM 2.0 fuses panels B and C and provides expansion, MSA (highlighted box), and apposition (*) bars combined in one picture.Expansion at the selected frame and LA values is also shown in panel A1.In addition, L and LA are displayed at the cross-sectional area (A) and at the proximal and distal references.

Heat maps
Qualitative review of heat maps showed fewer areas of focus in Ultreon TM 2.0 vs. AptiVue TM platforms and fewer areas of focus among experienced vs. inexperienced participants.A representative sample of heat maps for the stent sizing task is shown in Figure 4.Both experienced and inexperienced participants focused on key numbers in the Ultreon TM 2.0 platform but had more areas of focus in the cross section and longitudinal view in the AptiVue TM platform.

Discussion
This is the first study to use eye tracking to explore physician interpretation of intravascular imaging using an AI-enhanced AOI and sample eye-tracking data for Ultreon TM 2.0 and AptiVue TM platforms.(A) AOI and target creation for one of the stent result assessment tasks in the Ultreon TM 2.0 group.Right upper picture: AOI is defined by delimiting specific areas/boxes (Target and Target 1) that the software will recognize as areas where gaze/fixation data must be registered.Left and right bottom pictures: Fixation data of inexperienced and experienced operators, respectively, undertaking this stent assessment task without the overlying AOI and target boxes.(B) AOI and target creation for one of the calcium assessment tasks in the AptiVue TM group.Left upper picture: The image is imported into the eye-tracking dedicated computer and defined as a task image.Right upper picture: Areas of interest are defined by delimiting specific areas/boxes that the software will recognize for areas where gaze/fixation data must be registered.The "target" box defines the area of interest for the specific analysis.Left bottom picture: Fixation data of experienced operators undertaking this task with overlying AOI and target.Right bottom picture: Fixation patterns of experienced operators without the overlying AOI and target boxes.Eye tracking has been used to evaluate physician performance in non-invasive imaging interpretation tasks (13,15).Much of the literature comes from radiology where expertise is associated with decreased time on task and quicker focus on relevant areas of interest (13).Both metrics (TTT and TFF) were better among experienced vs. inexperienced physicians in this study.More importantly, the potential impact of comprehensively integrating AI into the image interpretation process was also assessed in this study.The Ultreon TM 2.0 software was associated with improvements in TTT, TFF, and DT for both inexperienced and experienced physicians suggesting that the streamlined interface and AI integration made both groups more proficiently behave in their viewing patterns.Thus, the phenomenon of "expertise reversal," in which experienced physicians who have already accustomed to the greater amount of visual information that the AptiVue TM software provides might have had their performance worse, did not occur (22).AI-enabled systems, such as Ultreon TM 2.0, hold a promise to improve time-sensitive decisions as part of a broader approach of collaboration between physicians and AI (23).Importantly, whether physicians can leverage AI involves a host of human and technological factors.In this study, physicians leveraged the AIenhanced OCT data with minimal training regardless of their levels of expertise.This ease of use relates to the intrinsic interpretability of the AI enhancements which are displayed as a visual overlay on top of raw cross-sectional imaging data, a concept termed explainability within the AI literature (24,25).Many motivations for incorporating AI explainability into the interface design including a desire to help physicians justify AI decisions, correct errors, and build trust through knowing why the AI output was produced are noted (26).This approach may account for the successes of integrating AI within the Ultreon TM 2.0 platform in contrast to mixed success in integrating AI in other areas (3).The enhanced viewing efficiency noted using Ultreon TM 2.0 holds relevance for clinical practice and the cognitive load of physicians seeking to use OCT datasets in planning PCI.Cognitive load theory suggests that effective learning and decision-making processes are limited by the amount of information that working memory can process at any given time (7).The high-resolution, cross-sectional dataset from OCT has a wealth of information to guide PCI.However, to take a full advantage of this dataset, physicians must process and interpret this additional information under the time pressures of clinical care, a scenario that may test the limits of their working memory (8).By standardizing the OCT workflow (21), and using AI-enhanced software to focus clinician attention on this standardized workflow, the use of cognitive load barriers could be reduced.In the LightLab experience, a standardized OCT workflow meaningfully impacted physician PCI decisions and reduced equipment use but increased procedure time by 9 min on average (27).Our eye-tracking findings suggest that Ultreon TM 2.0 may streamline the interpretation process by reducing complex information into manageable chunks, thus aligning with the principles of cognitive load theory.

Study limitations
Several limitations are worth mentioning.First, our study was conducted using static images that forced the interrogated physician to provide interpretation and answers without the ability to dynamically manipulate the image.This simplifies the task and would potentially underestimate the streamlining provided by Ultreon TM 2.0 compared with AptiVue TM .Second, while our participants had diverse training and experience with OCT, all were recruited from a single center.

Conclusions
Using OCT to make PCI decisions, physicians had improved the viewing efficiency with preserved accuracy using an AIenabled, visually streamlined Ultreon TM 2.0 software compared with the AptiVue TM platform.
Figure 3   compares average viewing efficiency and interpretation accuracy between the Ultreon TM 2.0 and AptiVue TM platforms.

FIGURE 3
FIGURE 3Comparison of viewing efficiency and interpretation accuracy metrics between Ultreon TM 2.0 and AptiVue TM platforms.The use of the Ultreon TM 2.0 platform compared with AptiVue TM while performing three interpretation tasks relevant to percutaneous coronary intervention (quantifying the arc of calcium, choosing a stent size, assessing a stent result) was associated with reduced time to first fixation in the most relevant target area of interest, reduced total task time, increased dwell time in the target area of interest, and preserved interpretation accuracy.

TABLE 1
Viewing efficiency stratified by the participant experience level.

TABLE 2
Viewing efficiency results stratified by task.

TABLE 3
Mixed multivariate models for viewing efficiency metrics and accuracy.