Development and usability testing of a patient digital twin for critical care education: a mixed methods study

Background Digital twins are computerized patient replicas that allow clinical interventions testing in silico to minimize preventable patient harm. Our group has developed a novel application software utilizing a digital twin patient model based on electronic health record (EHR) variables to simulate clinical trajectories during the initial 6 h of critical illness. This study aimed to assess the usability, workload, and acceptance of the digital twin application as an educational tool in critical care. Methods A mixed methods study was conducted during seven user testing sessions of the digital twin application with thirty-five first-year internal medicine residents. Qualitative data were collected using a think-aloud and semi-structured interview format, while quantitative measurements included the System Usability Scale (SUS), NASA Task Load Index (NASA-TLX), and a short survey. Results Median SUS scores and NASA-TLX were 70 (IQR 62.5–82.5) and 29.2 (IQR 22.5–34.2), consistent with good software usability and low to moderate workload, respectively. Residents expressed interest in using the digital twin application for ICU rotations and identified five themes for software improvement: clinical fidelity, interface organization, learning experience, serious gaming, and implementation strategies. Conclusion A digital twin application based on EHR clinical variables showed good usability and high acceptance for critical care education.


Introduction
Medical errors remain a major cause of morbidity, mortality, and cost in the US healthcare system (1).The intensive care unit (ICU) is particularly prone to preventable adverse events due to the complexity of care delivery and the patient severity of illnesses (2).The fast pace and high acuity of critical care practice can also limit opportunities for trainee autonomy.Providing a safe environment to practice decisionmaking in this setting may improve the ICU educational experience, care processes, and patient-centered outcomes (3).
Digital twins are virtual models that simulate the behavior of real objects in a digital environment.With the increasing availability of electronic health record (EHR) and sensor-derived patient data, digital twins hold significant potential applications within the healthcare sector (4,5).In particular, digital twin technology enables the creation of computerized patient replicas, simulating diverse clinical scenarios and intervention testing in silico to reduce avoidable risk in real patients (6).
Digital twins offer particular promise in critical care, where large quantities of data are continuously available, and the risk to patient safety posed by medical interventions is often significant (7,8).The benefits of a digital twin patient model to inform clinical decisionmaking in critical illness have been previously proposed (9)(10)(11)(12).Digital twins could also be adapted for critical care education, allowing learners to simulate the effects of various interventions and explore their potential outcomes in a controlled, virtual environment without negative patient impacts (13,14).Compared to conventional virtual patient simulation models, digital twins provide users with a more authentic experience in complex illness management by incorporating real-time, EHR-derived patient data into comprehensive computational models (15)(16)(17).
Our group has previously described the design and validation of a novel digital twin based on EHR clinical variables to model critically ill patients with sepsis for bedside decision support (11,13).In this model, major organ systems interact based on programmed expert rules to recreate and predict the future patient state in response to specific clinical interventions.In this work, we developed a novel application software utilizing this digital twin patient model to simulate clinical trajectories during the initial 6 h of critical illness.This study aimed to assess the usability, workload, and acceptance of the digital twin application software for critical care education in a cohort of internal medicine residents.This study comprised three sequential phases:

Materials and methods
1 Design and coding of a digital twin patient model based on EHR clinical variables and expert rules to simulate patient trajectories during the initial 6 h of critical illness. 2 Development of the user interface for an iOS digital twin application software designed for critical care education delivery.3 Usability testing of the digital twin application software with a cohort of internal medicine residents and collection of user feedback for iterative software improvement.

Digital twin patient model design and coding
The digital twin patient model tested in this study focused on physiologic interactions and medication effects relevant to the initial 6 h, or golden hours, of critical illness (18).Variables included in the model comprised clinical data commonly displayed in the ICU EHR.Expert rules describing the interactions between the seven major organ systems (neurologic, respiratory, cardiovascular, gastrointestinal, renal, immunologic, and hematologic) were developed using available literature and current clinical practice guidelines and refined using a modified Delphi panel of international critical care experts (11, 13,19,20).Medication effects and pharmacokinetic rules were derived from publicly available drug databases.The model was based on 70 total expert rules and iteratively improved based on feedback from the investigator group.A detailed description of model design and coding, together with two examples of expert rules, are presented in the Supplementary Materials and Methods.The rules that describe the physiologic interactions between the organ system variables are represented graphically in Supplementary Figure 1.

Digital twin application software development
The digital twin pilot application software tested in this study was developed on iOS using Swift programming language and Xcode integrated development environment version 14.2.User testing sessions were performed with a tablet version of the iOS digital twin application.
The user interface of the digital twin application software consists of a case selection screen, a patient room screen, an EHR screen, and an order entry screen.Users can select a case from a list of virtual clinical scenarios that include urosepsis, chronic obstructive pulmonary disease exacerbation, acute respiratory failure due to pneumonia, acute liver failure, gastrointestinal bleeding, myocardial infarction, and acute decompensated congestive heart failure.Each clinical scenario incorporates specific organ system variable alterations into the initial virtual patient presentation.The user can review the patient's history and physical examination findings on the patient room screen.The EHR screen displays the most relevant data for critical care decision-making, organized by organ systems and colorcoded based on the degree of abnormality (21).These data are divided into physical examination, laboratory testing, and other diagnostic findings.Clinical interventions performed by the user are displayed on the EHR screen, maintaining the organ system organization (Figure 2).
After using the order entry screen to initiate a diagnostic test or intervention, the user can advance the timeline (by 15-min intervals for the first hour, then by one-hour intervals until the 6-h endpoint of the simulation) to trigger the associated expert rules coded in the digital twin patient model.The expert rule engine determines which rules are executed based on the interventions ordered and the current value of each organ system variable, which defines the patient's clinical status.The effects of these rules are displayed as changes in the relevant clinical variables presented in the EHR, which reflect the patient's physiological response to the different interventions.

Study design and setting
To explore the usability of the digital twin application software as an educational tool in critical care, we collected both quantitative and qualitative data during seven user testing sessions with internal medicine resident volunteers performed at the Mayo Clinic, Rochester, from August 2022 to June 2023.Participants were compensated for their time with a gift card.The study protocol was evaluated and approved as exempt by the Mayo Clinic Institutional Review Board (IRB 21-010982; study title "Critical Care Coaching with an Electronic Health Record Digital Twin"; approval date 11/8/2021) after review by the Mayo Clinic Education Research Committee and the Mayo Clinic Internal Medicine Research in Education Group.The study was conducted in accordance with the ethical standards of the responsible institutional committee on human experimentation and with the Helsinki Declaration of 1975, as most recently amended.Verbal consent was obtained from the participants before each testing session.

Qualitative data collection and analysis
During user testing sessions, residents interacted for 15 min with a simulated case, describing their experience using a think-aloud and semi-structured interview format.The urosepsis case was used for all the user testing sessions to ensure consistency.Each case scenario and debriefing session was recorded, de-identified, transcribed, and analyzed for common themes.Qualitative data were used to refine the software and identify possible digital twin application implementation strategies in the current critical care curriculum.

Quantitative data collection and analysis
The System Usability Scale (SUS), NASA Task Load Index (NASA-TLX), and two survey questions were administered to each user at the end of the simulation session to collect quantitative information on software usability, workload, and learner acceptance.SUS is a measure of usability consisting of 10 questions with five options each (22).The final score ranges from 0 (low usability) to 100 (high usability).NASA-TLX measures perceived workload and evaluates six domains: mental demand, physical demand, temporal demand, performance, effort, and frustration (23).Each domain is scored from 0 (low workload) to 100 (high workload) in 5-point steps, then the unweighted average of the subscale scores is obtained.The survey questions explored how residents would consider using the digital twin application to prepare for or as part of their medical ICU rotation.De-identified data were collected and managed using Research Electronic Data Capture version 8.11.11 (REDCap, Vanderbilt University, Nashville, Tennessee, USA).Statistical analysis was performed using GraphPad Prism version 9.0.0 (GraphPad Software, San Diego, California, USA).To summarize the results, median (interquartile range, IQR) and counts (%) were used.Thirty-five post-graduate year one internal medicine residents participated in the user testing sessions of the digital twin application software.All residents were recruited during pre-planned central venous catheter procedural workshops conducted before the start of their medical ICU rotation.

Digital twin application software usability, workload, and acceptance
The average SUS score in our cohort was 70 (IQR 62.5-82.5),consistent with good software usability (22).The average NASA-TLX score was 29.2 (IQR 22.5-34.2),reflecting a low to moderate workload (24).The scores of each NASA-TLX domain are presented in Figure 3.The greatest perceived difficulty was the successful performance of required tasks, while physical and temporal demand and frustration levels were considered low.Mental demand and overall effort were rated as moderately high.More than 60% of residents indicated that they would use the digital twin application for a moderate amount or a great deal of time to prepare for and as part of their medical ICU rotation (Table 1).

User feedback for iterative software improvement
Resident comments for iterative software improvement were clustered in five domains, summarized in Table 2. Learners highlighted the importance of the digital twin application delivering a realistic clinical experience, including interactions with the virtual patient and simulated clinical environment and a plausible timeline for scenario progression.Residents also suggested that the EHR interface of the application software should be similar to the commercial product they use in the clinical environment.This would help them to learn to gather and interpret results and navigate the ordering process efficiently.They felt the digital twin application was most helpful in learning medication dosing and effects, enhancing pattern recognition, and improving their understanding of current guidelines through practice managing common ICU scenarios.Learners were mainly interested in a serious gaming experience to test their clinical skills in a safe environment, with a final evaluation reflected by a performance score attributed at the end of each scenario.Residents expressed a willingness to utilize the digital twin application before and during medical ICU rotations; however, they highlighted that their busy clinical schedules pose a significant obstacle to the implementation of the application, as they have limited free time available to use it.To address this issue, the internal medicine residents proposed incorporating practice sessions utilizing the digital twin application software into the current critical care education curriculum.

Discussion
This study presents the development and usability testing of a novel application software for critical care education built upon a digital twin patient model based on EHR clinical variables.The digital twin application allows physicians-in-training to test clinical interventions on virtual patients, fostering autonomy and advancing clinical skills in a safe environment that does not expose real patients to preventable harm.Digital twin application testing in a cohort of internal medicine residents suggests high software usability and learner willingness to use this tool to enhance their medical ICU rotation experience.
Although simulation-based education can improve learner confidence and knowledge, evidence supporting superior learning outcomes over more traditional educational delivery methods has varied based on the learning goals (25)(26)(27)(28).One notable advantage of simulation is its capacity to offer standardized, reproducible clinical scenarios within a risk-free learning environment, with clear patient safety benefits (29, 30).Emerging technologies, including medical simulation mobile applications and virtual reality, provide further opportunities for remote and on-demand training using simulated clinical cases, providing a consistent framework of residency training experiences that is more cost-effective than traditional high-fidelity simulation (31)(32)(33).In addition to providing flexible, efficient online opportunities for deliberate practice, digital twin technology can also integrate real-time patient data to create highly accurate and realistic  Residents acknowledged the potential of the digital twin application to enhance their critical care educational experience.However, they identified clinical schedule demands as the primary obstacle to effectively implementing this tool.In addition to dedicating time within the current critical care curriculum to practice using the digital twin application, residents suggested incorporating additional gamification features, such as a point and badge system, to increase user engagement.Serious gaming has been utilized in various medical education settings, including critical care and emergency medicine, and has been shown to improve knowledge retention and clinical competence (34,35).However, most studies to date have lacked welldefined control groups, and further research is needed to better understand the benefits of this educational delivery method on learning outcomes, together with the most appropriate learner group, educational context, and experience to achieve these goals (36,37).
Clinical data display was an important theme raised during software development and user testing sessions.Residents must rapidly learn to identify and review a significant volume of data associated with each patient in the ICU setting.Reviewing this clinical information takes significant time, and this task can feel overwhelming for new trainees without an organized approach (21, 38, 39).To address these challenges, the digital twin application interface displays only the most relevant data for treating critical illness.These data are also organized by organ system and colorcoded based on the degree of physiological disturbance and need for action (Figure 2).This user interface design has been shown to reduce time to clinical task completion, task load, and errors of cognition in the ICU when compared with standard EHR interfaces (40,41).During user testing sessions, residents acknowledged the potential usefulness of the system-based interface organization in the ICU context.However, they also emphasized the differences between this data display and the interface they regularly encounter in their clinical duties.They specifically highlighted the importance of practicing navigation within standard EHR systems at the beginning of their training.This situation creates a dilemma between two distinct learning objectives: the need for clear data presentation to minimize cognitive load and support deliberate practice in critical care decision-making versus data presentation that closely resembles the clinical EHR interface to enhance order entry efficiency through practice but potentially hinders the development of clinical reasoning in typical critical care scenarios.The challenges of adapting to the new interface might also have contributed to the moderately high NASA-TLX scores recorded in the domains of mental demand, successful task performance, and overall effort recorded during testing sessions.Additionally, the significant variations observed in the performance, mental demand, effort, and frustration domains of the score could indicate differences among residents in terms of their critical care knowledge and problem-solving capabilities rather than being attributed solely to the interface itself (42).This subject will require more targeted studies to qualify further.
The digital twin application software offers a convenient, low-cost alternative to enhance the current delivery of critical care education to learners at various levels of experience.This is the first time that digital twin technology has been applied to critical care education.The major strength of our digital twin patient model resides in using transparent pathophysiological relationships to derive expert rules, which have been refined using multinational and multi-specialty Delphi consensus (11,19).Digital twins can also be developed as purely data-driven models that do not consider causal pathways of diseases, but the lack of clarity in how these physiologic responses are derived creates significant barriers to their acceptance by bedside clinicians (43,44).To provide clinicians with a better understanding of how the underlying model reaches its output state, future iterations of the digital twin application will offer visualization of pathophysiological relationships using directed acyclic graphs in the user interface (45,46).The purpose of this methodology for digital twin model design and the user-centered software development process described in this work is to facilitate technology adoption and address the cognitive, emotional, and contextual concerns of clinicians who will utilize this tool (47)(48)(49).In the future, the digital twin model will be connected to the current EHR system, allowing continuous update based on real-time patient data to support clinical decisionmaking, clinical research, and medical education (Figure 1).This will allow clinicians at all experience levels to practice decision-making skills in a safe environment using actual, real-time cases encountered during daily ICU practice.When this step is accomplished, important ethical and regulatory issues must be considered before implementing this novel tool in daily clinical practice (44,50).This study has some limitations.First, the digital twin patient model described in this work has been tested on simulated clinical scenarios and on a relatively small cohort of patients with sepsis (11).We plan to prospectively validate this model on a larger cohort of critically ill patients importing real-time EHR data into the application software and further refine expert rules based on these and additional retrospective data.Second, only a limited number of users at a single center participated in the usability testing of the digital twin application software.In addition, all users belonged to a cohort of internal medicine residents with no previous ICU experience, which limit the generalizability of the results.We plan to continue the user testing sessions to iteratively improve the current digital twin application software, involving more senior residents, fellows, and staff intensivists with different experience levels to systematically validate this educational tool's performance and learning outcomes and compare it to more conventional educational techniques.

Conclusion
Our novel digital twin application software based on EHR clinical variables proved highly usable and well accepted by first-year internal medicine residents, and their feedback will inform further iterative improvement of its interface.The digital twin application software provides an attractive, realistic, low-cost option to teach critical care clinical decision-making.It offers opportunities for deliberate practice in a virtual environment, building experience and confidence on realtime ICU cases, which may result in greater opportunities for graduated learner autonomy at the bedside and reduced risk of medical errors.

Figure 1
Figure 1 provides an overview of the overall critical care patient digital twin project.This study comprised three sequential phases:

FIGURE 1
FIGURE 1Overview of the critical care patient digital twin project.The digital twin patient model was designed based on expert rules and electronic health record clinical variables.In this study, we focused on the development and usability testing of an iOS digital twin application for critical care education (solid arrows).After further prospective and retrospective validation with clinical data, future applications of the digital twin model include in silico clinical trials and bedside decision support (dashed arrows).This figure was created with BioRender.com.

FIGURE 2
FIGURE 2Electronic health record interface of the digital twin application software.Clinical variables included in the digital twin patient model are represented in the electronic health record screen and updated based on expert rules triggered by clinical interventions or changes in the patient's clinical status.White color indicates that a clinical variable is in its normal range and no intervention is needed, while yellow or red colors indicate a variable disturbance that would require urgent or emergent action.

FIGURE 3
FIGURE 3 Perceived workload of the digital twin application software as measured by the NASA Task Load Index.Overall and single-domain NASA Task Load Index (NASA-TLX) scores were obtained for each resident during user testing sessions (n = 35).Box plots represent median values (solid bar), interquartile range (IQR, margins of the box), and minimum and maximum values (whiskers).

TABLE 1
Results from the survey questions assessing the willingness of residents to use the digital twin application for medical ICU orientation and education.

TABLE 2
Main themes identified during user testing sessions.