Mitigating Latent Threats Identified through an Embedded In Situ Simulation Program and Their Comparison to Patient Safety Incidents: A Retrospective Review

Objective To assess the impact of service improvements implemented because of latent threats (LTs) detected during in situ simulation. Design Retrospective review from April 2008 to April 2015. Setting Paediatric Intensive Care Unit in a specialist tertiary hospital. Intervention Service improvements from LTs detection during in situ simulation. Action plans from patient safety incidents (PSIs). Main outcome measures The quantity, category, and subsequent service improvements for LTs. The quantity, category, and subsequent action plans for PSIs. Similarities between PSIs and LTs before and after service improvements. Results 201 Simulated inter-professional team training courses with 1,144 inter-professional participants. 44 LTs were identified (1 LT per 4.6 courses). Incident severity varied: 18 (41%) with the potential to cause harm, 20 (46%) that would have caused minimal harm, and 6 (13%) that would have caused significant temporary harm. Category analysis revealed the majority of LTs were resources (36%) and education and training (27%). The remainder consisted of equipment (11%), organizational and strategic (7%), work and environment (7%), medication (7%), and systems and protocols (5%). 43 service improvements were developed: 24 (55%) resources/equipment; 9 (21%) educational; 6 (14%) organizational changes; 2 (5%) staff communications; and 2 (5%) guidelines. Four (9%) service improvements were adopted trust wide. 32 (73%) LTs did not recur after service improvements. 24 (1%) of 1,946 PSIs were similar to LTs: 7 resource incidents, 7 catastrophic blood loss, 4 hyperkalaemia arrests, 3 emergency buzzer failures, and 3 difficulties contacting staff. 34 LTs (77%) were never recorded as PSIs. Conclusion An in situ simulation program can identify important LTs which traditional reporting systems miss. Subsequent improvements in workplace systems and resources can improve efficiency and remove error traps.

inTrODUcTiOn Simulation is an important method for improving patient safety and quality of care (1) at individual, team, and system level (2). Team-based simulation can improve patient safety (3)(4)(5)(6)(7)(8) by targeting team deficiencies, the greatest driver of lapses in patient safety (9). However, teamwork training alone will not overcome flaws in the fabric of the workplace. High reliability organizations (e.g., civil aviation) use in situ simulation to expose system vulnerability and hidden system flaws. These hidden threats are termed latent threats (LTs) (10). Proactive LT detection by in situ simulation is reported in emergency medicine, anesthesia, and pediatrics (11)(12)(13). In situ simulation is ideal for identifying LTs in the real work environment, using actual hospital systems, particularly during infrequent high stakes circumstances (14). Vulnerabilities in delivery settings deemed "safe" have been unveiled (11) and flaws in facility operational effectiveness detected (13,15). In situ simulation offers a systematic and realistic picture of work because system flaws are contextualized in real time and place (3). Patient safety incidents (PSIs) are unintended events with potential for patient harm, and all members of staff are encouraged to report these to Datix ® . PSI reporting is a statutory function of National Health Service (NHS) Improvement to reduce patient risks through the National Patient Safety Alerting System (16). Critiques of PSI reporting highlight a misplaced scrutiny on counting reports, rather than organizational learning from events (17,18) and the difficulties of feeding-back to frontline staff (19)(20)(21).
There is a paucity of research on the effect of LT detection on traditional safety models such as Datix ® .
We hypothesized that LTs identified during in situ simulation courses could occur in real life; therefore, mitigating LTs through service improvements might improve patient safety. We aimed to measure: 1. The quantity, category, and subsequent service improvements of LTs. 2. The quantity, category, and subsequent action plans for PSIs (reported on Datix ® ) occurring during the same period and including 1 year before the program inception in 2008. 3. The similarities between PSI Datix ® and LTs before and after service improvements.

MaTerials anD MeThODs
The Simulated interPRofessional Team Training (SPRinT) program at Royal Brompton Hospital is an embedded in situ inter-professional simulation team training program that runs courses in all areas of pediatric care (22). SPRinT faculty are frontline multidisciplinary staff including nurses, nurse educators, anesthetic and intensive care consultants, and medical and surgical trainees. There is a minimum of one nurse and one doctor faculty member at each SPRinT course. Participants are all staff involved with pediatric care including all grades of medical, surgical, and anesthetic staff (consultant and junior doctors), nursing staff (charge nurse and band five to seven nurses), and allied health professionals (health-care assistants, family liaison officers, and physiotherapists). The validated in situ 2-h SPRinT session (23) includes crisis resource management training, a simulated scenario followed by 45 min video-assisted debriefing [utilizing advocacy-inquiry methodology (24)]. Scenarios are derived from real events including serious untoward incidents. Identification, investigation, and mitigation of LTs uncovered during courses are an integral part of SPRinT's commitment to improving patient safety.

lTs and service improvements
During the debriefing after a simulation scenario, all faculty and participants identify any LTs that have occurred and suggest and discuss proposals for service improvement. These events are recorded at the time by two faculty members on a specific LT pro forma. LT severity is graded according to national Datix ® coding system. Datix ® is the online reporting system used by NHS organizations to report PSIs. They color code incidents from potential to cause harm (green), to causing minimal harm (yellow), to causing significant temporary harm (amber), and to permanent harm or death (red). LTs were uploaded to a SPRinT risk database from 2008 until February 2012 after which a specific LT reporting section on Datix ® was developed, the reporting criteria remained the same. Latent threats recorded from Paediatric Intensive Care Unit (PICU) SPRinT courses from April 2008 to April 15 were retrospectively reviewed. SPRinT clinical fellows PK and HM independently categorized LTs according to seven descriptive categories: education and training, resources, equipment, work and environment, medication, organization and strategic, and protocols and systems. All differences were resolved by agreement.
Service improvements for LTs discussed during the SPRinT debriefing session are implemented by the SPRinT and department lead. All service improvements were quantified and grouped independently (HM and PK) according to nine categories: no change; looking into; ongoing quality improvement project (QIP); staff communication; guidelines; education; equipment/resources; organizational change; and all events reviewed by root cause analysis, serious case review, or serious hazard of transfusion.

Psis and action Plans
All PSIs on PICU reported to Datix ® from April 2007 to April 2015 were reviewed and independently categorized by Philip Knight and Helen MacGloin according to the same seven descriptive categories used for LTs. All differences were resolved by agreement.
Hospital risk management, quality, and safety departments organize regular review of Datix ® and subsequent action plans are made. These action plans are instituted by the lead for incident reporting for that department. Action plans for PSIs were quantified and grouped independently (Helen MacGloin and Philip Knight) according to the same nine categories used for service improvements.

statistical comparison
The quantity, category, and grading of LTs and PSIs were compared to assess any impact of LT service improvements on PSIs over time. This review was registered as a quality improvement initiative exempt from ethics review.  with the potential to cause harm (green), 480 (25%) causing minimal harm (yellow), 60 (3%) causing significant temporary harm (amber), and 1 (0.05%) causing permanent harm or death (red). 1,140 (59%) PSIs could not be categorized because the precipitating cause could not be accurately identified, e.g., unanticipated surgery, initiation of extracorporeal membrane oxygenation.
Of 806 (41%) PSIs that could be categorized, medication PSIs accounted for 512 (64%), most reported errors in drug recording or controlled drug spillages. Other medication PSIs included prescribing errors, delayed or incorrect administration, or issues with expired and out of stock medications. 133 (17%) resource PSIs included missing resuscitation ventilator equipment, or infusion sets. 103 (13%) equipment PSIs included equipment failure requiring servicing or replacement. 29 (4%) work and environment PSIs included issues with arrest buzzers, blocked sinks, and bed-spaces. 13 (2%) PSI systems and protocols reports included provision of specialist services out of hours. Nine (1%) education and training PSIs included lack of knowledge of guidelines. Organization and strategic PSIs accounted for seven (0.9%) including incidents due to emergency admission pathways.
There were three scenarios that were particularly high yield, uncovering multiple high-risk LTs. The first was a catastrophic blood loss (CBL) dual location SPRinT course in PICU and the Blood Transfusion Laboratory (BTL) involving three teams (anesthesia, PICU, and cardiothoracic surgery). Real-time simultaneous video-recording enabled a unique perspective on LTs at the PICU and the BTL interface. A 15-point PICU checklist was developed for this dual site simulation and its adherence was assessed (adherence 30%) and a 14-point checklist in BTL (adherence 100%). LT's was uncovered including a delay contacting BTL technician, difficulty implementing emergency CBL management and collection of the wrong blood unit ( Table 2). Following these events, emergency blood tracking requirements were changed so that issued blood units could be handed directly from BTL staff to PICU staff, rather than collection of blood units from the blood refrigerator.
Retesting of the CBL protocol during a second SPRinT course highlighted unfamiliarity with the new CBL protocol and failure to order all blood products (FPP, platelets, and cryoprecipitate). Subsequent service improvements included changes to mandatory staff training requirements, repeated simulation and refinement   of CBL protocol with departmental education and subsequently implementation trust wide. The second scenario involved an emergency resternotomy post cardiac surgery and uncovered five LTs that would have caused harm. They included the inability to perform a cardiac echocardiogram on a deteriorating simulated patient with cardiac tamponade, because the echo machine was in use elsewhere. Difficulty distinguishing adult and pediatric chest re-opening sets, difficulty in locating items on chest opening trolley, difficulty in contacting cardiothoracic surgeons, and delay in administering treatment due to time searching for drug cupboard keys ( Table 2). Service improvements included a successful capital bid for an extra echo machine, purchase of mobile phone for nurse in charge and uploading The third scenario was of hyperkalaemia progressing to ventricular fibrillation and cardiac arrest. LTs uncovered: lack of insulin (an essential part of emergency treatment), lack of awareness of the hyperkalaemia protocol, and lack of knowledge on correct use of the defibrillator. To mitigate the latter, in situ simulated electrophysiology workshops and defibrillation simulation scenarios reviewing external pacing and using internal defibrillation paddles were developed and 70 staff members participated (May 2011-February 2012).

impact of lT service improvements on Future Psis
Latent threats matched 24 (1%) of 1,946 PSIs. The impact of service improvements for these LTs was investigated.
The LT of poor management of hyperkalaemia was preceded by four hyperkalaemia PSIs of different causes. After service improvement ( Table 1), one PSI occurred which was a record of the hyperkalaemia event with appropriate treatment and no adverse event.
The emergency buzzer failure LT was preceded by three similar PSIs with planned follow-up by the operational manager. After service improvement of implementation of regular system testing ( Table 1), there were no further PSIs.
The two LTs of difficulty calling essential staff to the simulated emergency were preceded by two similar PSIs. In 2009, simulation participants were unable to contact on-call PICU consultants via switchboard, following which a mobile phone programmed with all the PICU consultant contact details was obtained for the PICU charge nurse. However, difficulties contacting other essential staff resulted in the same LT recurring, therefore the mobile phone contact list was updated. No further LTs or PSIs from difficulties in calling essential staff recurred.
Three CBL PSIs preceded the first simulated CBL scenario (Figure 3) although before this, CBL was not audited; therefore, these data are not robust. One of these CBL PSIs highlighted poor team communication, recommending SPRinT simulations to improve team communication and emergency chest opening training, which was subsequently carried out.
Two CBL PSIs occurred between the first and second CBL simulations in 2012. The first CBL PSI reported incorrect patient identification resulting in transfusion with blood intended for another child. The second PSI reported incorrect activation of CBL protocol. LT system improvements were implemented subsequent to the second CBL simulation, and in 2013 four CBL events occurred without PSIs. CBL PSIs in 2014 and 2015 reported difficulties contacting the blood transfusion technician because a CBL arrest call had not been properly activated.
Resource LTs (ETCO2 lines and ECG paper) repeated as real events despite service improvements. Despite improved stock ordering after detecting lack of ETCO2 lines, this was followed by three similar PSIs. The LT from lack of ECG paper was preceded by one similar PSI and despite service improvement, recurred as two PSIs.

DiscUssiOn
In situ SPRinT courses identified LTs and implemented simple system adaptations rapidly with minimal cost or manpower. This echoes other in situ simulation studies reporting immediate risk mitigation and simple changes with few resources (12). In situ simulation coupled to system improvements is embraced by staff who witnessed the potential risk of LT to a real patient during simulation. Program engagement likely resulted from empowering staff to test hospital protocols, equipment, and environment for real system flaws. Such adaptive safety initiatives align with Berwick's plea for frontline staff to be enabled to identify problems, test changes, and lead service improvements (25).
The majority of LTs were due to lack of resources and education and training deficits. Of the 12 LTs which recurred despite service improvements, half were due to lack of resources. Other LT studies similarly unveiled poorly understood policies, staff training deficiencies, and issues with critical materials and supplies (26,27). In our review, seemingly improbable events during simulation (lack of insulin to treat hyperkalaemia and incorrect patient identification for blood transfusion) resurfaced in real life as PSIs. In addition, LTs uncovered significant training needs (e.g., defibrillation skills) unreported as PSI because education deficits are not well highlighted by incident reports.
Only 1% of PSIs were similar to LTs which might reflect the limitations of incident reporting and make PSIs poor measures of safety and quality improvement (28,29). We also felt that because few of the LTs had corresponding PSIs this actually showed the unique value of simulation over patient safety reporting and are treating this as a positive finding. In addition, incident reporting and in situ simulation provide different vistas on safety. Other studies have highlighted the need for different approaches to build a comprehensive picture of safety because of the complementary yet different information provided by each approach (30).
Some repeat simulation scenarios enabled development and testing of protocols and a shared perspective ordinarily unavailable. For example, the dual location CBL simulations enabled a unique observation of simulated error at the interface of care between PICU and blood transfusion. This perspective resulted in an interdepartmental collaborative approach to patient safety, facilitated learning, and supported the rationale for system improvement. Although statistically impossible to prove CBL events are safer (due to rarity), CBL PSI grading lessened after system improvements. Likewise trust wide introduction of an intubation box reduced the time to intubate in all areas. Subsequently, excessive time to prepare for intubation has not recurred as an LT in simulated events. However, as never reported as a PSI, system improvement impact cannot be assessed, although it is obvious that reduction in time to intubate improves patient care in a highrisk environment.
Comparing medication LTs and PSIs is interesting. Although most PSIs reported medication errors, emergency drug dose calculations never featured as a PSI possibly because of one of the LT service improvements (emergency drug charts).
The traditional definition of safety (safety 1) is the absence of harm (31). However, safety is not just non-events but a proactive sensitivity to the possibility of failure (32). This deeper understanding of work-as-done enables anticipation of events and team and system flexibility and resilience (33). In situ simulation can encompass both safety perspectives. To the best of our knowledge, this is the first study to review LT detection over time, providing a rich reflection of work-as-done. Measuring LTs from simulation could be developed similarly by other trusts as a proxy safety metric, in conjunction with traditional retrospective PSI reporting, as complementary approaches to safety.
There were several limitations to this study. PSIs on Datix ® were reviewed to assess the impact of service improvements on safety. However, voluntary reporting of adverse incidents captures only a minority of incidents and is subject to significant bias (30). The possibility of bias and missing CBL events was recognized prompting review of the blood transfusion audit. PSI limitations could have been overcome by triangulating them with other records of patient safety events such as patient complaint records, risk management databases, and safety walk-rounds. Retrospective categorization was another source of potential bias. There is no standard nomenclature for error analysis on PICUs although frameworks for incident analysis could have been adapted (34). Other simulation studies have used failure modes effects analysis (FMEA) coupled to simulation training to prioritize predicted risk to enable solution development in stratified order (35). In contrast, we addressed LTs as detected, according to the perceived patient risk because some FMEA critiques highlight the lack of evidence base and the resource heavy methodology involved (36). Iterative testing of system improvement efficacy was not performed which might have prevented later emergence of LTs as PSIs 2-3 years later. Follow-up interviews checking solution success might have enabled system improvement efficacy to be assessed (26). Another limitation was that the review was limited to PSIs and LTs from PICU, excluding LTs detected elsewhere in the directorate (e.g., PACU and the cardiac ward). Finally, participant psychological safety could have been threatened by reporting education and training LTs. Reassuringly, anonymous participant feedback never reported a negative impact on learning.
cOnclUsiOn To the best of our knowledge, this is the first study comparing LTs identified during in situ simulation to existing safety reporting systems and evaluate subsequent service improvements. The unique strengths of in situ simulation were highlighted. First, the identification of potential threats to patient safety particularly for training and knowledge gaps undetected elsewhere. Second, in situ simulation offered a real-time unbiased multi-professional approach to patient safety.
Modern health-care pressures require better models of safety (34) and perspectives reflecting health care's socio-technical system complexity. This cannot be achieved by one safety model but the best way to measure their individual or combined impact is unclear.
More work is required to integrate the strengths of in situ simulation into traditional models of patient care and safety and robustly measure its efficacy. This would enable in situ simulation to be harnessed by existing health-care systems and accepted as a valuable safety improvement modality.

aUThOr cOnTribUTiOns
Participation in the conception and design of the work: MB, PK, and HM. Execution of the work: MB, LL, ML, CK, PK, AD, HM, EH, DM, and PM. Analysis of the data and contribution of methodological expertise MB, HM, and PK.