1. Introduction
Surgeons operate in mentally and physically demanding workspaces where the impact of error is highly consequential (Christian et al., 2006; Modi et al., 2018). The cognitive and affective states of surgeons in the operating room (OR) are primary determinants of surgical performance (Schuetz et al., 2008; Yurko et al., 2010; Haji et al., 2015). Measuring changes in these states contributes to insightful performance analysis, training development and skills improvement (Eversbusch and Grantcharov, 2004; Carswell et al., 2005). However, these cognitive and affective state measurements, which can be derived from neurophysiological metrics, must be coupled with measures of surgical task success or failure to enable any conclusions about the correspondences between psychological indicators and surgical performance (Modi et al., 2017). Unfortunately, the majority of existing tools to measure surgical task performance are retrospective, reliant on subjective, manual expert annotation and often focused on entire procedures rather than on intraoperative performance (Maruthappu et al., 2014).
Examining the associations between neurophysiology and intraoperative performance requires detecting markers of intraoperative success or failure at millisecond to second resolutions and synchronizing those markers with neurophysiological data. In this study, we build and deploy an automated, objective error detection system for robotic-assisted surgery (RAS) simulations that is synchronized with an electroencephalogram (EEG) headset worn by trainee surgeons at an RAS operating console. We use this system to detect intraoperative errors and analyze the accompanying EEG changes during those errors.
Prior work examining EEG changes during RAS has primarily focused on correlating EEG measurements with retrospective, subjective surveys of perceived mental workload (Guru et al., 2015b; Yu et al., 2017; Zander et al., 2017; Zhou et al., 2018, 2020). While mental workload is a significant contributor to surgical performance (Sarker and Vincent, 2005; Healey et al., 2006; Balch et al., 2009), it is not a proxy for actual performance. Understanding the relationship between operator neurophysiology and actual surgical performance requires objective, high temporal-resolution measurements of intraoperative performance that can be synchronized with changes in neurophysiological indicators. Automated skill and error assessment algorithms for RAS platforms do exist (Chen et al., 2019) but are rarely used in conjunction with biometric data capture. Integrating intraoperative performance data with intraoperative physiological data enables investigation of the potential cognitive and affective features that influence surgical success or failure.
For neurophysiological assessment, we choose to focus on the EEG indicators of psychological features that have been shown to impact task performance in mentally and physically demanding workspaces as well as human-machine teams. “Focused attention” or “vigilance” is the ability of an individual to maintain sustained attention on a task (Oken et al., 2006). Decreases in focused attention increase error rates in human-machine teamwork (Gill, 2012). Fluctuations in levels of attention can be measured through changes in EEG spectral power analyses (Borghini et al., 2014). For this study, we examine global β: (α+θ) power ratio (Pope et al., 1995), occipital β power (Gola et al., 2013), occipital δ power (Harmony, 2013), occipital θ power (Putman et al., 2014), frontal θ power (Strijkstra et al., 2003), and global γ power (Kaiser and Lutzenberger, 2005) as quantitative indicators of attention.
Cognitive load and fatigue are similarly relevant to human-robot teaming. High levels of cognitive load lead to task errors (Sweller, 2011). Cognitive fatigue, a decrease in cognitive performance after sustained cognitive load (Ackerman, 2011), is of particular importance in safety-critical aviation, transportation, aerospace, military, medicine, and industrial settings where fatigued individuals routinely operate complex, automated systems (Zhang et al., 2009). EEG spectral power ratios (Ryu and Myung, 2005) can also be used as measures of cognitive load or fatigue. We focus on frontal θ power (Shou and Ding, 2013; So et al., 2017; Raufi and Longo, 2022), frontal α power (Palva and Palva, 2007) and parietal α power (Frey et al., 2015) as indicators of cognitive load. For cognitive fatigue, we examine global α power (Santamaria and Chiappa, 1987), occipital α power (Thut et al., 2006; Zumer et al., 2014), frontal θ power (Strijkstra et al., 2003), and θ:α power ratio at Fz (Berka et al., 2007).
Affective characteristics such as valence, arousal, and dominance (VAD) may also influence human-robot system performance. Valence, the positivity or negativity of emotion, modulates attention and has shown an inverted u-shaped relationship with human performance in various tasks (Cai and Lin, 2011). Arousal, often referred to as “stress,” has a similar inverted u-shaped relationship with cognitive load (Teigen, 1994; Staal, 2004). Dominance, feelings of being “in control” of a situation, has a positive relationship with performance during sustained tasks (Cohen et al., 2016). EEG correlates of valence include α power asymmetry (Ohme et al., 2010; Gordon et al., 2018), global β power (Liu and Sourina, 2013), global γ power (Oathes et al., 2008) and global α power (Ahern and Schwartz, 1985). Indicators of arousal include global θ power, global β power and global γ power (Liu and Sourina, 2013). EEG correlates of dominance include global α:β power ratio and parietal β power (Verma and Tiwary, 2017).
Finally, we examine the association of proposed error recognition-related EEG metrics with intraoperative error. We include global θ power (Cavanagh and Frank, 2014), and global β power (Ray and Cole, 1985) as quantitative EEG indicators of error recognition.
To correlate these EEG metrics with markers of intraoperative error, we require high temporal-resolution analysis of surgical performance data. Prior work in automated surgical performance analysis has primarily focused on kinematic, system event, and haptic data obtained from RAS platforms. This data is then used to find associations between features such as instrument traveling distance (kinematics), camera clutch engagement (system event), and grip force (haptics) and self-reported level of surgical experience (Chen et al., 2019). Other research has used surgical video analysis for competency assessment (Reiley and Hager, 2009; Tao et al., 2012; Wang and Majewicz Fey, 2018). While kinematic, system, haptic, and image data can be correlated with surgical experience and used for skills assessments, in order to examine the EEG correlates of actual task errors, automated surgical error detection is required. Prior work in intraoperative error detection largely relies on retrospective, manual annotations by expert surgeons (Chen et al., 2019). These manual annotations provide sub-task level resolution for error detection but sub-tasks within surgical procedures can take seconds to minutes. A sub-task that is labeled with an error then implies an error marker of seconds to minutes in length. The EEG signal associated with a long-duration error marker would include more than the neurophysiological response to error.
To more precisely examine the neurophysiology of intraoperative error, we use a video analysis pipeline to automatically label each video frame from all surgical simulation videos for the presence or absence of an error. Because RAS operating console video is captured at a frame rate of 30 frames-per-second (fps) and each frame has an error label, we achieve error detection resolution of 30 Hertz (Hz). Each of these error labels is associated with a timestamp that is then synchronized with raw EEG data. By creating non-overlapping EEG data windows that align with each video frame, we create a more accurate measure of the neurophysiology of intraoperative error.
Parameterizing the EEG-based neurophysiology of intraoperative error creates the possibility of designing training interventions to reduce or prevent these errors. Surgeons who exhibit high correlations between EEG indices that may be related to attention and error, cognitive load and error, or emotional valence and error, for example, may benefit from EEG neurofeedback training to improve attention (Egner and Gruzelier, 2001, 2004), cognitive load management (Gruzelier et al., 2006), memory (Vernon et al., 2003), or emotional regulation (Raymond et al., 2005). Other surgeons who have high correlations between EEG indicators of arousal or stress and error, or sensorimotor desynchronization and error, for example, may benefit from focused neurostimulation for stress reduction (Modi et al., 2019) and fine motor skill augmentation (Cox et al., 2020).
In this paper, we aim to characterize the neurophysiological response to intraoperative error during RAS. To achieve this, we build and deploy a system for intraoperative error detection and EEG signal synchronization. We then use this system in a series of experiments with trainee surgeons and analyze the results. Our motivation is a better understanding of the neurophysiology of intraoperative error to help guide more accurate human performance assessments and training for surgeons and other teleoperators.
4. Discussion
Potential cognitive and affective determinants of surgical performance, and their EEG correlates, have been explored in prior research (Gruzelier, 2014; Guru et al., 2015a; Ndaro and Wang, 2018). This work, however, does not couple EEG analysis with automated surgical error detection to capture high temporal-resolution indicators of intraoperative error. By integrating intraoperative error detection with synchronized EEG analysis, we aimed to characterize the neurophysiology of surgeons during robot-assisted surgery (RAS), particularly during intraoperative error.
To achieve this, we built and deployed a novel RAS data capture and analysis platform to examine the relationships among EEG spectral band powers at various channels, previously proposed EEG correlates of various cognitive and affective features in other contexts, and intraoperative error. Our primary statistical tools, linear mixed effects models (LMMs), are widely used in psychophysiological research particularly because of the non-independence of the data in hierarchical structures (Bagiella et al., 2000; Koerner and Zhang, 2017; Frömer et al., 2018). While sequential 30ms EEG samples may be correlated, we are not examining a temporal effect in our model, so the impact of potential residual error autocorrelation on the error effect is not meaningful for our hypotheses. For this reason, and to our knowledge, the standard LMMs in neurophysiological research do not use autoregressive residual covariance matrices (West et al., 2006). Additionally, when we randomly shuffle our datasets to remove any potential residual autocorrelation that may impact our results, our effect sizes and significance tests remain unchanged.
Our results suggest that relative band powers at most channels demonstrate significant changes during errors as compared to when errors are not being committed. We observe that EEG features which are proposed correlates of attention, cognitive load, cognitive fatigue, valence, arousal, dominance, and error recognition in other contexts (Klimesch, 1999; Strijkstra et al., 2003; Harmony, 2013; Liu and Sourina, 2013; Verma and Tiwary, 2017; Tempel et al., 2020) demonstrate significant changes during intraoperative error and may underlie the error-related signals detected in this study.
The direction of change in EEG features largely aligns with proposed associations between EEG statistics and cognitive or affective features in other contexts. Increases in occipital δ power have been associated with increases in attention (Harmony, 2013), and in our study, these increases are also observed during intraoperative error. Decreases in parietal α power have been associated with decreases in cognitive load (Klimesch, 1999). In our study, decreases in parietal α power were also observed during intraoperative error. Increases in frontal θ power have been associated with increases in cognitive fatigue (Strijkstra et al., 2003), and we observed an increase in frontal θ power during error. Left and right hemisphere α power asymmetry has been proposed as an indicator of emotional valence (Strijkstra et al., 2003) with higher asymmetry corresponding to more positive valence. We observed a decrease in α power asymmetry during error. Decreases in global β power have been associated with increased arousal (Liu and Sourina, 2013), and we observed a decrease in global β power during error. Increases in parietal β power have been associated with increases in dominance (Verma and Tiwary, 2017), and we observed decreases in parietal β power during error. For error recognition, increases in global β power have been associated with increased response inhibition during voluntary motor tasks (Tempel et al., 2020). In our study, global β power decreased during intraoperative error.
While this study attempts to characterize the physiological correlates of intraoperative error, this study is not designed to test the hypothesis that the cognitive and affective states correlate with physiological metrics that are causative of error. The EEG activations described in these results are exploratory and intended to generate hypotheses for future studies that can formally test hypotheses generated from the current findings. Moreover, correlations between physiological signals are not indicative of causation and therefore future studies may attempt to implement causal interventions to test the directional relationship between errors and physiological states.
There are several limitations of this study. First, the number of participants was neither large nor evenly distributed across skill level, potentially skewing cohort analysis. Second, using bandpass filters and RMS power calculations for frame-by-frame channel-band and EEG feature power measurements are accompanied by a tradeoff in the settling time of the filter. Third, the error detection video pipeline we use is based on color and text-based error indicators that appear in the operating console during surgical simulations. These error indicators are part of the built-in simulation software and cannot be turned off or altered during simulation tasks. This presents the possibility that the neurophysiological changes that accompany error may be confounded by a response to the specific error indicator in the stimulator. That said, we observed significant changes in parietal channels that would seem to be less likely if the neurophysiological changes were due to vision-based stimuli alone. Finally, these results may be affected by other variables such as movement at the console and time at the console. Based on our observations, low performers tended to move and shift body positions more frequently than high performers, potentially indicative of discomfort at the console. Low performers also required more time to complete the simulation tasks.
Despite these limitations, this study shows that a novel RAS data capture and analysis platform can enable the detection of distinct neurophysiological changes associated with intraoperative error. These changes are largely consistent with known relationships between EEG metrics and cognitive or affective factors that impact performance. This data highlights the possibility of using objective biometric data capture and error detection to better understand the neurophysiology of intraoperative error and help guide more accurate human performance assessments and precision training for surgeons and other teleoperators.