Neural Efficiency of Human–Robotic Feedback Modalities Under Stress Differs With Gender

Introduction

In human–robotic interactions (HRIs), operators require feedback on the current state of the process, or the status of the process being controlled by the robot (Wickens et al., 2015) to prevent mode confusion. Feedback has been applied in many domains including human–vehicle interaction (Politis et al., 2014), prosthetic limbs (Jimenez and Fishel, 2014) and assistive devices (Park et al., 2015). Feedback in robotic surgery training and robot-assisted minimally invasive surgical operations is essential for effective learning of operative skills (McKendy et al., 2017). Feedback allows surgeons to safely maneuver and manipulate surgical tools, including robots, during surgical operations (Abiri et al., 2018; Miyazaki et al., 2019).

One approach to providing feedback to surgeons is to employ multi-modal interfaces (Wickens et al., 2015) – using multiple sensory modalities to present information (Freeman et al., 2017) based on Wicken’s Multiple Resource Theory (MRT; Wickens, 2002). However, combining feedback modalities does not always lead to optimal performance (Benz and Nitsch, 2017); poorly presented or excessive feedback can be as bad as no feedback at all (Wickens et al., 2015). What is the optimal combination of feedback modalities (e.g., visual + haptic or visual + auditory or auditory + haptic or visual + auditory + haptic)? That depends, in part, on the task type (Burke et al., 2006), individual preferences (Koehn and Kuchenbecker, 2015), individual differences (Park et al., 2012), and context (van Huysduynen et al., 2016; Adebiyi et al., 2017). In the surgical robotics domain, surgeons have been found to better characterize tissue via palpation in multi-modal (visual + haptic) feedback than in a single modality (haptic) feedback (Abiri et al., 2018). Furthermore, haptic feedback has been found to reduce the effects of visual-perceptual mismatches (Abiri et al., 2017). However, mixed and inconsistent results on the effectiveness of haptic feedback exist. For example, haptic feedback was shown to be ineffective in transferring surgical skills to medical students in a virtual simulated environment (Våpenstad et al., 2017) and failed to provide significant improvement in surgical performance (Van der Meijden and Schijven, 2009). Potential contributors to such inconsistencies may include variations between individuals in perceiving feedback effectiveness, their preferences, and the load the modalities place on different user groups.

Even though previous studies (e.g., Van Erp and Verschoor, 2004; Burke et al., 2006; Lee and Spence, 2008) suggest that multimodal feedback enhances the performance of participants performing multiple-tasks under high demanding conditions, its effectiveness under stress conditions has never been studied. Gender is known to influence cognitive (e.g., Munro et al., 2012; Castonguay et al., 2015) and physical (e.g., Hunter, 2014) task performance. Also, men and women differ in the way they cope with and handle stress (Matud, 2004; Bonneville-Roussy et al., 2017; Goldfarb et al., 2019). Men have been found to exhibit higher prefrontal cortex (PFC) responses during stress than women (Seo et al., 2011; Goldfarb et al., 2019), whereas women have showed higher responses in the limbic regions (Wang et al., 2007; Kogler et al., 2015a; Goldfarb et al., 2019). However, whether multimodal feedback is moderated by the combined effect of gender and stress has never been studied. This is critically important in the surgical robotics applications, where surgeons and residents experience high levels of acute stress, which impairs their performance and impacts overall patient safety (Jones et al., 2015; Georgiou et al., 2017).

Indeed, residents and surgeons’ performance can be influenced by the interplay of their attributes (e.g., expertise, stress reactivity; Berguer and Smith, 2006; Shafiei et al., 2018), task (e.g., novel, activities that challenge different modalities, codes, and stages; Wickens, 2002), and context (e.g., laparoscopic, robot surgery, open surgery; Smith et al., 2003). Thus, it is crucial that several dimensions of surgeon capabilities and limitations, outside of task performance, be determined to understand the surgeon’s performance envelope. It is worth noting that human performance is not equal to task performance. Our position is that human (in this case, a surgeon) performance is the summation of task performance and neurophysiological cost of maintaining that performance (see Equation 1), which is of particular importance in safety-critical scenarios like surgical operations.

$$\text{Surgeon}\text{performance}={\displaystyle \sum (\mathrm{task}\text{performances}-\text{neurophysiological}\mathrm{cost})}(1)$$

Neurophysiological cost of maintaining performance can be measured with functional near-infrared spectroscopy (fNIRS), an optical imaging technique to monitor cortical brain activation (Gateau et al., 2018). fNIRS has been used in the surgical robotics domain to evaluate the impact of expertise on a knot-tying task (Leff et al., 2008), to compare impacts of robotic and conventional laparoscopic surgeries in surgeons performing a suturing task (Singh et al., 2018), and to evaluate arrangements aiding hand-eye coordination (Miura et al., 2018). However, very limited work has been published that examined the neural cost associated with different feedback modalities, as much of the current efforts utilize task performance metrics (e.g., accuracy, speed) and/or subjective assessments of workload and preferences. Recently, Curtin and Ayaz (2019) combined measures of neural activity and task-related performance measures into a novel metric –neural efficiency, that enabled them to establish a relationship between neural activity and performance. High neural efficiency is achieved when performance (e.g., accuracy) is high and the corresponding mental effort (e.g., level of concentration of oxygenated hemoglobin) is low, and vice versa (Curtin and Ayaz, 2019). Using fNIRs, Curtin and Ayaz (2019) found neural efficiency in the dorsolateral PFC of participants engaged in symbol digital substitution task consistently increased through practice.

Design of effective surgical feedback systems should seek to not only minimize the neural cost on surgeons during complex surgeries but also enhance their performance under added stress while considering gender differences. This is particularly important as surgeons are under extremely challenging socio-technical environments that burden them both physiologically and psychologically (Park et al., 2010; Sari et al., 2010). The objectives of the present study were to determine if (1) different feedback modalities (visual or visual + haptic feedback) place distinct neural costs on users, and whether this is associated with maintenance or enhancement of task performance, and (2) neural efficiency of different feedbacks differ between males and females under stress.

Materials and Methods

Participants

Twelve right-handed non-smokers (six males and six females) with mean (SD) age of 25.5 (2.68) years, height of 1.71 (0.08) m, and weight of 75.46 (16.78) kg participated in the study. Participants did not have previous experience using the Novint Falcon (Novint Technologies, Inc., Albuquerque, NM, United States) gaming interface. The study was approved by the Texas A&M University Institutional Review Board.

Experimental Design Conditions

The main factors under study were feedback modality (visual vs. visual + haptic), cognitive stress (no computation vs. computation), and gender (male vs. female). All participants underwent all feedback and stress conditions. To induce stress, we instructed participants to perform the serial-7 subtraction task concurrently with the experimental tasks.

Cognitive Stress Manipulation

The serial-n subtraction task, part of the Trier Social Stress Task (TSST; Kirschbaum et al., 1993), has been used extensively to induce cognitive stress (Taylor et al., 2006; Ritter et al., 2007; Bershad et al., 2018; Klatzkin et al., 2018, 2019). It involves repeatedly subtracting an n digit number from a 3- or 4- digit number. In the present study, we adopted the serial-7 subtraction test, requiring participants to repeatedly subtract 7 from a 3- or a 4-digit number provided to them at the start of each 30 s trial.

Human–Robotic Interaction

We used the Novint Falcon 3D joystick (Novint Technologies, Inc., Albuquerque, NM, United States), to implement interaction between participants and our virtual robotic system. The input tool of the 3D touch device was a 3D printed pinch grip made of polylactic acid (PLA) polymer, with box dimensions 5.04 cm × 6.04 cm × 2.13 cm, as shown in Figure 1A. It was important to ensure sufficient contact force between the user’s hand and the input tool. This ensures a 1:1 correspondence between the feedback (F_fb) delivered at the attachment plate and the gripper. For this reason, the pinch grip consisted of a compression spring in contact with an analog force-sensing resistor (FSR) (Thin Force Sensor 1131_0, Phidgets, Inc., Canada) that monitored user contact force (F_grip) during the experiment. The force-sensor was interfaced using an analog I/O board (1018_0 Interface Kit, Phidgets, Inc., Canada) to a Master PC that rendered a 3D virtual environment for user interaction. For the scope of this study, the FSR functioned as a binary switch that ensured sufficient contact force for object manipulation. During the experiments, the force data confirms consistent and firm contact between the gripper and fingers with an average value of F_grip > 20 N. Therefore, vibrotactile feedback was effectively delivered to the subjects. The Novint Falcon was the joystick that participants used to manipulate objects and traverse simple trajectories during the experiment. The visual interface was built using OpenGL and Visual C++, with the Falcon-compatible Force Dimension SDK (Force Dimension, Nyon, Switzerland), and Chai3D (SAIL, Stanford University, Stanford, CA, United States) haptic libraries. The virtual environment consisted of a three-dimensional cubic room, where the user was provided an isometric field-of-view.

FIGURE 1

Figure 1. (A) Schematic representation of user interaction with the haptic interface showing the directions of grip force (F_grip) and feedback (F_fb) (top). Participant interaction with the input tool of the Novint Falcon (bottom), and (B) The Virtual 3D environment and task path [on path (top), off path (bottom)].

Feedback Modalities

Visual feedback

We built a virtual interface using OpenGL, Visual C++, Falcon-compatible Force Dimension SDK (Force Dimension, Nyon, Switzerland), and Chai3D haptic libraries that provided participants with an isometric field-of-view of a three-dimensional cubic room (see Figure 1B). In the visual feedback condition, the object’s color changed to indicate whether the object was on or off the path.

Visual + haptic feedback

Participants used the Novint Falcon as the joystick to manipulate objects and traverse simple trajectories. Vibration motors located within the Falcon’s base rendered force and vibratory feedback (haptic stimuli). Passive links transduced these vibrations to the attachment plate where a 3D printed input tool (the gripper) was fixed. The three motors that actuated the arms of the parallel mechanism together generated a peak amplitude of 8.9 N at the end-effector. A software limit of 0.2 × peak force amplitude was set to provide feedback that did not impede user performance. In the visual + haptic feedback condition, the input tool provided vibratory feedback at 75 Hz, and peak amplitude of approximately 1.78 N when the user’s position was outside the prescribed path, along with the object color change that indicated on/off path feedback. Figure 1B shows what participants saw when an error occurs versus no error when completing the task.

Procedures

We obtained participants’ written informed consent, after which we collected their demographic information. After bio-instrumentation with the fNIRS head cap, discussed later, we instructed participants to sit on an adjustable chair with their backs and feet firmly supported. Next, we performed three baseline measurements protocol along with task familiarization.

First, we asked participants to focus their attention on a fixed point at eye level for 10 s, and then close their eyes for 20 s. Participants repeated these three times. Second, we instructed participants to close their eyes relax, breathe normally, and refrain from any activity for 2 min. Third, we instructed participants to perform 3 maximum voluntary contractions (MVCs) using a three-point pinch grip. Participants held a one-inch thick hard block. We instructed them to squeeze as hard as they could for 5 s, followed by 2 min of rest. Then, we instructed participants to use a three-point grasp to interact with the input tool of a haptic-enabled 3D touch device (Novint Falcon), located on a height-adjustable table. Participants used this device to manipulate objects and traverse simple trajectories during the experiment. We provided participants with a 10-min time window in order for them to get familiar with the input tool and virtual environment before the start of each experiment. This included two demo applications – object shape matching tasks and trajectory tracking tasks with increasing complexity.

The main experiment consisted of four conditions: visual feedback under no stress, visual feedback under stress, visual + haptic feedback under no stress, visual + haptic feedback under stress. We counterbalanced the order of the conditions to minimize any learning effects. In each experimental condition, we instructed participants to move a spherical object from a start point to an end point along a 3D helical path within a 30 s window. We instructed them to be as precise and accurate as possible. After the 30 s, the object returned to a default start position, during which time participants rested for 30 s to minimize any fatigue effects. This constituted one trial. Participants performed 10 trials in each condition, with a 5-min break between each condition to minimize any fatigue effects. At the end of the four conditions, participants were provided a 10-min break, following which this entire protocol was repeated. At the end of the experimental session, a total of 20 trials per condition were obtained from each participant.

Dependent Measures

Performance

We computed three task performance metrics: tracking error (TE), precision score (P_recS), and proximity score (P_roxS). We defined tracking error (TE) as the minimum Euclidean distance between the participant’s position and the general space curve using linear chordal approximations. We discretized the curve into 1000 equal segments of length less than 1–10th the object’s diameter and adopted a linear approximation to identify the minimum distance between a participant’s instantaneous position (reflected by the cursor tool’s coordinates) and each line segment that constitutes the helix. We accepted the minimum distance as TE. We defined P_recS, given by Eq. (2), as a function of TE. P_recS characterizes participant precision of a trial.

$$P_{\text{recS}}=\frac{TE-\min (TE)}{\max (TE)-\min (TE)}(2)$$

Also, we defined P_roxS, as a function of minimum distance (d_min) between the user’s instantaneous position and the target. This measures the participant’s task completion state based on a time history of the last n positions held by the subject for a given trial. We defined P_roxS as:

$$P_{\text{roxS}}=\frac{{\overline{d}}_{\text{n}}-\min ({\overline{d}}_{\text{n}})}{\max ({\overline{d}}_{\text{n}})-\min ({\overline{d}}_{\text{n}})}(3)$$

where ${\overline{d}}_{\text{n}}$ is the average of d_min for the last n positions. Figure 2 shows an example of user performance, where, in accordance with our expectations maximum error corresponds to low precision and high proximity scores (i.e., the user rushes to the finish), while minimum error corresponds to high precision, but low proximity scores (i.e., the user traces the path carefully but is slow in doing so).

FIGURE 2

Figure 2. An example of user performance data superimposed onto the reference trajectory (Left) and the absolute position error as a function of time for those attempts (Right) (A) maximum tracking error in correspondence with low precision but high proximity scores, (B) minimum tracking error in correspondence to high precision but poor proximity scores for a user.

Perceived workload

We assessed participants’ subjective workload using paper and pencil version of the NASA-Task Load Index (NASA-TLX) to assess workload across six dimensions: mental demand, physical demand, temporal demand, performance, effort, and frustration (Hart and Staveland, 1988). We presented NASA-TLX subscales to participants post each condition and asked them to rate each subscale in increments of the low, medium, and high estimates in 21 gradations. We employed a paired comparisons procedure wherein we presented 15 pairwise combinations to participants and asked them to select the subscale from each pair that had the most effect on the workload in each experimental condition. We used an overall workload score of the NASA-TLX as a measure of perceived workload.

Neural activity

Functional near-infrared spectroscopy employs near infrared absorption properties of hemoglobin to infer local concentration changes of cortical oxygenated and deoxygenated hemoglobin as correlates of functional brain activity (Gateau et al., 2018). Each fNIRS measurement channel is made up of two optodes, an optical emitter (source) and a receiver (detector), with an inter-optode distance of 2.5 to 4 cm that allows near infrared to penetrate the head of an adult human to a depth of 1.5 to 2.5 cm. The difference between near infrared absorption rates for oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR) allows for the change in oxygen concentration to be measured (Seraglia et al., 2011).

In the present study, we used continuous fNIRS (Techn, Inc., Milford, MA, United States, CW6 System) to record light intensity in two wavelengths (690 and 830 nm) discharged through 4 emitters and recorded at 8 detectors to obtain hemodynamic response at a total of 10 channels based on 10/20 international EEG system (Figure 3). We converted acquired light intensities into optical densities by taking the logarithm of input signals. As recommended by Huppert et al. (2009), signal noise originating from the equipment, experiment, and physiological sources were addressed and corrected for before converting optical density into hemoglobin concentration using the modified Beer-Lambert law (Delpy et al., 1988). In order to reduce high-frequency noise, we low pass filtered the optical densities at 3 Hz. We used a spline interpolation algorithm (Scholkmann et al., 2010) to detect and correct motion artifacts that showed abrupt changes, and wavelet algorithm (Chiarelli et al., 2015) to smoothen the signals. To reduce the effects of physiological noise and slow wave drifts, we bandpass filtered motion artifact corrected signals at 0.5 ∼ 0.016 Hz. Like Cheng et al. (2012), Foy and Chapman (2018), Modi et al. (2018), and Rhee and Mehta (2018), we used a cut-off frequency of 0.5 Hz (2 s) to reduce high-frequency instrument noise and physiological (e.g., heartbeat) noise. Indeed, Zhu et al. (2019) recommend band-pass frequency of 0.01 to 0.5 Hz, to remove physiological noise and possible slow wave drift caused by the fNIRS system. Like Rhee and Mehta (2018), we used the cut-off frequency of 0.016 Hz (∼1 min) to remove slow wave drift caused by the Techen fNIRS system. Finally, we calculated HbO and HbR levels at the 10 channels using the modified Beer-Lambert law (Delpy et al., 1988). Using Atlasview (Aasted et al., 2015), we determined the location of the 12 optodes (i.e., 4 emitters and 8 detectors) that created the 10 channels.

FIGURE 3

Figure 3. Functional near-infrared spectroscopy (fNIRS) probe design. Red dots represent sources, blue dots represent detectors. Green lines represent channels created between optodes. C1–C10 are channel labels.

We used HbO because it shows greater task-related changes that HbR (Rhee and Mehta, 2018). Similar to that reported by Hyodo et al. (2016) and Rhee and Mehta (2018), we first obtained task-related neural action within each trial by averaging 2 s around maximum HbO activation, from which mean HbO were computed across the 20 trials per condition. We then adopted a region of interest (ROI)-based approach (Soltanlou et al., 2018; Vanzella et al., 2019) to increase sensitivity to smaller effects (Powell et al., 2018). We defined four ROIs within the PFC – left lateral (C1 and C2), left medial (C3, C4, and C5), right lateral (C8, C9, and C10), and right medial (C6 and C7) PFC (see Figure 3). Mean HbO values from the channels in each ROI were averaged to obtain overall mean HbO per ROI per condition.

Neural efficiency

The neural efficiency (NE) hypothesis posits that intelligent individuals or experts efficiently utilize neural resources when engaged in a task better than less intelligent individuals or novices do (Haier et al., 1988). According to Curtin and Ayaz (2019), the NE concept can be extended to quantify the relationship between neural activity and performance, computed using Eq. (4).

$$NE=\frac{z(p)-z(ME)}{\sqrt{2}}(4)$$

where z(P) and z(ME) are z-score measures of performance and mental effort respectively. When NE = 0, performance and mental effort are balanced.

In the present study we standardized HbO levels, and the tracking error performance measure in the respective experimental conditions in order to compare the relative efficiency in those conditions. We computed the relationship between neural activation and TE in the ROIs over other performance measures, because TE is a widely used performance measure in HRI, in general, and robotic surgery, in particular (Sen et al., 2016; Enayati et al., 2018). Each standardized performance measure and standardized HbO levels were projected onto the identity line. We adopted an ROI-based approach to determine the relationship between neural activation in these regions and task performance for each of the four experimental conditions –visual feedback, visual feedback under stress, visual + haptic, visual + feedback under stress, using Eq. 5 below:

$$N{E}_{\text{ij}}=\frac{z(p_{\text{i}})-z(Hb{O}_{\text{ij}})}{\sqrt{2}}(5)$$

where i ∈ {visual, visual + haptic}, j = 1…4, P_i is the tracking error performance in the ith feedback modality, HbO_ij is oxygenated hemoglobin level in the jth ROI in feedback modality i.

Statistical Analyses

We conducted separate three-way mixed factors analysis of variance (ANOVAs) to test main and interactive effects of feedback (visual vs. visual + haptic), cognitive stress (no computation vs. computation), and gender (male vs. female) on the three task performance metrics. Furthermore, for each ROI, we conducted separate mixed ANOVAs on neural activity (i.e., HbO values) with gender as a between-subjects factor, and feedback (visual vs. visual + haptic) and cognitive stress (no computation vs. computation) as within-subjects factors. Finally, for each ROI we conducted separate gender × feedback × cognitive stress ANOVAs on NEs. We used Bonferroni corrections for post hoc comparisons and applied Greenhouse–Geisser corrections if the sphericity assumption was violated. We conducted post-analysis using Tukey’s test where needed and set the level of significance to 0.05.

Results

Performance

Tracking error was not significantly affected by stress, gender, or any two- or three-way interactions (all p’s > 0.219). There was a marginal feedback main effect, F(1,9) = 3.919, p = 0.079, ${\mathrm{\eta }}_{\text{p}}^2$ = 0.303, such that performance was better in the visual (M = 0.117, SD = 0.005) than visual + haptic (M = 0.125, SD = 0.008) feedback condition. Furthermore, there were no significant stress, feedback, gender, or any two- or three-way interactions on precision score (all p’s > 0.207). Moreover, precision score was not significantly affected by stress, gender, or two-or three-way interactions (all p’s > 0.166). There was a significant stress main effect, F(1,9) = 14.350, p = 0.004, ${\mathrm{\eta }}_{\text{p}}^2$ = 0.615, on proximity score, with performance under no stress (M = 0.702, SD = 0.032) better than under stress (M = 0.548, SD = 0.056). However, proximity score was not significantly affected by feedback, gender, or any two-or three-way interactions (all p’s > 0.186).

Perceived Workload

Perceived workload, measured as overall workload score of the NASA-TLX, was not significantly affected by feedback, gender, or two- or three-way interactions (all p’s > 0.224). There was a significant stress main effect, F(1,10) = 16.932, p = 0.002, ${\mathrm{\eta }}_{\text{p}}^2$ = 0.629. Participants perceived higher workload under stress (M = 68.368, SD = 3.733) than under no stress (M = 59.874, SD = 0.032).

Neural Activity

Using ROI-based group analysis, we found that two of the four ROIs, right medial and right lateral, were significantly activated. A three-way interaction between stress, feedback and gender approached significance [F(1,10) = 4.905, p = 0.051, ${\mathrm{\eta }}_{\text{p}}^2$ = 0.329] in the right medial PFC, and revealed a significantly higher activation in females under stress (M = 1.89, SD = 0.52) than no stress (M = 1.22, SD = 0.38) for visual feedback; t(10) = 2.23, p = 0.03 (see Figure 4). There were no main effects of stress, feedback, and gender, or any two-way interactions (all p’s > 0.202) in the right medial PFC.

FIGURE 4

Figure 4. Mean HbO in right medial PFC. Error bars represent standard error. The asterisk indicates a significant difference (p < 0.05) between no stress and stress conditions in females.

A significant stress and gender interaction was found in the right lateral PFC; F(1,10) = 6.257, p = 0.031, ${\mathrm{\eta }}_{\text{p}}^2$ = 0.385. Overall, there was greater activation under stress than no stress, but the effect of stress depended on the participant’s gender. Male participants demonstrated greater activation under stress (M = 1.67, SD = 0.22) than no stress (M = 1.22, SD = 0.33); t(10) = 2.711, p = 0.032. There were no main effects of feedback, stress and gender or their two-way or three-way interactions in the left lateral (all p’s > 0.162) and left medial (all p’s > 0.171).

Neural Efficiency

There was no significant stress, feedback, gender, or any two- or three-way interactions in the left lateral (all p’s > 0.406) and left medial (all p’s > 0.171) PFC.

There was a significant stress and gender interaction [F(1,10) = 7.794, p = 0.021, ${\mathrm{\eta }}_{\text{p}}^2$ = 0.464], with females (M = 0.124; SD = 0.445) exhibiting higher efficiency than males (M = −0.103, SD = 0.510) under stress in the right lateral PFC; t(10) = 2.262, p = 0.024 (see Figure 5A). Furthermore, a significant feedback and gender interaction [F(1,10) = 10.339, p = 0.011, ${\mathrm{\eta }}_{\text{p}}^2$ = 0.535] revealed higher efficiency for males (M = 0.241, SD = 0.873) than females (M = −0.290, SD = 0.486) in the visual feedback condition; t(10) = 2.306, p = 0.021 (see Figure 5B). There were no main effects of feedback, stress and gender or their three-way interactions (all p’s > 0.414).

FIGURE 5

Figure 5. Neural efficiency for males and females in right lateral PFC (A) Stress and gender interaction. The asterisk indicates a significant difference (p < 0.05) between males and female under stress. (B) Feedback and gender interaction. The asterisk indicates a significant difference (p < 0.05) between males and female in visual feedback condition. Error bars represent standard error.

A significant three-way interaction between stress, feedback and gender [F(1,10) = 6.418, p = 0.032, ${\mathrm{\eta }}_{\text{p}}^2$ = 0.416] in the right medial PFC revealed significantly higher neural efficiency for females under no stress (M = 0.215, SD = 0.408) than stress (M = −0.226, SD = 0.477) in the visual feedback condition; t(10) = 2.23, p = 0.03 (see Figure 6). There were no main effects or two-way interactions (all p’s > 0.407) in the right medial PFC.

FIGURE 6

Figure 6. Neural efficiency in right medial PFC. The asterisk indicates a significant difference (p < 0.05) in neural efficiency for visual feedback under no stress and stress for females. Error bars represent standard error.

Discussion

The present study found that during a simulated surgical HRI task, task performance (tracking error) is lower under multiple modalities (i.e., visual + haptic) than visual only feedback, and under stress than non-stress scenarios, but remained comparable between males and females. However, we found gender differences in regions of the PFC activated under stress; males exhibited increased activation of the right lateral PFC, while females exhibited increased activation of the right medial PFC. We found that the two feedback modalities were associated with varying neural efficiencies, and these were stress- and gender-dependent. Our findings engender further investigation into effectiveness of feedback modalities on males and females under stressful conditions in HRI.

The primary task in the present study was akin to a guided motor task (Feygin et al., 2002; Sigrist et al., 2013) with concurrent feedback (visual and visual + haptic), in which task performance was measured in terms of tracking error, precision, and task completion state. Consistent with Feygin et al. (2002), Liu et al. (2006), and Yang et al. (2008), we found that task performance (tracking error) was better with visual than visual + haptic feedback. Haptic guidance in the visual + haptic feedback condition was supposed to provide additional proprioceptive cues. However, our results suggest the visual + haptic feedback modality did not improve task performance, aligning with inferences made from prior studies that combining feedback modalities does not always lead to optimal performance (Benz and Nitsch, 2017).

Higher workload is associated with increased stress (Arora et al., 2010; Klein et al., 2012) which results in degraded performance (Yu et al., 2017). In the present study, higher perceived workload in the stress condition suggests that participants did find the computational aspect of the TSST stressful. Consistent with Wetzel et al. (2006) and Arora et al. (2010) task performance (proximity score) was impaired by stress. This also corroborates with findings reported in the larger motor control literature on the impact of stress on motor performance and coordination (Mehta and Agnew, 2012; Mehta, 2015; Mehta and Rhee, 2017).

Stress results in the activation of the hypothalamic-pituitary-adrenal (HPA) axis and autonomic nervous system (Ulrich-Lai and Herman, 2009; Felmingham et al., 2012), as well as activation or deactivation of various cortical structures (Kogler et al., 2015a). The PFC, which is known to integrate information from different modalities (Fuster et al., 2000), is the most implicated cortical structure in stress response (Arnsten, 2009; Kogler et al., 2015b; McKlveen et al., 2015). It is noteworthy that traditional research on feedback modalities (e.g., McMahan et al., 2011; Koehn and Kuchenbecker, 2015; Pacchierotti et al., 2015) has focused and based conclusions on task performance measures (such as time to completion, speed, total distance traveled) and subjective responses. In the present study, we posited that operator performance is the summation of task performance and neurophysiological cost of maintaining that performance (see Equation 1), and investigated that performance may be affected by feedback, gender and a contextual factor like stress. Whereas prior studies have associated stress with increased oxygenation bilaterally within the PFC (e.g., Yang et al., 2009; Takizawa et al., 2013; Brugnera et al., 2017), others have associated stress only with right hemisphere activation (e.g., Tanida et al., 2007; Sakatani et al., 2010). We found stress to be associated with significant activation in the right lateral and right medial PFC, suggesting the right hemisphere to be dominant in the control of sympathetic activity (Brugnera et al., 2017), particularly during motor tasks (Mehta and Parasuraman, 2014). The right medial PFC has been found to play a key role in facilitating stress hormone responses through interactions with the HPA axis (Cerqueira et al., 2008). The significant change in HbO may be due to increased use of regulatory capacities (Kalia et al., 2018). By extending the analysis of stress response to different regions of the brain, we provide a better picture of how the effects of stress are integrated in the brain.

Although stress response in both genders involves activation of the HPA and sympathetic system, men and women differ in their response under stress (Wang et al., 2007; Verma et al., 2011; Kalia et al., 2018). Findings from prior studies (Kajantie and Phillips, 2006; Jaušovec and Jaušovec, 2010; Hausmann, 2017) suggest that differences in stress response between males and females may be due to hormonal effects. Earlier studies have found higher HbO concentrations in males than females during verbal working memory (Li et al., 2010) and mental arithmetic (Yang et al., 2009) tasks. It is noteworthy that males and females differ significantly during resting state – men demonstrated a higher HbO (Jaušovec and Jaušovec, 2010; Chuang and Sun, 2014). In the present study, the two-way stress and gender interaction in the right lateral PFC with a medium effect size (η_p² = 0.385) revealed higher HbO in males than females under stress, results consistent with Li et al. (2010) and Cinciute et al. (2018). Greater HbO in males than females may have been due to increased neuronal activation or differences in brain and morphology (Ruigrok et al., 2014; Choleris et al., 2018). Aside from hormonal effects, particularly estrogen which mediates differences in the PFC’s stress sensitivity (Shansky et al., 2004), the difference in stress response between men and women in the present study may also be due to anatomical and physiological differences (Hunter, 2014). Furthermore, we found that females demonstrated higher HbO under stress than no stress in the visual feedback modality, suggesting that they recruited more neural resources to cope with stress and maintain performance (Mehta and Parasuraman, 2014; Mehta, 2016).

Generally, males outperform females in visuospatial tasks (Cutmore et al., 2000; Halpern et al., 2007; Thorson et al., 2011; Ali et al., 2015), although few studies have shown females outperform males in object rotation tasks (Silverman et al., 2007; McGivern et al., 2012). In the present study males outperformed females in the visual feedback condition, suggesting that males required relatively fewer neural resources to maintain task performance. Furthermore, in the visual feedback condition, women performed better under no stress than stress. It is evident from this study that the optimal combination of feedback modalities depends, in part, on individual characteristics (e.g., gender; Park et al., 2012), and context (e.g., stress; van Huysduynen et al., 2016; Adebiyi et al., 2017). Contrary to our expectations, the visual + haptic feedback modality did not distribute the neural cost across visual and haptic sensory modalities such that cognitive resources required for task execution will be reduced (Wickens, 2002; Sigrist et al., 2013).

Previous studies have shown that individuals vary in their response to stress, and that intra-individual variability over time may provide information about how individuals differ from each other (Sliwinski et al., 2009; Wang et al., 2012). Detection of group mean differences is enhanced when intra-individual variability is small compared to inter-individual variability (Wang et al., 2012). In the present study, the sample size was too small for meaningful analyses of individual differences. Nonetheless, we were able to detect differences in group means in the measures.

This study has some limitations. First, the task was not a pure representation of a surgical robotic task. However, given that there have been no studies that have investigated stress and gender effects, our first step was to conduct the study with a controlled robotic manipulation task. The next step is to investigate these factors with actual surgical robotics trials. Second, while the study sample size was small, this was a completely within-subject design that exhibited medium to big effect sizes. Future work that considers a larger and more diverse sample size (e.g., surgical population, age, expertise) are warranted. Third, the present study focused on a limited combination of modalities – future work should include multiple modalities with different stimulus salience parameters. Fourth, our fNIRS system covered only frontal portions of the head. Future work will utilize a whole-head fNIRS system to investigate other brain regions that may likely be recruited to preserve motor performance. Fifth, we did not record hormonal contraceptives and menstrual cycle phase for female participants. Since these are known to influence cortisol responses to stress in women, future work will investigate how they impact neural efficiencies of HRI feedback modalities under stress in women. Finally, we used perceived workload measured as the overall workload score of the NASA-TLX and known to be associated with stress (Arora et al., 2010; Klein et al., 2012), as proxy for stress. There were no physiological or subjective measures of how stressed the participants were. Future work should include measuring levels of stress, as the response may be individual specific.

Conclusion

To our knowledge, this is the first study to investigate how gender and stress moderate the effectiveness of multimodal feedback during HRI. We posited that operator performance is a summation of task performance and neurophysiological cost of maintaining that performance. We found that while task performance was higher with visual than visual + haptic feedback, it degraded under stress. Furthermore, the two feedback modalities were found to be associated with varying hemodynamic activations and neural efficiencies, and these were stress- and gender-dependent. Our findings suggest that it is crucial to consider stress and gender differences when designing feedback in HRI.

Data Availability

The datasets generated for this study are available on request to the corresponding author.

Ethics Statement

This study was carried out in accordance with the recommendations of the Federal Regulations for Protection of Human Research Subjects (45 CFR 46) with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the Institutional Review Board at Texas A&M University.

Author Contributions

WM and RK developed the study apparatus, conducted the data collection, performed the initial analysis, and drafted the initial manuscript. RM and SR conceptualized the study. RM analyzed the study results, and edited and revised drafted manuscript. JN analyzed the study results and revised the drafted manuscript.

Funding

The open access publishing fees for this article have been covered by the Texas A&M University Open Access to Knowledge Fund (OAKFund), supported by the University Libraries and the Office of the Vice President for Research.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References

Aasted, C. M., Yücel, M. A., Cooper, R. J., Dubb, J., Tsuzuki, D., Becerra, L., et al. (2015). Anatomical guidance for functional near-infrared spectroscopy: atlasviewer tutorial. Neurophotonics 2:020801. doi: 10.1117/1.NPh.2.2.020801

PubMed Abstract | CrossRef Full Text | Google Scholar

Abiri, A., Juo, Y. Y., Tao, A., Askari, S. J., Pensa, J., Bisley, J. W., et al. (2018). Artificial palpation in robotic surgery using haptic feedback. Surg. Endosc. 33, 1252–1259. doi: 10.1007/s00464-018-6405-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Abiri, A., Tao, A., LaRocca, M., Guan, X., Askari, S. J., Bisley, J. W., et al. (2017). Visual–perceptual mismatch in robotic surgery. Surg. Endosc. 31, 3271–3278. doi: 10.1007/s00464-016-5358-z

PubMed Abstract | CrossRef Full Text | Google Scholar

Adebiyi, A., Sorrentino, P., Bohlool, S., Zhang, C., Arditti, M., Goodrich, G., et al. (2017). Assessment of feedback modalities for wearable visual aids in blind mobility. PloS One 12:e0170531. doi: 10.1371/journal.pone.0170531

PubMed Abstract | CrossRef Full Text | Google Scholar

Ali, A., Subhi, Y., Ringsted, C., and Konge, L. (2015). Gender differences in the acquisition of surgical skills: a systematic review. Surg. Endosc. 29, 3065–3073. doi: 10.1007/s00464-015-4092-2

PubMed Abstract | CrossRef Full Text | Google Scholar

Arnsten, A. F. (2009). Stress signalling pathways that impair prefrontal cortex structure and function. Nat. Rev. Neurosci. 10:410. doi: 10.1038/nrn2648

PubMed Abstract | CrossRef Full Text | Google Scholar

Arora, S., Sevdalis, N., Aggarwal, R., Sirimanna, P., Darzi, A., and Kneebone, R. (2010). Stress impairs psychomotor performance in novice laparoscopic surgeons. Surg. Endosc. 24, 2588–2593. doi: 10.1007/s00464-010-1013-2

PubMed Abstract | CrossRef Full Text | Google Scholar

Benz, T. M., and Nitsch, V. (2017). Using multisensory cues for direction information in teleoperation: more is not always better. in Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA). Piscataway, NJ

Google Scholar

Berguer, R., and Smith, W. (2006). An ergonomic comparison of robotic and laparoscopic technique: the influence of surgeon experience and task complexity. J. Surg. Res. 134, 87–92. doi: 10.1016/j.jss.2005.10.003

PubMed Abstract | CrossRef Full Text | Google Scholar

Bershad, A. K., Miller, M. A., Norman, G. J., and de Wit, H. (2018). Effects of opioid-and non-opioid analgesics on responses to psychosocial stress in humans. Horm. Behav. 102, 41–47. doi: 10.1016/j.yhbeh.2018.04.009

PubMed Abstract | CrossRef Full Text | Google Scholar

Bonneville-Roussy, A., Evans, P., Verner-Filion, J., Vallerand, R. J., and Bouffard, T. (2017). Motivation and coping with the stress of assessment: gender differences in outcomes for university students. Contemp. Educ. Psychol. 48, 28–42. doi: 10.1016/j.cedpsych.2016.08.003

CrossRef Full Text | Google Scholar

Brugnera, A., Zarbo, C., Adorni, R., Tasca, G. A., Rabboni, M., Bondi, E., et al. (2017). Cortical and cardiovascular responses to acute stressors and their relations with psychological distress. Int. J. Psychophysiol. 114, 38–46. doi: 10.1016/j.ijpsycho.2017.02.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Burke, J. L., Prewett, M. S., Gray, A. A., Yang, L., Stilson, F. R., Coovert, M. D., et al. (2006). Comparing the effects of visual-auditory and visual-tactile feedback on user performance: a meta-analysis. in Proceedings of the 8th International Conference on Multimodal Interfaces. New York, NY.

Google Scholar

Castonguay, N., Lussier, M., Bugaiska, A., Lord, C., and Bherer, L. (2015). Executive functions in men and postmenopausal women. J. Clin. Exp. Neuropsychol. 37, 193–208. doi: 10.1080/13803395.2014.1000267

PubMed Abstract | CrossRef Full Text | Google Scholar

Cerqueira, J. J., Almeida, O. F., and Sousa, N. (2008). The stressed prefrontal cortex. Left Right Brain Behav. Immun. 22, 630–638. doi: 10.1016/j.bbi.2008.01.005

PubMed Abstract | CrossRef Full Text | Google Scholar

Cheng, R., Shang, Y., Hayes D., Jr. Saha, S. P., and Yu, G. (2012). Noninvasive optical evaluation of spontaneous low frequency oscillations in cerebral hemodynamics. Neuroimage 62, 1445–1454. doi: 10.1016/j.neuroimage.2012.05.069

PubMed Abstract | CrossRef Full Text | Google Scholar

Chiarelli, A. M., Maclin, E. L., Fabiani, M., and Gratton, G. (2015). A kurtosis-based wavelet algorithm for motion artifact correction of fNIRS data. NeuroImage 112, 128–137. doi: 10.1016/j.neuroimage.2015.02.057

PubMed Abstract | CrossRef Full Text | Google Scholar

Choleris, E., Galea, L. A., Sohrabji, F., and Frick, K. M. (2018). Sex differences in the brain: implications for behavioral and biomedical research. Neurosci. Biobehav. Rev. 85, 126–145. doi: 10.1016/j.neubiorev.2017.07.005

PubMed Abstract | CrossRef Full Text | Google Scholar

Chuang, C. C., and Sun, C. W. (2014). Gender-related effects of prefrontal cortex connectivity: a resting-state functional optical tomography study. Biomed. Optics Express. 5, 2503–2516. doi: 10.1364/BOE.5.002503

PubMed Abstract | CrossRef Full Text | Google Scholar

Cinciute, S., Daktariunas, A., and Ruksenas, O. (2018). Hemodynamic effects of sex and handedness on the wisconsin card sorting test: the contradiction between neuroimaging and behavioural results. PeerJ 6:e5890. doi: 10.7717/peerj.5890

PubMed Abstract | CrossRef Full Text | Google Scholar

Curtin, A., and Ayaz, H. (2019). Neural Efficiency Metrics in Neuroergonomics: Theory and Applications. Academic Press: Cambridge, MA.

Google Scholar

Cutmore, T. R., Hine, T. J., Maberly, K. J., Langford, N. M., and Hawgood, G. (2000). Cognitive and gender factors influencing navigation in a virtual environment. Int. J. Hum. Comput. Stud. 53, 223–249. doi: 10.1006/ijhc.2000.0389

CrossRef Full Text | Google Scholar

Delpy, D. T., Cope, M., van der Zee, P., Arridge, S. R., Wray, S., and Wyatt, J. S. (1988). Estimation of optical pathlength through tissue from direct time of flight measurement. Phys. Med. Biol. 33:1433. doi: 10.1088/0031-9155/33/12/008

PubMed Abstract | CrossRef Full Text | Google Scholar

Enayati, N., Ferrigno, G., and De Momi, E. (2018). Performance metrics for guidance active constraints in surgical robotics. Int. J. Med. Robotics Comput. Assisted Surg. 14:e1873. doi: 10.1002/rcs.1873

PubMed Abstract | CrossRef Full Text | Google Scholar

Felmingham, K. L., Tran, T. P., Fong, W. C., and Bryant, R. A. (2012). Sex differences in emotional memory consolidation: the effect of stress-induced salivary alpha-amylase and cortisol. Biol. Psychol. 89, 539–544. doi: 10.1016/j.biopsycho.2011.12.006

PubMed Abstract | CrossRef Full Text | Google Scholar

Feygin, D., Keehner, M., and Tendick, R. (2002). Haptic guidance: experimental evaluation of a haptic training method for a perceptual motor skill. in Proceedings 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. HAPTICS 2002. Piscataway, NJ

Google Scholar

Foy, H. J., and Chapman, P. (2018). Mental workload is reflected in driver behaviour, physiology, eye movements and prefrontal cortex activation. Appl. Ergonom. 73, 90–99. doi: 10.1016/j.apergo.2018.06.006.

PubMed Abstract | CrossRef Full Text | Google Scholar

Freeman, E., Wilson, G., Vo, D. B., Ng, A., Politis, I., and Brewster, S. (2017). “Multimodal feedback in HCI: haptics, non-speech audio, and their applications,” in The Handbook of Multimodal-Multisensor Interfaces, ed. S. Oviatt, (New York, NY: ACM Books).

Google Scholar

Fuster, J. M., Bodner, M., and Kroger, J. K. (2000). Cross-modal and cross-temporal association in neurons of frontal cortex. Nature 405:347. doi: 10.1038/35012613

PubMed Abstract | CrossRef Full Text | Google Scholar

Gateau, T., Ayaz, H., and Dehais, F. (2018). In silico versus over the clouds: on-the-fly mental state estimation of aircraft pilots, using a functional near infrared spectroscopy based passive-BCI. Front. Hum. Neurosci. 12:187. doi: 10.3389/fnhum.2018.00187

PubMed Abstract | CrossRef Full Text | Google Scholar

Georgiou, K., Larentzakis, A., and Papavassiliou, A. G. (2017). Surgeons’ and surgical trainees’ acute stress in real operations or simulation: a systematic review. Surgeon 15, 355–365. doi: 10.1016/j.surge.2017.06.003

PubMed Abstract | CrossRef Full Text | Google Scholar

Goldfarb, E. V., Seo, D., and Sinha, R. (2019). Sex differences in neural stress responses and correlation with subjective stress and stress regulation. Neurobiol. Stress. 11:100177. doi: 10.1016/j.ynstr.2019.100177

PubMed Abstract | CrossRef Full Text | Google Scholar

Haier, R. J., Siegel B. V. Jr., Nuechterlein, K. H., Hazlett, E., Wu, J. C., Paek, J., et al. (1988). Cortical glucose metabolic rate correlates of abstract reasoning and attention studied with positron emission tomography. Intelligence 12, 199–217. doi: 10.1016/j.neubiorev.2009.04.001

PubMed Abstract | CrossRef Full Text | Google Scholar

Halpern, D. F., Benbow, C. P., Geary, D. C., Gur, R. C., Hyde, J. S., and Gernsbacher, M. A. (2007). The science of sex differences in science and mathematics. Psychol. Sci. Public Interest 8, 1–51.

Google Scholar

Hart, S. G., and Staveland, L. E. (1988). Development of NASA-TLX (task load index): results of empirical and theoretical research. Adv. Psychol. 52, 139–183. doi: 10.1016/s0166-4115(08)62386-9

CrossRef Full Text | Google Scholar

Hausmann, M. (2017). Why sex hormones matter for neuroscience: a very short review on sex, sex hormones, and functional brain asymmetries. J. Neurosci. Res. 95, 40–49. doi: 10.1002/jnr.23857

PubMed Abstract | CrossRef Full Text | Google Scholar

Hunter, S. K. (2014). Sex differences in human fatigability: mechanisms and insight to physiological responses. Acta Physiol. 210, 768–789. doi: 10.1111/apha.12234

PubMed Abstract | CrossRef Full Text | Google Scholar

Huppert, T. J., Diamond, S. G., Franceschini, M. A., and Boas, D. A. (2009). HomER: a review of time-series analysis methods for near-infrared spectroscopy of the brain. Appl. Optics 48, D280–D298.

PubMed Abstract | Google Scholar

Hyodo, K., Dan, I., Kyutoku, Y., Suwabe, K., Byun, K., Ochi, G., et al. (2016). The association between aerobic fitness and cognitive function in older men mediated by frontal lateralization. Neuroimage 125, 291–300. doi: 10.1016/j.neuroimage.2015.09.062

PubMed Abstract | CrossRef Full Text | Google Scholar

Jaušovec, N., and Jaušovec, K. (2010). Resting brain activity: differences between genders. Neuropsychologia 48, 3918–3925. doi: 10.1016/j.neuropsychologia.2010.09.020

PubMed Abstract | CrossRef Full Text | Google Scholar

Jimenez, M. C., and Fishel, J. A. (2014). Evaluation of force, vibration and thermal tactile feedback in prosthetic limbs. in Proceedings of the 2014 IEEE Haptics Symposium (HAPTICS). Piscataway, NJ

Google Scholar

Jones, K. I., Amawi, F., Bhalla, A., Peacock, O., Williams, J. P., and Lund, J. N. (2015). Assessing surgeon stress when operating using heart rate variability and the State Trait Anxiety Inventory: will surgery be the death of us?. Colorectal Dis. 17, 335–341. doi: 10.1111/codi.12844

PubMed Abstract | CrossRef Full Text | Google Scholar

Kajantie, E., and Phillips, D. I. (2006). The effects of sex and hormonal status on the physiological response to acute psychosocial stress. Psychoneuroendocrinology 31, 151–178. doi: 10.1016/j.psyneuen.2005.07.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Kalia, V., Vishwanath, K., Knauft, K., Vellen, B. V., Luebbe, A., and Williams, A. (2018). Acute stress attenuates cognitive flexibility in males only: an fnirs examination. Front. Psychol. 9:2084. doi: 10.3389/fpsyg.2018.02084

PubMed Abstract | CrossRef Full Text | Google Scholar

Kirschbaum, C., Pirke, K. M., and Hellhammer, D. H. (1993). The ‘trier social stress test’–a tool for investigating psychobiological stress responses in a laboratory setting. Neuropsychobiology 28, 76–81. doi: 10.1159/000119004

PubMed Abstract | CrossRef Full Text | Google Scholar

Klatzkin, R. R., Baldassaro, A., and Rashid, S. (2019). Physiological responses to acute stress and the drive to eat: the impact of perceived life stress. Appetite 133, 393–399. doi: 10.1016/j.appet.2018.11.019

PubMed Abstract | CrossRef Full Text | Google Scholar

Klatzkin, R. R., Gaffney, S., Cyrus, K., Bigus, E., and Brownley, K. A. (2018). Stress-induced eating in women with binge-eating disorder and obesity. Biol. Psychol. 131, 96–106. doi: 10.1016/j.biopsycho.2016.11.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Klein, M. I., Warm, J. S., Riley, M. A., Matthews, G., Doarn, C., Donovan, J. F., et al. (2012). Mental workload and stress perceived by novice operators in the laparoscopic and robotic minimally invasive surgical interfaces. J. Endourol. 26, 1089–1094. doi: 10.1089/end.2011.0641

PubMed Abstract | CrossRef Full Text | Google Scholar

Koehn, J. K., and Kuchenbecker, K. J. (2015). Surgeons and non-surgeons prefer haptic feedback of instrument vibrations during robotic surgery. Surg. Endosc. 29, 2970–2983. doi: 10.1007/s00464-014-4030-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Kogler, L., Gur, R. C., and Derntl, B. (2015a). Sex differences in cognitive regulation of psychosocial achievement stress: brain and behavior. Hum. Brain Mapp. 36, 1028–1042. doi: 10.1002/hbm.22683

PubMed Abstract | CrossRef Full Text | Google Scholar

Kogler, L., Müller, V. I., Chang, A., Eickhoff, S. B., Fox, P. T., Gur, R. C., et al. (2015b). Psychosocial versus physiological stress—meta-analyses on deactivations and activations of the neural correlates of stress reactions. Neuroimage 119, 235–251. doi: 10.1016/j.neuroimage.2015.06.059

PubMed Abstract | CrossRef Full Text | Google Scholar

Lee, J. H., and Spence, C. (2008). Assessing the benefits of multimodal feedback on dual-task performance under demanding conditions. in Proceedings of the 22nd British HCI Group Annual Conference on People and Computers: Culture, Creativity Interaction. London

Google Scholar

Leff, D. R., Elwell, C. E., Orihuela-Espina, F., Atallah, L., Delpy, D. T., Darzi, A. W., et al. (2008). Changes in prefrontal cortical behaviour depend upon familiarity on a bimanual co-ordination task: an fNIRS study. Neuroimage 39, 805–813. doi: 10.1016/j.neuroimage.2007.09.032

PubMed Abstract | CrossRef Full Text | Google Scholar

Li, T., Luo, Q., and Gong, H. (2010). Gender-specific hemodynamics in prefrontal cortex during a verbal working memory task by near-infrared spectroscopy. Behav. Brain Res. 209, 148–153. doi: 10.1016/j.bbr.2010.01.033

PubMed Abstract | CrossRef Full Text | Google Scholar

Liu, J., Cramer, S. C., and Reinkensmeyer, D. J. (2006). Learning to perform a new movement with robotic assistance: comparison of haptic guidance and visual demonstration. J. Neuroeng. Rehabil. 3:20.

PubMed Abstract | Google Scholar

Matud, M. P. (2004). Gender differences in stress and coping styles. Pers. Indiv. Differ. 37, 1401–1415. doi: 10.1016/j.paid.2004.01.010

CrossRef Full Text | Google Scholar

McGivern, R. F., Adams, B., Handa, R. J., and Pineda, J. A. (2012). Men and women exhibit a differential bias for processing movement versus objects. PLoS One 7:e32238. doi: 10.1371/journal.pone.0032238

PubMed Abstract | CrossRef Full Text | Google Scholar

McKendy, K. M., Watanabe, Y., Lee, L., Bilgic, E., Enani, G., Feldman, L. S., et al. (2017). Perioperative feedback in surgical training: a systematic review. Am. J. Surg. 214, 117–126. doi: 10.1016/j.amjsurg.2016.12.014

PubMed Abstract | CrossRef Full Text | Google Scholar

McKlveen, J. M., Myers, B., and Herman, J. P. (2015). The medial prefrontal cortex: coordinator of autonomic, neuroendocrine and behavioural responses to stress. J. Neuroendocrinol. 27, 446–456. doi: 10.1111/jne.12272

PubMed Abstract | CrossRef Full Text | Google Scholar

McMahan, W., Gewirtz, J., Standish, D., Martin, P., Kunkel, J. A., Lilavois, M., et al. (2011). Tool contact acceleration feedback for telerobotic surgery. IEEE Trans. Haptics. 4, 210–220. doi: 10.1109/TOH.2011.31

PubMed Abstract | CrossRef Full Text | Google Scholar

Mehta, R. K. (2015). Impacts of obesity and stress on neuromuscular fatigue development and associated heart rate variability. Int. J. Obes. 39:208. doi: 10.1038/ijo.2014.127

PubMed Abstract | CrossRef Full Text | Google Scholar

Mehta, R. K. (2016). Stunted PFC activity during neuromuscular control under stress with obesity. Eur. J. Appl. Physiol. 116, 319–326. doi: 10.1007/s00421-015-3283-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Mehta, R. K., and Agnew, M. J. (2012). Influence of mental workload on muscle endurance, fatigue, and recovery during intermittent static work. Eur. J. Appl. Physiol. 112, 2891–2902. doi: 10.1007/s00421-011-2264-x

PubMed Abstract | CrossRef Full Text | Google Scholar

Mehta, R. K., and Parasuraman, R. (2014). Effects of mental fatigue on the development of physical fatigue: a neuroergonomic approach. Hum. Factors 56, 645–656. doi: 10.1177/0018720813507279

PubMed Abstract | CrossRef Full Text | Google Scholar

Mehta, R. K., and Rhee, J. (2017). Age-specific neural strategies to maintain motor performance after an acute social stress bout. Exp. Brain Res. 235, 2049–2057. doi: 10.1007/s00221-017-4949-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Miura, S., Kawamura, K., Kobayashi, Y., and Fujie, M. G. (2018). Using brain activation to evaluate arrangements aiding hand-eye coordination in surgical robot systems. IEEE Trans. Biomed. Eng. 66, 2352–2361 doi: 10.1109/tbme.2018.2889316

PubMed Abstract | CrossRef Full Text | Google Scholar

Miyazaki, R., Kanno, T., and Kawashima, K. (2019). Pneumatically driven surgical instrument capable of estimating translational force and grasping force. Int. J. Med. Robot. 15:e1983. doi: 10.1002/rcs.1983

PubMed Abstract | CrossRef Full Text | Google Scholar

Modi, H. N., Singh, H., Orihuela-Espina, F., Athanasiou, T., Fiorentino, F., Yang, G. Z., et al. (2018). Temporal stress in the operating room: brain engagement promotes “coping” and disengagement prompts “choking”. Ann. Sur. 267, 683–691. doi: 10.1097/SLA.0000000000002289

PubMed Abstract | CrossRef Full Text | Google Scholar

Munro, C. A., Winicki, J. M., Schretlen, D. J., Gower, E. W., Turano, K. A., Muñoz, B., et al. (2012). Sex differences in cognition in healthy elderly individuals. Aging Neuropsychol. Cogn. 19, 759–768. doi: 10.1080/13825585.2012.690366

PubMed Abstract | CrossRef Full Text | Google Scholar

Pacchierotti, C., Meli, L., Chinello, F., Malvezzi, M., and Prattichizzo, D. (2015). Cutaneous haptic feedback to ensure the stability of robotic teleoperation systems. Int. J. Robot. Res. 34, 1773–1787. doi: 10.1177/0278364915603135

CrossRef Full Text | Google Scholar

Park, A., Lee, G., Seagull, F. J., Meenaghan, N., and Dexter, D. (2010). Patients benefit while surgeons suffer: an impending epidemic. J. Am. College Surg. 306–313. doi: 10.1016/j.jamcollsurg.2009.10.017

PubMed Abstract | CrossRef Full Text | Google Scholar

Park, C. H., Ryu, E. S., and Howard, A. M. (2015). Telerobotic haptic exploration in art galleries and museums for individuals with visual impairments. IEEE Trans. Haptics 8, 327–338. doi: 10.1109/TOH.2015.2460253

PubMed Abstract | CrossRef Full Text | Google Scholar

Park, E., Kim, K. J., and Del Pobil, A. P. (2012). The effects of multimodal feedback and gender on task performance of stylus pen users. Int. J. Adv. Robot. Syst. 9:30. doi: 10.5772/50187

CrossRef Full Text | Google Scholar

Politis, I., Brewster, S., and Pollick, F. (2014). The effects of modality, urgency and message content on responses to multimodal driver displays. automotiveUI’14. in Proceedings of the Adjunct 6th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. New York, NY

Google Scholar

Powell, L. J., Deen, B., and Saxe, R. (2018). Using individual functional channels of interest to study cortical development with fNIRS. Dev. Sci. 21:e12595. doi: 10.1111/desc.12595

PubMed Abstract | CrossRef Full Text | Google Scholar

Rhee, J., and Mehta, R. K. (2018). Functional connectivity during handgrip motor fatigue in older adults is obesity and sex-specific. Front. Hum. Neurosci. 12:455

PubMed Abstract | Google Scholar

Ritter, F. E., Schoelles, M., Klein, L. C., and Kase, S. E. (2007). Modeling the range of performance on the serial subtraction task. in Proceedings of the 8th International Conference on Cognitive Modeling. Oxford,Google Scholar

Ruigrok, A. N., Salimi-Khorshidi, G., Lai, M. C., Baron-Cohen, S., Lombardo, M. V., Tait, R. J., et al. (2014). A meta-analysis of sex differences in human brain structure. Neurosci. Biobehav. Rev. 39, 34–50. doi: 10.1016/j.neubiorev.2013.12.004

PubMed Abstract | CrossRef Full Text | Google Scholar

Sakatani, K., Tanida, M., and Katsuyama, M. (2010). Effects of Aging on Activity of the Prefrontal Cortex and Autonomic Nervous System During Mental Stress Task. in Oxygen Transport to Tissue XXXI. Boston, MA: Springer

Google Scholar

Sari, V., Nieboer, T. E., Vierhout, M. E., Stegeman, D. F., and Kluivers, K. B. (2010). The operation room as a hostile environment for surgeons: physical complaints during and after laparoscopy. Mini. Invas. Ther. Allied Technol. 19, 105–109. doi: 10.3109/13645701003643972

PubMed Abstract | CrossRef Full Text | Google Scholar

Scholkmann, F., Spichtig, S., Muehlemann, T., and Wolf, M. (2010). How to detect and reduce movement artifacts in near-infrared imaging using moving standard deviation and spline interpolation. Physiol. Measure. 31:649. doi: 10.1088/0967-3334/31/5/004

PubMed Abstract | CrossRef Full Text | Google Scholar

Sen, S., Garg, A., Gealy, D. V., McKinley, S., Jen, Y., and Goldberg, K. (2016). Automating multi-throw multilateral surgical suturing with a mechanical needle guide and sequential convex optimization. in proceedings of the 2016 Ieee International Conference on Robotics and Automation (ICRA). Piscataway, NJ

Google Scholar

Seo, D., Jia, Z., Lacadie, C. M., Tsou, K. A., Bergquist, K., and Sinha, R. (2011). Sex differences in neural responses to stress and alcohol context cues. Hum. Brain Mapp. 32, 1998–2013. doi: 10.1002/hbm.21165

PubMed Abstract | CrossRef Full Text | Google Scholar

Seraglia, B., Gamberini, L., Priftis, K., Scatturin, P., Martinelli, M., and Cutini, S. (2011). An exploratory fNIRS study with immersive virtual reality: a new method for technical implementation. Front. Hum. Neurosci. 5:176. doi: 10.3389/fnhum.2011.00176

PubMed Abstract | CrossRef Full Text | Google Scholar

Shafiei, S. B., Hussein, A. A., and Guru, K. A. (2018). Dynamic changes of brain functional states during surgical skill acquisition. PloS One 13:e0204836. doi: 10.1371/journal.pone.0204836

PubMed Abstract | CrossRef Full Text | Google Scholar

Shansky, R. M., Glavis-Bloom, C., Lerman, D., McRae, P., Benson, C., Miller, K., et al. (2004). Estrogen mediates sex differences in stress-induced prefrontal cortex dysfunction. Mole. Psychiatry 9:531. doi: 10.1038/sj.mp.4001435

PubMed Abstract | CrossRef Full Text | Google Scholar

Sigrist, R., Rauter, G., Riener, R., and Wolf, P. (2013). Augmented visual, auditory, haptic, and multimodal feedback in motor learning: a review. Psychon. Bull. Rev. 20, 21–53. doi: 10.3758/s13423-012-0333-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Silverman, I., Choi, J., and Peters, M. (2007). The hunter-gatherer theory of sex differences in spatial abilities: data from 40 countries. Arch. Sex. Behav. 36, 261–268. doi: 10.1007/s10508-006-9168-6

PubMed Abstract | CrossRef Full Text | Google Scholar

Singh, H., Modi, H. N., Ranjan, S., Dilley, J. W., Airantzis, D., Yang, G. Z., et al. (2018). Robotic surgery improves technical performance and enhances prefrontal activation during high temporal demand. Ann. Biomed. Eng. 46, 1621–1636.

Google Scholar

Sliwinski, M. J., Almeida, D. M., Smyth, J., and Stawski, R. S. (2009). Intraindividual change and variability in daily stress processes: findings from two measurement-burst diary studies. Psychol. Aging 24:828. doi: 10.1037/a0017925

PubMed Abstract | CrossRef Full Text | Google Scholar

Smith, W. D., Berguer, R., and Rosser, J. J. (2003). Wireless virtual instrument measurement of surgeons’ physical and mental workloads for robotic versus manual minimally invasive surgery. Stud. Health Technol. Inform. 94, 318–324.

PubMed Abstract | Google Scholar

Soltanlou, M., Artemenko, C., Ehlis, A. C., Huber, S., Fallgatter, A. J., Dresler, T., et al. (2018). Reduction but no shift in brain activation after arithmetic learning in children: a simultaneous fNIRS-EEG study. Sci. Rep. 8:1707. doi: 10.1038/s41598-018-20007-x

PubMed Abstract | CrossRef Full Text | Google Scholar

Takizawa, R., Nishimura, Y., Yamasue, H., and Kasai, K. (2013). Anxiety and performance: the disparate roles of prefrontal subregions under maintained psychological stress. Cereb. Cortex 24, 1858–1866. doi: 10.1093/cercor/bht036

PubMed Abstract | CrossRef Full Text | Google Scholar

Tanida, M., Katsuyama, M., and Sakatani, K. (2007). Relation between mental stress-induced prefrontal cortex activity and skin conditions: a near-infrared spectroscopy study. Brain Res. 1184, 210–216. doi: 10.1016/j.brainres.2007.09.058

PubMed Abstract | CrossRef Full Text | Google Scholar

Taylor, S. E., Gonzaga, G. C., Klein, L. C., Hu, P., Greendale, G. A., and Seeman, T. E. (2006). Relation of oxytocin to psychological stress responses and hypothalamic-pituitary-adrenocortical axis activity in older women. Psychosom. Med. 68, 238–245. doi: 10.1097/01.psy.0000203242.95990.74

PubMed Abstract | CrossRef Full Text | Google Scholar

Thorson, C. M., Kelly, J. P., Forse, R. A., and Turaga, K. K. (2011). Can we continue to ignore gender differences in performance on simulation trainers?. J. Laparoendosc. Adv. Surg. Tech. 21, 329–333. doi: 10.1089/lap.2010.0368

PubMed Abstract | CrossRef Full Text | Google Scholar

Ulrich-Lai, Y. M., and Herman, J. P. (2009). Neural regulation of endocrine and autonomic stress responses. Nat. Rev. Neurosci. 10:397. doi: 10.1038/nrn2647

PubMed Abstract | CrossRef Full Text | Google Scholar

Van der Meijden, O. A., and Schijven, M. P. (2009). The value of haptic feedback in conventional and robot-assisted minimal invasive surgery and virtual reality training: a current review. Surg. Endosc. 23, 1180–1190. doi: 10.1007/s00464-008-0298-x

PubMed Abstract | CrossRef Full Text | Google Scholar

Van Erp, J. B., and Verschoor, M. H. (2004). Cross-modal visual and vibrotactile tracking. Appl. Ergon. 35, 105–112. doi: 10.1016/j.apergo.2003.12.004

PubMed Abstract | CrossRef Full Text | Google Scholar

van Huysduynen, H. H., De Valk, L., and Bekker, T. (2016). Tangible play objects: influence of different combinations of feedback modalities. in Proceedings of the TEI’16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction. New York, NY

Google Scholar

Vanzella, P., Balardin, J. B., Furucho, R. A., Zimeo Morais, G. A., Janzen, T. B., Sammler, D., et al. (2019). fNIRS responses in professional violinists while playing duets: evidence for distinct leader and follower roles at the brain level. Front. Psychol. 10:164. doi: 10.3389/fpsyg.2019.00164

PubMed Abstract | CrossRef Full Text | Google Scholar

Våpenstad, C., Hofstad, E. F., Bø, L. E., Kuhry, E., Johnsen, G., Mårvik, R., et al. (2017). Lack of transfer of skills after virtual reality simulator training with haptic feedback. Mini. Invasive Ther. Allied Technol. 26, 346–354. doi: 10.1080/13645706.2017.1319866

PubMed Abstract | CrossRef Full Text | Google Scholar

Verma, R., Balhara, Y. P. S., and Gupta, C. S. (2011). Gender differences in stress response: role of developmental and biological determinants. Indus. Psychiatr. J. 20:4. doi: 10.4103/0972-6748.98407

PubMed Abstract | CrossRef Full Text | Google Scholar

Wang, J., Korczykowski, M., Rao, H., Fan, Y., Pluta, J., Gur, R. C., et al. (2007). Gender difference in neural response to psychological stress. Soc. Cogn. Affect. Neurosci. 2, 227–239. doi: 10.1093/scan/nsm018

PubMed Abstract | CrossRef Full Text | Google Scholar

Wang, L. P., Hamaker, E., and Bergeman, C. S. (2012). Investigating inter-individual differences in short-term intra-individual variability. Psychol. Methods 17:567. doi: 10.1037/a0029317

PubMed Abstract | CrossRef Full Text | Google Scholar

Wetzel, C. M., Kneebone, R. L., Woloshynowych, M., Nestel, D., Moorthy, K., Kidd, J., et al. (2006). The effects of stress on surgical performance. Am. J. Surg. 191, 5–10.

PubMed Abstract | Google Scholar

Wickens, C. D. (2002). Multiple resources and performance prediction. Theor. Issues Ergon. Sci. 3, 159–177. doi: 10.1080/14639220210123806

CrossRef Full Text | Google Scholar

Wickens, C. D., Hollands, J. G., Banbury, S., and Parasuraman, R. (2015). Engineering Psychology & Human Performance. Psychology Press. Abingdon

Google Scholar

Yang, H., Wang, Y., Zhou, Z., Gong, H., Luo, Q., Wang, Y., et al. (2009). Sex differences in prefrontal hemodynamic response to mental arithmetic as assessed by near-infrared spectroscopy. Gender Med. 6, 565–574. doi: 10.1016/j.genm.2009.11.003

PubMed Abstract | CrossRef Full Text | Google Scholar

Yang, X. D., Bischof, W. F., and Boulanger, P. (2008). Validating the performance of haptic motor skill training. in Proceedings of the 2008 Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. Piscataway, NJ

Google Scholar

Yu, D., Dural, C., Morrow, M. M., Yang, L., Collins, J. W., Hallbeck, S., et al. (2017). Intraoperative workload in robotic surgery assessed by wearable motion tracking sensors and questionnaires. Surg. Endosc. 31, 877–886. doi: 10.1007/s00464-016-5047-y

PubMed Abstract | CrossRef Full Text | Google Scholar

Zhu, Y., Rodriguez-Paras, C., Rhee, J., and Mehta, R. K. (2019). Methodological approaches and recommendations for functional near-infrared spectroscopy applications in HF/E research. Hum. Factors. doi: 10.1177/0018720819845275 [Epub ahead of print].

CrossRef Full Text | PubMed Abstract | Google Scholar