The last decade has seen important research toward road automation (Badue et al., 2019). Promised as a revolution for users to gain flexibility, leisure time, and safety (Harb et al., 2018; Correia et al., 2019), self-driving cars have nonetheless several important technological gaps that must be filled before becoming a reality. On the way toward level 5 automation (full automation anywhere, see SAE International, 2018), teleoperation could represent an important trade-off to maintain safety while developing system capabilities. Teleoperation, literally operating a vehicle at a distance, is already used in environments unreachable or dangerous to humans, such as war theaters, nuclear environments, and space (Lichiardopol, 2007). Tomorrow, teleoperation could be performed by algorithms in the cloud and allow any vehicle to reach level 5 automation with minimal modifications (Zhang, 2020). However, the technology could also use human intervention today to enhance partial automation and widen its operational design domain (Kang et al., 2018). Operators would then monitor a set of vehicles, taking control whenever necessary, such as in the event of snow or in an emergency. Specifically, an important advantage of human teleoperation is the assumption that there could be more vehicles to monitor than operators, as not all vehicles would require assistance at the same time (Zhang, 2020).

Aside from technical challenges like latency (Neumeier et al., 2019), the possibility of jumping into a specific situation only when needed raises important interrogations regarding the ability of operators to assume manual control when needed. Humans would then only have to monitor, presumably ever-alert, for deviations and problems. Situations where operators are supervising automated control loop are called out-of-the-loop (OOTL) situations (Norman and Orlady, 1988; Endsley and Kiris, 1995). Unfortunately, OOTL situations reduce the operators' ability to intervene, if necessary, and to assume manual control, i.e., to come back in the control loop (Kurihashi et al., 2015; Louw et al., 2015a; Naujoks et al., 2016). Supervisors at this point seem dramatically powerless to diagnose the situation, determine the appropriate solution, and execute it before the accident happens. Accident reports may contain the terms “total confusion” (National Transportation Safety Board, 1975, 17; Bureau d'Enquête et d'Analyse, 2002, 167), “surprise effect” (Bureau d'Enquête et d'Analyse, 2012a, 44, 2016, 10), or “no awareness of the current mode of the system” (Bureau d'Enquête et d'Analyse, 2012b, 178). These negative side effects on overall performance are commonly referred to as OOTL performance problems.

Nowadays, it is assumed that OOTL performance problem is fundamentally a matter of human–automation interaction arising from both operators' internal states and system properties, which ultimately spoils performance (Berberian et al., 2017). From this definition, one way to mitigate related performance drops may be to monitor operators' internal states and look for precursors to OOTL performance problems. Among others, it has been demonstrated that non-challenging tasks, such as passive monitoring of automation, can promote episodes of mind wandering, whereby attention drifts away from the task at hand (Smallwood et al., 2008; Durantin et al., 2015; Smallwood and Schooler, 2015; Gouraud et al., 2018a,b; Dehais et al., 2020). Mind wandering (MW) is a family of experiences unrelated to the here and now (Seli et al., 2018). When MW happens during a task, it moves operators' minds away from their tasks toward matters not directly related to their current works. Although such uncontrolled thoughts could be beneficial as long-term planning and mind refreshment (McMillan et al., 2013; Ottaviani and Couyoumdjian, 2013; Terhune et al., 2017), it may thwart short-term performances (He et al., 2011; Galera et al., 2012; Cowley, 2013; Casner and Schooler, 2014; Dündar, 2015). Therefore, real-time tracking of MW is an important goal within safety-critical industries, particularly when automation supervision fills a significant part of the job. Indeed, real-time tracking of internal states like MW would allow detecting problems before performance drops and accidents happen. However, a better understanding of the emergence of this attentional decoupling remains essential to achieve such a goal. This is precisely the objective of this study.

Many physiological tools have already demonstrated sensitivity to several aspects of MW; however, electroencephalography (EEG) is among the most promising. EEG signal has already helped uncover an important facet of MW: attentional decoupling. People subject to MW experience a drop in the cortical processing of the external environment, as their attention is redirected to inner thoughts (Schooler et al., 2011). Neurologically, attentional decoupling is characterized by weaker neuronal responses to external stimuli and greater deactivation of the regions dedicated to their processing. During GO/NOGO tasks, researchers (Kam et al., 2011) showed that the amplitude of P1, N1, and P3 components (respectively associated with visual perception, auditory perception, and external stimuli processing) were all lower during task-unrelated MW. This effect held true whether stimuli were the SART (Sustained Attention to Response Task) stimuli or irrelevant to the task. Such results were replicated in two other settings: a time-estimation task (Kam et al., 2012) and during monotonous manual driving (Baldwin et al., 2017). It was also highlighted through ERPs that attentional decoupling involved lower emotional reactions (Kam et al., 2014). Experiments also uncovered the signature of MW on alpha waves in occipital, i.e., visual stimuli processing, areas (O'Connell et al., 2009; Braboszcz and Delorme, 2011; Baird et al., 2014; Atchley et al., 2017; Baldwin et al., 2017; Arnau et al., 2020), although the exact way is still debated as explicated in the next sections. Nevertheless, changes in alpha activity during MW are in line with the alpha band being involved in the deactivation of the concerned areas (Bonnefond and Jensen, 2012; Benedek et al., 2014; Villena-González et al., 2016).

Even though MW has a strong influence on the neuronal signal, the factors modulating the attentional decoupling remain unidentified. A first important question is the exact degree of attentional decoupling. Put differently, do all MW have the same potential for attentional decoupling? Is “depth” a feature of MW episodes? Several studies provide insight into depth as a feature of MW episodes. Cheyne et al. (2009) used a SART to investigate the validity of their bi-directional model of inattention. They obtained converging measures supporting three postulated states of inattention: level 1 characterized by more erratic reaction time, level 2 by anticipations, and level 3 by omissions. Following the same path, Schad et al. (2012) detailed the “levels of inattention hypothesis” based on the assumption that our mind processes information sequentially, involving greater complexity at each step. MW could then thwart information processing at different stages, depending on the depth of the episode. While some MW episodes could be superficial, only impacting higher cognition, others could completely decouple from the task by blocking external information encoding and “cascade through the cognitive system” to impact more complex processing (Smallwood, 2011).

A second issue refers to the impact of MW on non-relevant stimulation. It was initially assumed that MW involves a specific impairment in the processing of task-relevant events (e.g., Smallwood et al., 2003, 2004). Studies using ERPs have shown that MW dampens the processing of sensory information, regardless of the relevance of this information to the task (Barron et al., 2011; Kam et al., 2011). However, the fact that MW can impact mechanisms of selective attention does not mean that all stages of sensory processing are turned off. Rather, it signifies that the highlighting of specific sensory inputs for higher levels of cognitive analysis is attenuated. After all, we are able to perform most of our daily tasks without any errors, even during MW episodes. In this context, steady-state responses (SSR) may highlight the exact impact of MW on cognition. An SSR is an evoked potential emerging from external periodical stimulus and whose phase and amplitude remain constant (Picton et al., 2003). Multiple studies have highlighted that in environments with multiple SSR competing for attention, focusing on one SSR increases its amplitude to the detriment of the others (Skosnik et al., 2007; Müller et al., 2009; Saupe et al., 2009a; Diesch et al., 2012; Mahajan et al., 2014). However, it has been shown that this effect is highly dependent on experiments' features: paying attention to a 20-Hz ASSR presented on one ear showed increase amplitude ipsilaterally, but not for a 40-Hz stimulus (Müller et al., 2009); in another study, the attention-competition effect decreased SSR amplitude only when concurrent SSR were presented on the same sensory modality (Porcu et al., 2014). These results highlight the complexity of the different stages of perception and attention, and SSR may help to understand the influence of MW on them. Moreover, if SSR were to be impacted by MW, it could reveal extraordinarily useful to study the features of attentional decoupling. Indeed, it would allow continuous monitoring, contrary to ERPs, while being fully controlled in frequency, in contrast to natural brain waves. O'Connell et al. (2009) has already investigated the influence of lapses of attention on a visual SSR without finding significant results regarding its amplitude. However, they did not use a questionnaire to track MW episodes. To our knowledge, no research has specifically addressed the impact of internally directed attention on SSR amplitude.

Our purpose in this experiment is to evaluate the viability of MW neuronal markers in complex laboratory tasks mimicking automated ecological environments, as well as help to characterize features of the attentional decoupling in these environments. Our hypotheses are (1) the evolution of MW can be tracked in complex environments through a decrease in ERPs and ASSR amplitude coupled with an increase in alpha power during MW episodes compared with focus moments, (2) MW attentional decoupling demonstrates a gradual nature on EEG measures (ERP, alpha, ASSR) correlated to the proximity of the thoughts content to the task at hand; more precisely, a MW episode with thoughts closer to the immediate environment will have less influence on EEG measures than another MW episodes with thoughts totally unrelated to the here and now.

We performed an a priori analysis to estimate the required sample size. Most publications investigating the links between MW and EEG did not report effect size explicitly. However, as repeated-measure ANOVA was often used, we could calculate from these publications Cohen's f using F-value, CI, and degrees of freedom. The lower value computed, which we retained to adopt a conservative view, was 0.54 (Kelley, 2007a,b, 2020; Uanhoro, 2017). We then used G^*Power (Faul et al., 2007, 2009) to calculate the sample size, which yielded a minimum of 14 participants.

Eighteen participants (12 females, all right-handed) performed the experiment (age ranging from 21 to 45 years; M = 25, 95% CI = [22; 29]). After pre-processing the data, we discarded three subjects:

This resulted in 15 subjects in the analysis. The participants in this study were volunteers from the ONERA Company (ONERA, the French Aerospace Lab) or Marseille University. They received 20€ vouchers (cards for online payment) for the experiment. All the participants had normal or corrected-to-normal visual acuity and hearing, had no neurological or psychiatric disorders, and were not under any medication. All participants signed a written declaration of informed consent. The procedure was approved by ONERA ethical committee and was conducted in accordance with the World Medical Association Declaration of Helsinki.

Sessions started with an explanation of the two tasks, followed by a 10-min training period and a 55-min session. During this study, participants had to perform the visual task (supervise the UAV avoiding obstacles and acknowledge or correct any mistake, see Visual task) and the auditory task (press a button as fast as possible when hearing a beep, see Auditory task) at the same time. The session contained 70 clusters of obstacles for a total of 210 obstacles. Clusters were separated by 45 s on average. All autopilot decisions and collisions were predefined and, therefore, they were the same for all subjects. The autopilot made two errors initially placed randomly (3% errors; errors on trials 31 and 52 for all subjects). This low error rate was chosen to have a relatively safe system and reproduce ecological OOTL conditions.

Parallel to the visual task, participants performed the auditory task and had to react to infrequent beeps by pushing “Enter” button as fast as possible with their left hand. This secondary task served as a way to measure attention (see Measures and analysis for the exact measures reported). They were explicitly told that beeps and experience-sampling probes were to be treated as fast as possible, whatever was happening on the obstacle-avoidance task. Beeps were presented every 20–40 s. On average, one out of three beeps was followed by an attentional probe. In total, 32 probes were displayed during the whole session. The distribution of the experience-sampling probes was not correlated with events on the obstacle-avoidance task, to minimize performance influence on experience-sampling reports. We instructed participants not to pay attention to the ASSR background sound.

We used R-Studio 1.1.456, R 3.5.1 (RStudio Team, 2015; R Core Team, 2016) for statistical analysis, and Matlab 2018a (The Mathworks Inc., 1992), EEGLAB (Delorme and Makeig, 2004), and FieldTrip (Oostenveld et al., 2010) to filter and analyze EEG data. All 95% CIs reported hereafter were computed using the boot R package with 10,000 iterations with normal bootstrap approximation (Canty and Ripley, 2017).

All linear mixed-effect analyses used the R lme function to create the models (Bates et al., 2017), with a random intercept for subjects to account for our repeated-measure design. Each time, we visually inspected residual plots to spot any obvious deviations from normality or homoscedasticity. We assessed the influence of predictors by creating a baseline model and then added each predictor in turn; we compared each model with the previous one to verify if adding a predictor significantly reduced uncertainty. The R Anova function was used to compare models by performing likelihood-ratio tests between given models and report the χ² value (R Core Team, 2016). We chose type 2 sum of squares or type 3 sum of squares when there were interactions to consider between predictors. Post hoc tests were conducted using the glht and mes functions on the complete model (R Core Team, 2016).

Participants reported on average 31.3% task-related MW (SD = 4.4%) and 36.6% task-unrelated MW (SD = 5.0%, see Figure 3, Supplementary Data Sheet 1). This rate is consistent with previous studies (Smallwood et al., 2006; Smallwood and Schooler, 2015; Gouraud et al., 2018a,b). Each participant reported on average 1.5% “External distraction” reports (SD = 1.21). Considering this low rate, we discarded “External distraction” reports and adopted the ternary approximation of attentional states (i.e., either focused, task-related MW, or task-unrelated MW). All participants answered all 32 probes, except one participant who did not answer four probes.

Blocks did not significantly influence task-related MW. On the contrary, blocks significantly influenced task-unrelated MW rates, χ² = 12.13, p = 0.007. Post-hoc tests revealed that task-unrelated MW rate were significantly higher under the second block compared with the first and third blocks, p = 0.021, d = 0.55, p = 0.010, d = 0.62, respectively. All results from model comparisons are gathered in Table 1, bold values being significant.

The auditory task performance was investigated using reaction time when presented a beep followed by a probe. Participants reacted to on average 31.3 beeps out of the 32 presented. Attentional states did not influence reaction time. On the contrary, there was a significant influence of blocks on reaction time, χ² = 25.52, p < 0.001. Post-hoc tests revealed that participants were significantly slower during the fourth block compared with the first and third blocks, respectively (p = 0.007, d = 0.48 and p = 0.016, d = 0.28). All results from model comparisons are gathered in Table 2 and illustrated in Figure 4.

The amplitude evolution of ERPs elicited by the auditory task (beeps) was investigated. Attentional states significantly influenced both N1 and P3 components (see Table 3 and Figure 5). Post-hoc tests revealed that for the N1 component, reports of task-unrelated MW were accompanied with a lower amplitude (M = −6.06 μV, 95% CI = [−8.01; −4.12] μV) compared with periods of focus (M = −9.39 μV, 95% CI = [−12.21; −6.60] μV), p = 0.024, d = 0.36. For the P3 component, the statistics showed a significantly higher amplitude for task-related MW (M = 12.69 μV, 95% CI = [9.28; 16.13] μV) compared with focus periods (M = 8.20 μV, 95% CI = [5.54; 10.85] μV), p = 0.009, d = 0.16.

Alpha wave power evolution before experience-sampling probes was investigated. Results showed a significant influence of attentional states on alpha amplitude (see Figure 6 and Table 4, bold values being significant), χ² = 8.35, p = 0.015. Post-hoc tests showed significantly higher alpha amplitude during task-unrelated MW (M = 53.83 μV²/Hz, 95% CI = [52.35; 55.31] μV²/Hz) compared with focus episodes (M = 53.03 μV²/Hz, 95% CI = [51.90; 54.16] μV²/Hz), p = 0.014, d = 0.27. Other comparisons (task-related MW vs. focus, task-related MW vs. task-unrelated MW) were not significant.

No influence of attentional states on ASSR amplitude was uncovered (Figure 7). However, spectral plots still revealed a peak at 40 Hz, showing that the ASSR was visible on participants' spectrum even during this complex task (see Figures 8, 9). Should anyone want to reuse this background noise for other ASSR activities within aeronautical-inspired environments, we mention that 12 participants out of 18 reported that they felt the noise was similar to a propeller airplane.

Taken together, the observed effects support a reduction in cortical processing of the external environment during task-unrelated MW. First, for the auditory task, N1 component elicited by the beeps had a lower amplitude during task-unrelated MW, indicating a state of reduced perception of stimuli already identified by Kam et al. (2011). Participants who experienced task-unrelated MW were less receptive to the beeps. Nevertheless, only a non-significant trend could be observed in reaction times (Figure 4), with subjects being faster during the fourth block for task-unrelated MW compared with other attentional states. Subjects may have focused, maybe even attention-tunneling, on the visual task when being focused or in task-related MW. On the contrary, being in task-unrelated MW may have led participants to use strategies favoring speed over precision, without significant impact on the accuracy due to the low difficulty of the task (Salomone et al., 2021).

Second, regarding the visual task, the increase in alpha power in the parieto-occipital lobe shows that participants inhibited visual perception during MW episodes (Foxe and Snyder, 2011; Benedek et al., 2014; Clayton et al., 2015). Although the debate still exists on alpha power, both analyses are congruent and consistent with research sharing the same features, i.e., probe-caught MW (Baird et al., 2014), visual (Compton et al., 2019), and ecological task (Baldwin et al., 2017). MW creates a decoupling from the task at hand, even in complex bimodal environments. Our results are a first step toward filling the gap between real consequences of MW (Galera et al., 2012; Berthié et al., 2015) and EEG research in laboratory settings (Kam, 2010; Kam et al., 2019). Taken together, visual and auditory analyses support the multimodal influence of MW in complex environments (Kam et al., 2011), although our setup does not allow us to make quantified claims and compare modalities.

We observed no effect of attention on ASSR amplitude, even though its evoked power was visible on the EEG spectrum of the participants. This outcome is in line with the results of O'Connell et al. (2009) regarding the absence of amplitude modulation of MW on SSR. It is possible that our experiment did not succeed because of its features, such as the use of amplitude modulation instead of clicks (Voicikas et al., 2016) or the insufficient number of participants. Another possibility may be that SSR produced by non-target background noise is already being reduced by participants instructed to ignore it from the start; it may therefore not be further influenced by MW. However, this hypothesis is in contradiction with both literature on ASSR in attention modulation settings (Skosnik et al., 2007; Müller et al., 2009; Mahajan et al., 2014) and our own results regarding lower N1 amplitude during task-unrelated MW. To account for this observation, a final explanation may be that internally directed attention like MW is fundamentally different from the evolution of external direction between sensory modalities. In this case, the absence of amplitude modulation would show that MW does not impact the earliest stages of perception, allowing for a basic processing of external stimuli. Further work in this area is needed to provide robust conclusions.

Important differences were highlighted between task-related and task-unrelated MW, supporting the existence of “depth” or “intensity” (related to the decoupling) in MW episodes. During task-unrelated MW, participants inhibited perception of auditory stimuli (as shown by the N1 amplitude), but not during task-related MW compared with focus moments. On the contrary, auditory information processing (P3 amplitude) was higher during task-related MW than during focus intervals. Participants reporting being focused may actually focus on the visual part of the task (the most cognitively demanding) while inhibiting all auditory stimuli, whether relevant to the task or not. On the other hand, task-related MW may create a more superficial decoupling than task-unrelated MW. This mental state may redirect attentional resources from the exhausting visual task to listening to auditory cues, thus participating in a more balanced resource allocation independently of task demand. Unfortunately, we did not observe differences in performance, i.e., reaction time during the auditory task. It is likely that because the processing of auditory stimuli did not require much cognitive resources, superficial perception was enough to perform it.

Previous explanation remains very conditional, as the available observations are not sufficient to definitely establish the depth of MW. A graded MW with a different decoupling could explain why we are most of the time able to perform tasks while being in MW, while sometimes we make clear errors that could have been avoided with our full attention (Cheyne et al., 2006; Carriere et al., 2008; Farley et al., 2013). Two protocols may complete the present study in relation to MW depth: using the same experiment, but asking the participant to ignore the beeps; the irrelevance of beeps may thwart interesting results when analyzing the influence of task-related MW on ERPs. Another possibility would be to use the same experiment once again, but this time participants would have two different beeps to react to, each associated with a different button. The needs for more processing of auditory stimuli could link the performance data to MW decoupling depth. Nevertheless, more data are needed to rule over the depth dimension.