Improvements to ship pilots’ situation awareness (SA) in maritime navigation are critical to reducing human errors, which have caused 75–96% of marine accidents over the last few years (Hetherington et al., 2006; Mohammadfam et al., 2019). In recent years, growth in traffic densities, ship speeds, and ship sizes have led to the need to improve pilots’ operational safety (Weng and Yang, 2015). Due to the complexity of marine systems and the increased use of automation and intelligence, it is becoming more difficult for pilots to fully percept and understand the current environmental state and predict changes in near future (Wild, 2011; Stirling et al., 2019). To prevent pilots’ unsafe behaviors in a more effective and practical manner, it is indispensable to identify at-risk cognitive competency combined with emergency situations in pilotage (Mosier et al., 2013). However, the measurement gap is increased by the requirement of pilots, as the task executors, to maintain high SA levels, as well as the individual differences and experiential properties of pilotage (Darbra et al., 2007). Therefore, identifying at-risk cognitive competency (i.e., low SA levels) as a means of preventing human errors is complicated, and is worthy of further investigation in operational situations.

At present, the collision of poor visibility situations continues to account for a major proportion of all accidents resulting in casualties, and studies are focused on determining their relevance through statistical methods (Antao and Soares, 2019). Chauvin et al. (2013) verified that collision accidents were significantly affected by visibility, which is a variable in 56.51% of environmental factors in the human factors analysis and classification system (HFACS). Bye and Aalberg (2018) supported this finding by calculating the ability of different visibility conditions to explain accidents, and concluded that the most significant effect occurs when the visibility drops to less than 0.25 nautical miles. Existing research has effectively evaluated whether poor visibility conditions are more likely to cause accidents than other environmental factors (e.g., water depth, flow rate, etc.), but has not assessed changes in the cognitive state of the pilots in preparation for potential hazards in poor visibility (Endsley, 2015). In this study, the poor visibility was selected as a typical situation, in which the identification of pilots’ low SA levels is necessary. According to the literature, the SA three-level hierarchical structure is one of the most popular conceptual cognitive frameworks, which frames SA into three stages: perception, comprehension, and prediction (Lo et al., 2016). Therefore, the first problem to solve is how to accurately identify pilots’ low SA level with the three-level cognitive frameworks, taking poor visibility scenes as an example (Zhang et al., 2020).

According to the theoretical interpretation of the SA three-level model, eye movements and EEG parameters among physiological indicators are the most effective reflection of an individual’s intrinsic state from the cognitive processes of perception, understanding and prediction, providing the possibility to essentially prevent unsafe behaviors by detecting SA levels dynamically (Zhang et al., 2020; Pei et al., 2022). The correlation between pilots’ eye-tracking metrics and SA levels has been validated in a previous study (Jiang et al., 2021). However, eye-tracking metrics are only acquired through tangible visual behaviors, and there are no consistent conclusions on whether such visual behaviors truly meet the perceptual requirements in the SA model, or whether the information gazed at is effectively understood by the pilot and the actual behavior is taken (di Flumeri et al., 2018). Therefore, it is necessary to conduct research based on the application of EEG acquisition technology to further assist in quantifying the cognitive processes of unsafe behaviors, and the use of pilots’ EEG features for SA identification is the first problem that needs to be addressed.

To ensure consistency in the study of EEG signals in cross-domain applications, the brain is usually divided into different electrode sites based on five regions [frontal (F), central (C), parietal (P), temporal (T), and occipital (O)] (Aggarwal and Chugh, 2022). EEG is an active map of the brain displayed by converting EEG data to the time-frequency by appropriate methods such as Fourier transform (FT) (Srinivasan et al., 2019). In time-frequency analysis, information about the cognitive state of an individual is usually obtained based on the data in different frequency bands, commonly including delta (δ) (1–4 Hz), theta (θ) (5–8 Hz), alpha (α) (9–14 Hz), beta (β) (15–30 Hz), and gamma (γ) (31–59 Hz) (Pan et al., 2021). Therefore, it has become a common trend to analyze frequency features for the performance evaluation in EEG-based studies (Gutiérrez and Ramírez-Moreno, 2016). Puma et al. (2018) explored the effect of task difficulty on the θ and α bands by varying the number of sub-tasks and showed that the higher the level of the operator’s behavioral performance, the lower the power spectral density (PSD) of the θ and α bands. Dimitriadis et al. (2010) found that the δ band was correlated with the difficulty of the operator’s perceptual task, i.e., the PSD of the δ band increased when the operator perceived the experimental task as too easy. However, previous studies not fully utilizing EEG time-frequency features for cognitive studies.

No common agreement has been made for the EEG features correlated with SA as EEG acquisition techniques were applied in different tasks and workplaces. Regarding the relationship between time-frequency features and SA, there have been attempts to validate the association (Klaproth et al., 2020; Iqbal et al., 2021). Kästle et al. (2021) identified the correlations between β and γ in parietal and temporal regions and SA levels during a control task. Notably, only a few studies (Saini et al., 2020) have adopted multiple EEG spatial-frequency features, but there is no consensus on which features are associated with SA due to different task conditions and application purposes, let alone the identification of pilots’ SA levels based on EEG association metrics.

Therefore, the aims in this paper are twofold: First, the primary goal is to assess the relationships between EEG time-frequency features and SA levels. Correlation analysis indicated that α/β, θ/(α + θ), and (α + θ)/β frequency bands are significantly correlated with SA levels. The ultimate intention of this work is to explore the SA identification method based on associated EEG metrics, which is expected to reduce piloting risks through improved pilot selection and training.

The research framework was constructed to identify pilots’ SA levels using EEG features after the EEG acquisition experiment in bridge simulator was implemented (Figure 1). Firstly, it was assumed that EEG features and SA were significantly correlated in specific scenes (including poor visibility situation). Consequently, based on SART questionnaire and EEG acquisition technology, the different SA level groups and EEG metrics were separately defined as independent and dependent variables. To ensure the professionalism of questionnaires and measurement accuracy, the SART questionnaires were confirmed through safety engineering and management, maritime supervision, and senior pilot experts to adjust existing measurement items (Jiang et al., 2021).

Secondly, EEG acquisition technology was used to capture pilots’ brain waves, including the θ/β, α/β, θ/(α + θ), (α + θ)/β, and (α + θ)/(α + β) in different brain regions (e.g., F and C), which were used to indirectly infer SA-related constructs (e.g., understanding and prediction). And then the visualization methods (i.e., brain regions activity and time-frequency analysis) are illustrated to preliminarily understand the common EEG patterns and differences between the various SA groups. Although visualization techniques allow analyzing EEG data in an explorative way, a statistical analysis must be performed to determine differences in spatial-frequency analysis as caused by variations in SA levels. Quantitatively, EEG frequency metrics are calculated for pilots in each SA group (high and low) within different regions. The average power-based values of EEG metrics among pilots with high and low SA levels were input into the permutation simulation to compare their associations. After digital and time-domain filtering, the FT-based EEG frequency features were divided into training and testing sets for input to the RF-CNN (Random forest-convolutional neural network) method, which provides the new avenue of classifying the SA for pilots’ screening.

For the purpose of constructing the SA identification model using EEG frequency metrics in different brain regions, groups with different SA levels were first established. To facilitate the analysis, the research hypotheses divide pilots’ SA into two levels based on their SART score: high (above average SART score) and low (below average SART score). According to the SART scores (mean = 20.13, standard deviation = 5.83), the pilots were divided into a high-SA group containing 13 participants (mean = 24.5, standard deviation = 5.13) and a low-SA group containing 12 participants (mean = 15.2, standard deviation = 4.37).

The results highlight that the RF-CNN method achieves good SA identification using six principal components of EEG features (Table 5). The EEG data of 25 pilots were collected during the whole experimental process based on pilots’ competency in poor visibility situations. Moreover, the poor visibility was designed to appear randomly in normal navigation during the simulation experiment. With respect to the fundamental analysis of EEG data, a permutation simulation was used to quantify the significance of the correlation between EEG cognitive state, as visualized in the form of brain activity and time-frequency analysis, and SA levels, as measured by SART. As the window width of short-time Fourier transform (STFT) is fixed and cannot be adaptively adjusted, it is usually applicable to the analysis of stable signals. WT is then proposed to observe the partial features of the signal through time windows of adaptive widths varying with frequency, but is often considered as a single power signal in the spatial dimension. Therefore, combining the time-frequency and spatial features of EEG signals, the frequency combination features of different brain regions in the time dimension are extracted for SA identification. The EEG frequency features were then extracted from the associated metrics and divided into overlapping 5 s epochs after digital and time-domain filtering. To prevent possible data overfitting and information overlap, the initial number of input features was determined to be 10 based on the minimum RMSE and RE values of the RF model, and the PCA algorithm was used for further EEG feature combination to obtain six major components as the final input set of the CNN model. Moreover, PCA was applied to optimize the network structure of the CNN model to improve the training convergence efficiency and accuracy, and the feasibility of the above methods has been verified by comparing the results with traditional methods (see 3.3 for details). The RF-CNN method was then used to obtain the identification accuracy of the at-risk cognitive competency (i.e., low SA level). Comparative analysis of the three-performance metrics shows that the proposed method outperforms two other traditional methods that do not use feature optimization.

The results of this study confirm that the at-risk cognitive state can be effectively detected with high accuracy, sensitivity, and specificity using EEG frequency features combined with an appropriate identification framework. The identification results obtained by the RF-CNN, RF, and CNN algorithms with different types of features are shown in Figure 8. We can observe that a combination of the top-nine most important features facilitates an improvement in classification accuracy, from 78.1% using the RF without feature optimization to 84.8% using RF-CNN. Comparing the RF and CNN algorithms, which used the same feature data as their inputs, the latter produced better ACC, TPR, and TNR values, which verifies the applicability of the CNN for nonlinear multidimensional sample sets of EEG data in poor visibility situations. Therefore, we can conclude that an effective RF-CNN method for identifying at-risk cognitive state should include a strong classifier after PCA optimization (i.e., CNN modified module) and input data that have been refined by the combination of salient EEG frequency features in different brain regions (i.e., RF module).

An accurate comparison with the results from other studies in which EEG was used is difficult, because each study used a different method of simulation, a different set of EEG features, and a different classification of SA groups. The main limitation of the current study involves the performance issue of EEG acquisition devices. When the pilot is in a simulation experiment, the EEG devices are susceptible to environmental factors and physiological artifacts that generate large amounts of clutter. Although digital and time-domain filtering helps with feature extraction, it inevitably affects the significance level of the correlation results. Moreover, regarding feature extraction, the identification accuracy is associated with the correlation metrics and epoch lengths of EEG data in poor visibility situations. This study shows that pilots with low SA levels are susceptible to environmental risk factors resulting in significant EEG fluctuations, as expressed by a total of 14 EEG frequency combination features in F and C regions with an epoch length of 5 s. However, the SA level is not only correlated with EEG time-frequency features but is also related to other physiological measurement metrics, as confirmed in previous studies (Mehta et al., 2018; Zahabi et al., 2021). Therefore, the identification processes of at-risk cognitive states are complicated, and require further investigation considering multiple fusion metrics such as heart rate variability (HRV), electrocardiograph (ECG) and eye-tracking signals with different epoch lengths, as well as multiple classifications of SA groups (Paulus and Remijn, 2021; Xiong et al., 2021; Ren et al., 2022).

The results using the proposed RF-CNN method confirm that it is feasible to identify pilots’ at-risk cognitive competency (i.e., low SA levels) using EEG features. Specifically, the EEG data of 25 ship pilots were obtained from bridge simulation experiments in poor visibility situations for the training of the identification model, containing RF, modified CNN and validation modules. Six EEG principal components were produced by the PCA after RMSE and RF correction to obtain the optimal model training sets, with the cumulative contribution rate of more than 87.7%. The experimental results demonstrate that the proposed feature combinations enhance the classification performance over that of RF and CNN, which do not employ features optimization, demonstrating the potential for our approach to be used in the computer-aided screening of pilots’ cognitive competency. Considering the universality of the proposed method, future research will focus on the verification and application of the identification model in different emergency situations (e.g., ship departure, two-ship crossing, and anchoring) with multiple fusion metrics (e.g., HRV, eye-tracking). Therefore, this study yields not only immediate benefits in monitoring cognitive competency and preventing unsafe behaviors, but also long-term benefits in opening new avenues for the construction of evaluation systems for the physical and mental competency of ship pilots.

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The patients/participants provided their written informed consent to participate in this study.

All authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.

This work was supported by the National Natural Science Foundation of China (Grant No. 71503166) and the Jinhua municipal public welfare technology application research project (Grant No. 2022-4-014).

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.