Event Abstract

Back to Event

fNIRS Feature Importance for Attentional State Prediction

Angela R. Harrivel1*, Robert Milletich2, Chad L. Stephens1, Christina Heinich1, Nicholas Napoli3, Mary Carolyn Last4 and Alan Pope1
  • 1 Langley Research Center, United States
  • 2 Booz Allen Hamilton (United States), United States
  • 3 University of Virginia, United States
  • 4 Analytical Mechanics Associates, United States

Introduction: Cognitive state prediction is attractive for the detection of human performance decrement and the mitigation of errors during safety-critical operational activities. Such methods are applicable in ground-based verification studies, and to support countermeasures during training or operational environments such as commercial piloting, air traffic control, or astronaut extra-vehicular activity. In particular, we aim to improve the discrimination of optimal from non-optimal states of attention by fusing data from multiple passive physiological sensors, down-selecting features and tuning machine learning algorithms. Prior work has explored the use of electroencephalography, functional Near Infrared Spectroscopy (fNIRS), galvanic skin response, eye tracking, heart rate, and respiration, but has not fully investigated classifier state prediction accuracy for various types of features extracted from fNIRS signals. Here we perform cognitive state prediction with, and assess the importance of, fNIRS features based on functionally-connected brain networks to inform future real-time cognitive state prediction methods. Methods: Six commercial aviation pilots were asked to perform “benchmark” tasks and motion-based flight simulation while wearing fNIRS and other physiological sensors. All participants consented to the study as approved by the Institutional Review Board of the NASA Langley Research Center. Here we report on the analysis of fNIRS benchmark task data to assess the usefulness of various fNIRS-derived features to discriminate cognitive states. Results of analysis of other physiological data recorded during benchmark tasks and flight simulation were reported elsewhere (Harrivel et al., 2017; Stephens et al., 2017). Use of the benchmark tasks was modeled after Hirshfield, et al. (2009). Selected tasks were used to induce five cognitive states for 6 minutes each: high workload, low workload, startle, channelized attention, and diverted attention (Harrivel et al., 2016a). Two fiber optic sensor arrays were used to record fNIRS signals at 6.25 Hz (Imagent, ISS, Inc.): one with 2 brain-sensing optodes placed over the Medial Frontal Gyrus (MFG, a node of the default mode or task-negative network), and another with 5 brain-sensing optodes placed over the right Dorsolateral Pre-Frontal Cortex (DLPFC, a node of an attentional or task-positive network). Additionally, a shallow physiology-sensing optode was employed on each sensor array. Time traces of relative hemoglobin concentration changes for both oxygenated and deoxygenated species were calculated for each channel. These were band-pass filtered to include 0.003 Hz to 0.1Hz to capture both functional hemodynamic responses, and low frequency spontaneous activity in functionally-connected default mode and attentional networks. Signals measured by the physiology-sensing optodes were filtered to include higher frequencies (0.008 Hz to 3 Hz) to capture a wide range of cardiac and respiratory fluctuations potentially to inform the classifiers. The “as-measured” traces were used as previously described (Harrivel et al., 2016b), without a-priori knowledge of task performance for processing, and without cleaning of physiological noise. Various feature types were extracted from the time series to assess their usefulness for improving classification accuracy. Features were based on spectral information (POWER), statistical summaries (SS), engagement indices (ENG), and correlation (COR). Spectral features included power up to 0.5Hz and the average, variance, and max power within this band. Statistical summaries included moments, coefficient of variation, percentiles, and multivariate and permutation entropy. “Engagement” indices were created using ratios or differences between the MFG and DLPFC traces, quantifying external task engagement using task-positive vs. task-negative network activity (Harrivel et al., 2013; Fox et al., 2005). Correlation-based features include Pearson correlations and eigenvalues of the correlation matrices designed to quantify more complicated network activations as potential indications of mind wandering (Smallwood et al., 2008; Christoff et al., 2009; Durantin et al., 2015). Data were partitioned into training and holdout sets using a 50/50 split accounting for the temporal nature of the data and class imbalance. Feature selection was performed on the training set to find an optimal subset of features using two models: (1) Logistic regression with L1 penalty, and (2) random forests. For logistic regression, one-vs-all models were fit for a total of five models and five sets of coefficients. An embedded feature selection technique was performed where features with non-zero coefficients were selected. For random forests, features with mean decrease importance scores (using the Gini Index) greater than the median importance were selected. For both classifiers, a recursive feature elimination (RFE) wrapper method was used to recursively identify the top 50% of features. Finally, an ensemble feature selector was also implemented by combining greater-than-median features for each model. Each feature subset was evaluated on the holdout data using a 4-fold stratified, temporal cross-validation. For comparison purposes, classifiers were also tested using all features. The evaluation metric was the weighted area under the curve, and weights were adjusted account for class imbalance. Results: Generally, the best results were obtained by performing feature selection prior to model evaluation. The embedded L1 feature selection method resulted in similar performance as the RFE method (6-pilot-average AUC of 0.711 vs 0.713) but used approximately one-fifth of the original feature set, whereas the RFE algorithm using half of the original feature set. For random forests, the best results were obtained using the ensemble selection method. The highest 6-pilot-average AUC was 0.784 +/- 0.110, which resulted from employing the random forest classifier with the ensemble-derived feature subset. The top selected features across all pilots were the spectral power-based features in four of the six pilots (Figure 1). Moreover, across all feature types and all pilots, the oxy-derived features were selected more often than deoxy-derived features. Conclusion: The results of the present study warrant the prioritization of SS, ENG and POWER features in future analyses. Importantly, the power features also include spectral respiratory information, perhaps partly explaining their superior ability to increase AUC. Future work includes state prediction using data collected during the flight simulation. Further, fNIRS will be combined with other physiological sensors as previously reported (Harrivel et al., 2017; Stephens et al., 2017). Real-time operator cognitive state information may be relayed to flight instructors during training, or to semi-autonomous systems to optimize human-automation interaction and teaming.

Figure 1
Image

Acknowledgements

We thank our pilot participants, the Data Science Team and the Simulation and Data Analysis Branch of the NASA Langley Research Center, and fellow NASA Airspace Operations and Safety Program colleagues in the Technologies for Airplane State Awareness sub-project who supported data collection during this study.

References

Christoff, K., Gordon, A. M., Smallwood, J., Smith, R., and Schooler, J. W. (2009). Experience sampling during FMRI reveals default network and executive system contributions to mind wandering. Proc. Natl. Acad. Sci. USA, 106, 8719-8724. http://doi: 10.1073/pnas.0900234106

Durantin, G., Dehais, F. and Delorme, A. (2015). Characterization of mind wandering using fNIRS. Front. Syst. Neurosci. 9, 45. http://doi: 10.3389/fnsys.2015.00045

Fox, M. D. Snyder, A. Z. Vincent, J. L. Corbetta, M. Van Essen, D. C. Raichle, M. E. (2005). The human brain is intrinsically organized into dynamic, anticorrelated functional networks. Proc. Natl. Acad. Sci. USA, 102(27), 9673–9678.

Harrivel, A., Stephens, C., Milletich, R., Heinich, C., Last, M.C., Napoli, N., Abraham, N., Lance Prinzel, L., Motter, M., Pope, A. (2017). Prediction of Cognitive States during Flight Simulation using Multimodal Psychophysiological Sensing. AIAA Science and Technology Forum and Exposition. Grapevine, TX.

Harrivel, A., Liles, C., Stephens, C., Ellis, K., Prinzel, L., Pope, A. (2016a). Psychophysiological Sensing and State Classification for Attention Management in Commercial Aviation. AIAA Science and Technology Forum and Exposition. San Diego, CA.

Harrivel, A., Weissman, D., Noll, D., Huppert T., Peltier, S. (2016b). Dynamic filtering improves attentional state prediction with fNIRS. Biomedical Optics Express, 7(3), 979-1002. http://doi: 10.1364/BOE.7.000979

Harrivel, A., Weissman, D., Noll, D., Peltier, S. (2013). Monitoring attentional state with fNIRS. Frontiers in Human Neuroscience, 7, 861. http://doi: 10.3389/fnhum.2013.00861

Hirshfield, L. M., Solovey, E. T., Girouard, A., Kebinger, J., Jacob, R. J., Sassaroli, A. (2009). Brain Measurement for Usability Testing and Adaptive Interfaces: An Example of Uncovering Syntactic Workload with Functional Near Infrared Spectroscopy. Computer-Human Interaction. Boston, MA.

Smallwood, J., Beach, E., Schooler, J.W., Handy, T.C. (2008). Going AWOL in the Brain: Mind Wandering Reduces Cortical Analysis of External Events. The Journal of Cognitive Neuroscience, 20(3), 458–469.

Stephens, C., Prinzel, L., Harrivel, A., Comstock, J., Abraham, N., Pope, A., Wilkerson, J., Kiggins, D. (2017). Crew State Monitoring and Line-Oriented Flight Training for Attention Management. International Symposium on Aviation Psychology. Dayton, OH.

Keywords: fNIRS, machine learning, Crew monitoring, human performance, Psychological factors, Behavioral Sciences

Conference: 2nd International Neuroergonomics Conference, Philadelphia, PA, United States, 27 Jun - 29 Jun, 2018.

Presentation Type: Oral Presentation

Topic: Neuroergonomics

Citation: Harrivel AR, Milletich R, Stephens CL, Heinich C, Napoli N, Last M and Pope A (2019). fNIRS Feature Importance for Attentional State Prediction. Conference Abstract: 2nd International Neuroergonomics Conference. doi: 10.3389/conf.fnhum.2018.227.00067

Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters.

The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated.

Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed.

For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions.

Received: 07 Mar 2018; Published Online: 27 Sep 2019.

* Correspondence: Dr. Angela R Harrivel, Langley Research Center, Hampton, Virginia, 23681, United States, angela.r.harrivel@nasa.gov