Event Abstract

Back to Event

Towards a Multimodal Model of Cognitive Workload through Synchronous Optical Brain Imaging and Eye Tracking Measures

Erdinc Isbilir1*, Murat P. Cakir2, Cengiz Acarturk2 and Simsek Tekerek1
  • 1 Roketsan Missiles Industries Inc., Türkiye
  • 2 Middle East Technical University, Cognitive Science, Türkiye

Introduction Recent advances in neuroimaging and eye tracking technologies have enabled the simultaneous monitoring of brain activity in unison with eye movements through wearable sensors in the field. Neuroergonomics emerged as a research discipline that emphasizes the use of such methodologies in the field with real operators during real tasks to attain improved ecological validity. In this study we focus on the relationship between hemodynamic changes observed in the prefrontal cortex and the eye movements of operators in a real-world setting to explore the neural and ocular correlates of cognitive workload in the context of a simulated human-machine interaction task. Materials and Methods An experiment was conducted with 15 operators 7 of whom were experts and 8 were novices in terms of their familiarity and experience with the human-machine interface of a platform. The participants were asked to follow the regular operation sequence including data entry, system checks and task execution in normal and adverse conditions while their prefrontal cortices and eye movements were monitored with a portable functional near-infrared spectroscopy (fNIRS) system and a mobile eye tracker, respectively. During the adverse condition the operators were prompted with a system failure message at a predefined state and were asked to provide an account regarding the underlying causes of the problem. The goal of the study is to observe in what ways experts and novices differ in terms of their hemodynamic responses and eye movement patterns during regular and adverse task conditions. The operators’ brain activations at the prefrontal cortex were measured with an fNIRS system that is composed of a flexible headpiece, which holds 4 light sources and 10 detectors to obtain oxygenation measures at 16 optodes on the prefrontal cortex; a control box for hardware management; and a computer that runs COBI Studio software (Ayaz et al., 2011) for data acquisition. The sensor has a source-detector separation of 2.5cm, which allows for approximately 1.25cm penetration depth. This system can monitor changes in relative concentrations of oxy-hemoglobin (HbO) and deoxy-hemoglobin (HbR) at a temporal resolution of 2Hz. The 16 optodes correspond to regions in the Broadmann areas 9, 10, 44 and 45 (Figure 1). fNIRS is a neuroimaging modality that enables continuous, noninvasive, and portable monitoring of changes in blood oxygenation and blood volume related to human brain function (Ayaz et al., 2013). Over the last decade, studies in the laboratory have established that fNIRS provides a veridical measure of oxygenation and blood flow in the brain. fNIRS is not only non-invasive, safe, affordable and portable, it also provides a balance between temporal and spatial resolution which makes fNIRS a viable option for in-the field neuroimaging. fNIRS technology uses infrared light within 700 to 900nm, introduced at the scalp, to enable the non-invasive measurement of changes in the relative ratios of HbR and HbO in the capillary beds during brain activity. Near-infrared light is known to diffuse through the intact scalp and skull, which makes it suitable for tracing relative changes in the concentration of specific chromophores in the neural tissue with non-invasive, spectroscopic methods (Jobsis, 1977). Whereas most biological tissue (including water) are relatively transparent to light in the near infrared range between 700 to 900nm, hemoglobin is a strong absorber of light waves in this range of the spectrum. This provides an optical window into neural tissue where one can approximate relative changes in HbR and HbO concentrations based on how IR light is attenuated in neural tissue (Figure 2). The participants’ eye movements were recorded by a wearable eye-tracking device in the form of glasses that has 120Hz binocular eye cameras that provides IR illumination to track the operators’ pupils and a 120Hz high speed world camera to record the scene that the operator sees. Results Our preliminary findings indicate that experts carry out the task sequence more smoothly and faster than novices, which correspond to lower prefrontal oxygenation levels and more structured scan paths over relevant parts of the interface. Novices’ scan paths during normal operation evidence visual search behavior, possibly due to their unfamiliarity with the interface elements and task sequences, which result in more cognitive processing and a higher oxygenation trend. In the adverse condition the moment of failure tended to trigger an increase in prefrontal oxygenation, together with long fixations on the error message and its proximity. Experts’ hemodynamic responses for the error prompts were higher than novices, possibly due to their higher level of engagement with the task. By the time of the conference we plan to finalize the analysis of fNIRS and eye tracking data to devise a statistical model of the observed patterns in an effort to arrive at a multimodal account of cognitive workload.

Figure 1
Image
Figure 2
Image

Acknowledgements

The researchers acknowledge Roketsan Missiles Industries Inc. for the facilitation of this study.

References

Ayaz, H., Shewokis, P.A., Curtin, A., Izzetoglu, M., Izzetoglu, K., & Onaral, B. (2011). Using MazeSuite and functional near infrared spectroscopy to study learning in spatial navigation. Journal of Visualized Experiments, 56.
Ayaz, H., Onaral, B., Izzetoglu, K., Shewokis, P. A., McKendrick, R., & Parasuraman, R. (2013). Continuous monitoring of brain dynamics with functional near infrared spectroscopy as a tool for neuroergonomic research: Empirical examples and a technological development. Frontiers in Human Neuroscience, 7, 1-13.
Izzetoglu, K., Ayaz, H., Hing, J. T., Shewokis, P. A., Bunce, S. C., Oh, P., & Onaral, B. (2015). UAV operators workload assessment by optical brain imaging technology (fNIR). In Handbook of Unmanned Aerial Vehicles (pp. 2475-2500). Springer Netherlands.
Jobsis, F. F. (1977). Noninvasive, infrared monitoring of cerebral and myocardial oxygen sufficiency and circulatory parameters. Science, 198(4323), 1264-1267.

Keywords: neuroergonomics, fNIRS, eye tracking, human factors, cognitive workload, human-machine interaction

Conference: 2nd International Neuroergonomics Conference, Philadelphia, PA, United States, 27 Jun - 29 Jun, 2018.

Presentation Type: Oral Presentation

Topic: Neuroergonomics

Citation: Isbilir E, Cakir MP, Acarturk C and Tekerek S (2019). Towards a Multimodal Model of Cognitive Workload through Synchronous Optical Brain Imaging and Eye Tracking Measures. Conference Abstract: 2nd International Neuroergonomics Conference. doi: 10.3389/conf.fnhum.2018.227.00089

Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters.

The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated.

Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed.

For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions.

Received: 09 Apr 2018; Published Online: 27 Sep 2019.

* Correspondence: Dr. Erdinc Isbilir, Roketsan Missiles Industries Inc., Ankara, Türkiye, erdinc.isbilir@roketsan.com.tr