Original Research ARTICLE
Theories and Methods for Labelling Cognitive Workload: Classification and Transfer Learning
- 1Northrop Grumman (United States), United States
There are a number of key data-centric questions that must be answered when developing classifiers for operator functional states. “Should a supervised or unsupervised learning approach be used? What degree of labelling and transformation must be performed on the data? What are the trade-off between algorithm flexibility and model interpretability, as generally these features are at odds?” Here we focus exclusively on the labelling of cognitive load data for supervised learning.
We explored three methods of labelling cognitive states for three-state classification. The first method labels states derived from a tertiary split of the number of items an individual had to hold on in mind on each trial of a spatial memory task. The second method was more adaptive, it employed mixed effects stress-strain curve and estimated an individual’s performance asymptotes with respect to the same spatial task. The final method was similar to the second approach, yet it employed a mixed effects Rasch model to estimate individual capacity limits context of item response theory for the memory task.
To assess the strength of each of these labelling approaches we compared area under the curve (AUC) for receiver operating curves (ROC) as well the AUC of precision-recall ROCs (PR-ROC) from elastic net and random forest classifiers. We chose these classifiers based on a combination of interpretability, flexibility and past modeling success. We tested these techniques across two groups of individuals and two tasks to test the effects of different labelling techniques on cross-person and cross-task transfer. Overall, we observed that Rasch models paired with random forests lead to the best model fits, and showed evidence of cross-person and cross-task transfer.
Keywords: Mental Workload, Memory, Prefrontal Cortex, fNIRS (functional near infrared spectroscopy), brain-computer interface, Transfer Learning
Received: 25 Mar 2019;
Accepted: 12 Aug 2019.
Copyright: © 2019 McKendrick, Feest, Harwood and Falcone. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
* Correspondence: Dr. Ryan McKendrick, Northrop Grumman (United States), Falls Church, United States, email@example.com