AUTHOR=McKendrick Ryan , Feest Bradley , Harwood Amanda , Falcone Brian 

TITLE=Theories and Methods for Labeling Cognitive Workload: Classification and Transfer Learning

JOURNAL=Frontiers in Human Neuroscience

VOLUME=Volume 13 - 2019

YEAR=2019

URL=https://www.frontiersin.org/journals/human-neuroscience/articles/10.3389/fnhum.2019.00295

DOI=10.3389/fnhum.2019.00295

ISSN=1662-5161

ABSTRACT=There are a number of key data-centric questions that must be answered when developing classifiers for operator functional states. “Should a supervised or unsupervised learning approach be used? What degree of labelling and transformation must be performed on the data? What are the trade-off between algorithm flexibility and model interpretability, as generally these features are at odds?” Here we focus exclusively on the labelling of cognitive load data for supervised learning. 
We explored three methods of labelling cognitive states for three-state classification. The first method labels states derived from a tertiary split of the number of items an individual had to hold on in mind on each trial of a spatial memory task. The second method was more adaptive, it employed mixed effects stress-strain curve and estimated an individual’s performance asymptotes with respect to the same spatial task. The final method was similar to the second approach, yet it employed a mixed effects Rasch model to estimate individual capacity limits context of item response theory for the memory task. 
To assess the strength of each of these labelling approaches we compared area under the curve (AUC) for receiver operating curves (ROC) as well the AUC of precision-recall ROCs (PR-ROC) from elastic net and random forest classifiers. We chose these classifiers based on a combination of  interpretability, flexibility and past modeling success. We tested these techniques across two groups of individuals and two tasks to test the effects of different labelling techniques on cross-person and cross-task transfer. Overall, we observed that Rasch models paired with random forests lead to the best model fits, and showed evidence of cross-person and cross-task transfer.