Assessing neuroelectrical markers of emotional appraisal during the interaction with adaptive user interfaces
-
1
Institut für Arbeitswissenschaft und Technologiemanagement, Universität Stuttgart, Germany
-
2
Fraunhofer-Institut für Arbeitswirtschaft und Organisation (FHG), Germany
Neuro-adaptive interaction paradigms are being discussed as a strategy to make assisitive technologies more user-oriented by recognizing mental states and adapting assistance accordingly (Zander, Krol, Birbaumer, & Gramann, 2016). Still, so far, little empirical evidence exists about the neuroelectrical makers of emotional user appraisal in realistic scenarios that can be used for real-time classification. We recently presented a new experimental paradigm, AFFINDU, for research in the domain of neuro-adaptive technologies (Pollmann, Ziegler, Peissner, & Vukelić, 2017). We simulated an adaptive system that induces positive and negative affective user states by supporting or impeding the completion of a navigation task. AFFINDU roots in a prominent appraisal model of emotions (Sander, Grandjean, & Scherer, 2005) which states that appraisal is a cognitive mechanism that considers a rapid evaluation, personal judgement and implications of events. In the previous study, we included the AFFINDU system with a simple grid-based menu, through which the user had to navigate in a controlled step-by-step sequence. Participants’ affective reactions to adaptive rearrangements of the menu that supported or impeded the navigation task were recorded using electroencephalography (EEG). The event-related potential (ERP) analysis showed that distinct neuroelectrical markers of affective states can be found that are induced by supportive adaptations (SA) and impeding adaptations (IA). The current study investigates whether similar markers of emotional appraisal can be found in a more realistic, ecologically valid scenario. To do so, we included the AFFINDU system in a more complex interaction task: The user could freely navigate through a whole website with different pages instead of navigating step-by-step through the grid-based menu. The system performed SA and IA that changed the graphical UI as well as graphical changes without any consequence for task achievement (neutral adaptations; NA). Similarly to a study by (Peissner & Edlin-White, 2013), for each adaptation the user could provide feedback to the adaptation (keep, revoke, I don’t care).
24 participants (18 to 42 years, 12 females) were recruited for the 75-minute experimental session (Fig.1). A ten-minutes testing period, during which participants could explore the website and navigation, was followed by six measurement blocks (Fig.1a). In each block, they had to perform three search tasks. They were confronted with 90 adaptations (30 SA, 30 IA and 30 NA) in random order. Each task started with an overview of 16 different apps. Participants had to open one specific app and complete the search tasks by navigating through different website pages (e.g. find the information for the weather forecast for a specific day for a specific city in the weather app). Within the app, they had to narrow their information down by navigating through different website pages (e.g. starting with weather app>selecting the country>selecting the city>selecting the day). The website pages were designed with different UI layout patterns, e.g. grid menu, dynamic tabular menu or scroll menu. These layout patterns were altered by the underlying AFFINDU system to produce SA (make it easier to find target, Fig.1c), IA (make it harder to find target, Fig.1d) and NA (irrelevant for finding target, Fig.1e). Each adaptation was followed by a pop-up asking for participants’ feedback (keep adaptation(“Yes”), revoke adaption(“No”), I don’t care(“Never mind”).The interaction was implemented as a controllable event-based procedure including different time windows (Fig.1f) to enable simultaneous recordings of EEG and near-infrared spectroscopy. Here, we only present results for the EEG analysis.
For the EEG analysis we grouped the SA, IA and NA for all blocks together. Stimulus-locked epochs (200 msec before to 1000 msec after the beginning of the adaptation) were created separately for SA, IA and NA. The pre-processing and artefact correction procedure and calculation of signed signed r-square values was done in the same way as in our previous study (Pollmann et al., 2017). For each ERP we conducted a separate multiple dependent sample t-test (SA vs. IA, SA vs. NA and IA vs. NA) on the level of individual electrodes to identify possible spatial differences of adaptation-induced modulations. Here, a cluster-based, non-parametric randomization approach which included correction for multiple comparisons (FieldTrip toolbox; (Oostenveld, Fries, Maris, & Schoffelen, 2011) was performed.
Fig.2 shows participants’ overall subjective feedback to the three different adaptations . The results serve as a manipulation check, clearly showing that the IA were disapproved while the SA were approved by the participants. NA resulted in a more or less equal distribution of yes, no, or never mind answers. By studying the ERP results, we found discriminative separability in the most relevant ERP-time windows for emotional appraisal, such as N200, P300 and the late positive potential (LPP). The results from the signed r-square values revealed the following TOIs to be most discriminative between adaptations: N200=190-250 msec, P300=325–410 msec, LPP=575–800 (Fig. 3). The non-parametric randomization test revealed significant differences for the ERP topographical changes among the three different adapations (Fig. 4). Electrode clusters, showing significant differences are indicated by filled black circles. We found that emotional appraisal, approve SA and disapprove IA, significantly distinguishes clusters from the NA at frontal and parietal sites. Furthermore, electrode clusters at frontal and parietal regions also significantly discriminate positive (SA) and negative (IA) appraisal checks within the N200, P300 and LPP-components. These different modulations of ERPs might index different stages of emotional appraisal including perceptual and semantic encoding for the N200-component, possible attentional processes for the P300-component and a more elaborate conscious evaluation for the LPP-complex (Olofsson, Nordin, Sequeira, & Polich, 2008). To sum up, we showed that positive, negative and neutral emotional appraisal can be distinguished in a realistic and plausible interaction scenario based on ERP-analysis. This helped us to expand our knowledge of the neuroelectrical mechanisms underlying emotional appraisal to automatic UI adaptation principles. Future research can make use of these ERP-components for real-time classification in a closed-loop neuro-adaptive interaction cycle for usage in web-based UI adaptive systems.
Figure 1: Overview of the whole experiment
Figure 2: Results for the subjective evaluation of the adaptation dialogue box
Figure 3: Grand averages of signed r-square values
Figure 4: Statisical comparisons of event-related potential topographies for different adaptations
Acknowledgements
This research was supported by a grants from the German Federal Ministry for Education and Research (BMBF: 16SV7195K) and the European Union’s Seventh Framework Program under FP7 Grant #610510.
References
Olofsson, J. K., Nordin, S., Sequeira, H., & Polich, J. (2008). Affective picture processing: An integrative review of ERP findings. Biological Psychology, 77(3), 247–265. https://doi.org/10.1016/j.biopsycho.2007.11.006
Oostenveld, R., Fries, P., Maris, E., & Schoffelen, J.-M. (2011). FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological Data. Computational Intelligence and Neuroscience, 2011, 1–9. https://doi.org/10.1155/2011/156869
Peissner, M., & Edlin-White, R. (2013). User Control in Adaptive User Interfaces for Accessibility. In P. Kotzé, G. Marsden, G. Lindgaard, J. Wesson, & M. Winckler (Eds.), Human-Computer Interaction – INTERACT 2013 (Vol. 8117, pp. 623–640). Berlin, Heidelberg: Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-40483-2_44
Pollmann, K., Ziegler, D., Peissner, M., & Vukelić, M. (2017). A New Experimental Paradigm for Affective Research in Neuro-adaptive Technologies (pp. 1–8). ACM Press. https://doi.org/10.1145/3038439.3038442
Sander, D., Grandjean, D., & Scherer, K. R. (2005). A systems approach to appraisal mechanisms in emotion. Neural Networks, 18(4), 317–352. https://doi.org/10.1016/j.neunet.2005.03.001
Zander, T. O., Krol, L. R., Birbaumer, N. P., & Gramann, K. (2016). Neuroadaptive technology enables implicit cursor control based on medial prefrontal cortex activity. Proceedings of the National Academy of Sciences, 201605155. https://doi.org/10.1073/pnas.1605155114
Keywords:
Neuro-adaptive system,
Electroencephalography (EEG),
Emotions,
Affect,
appraisal,
web-based adaptive user interfaces,
Assistive Technology
Conference:
2nd International Neuroergonomics Conference, Philadelphia, PA, United States, 27 Jun - 29 Jun, 2018.
Presentation Type:
Oral Presentation
Topic:
Neuroergonomics
Citation:
Malik
FT,
Pollmann
K,
Peissner
M and
Vukelić
M
(2019). Assessing neuroelectrical markers of emotional appraisal during the interaction with adaptive user interfaces.
Conference Abstract:
2nd International Neuroergonomics Conference.
doi: 10.3389/conf.fnhum.2018.227.00057
Copyright:
The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers.
They are made available through the Frontiers publishing platform as a service to conference organizers and presenters.
The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated.
Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed.
For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions.
Received:
11 Mar 2018;
Published Online:
27 Sep 2019.
*
Correspondence:
Dr. Mathias Vukelić, Institut für Arbeitswissenschaft und Technologiemanagement, Universität Stuttgart, Stuttgart, Germany, mathias.vukelic@iao.fraunhofer.de