Edited by: Antonio Lanatà, University of Pisa, Italy
Reviewed by: Hun-Kuk Park, Kyung Hee University, South Korea; Giovanni Mirabella, University of La Sapienza, Italy
*Correspondence: Francesco Onorati, Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano, Via Ponzio 34/5, 20133 Milan, Italy Behavior and Brain Lab, IULM University, Via Carlo Bo 1, 20143 Milan, Italy e-mail:
This article was submitted to the journal Frontiers in Neuroengineering.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
With the recent advent of new recording devices and an easier access to signal processing tools, researchers are increasingly exploring and studying the Pupil Dilation (PD) signal. Recently, numerous studies pointed out the relations between PD dynamics and psychophysiological states. Although it is well known that PD is controlled by the Autonomic Nervous System (ANS), and ANS responses are related to emotional events/stimuli, the relationship between emotional states and PD is still an open issue. The aim of this study is to define the statistical properties of the PD signal, to understand its relation with ANS correlates such as Heart Rate Variability (HRV) and respiration (RESP), and to explore if PD could provide information for the evaluation of the psychophysiological response of ANS to affective triggering events. ECG, RESP, and PD data from 13 normal subjects were recorded during a memory recall paradigm, and processed with spectral and cross-spectral analysis. Our results demonstrate that variability indices extracted from fast PD oscillations, not observable through standard cardiorespiratory identification in the frequency domain, would be able to discern psychophysiological responses elicited by basic emotional stimuli. A strong linear coupling was found between the variables, due to the influence of RESP on both PD and HRV within the High Frequency (HF) band, from 0.15 to 0.45 Hz. Most importantly, our results point at PD features as possible candidates for characterizing basic emotional stimuli.
The Autonomic Nervous System (ANS) primarily innervates the smooth musculature of all organs, the heart and the glands, and mediates the neuronal regulation of the internal environment to keep a proper balance, a process in general not under direct voluntary control (Jänig,
The Sympathetic Nervous System (SNS) and the Parasympathetic Nervous System (PNS) both innervate the heart (Berntson et al.,
One of the most recent ANS correlate introduced in scientific literature is Pupil Dilation (PD) (Beatty and Lucero-Wagoner,
The aim of this study is to apply mathematical methods to process PD dynamics during an emotionally characterized protocol in order to define descriptive statistical indices of PD and to explore their relation with ANS correlates such as HRV and respiration (RESP). In addition, we verify if PD could provide information in the evaluation of ANS responses during a psychophysiological affective protocol.
Personal feelings are proved to produce large responses in measurements based on self-reports or physiological features (Bond,
The target emotions were chosen within the classical discrete categorical model of emotions, according to which they are represented as discernible but fuzzy bounded entities (Russell,
Healthy volunteers were recruited from the student body of IULM University of Milan. The subjects did not suffer from mental pathologies. The experimental protocol was divided in two phases. In the first phase, subjects were scheduled for an interview where they were asked to recall and loudly tell two recent autobiographical episodes for each of the target emotions. Then the psychologist, in agreement with the subject, chose the most vivid and intense episode for each target emotion. These episodes are then used in the second phase, as described below. In Table
Happiness | To meet a close relative, a friend, or partner after a long time; an important sport success |
Anger | To be cheated on; to fail a test or a school exam |
Sadness | The grief for the death of a close relative or a friend; the end of a love affair |
Subjects who could not recall vivid recent episodes for each of the target emotions were excluded from the second phase of the experimental protocol. In total, 13 subjects participated in the second phase of the experiment: they were scheduled for a second appointment and admonished not to consume coffee or caffeinated products at least 4 h before it.
The second phase of the experiment is the recording session, during which the subjects were helped in recalling the same autobiographical episodes chosen with the psychologist during the first phase of the experiment.
Figure
The subjects were asked to sit in front of the SensoMotoric Instruments RED250™ Eye-tracker monitor provided by a color gray screen at a fixed distance of 70 cm, in a room with constant illumination conditions. The PD signals were recorded at a sample frequency of 250 Hz; prior to start the “Baseline” and the single emotion recalls, calibration of the eye-tracker was performed. For computation purposes, the signals were then low-passed and resampled at 50 Hz.
ECG and the RESP signals were recorded using a Flexcomp Infinity™ encoder (Thought Technology Ltd.; Montreal, Canada) at a sampling rate of 2048 Hz, then resampled at 256 Hz. Relative changes in thoracic expansion were measured using a band provided with a tension-sensitive latex transducer; the thoracic band was placed over the upper part of the chest, individually adjusted to produce the maximal deflection during normal breathing; in the pre-experimental phase the subject was asked to exhale and inhale in a sealed reservoir bag: this procedure was designed to calibrate the RESP signal and to cancel the effects of the differences due to the band positioning and to the different thoracic expansions among the subjects. ECG was recorded using a standard 3 leads montage (Einthoven lead 2 configuration) on the right and left forearms. R-waves were detected and corrected from ectopic beats with a specific detection and correction program (Citi et al.,
Before performing the PD Analysis, a PD Reconstruction phase was needed to fill the missing data due to eye-blinking events and artifacts, to obtain an evenly sampled signal. Eye-blinking events were automatically recognized by the eye-tracker and reviewed offline to correct misdetection or missed events. A temporal window from 100 ms before to 100 ms after each eye-blinking event was clipped from the data (Einhäuser et al.,
An iterative method based on Singular Spectrum Analysis (SSA), called Iterative-SSA, was implemented to fill the gap generated on a PD signal by blink events (Sassi et al.,
The SSA is a powerful signal processing technique introduced by Broomhead and King (
Once the dimension
A singular value decomposition (SVD) is carried on
The original time series is expanded in an optimal way as the sum of its
The choice of
For the estimation of missing data of the time series, Schoellhamer (
Figure
As previous studies reported a spectral content for the PD signal up to 4–5 Hz (Nakayama and Shimizu,
A parametric spectral analysis, via autoregressive (AR) model coefficients estimation was performed to compute the spectral components. The order of the model was chosen according to the Akaike Information Criterion (AIC) (Akaike,
Given two time series
To compute
Once the AR coefficients and the covariances are obtained, it is possible to estimate the cross-spectral matrix
Estimated the auto-spectra
Finally, according to a bivariate closed-loop model (Barbieri et al.,
PD signals were sampled at the occurrences of the R-waves. The Coherence functions γ2 between the signals were computed by Equations (6) and (7).
For assessing the significance zero level of the Coherence (and for nDC) a surrogate data analysis procedure was performed (Faes et al.,
The corresponding value of causal gains are evaluated at the frequency values of the peaks of nDC.
We computed all the above indices from segments long at least 90 s for the available signals (PD, RR, RESP) during the “Baseline” condition and the three emotional events “Happiness,” “Anger” and “Sadness”. For each index and epoch, we performed a Lilliefors test (Lilliefors,
We performed a discriminant analysis to test the ability of the most relevant indices to potentially distinguish “Baseline”from the emotionally characterized events. We computed true positive rate (TPR) and false positive rate (FPR) varying the discriminant threshold. Statistical indices such as Sensitivity (
Table
μ | 3.998 ± 0.397 | 3.981 ± 0.451 | 3.916 ± 0.498 | 3.990 ± 0.460 |
σ | 0.282 ± 0.057 | 0.240 ± 0.051 | 0.237 ± 0.073 | 0.239 ± 0.070 |
0.071 ± 0.015 | 0.061 ± 0.013 | 0.060 ± 0.017 | 0.060 ± 0.015 |
We observe a decrease of the overall variability during triggering events. The reduction of σ, although not significant, occurs during emotional events regardless of mean pupil size μ, which is possibly dependent on other causes, such as accommodation or brightness level. As a consequence, the
Table
μRR | 0.826 ± 0.115 | 0.811 ± 0.111 | 0.805 ± 0.121 | 0.797 ± 0.104 |
LF norm | 0.453 ± 0.139 | 0.532 ± 0.250 | 0.588 ± 0.246 | 0.570 ± 0.185 |
HF norm | 0.418 ± 0.203 | 0.352 ± 0.280 | 0.271 ± 0.248 | 0.260 ± 0.243 |
LF/HF | 1.281 ± 0.556 | 4.550 ± 9.251 | 8.370 ± 14.070 | 5.775 ± 6.998 |
0.271 ± 0.060 | 0.262 ± 0.093 | 0.267 ± 0.092 | 0.241 ± 0.110 | |
σ2 | 0.012 ± 0.023 | 0.020 ± 0.038 | 0.020 ± 0.042 | 0.028 ± 0.059 |
HF% | 0.765 ± 0.157 | 0.656 ± 0.281 | 0.752 ± 0.188 | 0.536 ± 0.278 |
The Friedman test didn't show any statistical significance among the psychophysiological conditions in the cardiorespiratory features presented in Table
PD spectral indices at both low frequencies (LF and HF) and high frequencies (VHF) are shown in Table
LFPD | 0.030 ± 0.023 | 0.023 ± 0.017 | 0.026 ± 0.026 | 0.019 ± 0.013 |
LFPD norm | 0.467 ± 0.189 | 0.465 ± 0.148 | 0.490 ± 0.213 | 0.459 ± 0.161 |
HFPD | 0.016 ± 0.010 | 0.012 ± 0.007 | ||
HFPD norm | 0.313 ± 0.223 | 0.319 ± 0.214 | 0.216 ± 0.075 | 0.275 ± 0.160 |
LF/HFPD | 2.268 ± 1.677 | 2.851 ± 3.336 | 3.019 ± 3.486 | 2.400 ± 2.141 |
VHF[0.45–1]( |
0.464 ± 0.192 | 0.486 ± 0.378 | 0.374 ± 0.260 | 0.425 ± 0.293 |
VHF[1–2.5]( |
0.204 ± 0.115 | 0.204 ± 0.204 | 0.158 ± 0.137 | 0.208 ± 0.182 |
VHF[2.5–5]( |
0.484 ± 0.174 | 0.442 ± 0.190 | 0.409 ± 0.214 |
Notably, according to the Friedman test, there are significant inter-group differences due to the different experimental conditions for HFPD [
Similarly, the absolute power in both LF and HF bands shows a clear decrease. In particular, this decrease is statistically significant (
An example of Coherence analysis between PD, RR intervals and RESP is presented in Figure
In Table
RR-PD, LF | 0.502 ± 0.157 | 0.559 ± 0.225 | 0.547 ± 0.172 | 0.491 ± 0.213 |
3/13 | 4/13 | 5/13 | 4/13 | |
RR-PD, HF | 0.647 ± 0.136 | 0.574 ± 0.133 | 0.625 ± 0.129 | 0.612 ± 0.124 |
4/13 | 9/13 | 9/13 | 10/13 | |
RESP-PD | 0.688 ± 0.128 | 0.605 ± 0.146 | 0.617 ± 0.174 | 0.636 ± 0.192 |
10/13 | 9/13 | 10/13 | 10/13 | |
RR-RESP | 0.888 ± 0.145 | 0.858 ± 0.113 | 0.774 ± 0.217 | 0.860 ± 0.083 |
12/12 | 4/12 | 5/13 | 13/13 |
As an overall results, it is possible to see that during “Baseline” the average Coherence shows higher values at HF and lower values at LF. The Coherence analysis between PD and the RR signal shows on average 4 Coherences above threshold in the LF band out of 13 subjects. In the HF band we reported 10 subjects out of 13 showing a Coherence above threshold during “Sadness,” and 9 subjects out of 13 during “Happiness” and “Anger”. For “Baseline” we found an above threshold Coherence in the HF band for 4 subjects out of 13, even though the average Coherence is higher than the other conditions: this result might be due to the method used to assess the significance zero level, which in some cases seems to be too conservative. In the analysis between PD and RESP, a higher Coherence has been reported (10 out of 13 above threshold for “Baseline,” “Anger,” and “Sadness,” and 9 out of 13 for “Happiness”).
We computed also the nDC and the Gain of the related transfer functions. A graphical representation of the performed analysis is in Figure
RR → PD, HF | 0.797 ± 0.054 | 0.623 ± 0.065 | 0.722 ± 0.121 | 0.646 ± 0.103 |
1.496 6/13 | 2.366 7/13 | 1.498 5/13 | 1.942 8/13 | |
PD→RR, HF | 0.685 ± 0.139 | 0.614 ± 0,180 | 0.706 ± 0.090 | 0.711 ± 0.146 |
0.171 6/13 | 0.124 7/13 | 0.374 6/13 | 0.193 6/13 | |
RESP→PD | 0.756 ± 0.143 | 0.626 ± 0.167 | 0.662 ± 0.168 | 0.657 ± 0.145 |
0.780 10/13 | 0.735 9/13 | 0.612 8/13 | ||
RESP→RR | 0.927 ± 0.071 | 0.879 ± 0.101 | 0.841 ± 0.201 | 0.904 ± 0.082 |
0.434 13/13 | 0.365 13/13 | 0.305 12/13 | 0.351 13/13 |
Only
We noticed that HFPD and
In accordance to the results just presented, we chose the indices with highest statistical power. In Table
HFHRV norm | 0.538, 0.923 | 0.615, 0.846 | 0.769, 0.692 |
LF/HFHRV | 0.308, 0.846 | 0.385, 0.387 | 0.077, 1.000 |
HFPD | 0.461, 0.692 | 0.615, 0.769 | 0.538, 0.846 |
VHF[2.5–5] | 0.769, 0.461 | 0.615, 0.769 | 0.615, 0.615 |
0.461, 0.385 | 0.615, 0.615 | 0.692, 0.385 | |
Lin. Comb.PD | 0.769, 0.538 | 0.769, 0.692 | 0.692, 0.615 |
This work focuses on mathematical methods for PD signal processing aimed at estimating novel markers of autonomic activity and investigates PD dynamic changes during a psychophysiological study, in particular during emotionally characterized events compared with a general relaxation/deactivation condition. The analysis was performed through the following multiple steps: (1) we implemented an
The first step was to provide an algorithm for the reconstruction of the PD signal during blinking events and movement artifacts. The adopted procedure allowed to analyze PD at high frequencies, hence exploring the information carried by fast oscillations, and to improve the analysis of the signal at low frequencies, providing an estimation of the missing data as close as possible to the underlying dynamics of the observable data. From the reconstructed signals we computed time domain aggregate features. We observed an overall decrease in variability during triggering events. Although it was not sufficient to significantly distinguish emotional conditions from “Baseline,” this trend is in agreement with a common behavior of other physiological signals during psychophysiological events (Caldirola et al.,
As firstly hypothesized by Borgdorff (
In this work we therefore explored the spectral components of PD in frequency ranges classically related to the ANS control of cardiorespiratory activity, i.e., LF and HF, computing widespread features. The observed trends of PD and RR series features confirm similar aggregate behavior for PD and RR signal. These patterns reflects similar autonomic modulation at HRV frequency bands for both pupillary and cardiovascular systems during emotional triggering events. The analysis of Coherence clarifies the nature of this behavior.
The results of Coherence analysis support the idea of a coupling between PD and cardiorespiratory activity, mostly for the respiratory influence on PD. In particular, results regarding the Coherence in the HF band are consistent with previous findings (Calcagnini et al.,
The low value of Coherence and nDC, as well as the unclear directionality between HRV and PD at LF could be explained by different hypotheses. Although a large part of the power content of PD is at low frequencies, these dynamics might not reflect autonomic control on the pupillary system, or at least an autonomic response to particular triggering events such as emotionally characterized events. Moreover, these dynamics show a weak linear coupling with HRV LF rhythms, as suggested by Calcagnini et al., (
As further outcome, we explored PD dynamics at higher frequencies (from 0.45 to 5 Hz). These oscillations have trends comparable to those in PD total variance and HF power. The highly significant differences between “Baseline” and “Anger” and the similar behavior of the signal at higher frequencies support the hypothesis of a direct central autonomic control on PD, which reflects also the response of the ANS to emotional triggering events. The HFPD and VHFPD components provides characterizing features, which might lead to propose PD-based markers in stressful, emotional or arousing events. For this reason, it would be interesting to explore more deeply the role of the central autonomic network as reflecting in PD responses during emotional events.
In addition, to deepen the classification prospect of PD features might improve nowaday classification performances and candidate PD as a new important signal in recognition and classification of emotional conditions in different research fields. In particular, the possibility to evaluate affective states from PD might lead to interesting developments for communication applications: as a contactless autonomic correlate, PD could cover a major role in the detection of arousing events elicited by audio-visual contents.
The results of our analysis show that the differences between the experimental conditions were reflected on PD indices and confirmed by the level of sensitivity and specificity of these indices in distinguishing a baseline state from the emotional events. For this reason, classification performances of these new indices need to be explored in future works. The influence of RESP to PD and the weak or absent linear coupling between HRV and PD in the LF band are other relevant outcomes of this work and confirm the findings previously reported in the literature. An important advance would be to evaluate the effect of known and controlled ANS changes on RR intervals, RESP and PD coupling. Importantly,
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
1The valence represents the degree of pleasure or displeasure an event or a stimulus is able to elicit. The arousal determines the degree of activation in response of an event or a stimulus (Lang,