Abstract
Introduction:
Fear is a fundamental emotion essential for survival; however, excessive fear can lead to anxiety disorders and other adverse consequences. Monitoring fear states is crucial for timely intervention and improved mental well-being. Although functional magnetic resonance imaging (fMRI) has provided valuable insights into the neural networks associated with fear, its high cost and environmental constraints limit its practical application in daily life. Electroencephalography (EEG) offers a more accessible alternative but struggles to capture deep brain activity. Physiological measures such as pupil dynamics and heart rate can provide indirect insights into these deeper processes, yet they are often studied in isolation. In this context, we aimed to evaluate the practical effectiveness and limitations of a multimodal approach that combines pupil dynamics and heart rate—indirect indicators of deep brain activity—with EEG, a temporally precise but spatially limited measure of cortical responses.
Methods:
We simultaneously recorded EEG, pupillometry, and heart rate in 40 healthy male participants exposed to fear-inducing and neutral visual stimuli, while also assessing their psychological states.
Results:
Fear-inducing stimuli elicited distinct physiological responses, including increased occipital theta power, pupil dilation, and decreased heart rate. Notably, pupil size was the most sensitive discriminator of emotional state, though the integration of modalities yielded only limited improvement in classification accuracy.
Discussion:
These findings provide empirical support for the feasibility of multimodal physiological monitoring of fear and underscore the need for further refinement for real-world applications.
1 Introduction
Fear is a fundamental emotion triggered by realistic and severe threats [reviewed in Blanchard and Blanchard (); Davis et al. (); Grupe and Nitschke (); Malezieux et al. ()]. Its neural basis involves the transmission of environmental information from primary and associative sensory cortices and the thalamus to the amygdala, where this information is evaluated for threat detection. When a threat is detected, the outcome of this evaluation triggers behavioral and physiological responses through connections to the hypothalamus and brainstem. Additionally, the amygdala projects to extensive cortical regions and the hippocampus, adjusting cognitive functions such as attention, memory, and decision-making. Thus, fear plays a vital role in survival by facilitating rapid responses and enhancing learning and memory (Phelps and LeDoux, ; LeDoux, ). However, excessive fear can negatively affect the body and lead to anxiety disorders, phobias, avoidance behaviors, and activity limitations [reviewed in Pittig et al. ()]. Therefore, accurately and efficiently monitoring an individual's fear state—especially through non-invasive, real-time techniques—is essential for enabling appropriate interventions and adaptive support in daily life. Multimodal physiological monitoring, combining neural and autonomic indicators, holds promise as a practical approach for this purpose (Sheikh et al., ; Williams and Pykett, ).
One method of monitoring fear involves assessing the activity of neural networks centered on the amygdala, a deep brain structure involved in fear evaluation [reviewed in Domínguez-Borràs and Vuilleumier ()]. Over the past few decades, functional magnetic resonance imaging (fMRI) studies have revealed the characteristics of these neural networks, particularly the interactions between the amygdala and the cerebral cortex, hippocampus, thalamus, and hypothalamus (Richardson et al., ; Kral et al., ). However, this approach is limited by high costs and environmental constraints, making it challenging to monitor mental states in everyday settings. Alternatively, electroencephalography (EEG)-based approaches are more cost-effective and limited by fewer environmental constraints. Although EEG has inherent limitations in spatial resolution and direct access to deep brain structures, its high temporal resolution and portability make it a promising tool for monitoring cortical responses related to emotional states in real-world settings (Chien et al., ; Sperl et al., ; Chen et al., ). Recent advancements in wearable EEG technology have spurred increasing interest in their application for monitoring mental states in real-world environments (Anders and Arnrich, ).
In addition to EEG, other physiological data, such as pupil dilation and heart rate, serve as powerful tools for monitoring mental states related the neural activities of amygdala and locus coeruleus (LC), a central hub for the noradrenergic (NA) [reviewed in Sheikh et al. (); Johansson and Balkenius ()]. Through these neural activities, emotions, such as fear, influence both behavioral and physiological responses. Furthermore, these physiological responses reflect activity in deep brain regions that cannot be captured by EEG alone. Therefore, combining EEG with behavioral and physiological data in a multimodal approach can complement the spatiotemporal coverage of each modality, enhancing the accuracy of mental state estimation (Anders and Arnrich, ). Although this approach offers potential advantages, multimodal strategies for monitoring mental states remain underexplored, as most studies focus on either EEG alone (Chien et al., ; Sperl et al., ; Anders and Arnrich, ) or single-modality approaches using behavioral or physiological data (Leuchs et al., ; Rafique et al., ).
In this context, we aimed to evaluate the practical effectiveness and limitations of a multimodal approach that combines pupil dynamics and heart rate—which are considered indirect indicators of deep brain activity—with EEG, a temporally precise but spatially limited measure of cortical activity. Rather than seeking to uncover novel neural mechanisms, the present study focuses on validating whether integrating these complementary modalities improves the accuracy of emotional state detection, particularly for fear. To this end, we conducted simultaneous EEG, pupillometry, and heart rate measurements in 40 male participants while presenting fear-inducing and neutral visual stimuli. The participants were also assessed for trait and state anxiety, enabling us to evaluate multimodal indicators in relation to individual emotional characteristics. By evaluating the relative utility and limitations of each physiological modality, this study seeks to contribute to the development of practical, real-world emotional monitoring systems. Our results revealed that pupil dilation following the stimuli was the most sensitive indicator distinguishing between fear and neutral conditions. However, integrating multiple modalities yielded limited classification improvements. These findings offer empirical insights into the relative utility of each physiological signal and suggest future directions for real-world emotional state monitoring.
2 Materials and methods
2.1 Participants
Forty male participants (aged 18–25 years) were recruited from the student body at Chiba Institute of Technology for this study. None of the participants had neurological impairments or behavioral difficulties, and all had normal vision or vision corrected to normal with contact lenses. Here, based on an a priori power analysis assuming a medium effect size [partial η2 = 0.06; (Cohen, )], statistical power of 0.80, and two repeated measures, the minimum required sample size was estimated to be 33 participants. Female participants were excluded to avoid the potential influence of menstrual-related psychological stress on the state anxiety test. Participants were instructed to refrain from consuming caffeinated or alcoholic beverages and engaging in strenuous exercise on the day before and the day of the measurements. Written informed consent was obtained prior to participation. The Ethics Committee of Chiba Institute of Technology approved all experimental protocols and methods for this study (approval number: 2023-05-01). All procedures were conducted in accordance with the principles outlined in the Declaration of Helsinki.
Table 1 presents the data for all participants. In this study, the Japanese version of the State-Trait Anxiety Inventory (STAI), developed by Spielberger and colleagues, was used to measure participants' anxiety levels (Spielberger et al., ). The STAI is widely recognized for its reliability and validity and is frequently used in educational and clinical settings. The STAI effectively captures trait anxiety, reflecting an individual's general tendency to experience anxiety and state anxiety, which varies according to situational factors, providing a comprehensive measure of psychological state. In this study, the STAI was employed to assess participants' baseline anxiety levels. The STAI evaluates anxiety across two distinct dimensions: state anxiety (A-State) and trait anxiety (A-Trait). State anxiety (A-State) assesses participants' transient feelings of anxiety, while trait anxiety (A-Trait) indicates their general tendency toward anxiety over time. Each anxiety dimension consists of 20 items. Participants rated their own psychological state on a 4-point Likert scale, ranging from 1 (rarely) to 4 (almost always). Scores for each item were summed, resulting in a total score for each dimension, with possible scores ranging from 20 to 80. Higher scores indicate higher levels of anxiety. The STAI was administered before the visual stimulus task to assess baseline anxiety. The STAI scores were calculated by aggregating participants' responses for each item, resulting in total scores for A-State and A-Trait.
Table 1
| Number of participants | N = 40 |
|---|---|
| Age (years) | 20.575 (18-25, 1.693) |
| Sex (male/female) | 40/0 |
| A-state | 40.375 (23–55, 6.758) |
| A-trait | 45.075 (23–60, 10.307) |
Demographic characteristics of all participants are represented as means (range, standard deviation [SD]).
The STAI was administered before the visual stimulus task and its scores were composed of scores for A-state and A-trait.
2.2 Experimental procedures
Figure 1A presents an overview of the experimental setup. The measurement equipment included a glasses-type eye tracker (Tobii Pro Glasses 3, Tobii Technology K.K.), an 8-channel wearable EEG device (NB1-EEG, East Medic Corporation), and a wearable heart rate monitor (my Beat, WHS-1, UNION TOOL CO.). Participants sat in a chair with a backrest while wearing the eye tracker, EEG device, and heart rate monitor during the experiments. An LCD monitor with a vertical size of 597.6 cm (1920 pixels) and a horizontal size of 336.2 cm (1,080 pixels) was used for visual stimulus presentation. The effective field of view for the participants was ±26.47° horizontally and ±6.48° vertically. The experiment was conducted in a laboratory with an illuminance range of 300 lx to 600 lx, measured using a lux meter compliant with JIS C 1609-1:2006 Class A linear type. In Figure 1B, the measurement scene is shown while presenting a gray blank image.
Figure 1
For the visual stimulation task, we selected 36 images from the International Affective Picture System (IAPS) based on each image's scores (mean pleasure/arousal), consisting of 18 neutral images (mean pleasure/arousal = 4.5, 6.2) and 18 fear images (mean pleasure/arousal = 2.1, 6.6). The fear images included depictions of weapons, corpses, and murder scenes, while the neutral images featured items like snakes, spiders, and sharks. Additionally, we prepared gray (neutral) images that only displayed a gray blank image with R = 0, G = 0, B = 0 in the RGB scale. In the experiment, to balance the presentation order of the images, participants were randomly divided into two groups: group sequence-A (22 participants) and group sequence-B (18 participants) (see demographic characteristics of both groups in Table 2). As demonstrated in Figure 1C, group sequence-A was first shown a gray blank image for 5 min, followed by a 10-min break, and then shown fear images for 216 s. After another 10-min break, participants were shown neutral images for 216 s, followed by a final 10-min break. Group sequence B was first shown gray images for 5 min, followed by a 10-min break, then neutral images for 216 s, another 10-min break, and finally fear images for 216 s, ending with a 10-min break. Additionally, gray images were presented for 2 min before and after each presentation of fear and neutral images. During each presentation, participants viewed each image for 10 s, followed by a 2-s gray image, and then another 10-s display, repeating this cycle for a total of 216 s. A 10-min resting period was given before and after the visual stimulation task to ensure participants had ample rest.
Table 2
| Evaluation indexes | Group sequence-A | Group sequence-B | p-value |
|---|---|---|---|
| Number of participants | N = 22 | N = 18 | — |
| Age (years) | 20.545 (18-23, 1.535) | 20.611 (18–25, 1.914) | 0.905 |
| Sex (male/female) | 22/0 | 18/0 | — |
| A-state | 39.000 (23–55, 7.584) | 42.056 (31-52, 5.319) | 0.157 |
| A-trait | 43.955 (23–60, 11.303) | 46.444 (26-59, 9.070) | 0.454 |
Demographic characteristics of sequence-A and sequence-B participants are represented as means (range, standard deviation [SD]).
There are no significant differences in age, sex ratio, A-state, and A-trains between groups of sequence A and sequence B.
Furthermore, to evaluate participants' momentary mood states during each resting period, we used the Two-Dimensional Mood Scale (TDMS) (Sakairi et al., ). The TDMS is a psychological assessment tool developed for self-monitoring and self-regulation, assessing mood states based on two dimensions: pleasure and arousal. This scale allows participants to self-assess their mood using eight adjectives, such as “energized” and “relaxed,” rated on a 6-point Likert scale (0: not at all, 5: very much). Each mood state is classified into one of the following four quadrants based on the combination of pleasure and arousal: high-energy pleasure (e.g., lively), high-energy discomfort (e.g., tense), low-energy pleasure (e.g., calm), and low-energy discomfort (e.g., lethargic). This approach visually represents momentary psychological states, enabling a quantitative understanding of mood changes. In this study, TDMS was administered at three time points: before the task, after the first image presentation, and after the second image presentation. To evaluate mood fluctuations, difference scores were calculated by subtracting the pre-task baseline from each post-task score. This approach allowed us to capture short-term, task-induced changes in pleasure and arousal in response to specific psychological stimuli. For example, a decrease in energy levels and a decline in emotional stability following exposure to fear-inducing stimuli may indicate a strong psychological impact.
2.3 Recording
2.3.1 Pupil size
Pupil diameter was measured using the glasses-type eye tracker (Tobii Pro Glasses 3) with a sampling frequency of 100 Hz. For segments missing due to blinking, linear interpolation was performed, treating the range, including 0.1 s before and after the missing segment, as missing values. Furthermore, all segments of the pupil responses for the gray blank images, fear images, and neutral images in any participant that deviated by more than three SDs from that participant's mean for the corresponding condition were treated as missing values, and linear interpolation was performed accordingly. Finally, a 10 Hz low-pass filter was applied. The average pupil diameter during the 5-min presentation of a gray blank image was used as the baseline for each participant. The pupil size was then calculated as the difference between this baseline and the time-averaged pupil diameter during the 3-min presentation of a gray blank image following the visual stimulus task.
2.3.2 Heart rate
The heart rate measured by the wearable heart rate sensor (my Beat, WHS-1/RRD-1) was used to calculate the average R-R interval (RRI) during a 5-min presentation of a gray blank image, which served as the baseline for each participant. The difference from this baseline was then calculated for the average RRI during the visual stimulus tasks with fear and neutral image groups, as well as during the gray blank image presentation following the visual stimulus tasks.
2.3.3 Electroencephalography
EEG data were recorded using a wearable 8-electrode EEG device (NB1-EEG) with a sampling rate of 121.37 Hz. Using EEGLAB 2023.1, the data from each segment, consisting of a 5-min gray blank image, the visual stimulation task, and a gray blank image after the task, were resampled to 120 Hz and filtered with a 1-40 Hz band-pass filter. Independent Component Analysis (ICA) was then performed on each segment to classify each component into Brain, Muscle, Eye, Heart, or Other, according to the highest probability. Components classified as categories other than the Brain were removed. For the analysis, electrodes from the frontal (F3, F4) and parieto-occipital (PO3, PO4) regions were selected because these sites have been reliably associated with emotional and visual processing (Keil et al., ; Hajcak et al., ). The relative powers of the theta, alpha, and beta waves were calculated for each electrode, and averaged within the frontal and occipital regions.
2.3.4 Simultaneous measurements
The start timing of pupil size and EEG measurements was controlled using Python, and timestamps of visual stimuli during the measurements were also recorded. For the heart rate monitor, the start switch was pressed within 1 s before or after the start of pupil diameter and EEG measurements.
2.4 Evaluation index
The evaluation indices for pupil size, heart rate, and EEG were calculated using the average of the left and right pupil diameters, the average RRI value, and the relative power from the frontal (F3, F4) and occipital (PO3, PO4) electrodes, respectively. For EEG, the relative power of theta (2–4 Hz), alpha (8–13 Hz), and beta (13–30 Hz) bands was calculated for each electrode and averaged for the frontal and occipital regions. For pupil size, deviation from baseline was determined by subtracting the baseline value from the time-averaged left and right pupil diameters during the presentation of the gray blank image following the visual stimulation task. During the visual stimulation task, image luminance was not constantly maintained to ensure a vivid and naturalistic presentation. As a result, pupil diameter was affected not only by arousal and emotional valence, but also by luminance fluctuations. To minimize this confounding influence, we excluded the stimulation period from the pupil-based evaluation. In contrast, RRI values and EEG relative power were calculated by averaging data from both the visual stimulation period and the subsequent gray blank image.
2.5 Statistical analysis
The difference between neutral- and fear-induced responses was evaluated using repeated measures analysis of variance (ANOVA) for crossover trial assessment. The between-subject factor was sequence (sequence-A and sequence-B), while the within-subject factors were image type (neutral or fear) and presentation period. Each response was evaluated based on the change from the baseline value measured during the initial gray image presentation. Statistical significance was set at p < 0.05 using the Greenhouse-Geisser correction for both psychological responses (arousal and comfort levels) and physiological responses (pupil size, RRI, EEG power). To control for the risk of Type I error due to multiple comparisons among physiological indices, Bonferroni correction was additionally applied.
We utilized the receiver operating characteristic (ROC) curve and its area under the curve (AUC) based on logistic regression to assess the ability of biosignals to distinguish which stimulus (fear-induced or neutral) was presented to participants. For cross-validation, leave-one-out cross-validation (LOOCV) was applied. A random classification corresponds to AUC = 0.5, while a perfect classification corresponds to AUC = 1.0.
All statistical analyses were conducted using IBM SPSS Statistics (Version 29.0.2) or MATLAB (Version R2023b).
3 Results
3.1 Psychological responses and their relation to trait and state anxiety
For each group, sequence A and sequence B, comfort and arousal levels following the visual stimulus presentation were measured using TDMS and evaluated through a crossover trial. The results of comfort and arousal evaluations are presented in Tables 3, 4, respectively. Both comfort and arousal levels showed a significant main effect related to image type (fear/neutral). Additionally, Figure 2 displays violin plots and boxplots showing the distribution of TDMS alternated scores after the stimuli compared to the baseline (beginning of the tasks) in the sequence A and sequence B groups. Subplots (A) and (B) correspond to the comfort level and arousal level, respectively. The presentation of fear-inducing images resulted in a significant decrease in comfort compared to neutral images. Conversely, fear-inducing images led to a significant increase in arousal compared to neutral images.
Table 3
| Factors | F-value (p-value, η2) |
|---|---|
| Image type | 17.736 (< 0.001, 0.318) |
| Sequence | 0.000 (0.995, 0.000) |
| Period | 0.338 (0.564, 0.009) |
Results of repeated-measures analysis of variance (ANOVA) for crossover trial on comfort levels with factors of image type, presentation sequence, and measurement period.
Significant results (p < 0.05) are shown in bold.
Table 4
| Factors | F-value (p-value, η2) |
|---|---|
| Image type | 7.832 (0.008, 0.171) |
| Sequence | 0.496 (0.486, 0.013) |
| Period | 0.034 (0.855, 0.001) |
Results of repeated-measures analysis of variance (ANOVA) for crossover trial on arousal levels with factors of image type, presentation sequence, and measurement period.
Significant results (p < 0.05) are shown in bold.
Figure 2
Moreover, correlations between A-state and A-trait in the STAI scores were analyzed. The results showed a significant and strong Pearson's correlation, r = 0.66 (p < 0.001). Regarding the relationship between alternated scores in the TDMS and the STAI scores, Table 5 presents the results of the correlation analysis, combining both groups for each type of image. A strong negative correlation was observed between the A-state and alternated arousal scores. This pattern suggests that participants with higher A-state scores tended to exhibit smaller changes in arousal, which is consistent with previous findings indicating reduced autonomic reactivity and restricted autonomic flexibility in high-anxiety individuals (Thayer and Lane, ; Friedman, ; Porges, ; Asbrand et al., ).
Table 5
| Factors | A-state | A-trait |
|---|---|---|
| Alternated comfort score by fear-image | 0.242(0.133) | 0.244(0.129) |
| Alternated arousal score by fear-image | −0.361(0.022) | −0.202(0.210) |
| Alternated comfort score by neutral-image | 0.148(0.363) | 0.034(0.835) |
| Alternated arousal score by neutral-image | −0.134(0.410) | 0.068(0.678) |
Pearson's correlation r-value (p-value) between STAI scores and TDMS alternated scores based on the score after representing a gray blank image.
Significant results (p < 0.05) are shown in bold.
3.2 Physiological responses for pupil size, heart rate, and electroencephalography
For each group, sequences A and B, the alternated pupil size, RRI, and frontal/occipital relative powers during and/or after the visual stimulus presentation were evaluated in a crossover trial, as shown in Table 6. Significant main effects of image type (fear vs. neutral) were observed in left and right pupil diameters following stimulus presentation, with corresponding Cohen's f values of f = 0.789 (left) and f = 0.939 (right), indicating large effect sizes. Similarly, significant effects were found for RRI and occipital theta relative power during stimulus presentation, with Cohen's f values of f = 0.720 and f = 0.606, respectively, also indicating large effect sizes. Figures 3A–C display violin plots and boxplots that illustrate the distribution of these alternated physiological responses. The results indicate that the average alternated values of these physiological responses across groups (sequence A and sequence B) significantly increased following fear-inducing images compared to neutral images.
Table 6
| Evaluation indexes | Factors | F-value (p-value, η2) during stimulus | F-value (p-value, η2) after stimulus |
|---|---|---|---|
| Size of left pupil | Image type | — | 23.659 (< 0.001, 0.384) |
| Sequence | — | 0.422 (0.520, 0.011) | |
| Period | — | 0.664 (0.420, 0.017) | |
| Size of right pupil | Image type | — | 33.573 (< 0.001, 0.469) |
| Sequence | — | 0.697 (0.409, 0.018) | |
| Period | — | 2.100 (0.155, 0.052) | |
| RRI of heart rate | Image type | 19.752 (< 0.001, 0.342) | 0.068 (0.795, 0.002) |
| Sequence | 0.301 (0.586, 0.008) | 0.036 (0.850, 0.001) | |
| Period | 3.869 (0.057, 0.092) | 0.644 (0.427, 0.017) | |
| Frontal theta relative power | Image type | 0.203 (0.655, 0.005) | 0.656 (0.423, 0.017) |
| Sequence | 0.325 (0.572, 0.008) | 0.042 (0.838, 0.001) | |
| Period | 0.014 (0.907, 0.000) | 2.288 (0.139, 0.057) | |
| Occipital theta relative power | Image type | 13.969 (< 0.001, 0.269) | 0.013 (0.911, 0.000) |
| Sequence | 0.296 (0.589, 0.008) | 0.203 (0.655, 0.005) | |
| Period | 0.187 (0.668, 0.005) | 0.589 (0.448, 0.015) | |
| Frontal alpha relative power | Image type | 0.384 (0.539, 0.010) | 0.578 (0.452, 0.015) |
| Sequence | 0.522 (0.474, 0.014) | 1.471 (0.233, 0.037) | |
| Period | 0.375 (0.544, 0.010) | 0.523 (0.474, 0.014) | |
| Occipital alpha relative power | Image type | 0.150 (0.701, 0.004) | 0.686 (0.413, 0.018) |
| Sequence | 0.364 (0.550, 0.009) | 0.023 (0.880, 0.001) | |
| Period | 0.680 (0.415, 0.018) | 0.463 (0.500, 0.012) | |
| Frontal beta relative power | Image type | 0.518 (0.476, 0.013) | 2.083 (0.157, 0.052) |
| Sequence | 0.396 (0.533, 0.010) | 1.604 (0.213, 0.040) | |
| Period | 1.652 (0.206, 0.042) | 0.000 (0.983, 0.000) | |
| Occipital beta relative power | Image type | 1.770 (0.191, 0.045) | 0.725 (0.400, 0.019) |
| Sequence | 0.171 (0.682, 0.004) | 0.729 (0.398, 0.019) | |
| Period | 0.074 (0.786, 0.002) | 0.752 (0.391, 0.019) |
Results of repeated measures ANOVA for crossover trial regarding physiological responses for pupil size, heart rate, and EEG.
Significant results (p < 0.05) (Bonferroni-corrected) are shown in bold.
Figure 3
Further, we evaluated the ability to distinguish which stimulus (fear-induced or neutral) was presented to participants, based on pupil size after the stimulus, as well as RRI and occipital theta relative power during the stimulus. Here, the right and left pupil sizes showed a high correlation (Pearson's correlation r = 0.930); therefore, their values were averaged to obtain a single pupil size measure. In Figure 4, the ROC curve and AUC values obtained using logistic regression are presented. As a result, pupil size demonstrated the highest ability to distinguish stimuli (AUC = 0.69) among the other unimodal biosignals. Regarding combinations of these biosignals, combining RRI with pupil size slightly improved the detection ability (AUC = 0.70).
Figure 4
4 Discussion
In this study, we aimed to investigate the neural activity associated with fear-induced cortical and deep brain responses, as reflected in heart rate and pupil changes. We simultaneously measured EEG, pupillometry, and heart rate in participants exposed to fear-inducing and neutral visual stimuli alongside assessments of their psychological responses. The results showed that fear-inducing stimuli decreased comfort levels and increased arousal, and these changes correlated more strongly with state anxiety than with trait anxiety. Furthermore, fear-inducing stimuli led to increased pupil size after the stimulus, a rise in RRI during the stimulus, and occipital theta band power during the stimulus.
4.1 Psychological responses
We observed that viewing the fear-inducing images lowered pleasantness and increased arousal, with a strong positive correlation between trait and state anxiety. This result is consistent with prior findings that an individual's enduring predisposition toward anxiety (trait anxiety) strongly influences their short-term anxiety in specific situations (state anxiety) (Leal et al., ). Regarding the effect of trait anxiety on responses to neutral- and fear-inducing images, no significant correlation was observed in either comfort or arousal. These results suggest that trait anxiety may not strongly influence physiological arousal responses to intense fear stimuli. In contrast, individuals with higher state anxiety exhibited lower changes in arousal (see Table 5). As noted in the Methods section (Section 2.2), the fear-inducing images were selected based on normative ratings from IAPS, with low valence (mean = 2.1) and high arousal (mean = 6.6), indicating strong emotional unpleasantness and intensity. Given the high baseline arousal induced by these stimuli, it is possible that their effects were already near ceiling levels, thus reducing observable modulation by individual differences in trait anxiety.
4.2 Physiological responses
4.2.1 Individual response
We discuss the reasons underlying the observed individual physiological responses. First, regarding the increase in RRI during the presentation of fear-inducing images, previous studies have reported a decrease in heart rate due to vagal activation triggered by fear stimuli (Bradley et al., ; Hermans et al., ). Our findings are consistent with those of earlier studies. Second, concerning the increase in pupil size following the presentation of fear-inducing stimuli, it is notable that the LC, the origin of the neural pathway controlling pupil size, strongly influences pupil dynamics via the NA system (Aston-Jones and Cohen, ). The heightened arousal induced by fear stimuli likely caused pupil dilation, which aligns with the psychological response of increased arousal measured using the TDMS (see Figure 2). Finally, the enhancement of occipital theta power during fear-inducing stimuli can be explained by the following mechanism: when the amygdala evaluates a stimulus as a threat, it provides attentional connections to the basal forebrain and various cortical areas (Pinto et al., ; Peck and Salzman, ; Chaves-Coira et al., ), leading to increased theta power in the temporal and occipital regions (Torrence et al., ). Our results align with this mechanism, showing enhanced occipital theta power in response to fear-inducing stimuli.
We discuss why pupil dilation exhibited the highest ability to distinguish between fear-induced and neutral stimuli, compared to RRI and occipital theta relative power. First, pupil size reflects neural activity in deep brain regions related to fear and arousal, such as the LC and amygdala (Johansson and Balkenius, ), which cannot be detected by EEG signals. Furthermore, the LC is directly involved in the neural pathway controlling pupil diameter, serving as its primary source. Therefore, pupil dilation more accurately reflects fear and arousal compared to other physiological signals.
It should be noted that, unlike EEG and RRI, pupil dilation was analyzed only during the post-stimulus blank period. This decision was made because the luminance of the fear-inducing and neutral images varied in order to maintain naturalistic and vivid presentation, which can significantly influence pupil size. Although some studies have attempted to correct for luminance-related effects using regression-based models (Hermans et al., ), such methods require careful calibration and validation, especially in dynamic visual environments. In this initial study, we prioritized isolating emotionally driven pupil responses by excluding the stimulation period altogether. Nevertheless, future work should consider incorporating luminance correction models or presenting stimuli with controlled brightness to enable more direct cross-modal comparisons of temporal dynamics.
One limitation of this study is the assumption that all images categorized as “neutral” or “fear-inducing” would elicit consistent emotional responses across participants. In particular, certain stimuli considered neutral (e.g., snakes or spiders) may evoke fear in some individuals, while others may not perceive fear-inducing images as strongly aversive. This variability in subjective perception could have influenced the distribution of behavioral responses, as reflected in the skewness observed in Figure 3. Future studies should include individual-level ratings of emotional valence for each image to better account for such inter-individual differences. Another limitation concerns the analysis of hemispheric asymmetries in neural and physiological signals. While we focused on averaged activity across hemispheres in the current study, previous research suggests that emotional processing may involve lateralized neural responses (Harmon-Jones and Gable, ). A thorough investigation of hemispheric effects would also require examining potential lateralization in other modalities such as pupil dynamics, which may be influenced by lateralized neural circuits such as the bilateral LCs (Samuels and Szabadi, ; Liu et al., ; Nobukawa et al., ; Kumano et al., ). However, clarifying these possibilities requires comprehensive multimodal analyses, such as examining hemispheric asymmetries in neural activity and pupil dynamics, which were beyond the scope of the present study. Future research should address this issue using a multimodal approach that explicitly incorporates hemisphere-specific analyses.
4.2.2 Multimodal response
We discuss why pupil dilation was observed exclusively during the post-stimulus period following the presentation of fear-inducing images. The LC-NA system strongly influences pupil size. In contrast, the enhancement of theta power in brain waves is affected by cortical feedback from the amygdala, while the increase in RRI is driven by vagal activity. Recent studies have shown that LC neurons exhibit two distinct firing modes: tonic discharge, which refers to a constant firing frequency, and phasic discharge, which occurs in response to specific stimuli or events. Under conditions of sustained stress, the LC transitions from a moderate tonic mode to a high tonic mode, and phasic mode activity is suppressed (Janitzky, ). Additionally, stress responses have been reported to prolong the activity of the NA system (Valentino and Van Bockstaele, ). Based on these findings, the sustained pupil dilation observed during the gray image period following the fear-inducing stimuli may be attributed to an acute stress state induced by the stimuli, which triggered a strong tonic mode in the LC. This sustained dilation likely elevated the baseline pupil size, with effects persisting for a relatively long duration. However, further studies are necessary to validate this hypothesis under experimental conditions that better isolate LC-NA system activity. Such investigations will be essential for a deeper understanding of this phenomenon.
Our findings confirmed that pupil dilation was the most sensitive single indicator of fear, consistent with previous literature on emotional arousal and sympathetic activation (Bradley et al., ; Laeng et al., ; Mathôt, ). While this result does not represent a novel discovery, its replication within a multimodal monitoring context provides empirical support for the robustness of pupillary measures (Mathôt, ). Importantly, the study contributes by systematically comparing pupil-based responses with EEG and heart rate measures under controlled conditions, highlighting the practical implications of modality selection for emotional state monitoring.
Regarding additional limitations of this study, although pupil dilation exhibited the highest accuracy in distinguishing which stimulus (fear-induced or neutral) was presented, combining it with EEG and RRI resulted in only limited accuracy improvement (see Figure 4). One possible reason for this limited synergy is that the three modalities may partially reflect overlapping physiological processes related to fear responses, reducing the incremental value of integration. Additionally, differences in temporal dynamics across modalities may have complicated effective integration. From an analytical perspective, our EEG analysis focused primarily on power changes at the sensor level. Importantly, we did not employ source localization or connectivity-based analyses, which could have provided deeper insights into the cortical or large-scale network dynamics associated with fear processing. As EEG and RRI reflect distinct aspects of neural activity beyond autonomic arousal, future studies should consider employing more advanced analytical approaches—such as source-level inference, functional connectivity analyses (ideally using high-density EEG), and complexity measures that capture non-linear dynamics indicative of large-scale neural interactions (Nobukawa et al., )—to better delineate the complementary contributions of these modalities in emotional state monitoring. In addition, heart rate variability (HRV) indices may offer more informative features than simple RRI measures and should be incorporated into future multimodal designs. More limitation of this study is that only male participants were included. This decision was made to control for hormonal fluctuations related to the menstrual cycle, which can affect stress reactivity and emotional processing. Previous studies have shown that hormonal phases can modulate amygdala activity and autonomic responses to emotional stimuli (Andreano and Cahill, ). To reduce variability in physiological responses, we chose a homogeneous sample in this initial investigation. However, future studies should incorporate female participants and consider controlling for menstrual cycle phase or hormonal levels to enhance the generalizability and inclusivity of the findings. Furthermore, while we evaluated classification performance using LOOCV, we did not report confidence intervals for the AUC scores. This is primarily due to the small sample size (N = 40), which limited the feasibility of reliable resampling-based confidence interval estimation. LOOCV was chosen to maximize training data per fold, a common approach in low-N settings, but it inherently lacks variance estimation across folds. Future studies with larger cohorts should adopt cross-validation schemes that allow for statistical resampling (e.g., k-fold cross-validation with bootstrapping) to quantify uncertainty and improve the robustness of classification metrics.
4.2.3 Potential applications of multimodal fear detection
Despite the limited accuracy gains from multimodal integration observed in this study, the use of multiple physiological measures remains promising for future applications in emotional state monitoring. While each modality—pupil size, EEG, and heart rate—may partly reflect overlapping processes (Nobukawa et al., ), they also capture distinct neurophysiological dimensions of fear, such as sympathetic activation, cortical arousal, and parasympathetic regulation. These complementary features may prove advantageous in more ecologically valid or high-stakes settings where robustness is critical.
Potential applications of such multimodal monitoring include clinical domains (e.g., assessment of preoperative anxiety or PTSD) (Senaratne et al., ), or safety-critical environments (e.g., driver monitoring, aviation) (Rastgoo et al., ). Furthermore, with advancements in wearable technologies and data fusion algorithms, it may become feasible to implement such systems in real-world contexts, allowing for continuous, adaptive support based on an individual's affective state. Future research should explore how to optimally leverage the complementary features of each modality to improve emotion monitoring in real-world applications.
5 Conclusions
In this study, we evaluated psychological and multimodal physiological responses to fear-induced stimuli. The results revealed characteristic responses in each modality, including heart rate, pupil size, and EEG, which likely reflect processes occurring in different neural systems. These findings suggest the potential for future applications in monitoring an individual's fear state by integrating such multimodal neural responses.
Statements
Data availability statement
The datasets presented in this article are not readily available because the informed consent did not include the declaration regarding publication of data. Requests to access the datasets should be directed to nobukawa@cs.it-chiba.ac.jp.
Ethics statement
The studies involving humans were approved by Ethics Committee of the Chiba Institute of Technology. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.
Author contributions
YE: Visualization, Formal analysis, Writing – original draft, Data curation, Software, Investigation, Validation, Methodology, Resources, Conceptualization, Writing – review & editing. TS: Data curation, Software, Investigation, Writing – review & editing, Resources, Conceptualization, Validation, Writing – original draft, Formal analysis, Visualization, Methodology. IW: Software, Data curation, Writing – review & editing. AU: Visualization, Writing – review & editing, Formal analysis. TI: Investigation, Writing – review & editing, Conceptualization. TT: Investigation, Validation, Writing – review & editing, Methodology, Funding acquisition, Project administration, Conceptualization. SN: Resources, Validation, Project administration, Funding acquisition, Formal analysis, Supervision, Writing – review & editing, Data curation, Investigation, Visualization, Methodology, Writing – original draft, Conceptualization.
Funding
The author(s) declare that financial support was received for the research and/or publication of this article. This work was supported by JSPS KAKENHI: Grant-in-Aid for Scientific Research (C) (JP22K12183 [SN], JP23K06983 [TT]), Grant-in-Aid for Scientific Research (B) (JP25K03198 [SN]), and Grant-in-Aid for Transformative Research Areas (A) (JP20H05921 and JP25H02626 [SN]). Additional financial support was provided by TOPPAN Edge Inc. under a collaborative research agreement. TOPPAN Edge Inc. was not involved in the study design, collection, analysis, interpretation of data, the writing of this article, or the decision to submit it for publication.
Acknowledgments
At the final proofing stage of this manuscript, we utilized the services of Editage for English language editing (www.editage.com).
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Generative AI statement
The author(s) declare that Gen AI was used in the creation of this manuscript. Portions of the text, including language refinement and editing, were generated with the assistance of ChatGPT-4, an AI language model developed by OpenAI.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
1
AndersC.ArnrichB. (2022). Wearable electroencephalography and multi-modal mental state classification: a systematic literature review. Comput. Biol. Med. 150:106088. 10.1016/j.compbiomed.2022.106088
2
AndreanoJ. M.CahillL. (2010). Menstrual cycle modulation of medial temporal activity evoked by negative emotion. Neuroimage53, 1286–1293. 10.1016/j.neuroimage.2010.07.011
3
AsbrandJ.VögeleC.HeinrichsN.NitschkeK.Tuschen-CaffierB. (2022). Autonomic dysregulation in child social anxiety disorder: an experimental design using CBT treatment. Appl. Psychophysiol. Biofeedback 47m 199–212. 10.1007/s10484-022-09548-0
4
Aston-JonesG.CohenJ. D. (2005). An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annu. Rev. Neurosci. 28, 403–450. 10.1146/annurev.neuro.28.061604.135709
5
BlanchardD. C.BlanchardR. J. (2008). Defensive behaviors, fear, and anxiety. Handbook Behav. Neurosci. 17, 63–79. 10.1016/S1569-7339(07)00005-7
6
BradleyM. M.MiccoliL.EscrigM. A.LangP. J. (2008). The pupil as a measure of emotional arousal and autonomic activation. Psychophysiology45, 602–607. 10.1111/j.1469-8986.2008.00654.x
7
Chaves-CoiraI.Rodrigo-AnguloM. L.Nu nezA. (2018). Bilateral pathways from the basal forebrain to sensory cortices may contribute to synchronous sensory processing. Front. Neuroanat. 12:5. 10.3389/fnana.2018.00005
8
ChenS.TanZ.XiaW.GomesC. A.ZhangX.ZhouW.et al. (2021). Theta oscillations synchronize human medial prefrontal cortex and amygdala during fear learning. Sci. Adv. 7:eabf4198. 10.1126/sciadv.abf4198
9
ChienJ.CollocaL.KorzeniewskaA.ChengJ. J.CampbellC.HillisA.et al. (2017). Oscillatory eeg activity induced by conditioning stimuli during fear conditioning reflects salience and valence of these stimuli more than expectancy. Neuroscience346, 81–93. 10.1016/j.neuroscience.2016.12.047
10
CohenJ. (2013). Statistical Power Analysis for the Behavioral Sciences. London: Routledge.
11
DavisM.WalkerD. L.MilesL.GrillonC. (2010). Phasic vs sustained fear in rats and humans: role of the extended amygdala in fear vs anxiety. Neuropsychopharmacology35, 105–135. 10.1038/npp.2009.109
12
Domínguez-BorràsJ.VuilleumierP. (2022). Amygdala function in emotion, cognition, and behavior. Handb. Clin. Neurol. 187, 359–380. 10.1016/B978-0-12-823493-8.00015-8
13
FriedmanB. H. (2007). An autonomic flexibility-neurovisceral integration model of anxiety and cardiac vagal tone. Biol. Psychol. 74, 185–199. 10.1016/j.biopsycho.2005.08.009
14
GrupeD. W.NitschkeJ. B. (2013). Uncertainty and anticipation in anxiety: an integrated neurobiological and psychological perspective. Nat. Rev. Neurosci. 14, 488–501. 10.1038/nrn3524
15
HajcakG.MacNamaraA.OlvetD. M. (2010). Event-related potentials, emotion, and emotion regulation: an integrative review. Dev. Neuropsychol. 35, 129–155. 10.1080/87565640903526504
16
Harmon-JonesE.GableP. A. (2018). On the role of asymmetric frontal cortical activity in approach and withdrawal motivation: an updated review of the evidence. Psychophysiology55:e12879. 10.1111/psyp.12879
17
HermansE. J.HenckensM. J.RoelofsK.FernándezG. (2013). Fear bradycardia and activation of the human periaqueductal grey. Neuroimage66, 278–287. 10.1016/j.neuroimage.2012.10.063
18
JanitzkyK. (2020). Impaired phasic discharge of locus coeruleus neurons based on persistent high tonic discharge—a new hypothesis with potential implications for neurodegenerative diseases. Front. Neurol. 11:371. 10.3389/fneur.2020.00371
19
JohanssonB.BalkeniusC. (2018). A computational model of pupil dilation. Conn. Sci. 30:5–19. 10.1080/09540091.2016.1271401
20
KeilA.BradleyM. M.HaukO.RockstrohB.ElbertT.LangP. J. (2002). Large-scale neural correlates of affective picture processing. Psychophysiology39, 641–649. 10.1111/1469-8986.3950641
21
KralT.WilliamsC.WylieA.McLaughlinK.StephensR.Mills-KoonceW.et al. (2024). Intergenerational effects of racism on amygdala and hippocampus resting state functional connectivity. Sci. Rep. 14:17034. 10.1038/s41598-024-66830-3
22
KumanoH.NobukawaS.ShiramaA.TakahashiT.TakedaT.OhtaH.et al. (2022). Asymmetric complexity in a pupil control model with laterally imbalanced neural activity in the locus coeruleus: a potential biomarker for attention-deficit/hyperactivity disorder. Neural Comput. 34, 2388–2407. 10.1162/neco_a_01545
23
LaengB.SiroisS.GredebäckG. (2012). Pupillometry: A window to the preconscious?Perspect. Psychol. Sci. 7, 18–27. 10.1177/1745691611427305
24
LealP. C.GoesT. C.da SilvaL. C. F.Teixeira-SilvaF. (2017). Trait vs. state anxiety in different threatening situations. Trends Psychiat. Psychother. 39, 147–157. 10.1590/2237-6089-2016-0044
25
LeDouxJ. E. (2013). Emotion circuits in the brain. Fear Anxiety. 2013, 259–288. 10.1146/annurev.neuro.23.1.155
26
LeuchsL.SchneiderM.CzischM.SpoormakerV. I. (2017). Neural correlates of pupil dilation during human fear learning. Neuroimage147, 186–197. 10.1016/j.neuroimage.2016.11.072
27
LiuY.RodenkirchC.MoskowitzN.SchriverB.WangQ. (2017). Dynamic lateralization of pupil dilation evoked by locus coeruleus activation results from sympathetic, not parasympathetic, contributions. Cell Rep. 20, 3099–3112. 10.1016/j.celrep.2017.08.094
28
MalezieuxM.KleinA. S.GogollaN. (2023). Neural circuits for emotion. Annu. Rev. Neurosci. 46, 211–231. 10.1146/annurev-neuro-111020-103314
29
MathôtS. (2018). Pupillometry: psychology, physiology, and function. J. Cognition1:1. 10.5334/joc.18
30
NobukawaS.ShiramaA.TakahashiT.TakedaT.OhtaH.KikuchiM.et al. (2021). Pupillometric complexity and symmetricity follow inverted-u curves against baseline diameter due to crossed locus coeruleus projections to the edinger-westphal nucleus. Front. Physiol. 12:92. 10.3389/fphys.2021.614479
31
NobukawaS.ShiramaA.TakahashiT.TodaS. (2024). Recent trends in multiple metrics and multimodal analysis for neural activity and pupillometry. Front. Neurol. 15:1489822. 10.3389/fneur.2024.1489822
32
PeckC. J.SalzmanC. D. (2014). The amygdala and basal forebrain as a pathway for motivationally guided attention. J. Neurosci. 34, 13757–13767. 10.1523/JNEUROSCI.2106-14.2014
33
PhelpsE. A.LeDouxJ. E. (2005). Contributions of the amygdala to emotion processing: from animal models to human behavior. Neuron48, 175–187. 10.1016/j.neuron.2005.09.025
34
PintoL.GoardM. J.EstandianD.XuM.KwanA. C.LeeS.-H.et al. (2013). Fast modulation of visual perception by basal forebrain cholinergic neurons. Nat. Neurosci. 16, 1857–1863. 10.1038/nn.3552
35
PittigA.TreanorM.LeBeauR. T.CraskeM. G. (2018). The role of associative fear and avoidance learning in anxiety disorders: gaps and directions for future research. Neurosci. Biobehav. Rev. 88, 117–140. 10.1016/j.neubiorev.2018.03.015
36
PorgesS. W. (2007). The polyvagal perspective. Biol. Psychol. 74, 116–143. 10.1016/j.biopsycho.2006.06.009
37
RafiqueS.KanwalN.KaramatI.AsgharM. N.FleuryM. (2020). Towards estimation of emotions from eye pupillometry with low-cost devices. IEEE Access9, 5354–5370. 10.1109/ACCESS.2020.3048311
38
RastgooM. N.NakisaB.RakotonirainyA.ChandranV.TjondronegoroD. (2018). A critical review of proactive detection of driver stress levels based on multimodal measurements. ACM Comp. Surv. 51, 1–35. 10.1145/3186585
39
RichardsonM. P.StrangeB. A.DolanR. J. (2004). Encoding of emotional memories depends on amygdala and hippocampus and their interactions. Nat. Neurosci. 7, 278–285. 10.1038/nn1190
40
SakairiY.NakatsukaK.ShimizuT. (2013). Development of the two-dimensional mood scale for self-monitoring and self-regulation of momentary mood states. Jap. Psychol. Res. 55, 338–349. 10.1111/jpr.12021
41
SamuelsE. R.SzabadiE. (2008). Functional neuroanatomy of the noradrenergic locus coeruleus: its roles in the regulation of arousal and autonomic function part i: principles of functional organisation. Curr. Neuropharmacol. 6, 235–253. 10.2174/157015908785777229
42
SenaratneH.OviattS.EllisK.MelvinG. (2022). A critical review of multimodal-multisensor analytics for anxiety assessment. ACM Trans. Comp. Healthc. 3, 1–42. 10.1145/3556980
43
SheikhM.QassemM.KyriacouP. A. (2021). Wearable, environmental, and smartphone-based passive sensing for mental health monitoring. Front. Digital Health3:662811. 10.3389/fdgth.2021.662811
44
SperlM. F.PanitzC.RossoI. M.DillonD. G.KumarP.HermannA.et al. (2019). Fear extinction recall modulates human frontomedial theta and amygdala activity. Cereb. Cortex29, 701–715. 10.1093/cercor/bhx353
45
SpielbergerC. D.Gonzalez-ReigosaF.Martinez-UrrutiaA.NatalicioL. F.NatalicioD. S. (1971). The state-trait anxiety inventory. Interamerican J. Psychol. 5, 3–4.
46
ThayerJ. F.LaneR. D. (2000). A model of neurovisceral integration in emotion regulation and dysregulation. J. Affect. Disord. 61, 201–216. 10.1016/S0165-0327(00)00338-4
47
TorrenceR. D.TroupL. J.RojasD. C.CarlsonJ. M. (2021). Enhanced contralateral theta oscillations and n170 amplitudes in occipitotemporal scalp regions underlie attentional bias to fearful faces. Int. J. Psychophysiol. 165, 84–91. 10.1016/j.ijpsycho.2021.04.002
48
ValentinoR. J.Van BockstaeleE. (2008). Convergent regulation of locus coeruleus activity as an adaptive response to stress. Eur. J. Pharmacol. 583, 194–203. 10.1016/j.ejphar.2007.11.062
49
WilliamsJ. E.PykettJ. (2022). Mental health monitoring apps for depression and anxiety in children and young people: A scoping review and critical ecological analysis. Soc. Sci. Med. 297:114802. 10.1016/j.socscimed.2022.114802
Summary
Keywords
electroencephalography, fear monitoring, multimodal measurement, pupil dynamics, heart rate
Citation
Ebato Y, Saki T, Wakita I, Ueno A, Ishibashi T, Takahashi T and Nobukawa S (2025) Evaluating multimodal physiological signals for fear detection: relative utility of pupillometry, heart rate, and EEG. Front. Hum. Neurosci. 19:1605577. doi: 10.3389/fnhum.2025.1605577
Received
03 April 2025
Accepted
30 June 2025
Published
29 July 2025
Volume
19 - 2025
Edited by
Kuldeep Singh, Guru Nanak Dev University, India
Reviewed by
Amadeu Quelhas Martins, European University of Lisbon, Portugal
Fatemeh Hadaeghi, University Medical Center Hamburg-Eppendorf, Germany
Updates
Copyright
© 2025 Ebato, Saki, Wakita, Ueno, Ishibashi, Takahashi and Nobukawa.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Sou Nobukawa nobukawa@cs.it-chiba.ac.jp
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.