Edited by: Antonio Fernández-Caballero, University of Castilla-La Mancha, Spain
Reviewed by: Rajesh Singla, Dr. B. R. Ambedkar National Institute of Technology Jalandhar, India; Zafer Iscan, Bahçeşehir University, Turkey
†These authors share first authorship
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
Neurocinematics is an emerging discipline in neuroscience, which aims to provide new filmmaking techniques by analyzing the brain activities of a group of audiences. Several neurocinematics studies attempted to track temporal changes in mental states during movie screening; however, it is still needed to develop efficient and robust electroencephalography (EEG) features for tracking brain states precisely over a long period. This study proposes a novel method for estimating emotional arousal changes in a group of individuals during movie screening by employing steady-state visual evoked potential (SSVEP), which is a widely used EEG response elicited by the presentation of periodic visual stimuli. Previous studies have reported that the emotional arousal of each individual modulates the strength of SSVEP responses. Based on this phenomenon, movie clips were superimposed on a background, eliciting an SSVEP response with a specific frequency. Two emotionally arousing movie clips were presented to six healthy male participants, while EEG signals were recorded from the occipital channels. We then investigated whether the movie scenes that elicited higher SSVEP responses coincided well with those rated as the most impressive scenes by 37 viewers in a separate experimental session. Our results showed that the SSVEP response averaged across six participants could accurately predict the overall impressiveness of each movie, evaluated with a much larger group of individuals.
Neurocinematics is an emerging interdisciplinary research field that aims to provide a new method to quantitatively evaluate films or movie contents by employing neuroscientific techniques to analyze viewers’ brain activities during movie screening (
Although fMRI-based neurocinematics studies have shown promising results and new perspectives, these studies pose numerous limitations for use in practical applications (
In this study, we propose a novel method to continuously track temporal changes in emotional arousal during movie screening using only three electrodes. We employed a characteristic brain response known as steady-state visual evoked potential (SSVEP), a train of periodic EEG waves elicited by visual stimuli flickering or reversing at a specific frequency. SSVEP has been widely used in brain-computer interface (BCI) applications since the early days of BCI studies (
Six healthy right-handed adults with normal or corrected-to-normal vision served as paid volunteer participants. All participants were adult males, with an average age of 22.3 ± 1.7 years. None of the participants had a history of neurological or psychiatric disease. Following a thorough explanation of the study protocol, all participants signed an informed consent form. All experimental protocols were approved by the institutional review board of Hanyang University (IRB No. HYI-14-167-11) according to the Declaration of Helsinki.
Two 5-min video clips were extracted from emotionally arousing movies: “Ju-on 2” (2003) and “Bang! You’re Dead” (1985) to elicit fearful and nervous emotions, respectively. First, “Ju-on 2,” a world-renowned Japanese horror film, was edited to include both scary and non-scary scenes for this experiment. To maximize the fearful emotions of the viewers, the scary scenes included the sudden appearance of specters with thrilling background music. Non-scary scenes, mostly consisting of everyday scenes that do not contain any specter or thrilling background music, played a role in bridging the events required to form a complete story. “Bang! You’re Dead” was a famous TV series of a film director Alfred Hitchcock, which has been employed in previous neurocinematics studies (
In our experiments, a new visual stimulus was used to continuously evoke the SSVEP response at a specific frequency, while the participants watched the designated video clips. The visual stimulus consisted of small-sized colored dots randomly scattered to cover the entire screen. The size of each dot was set as small as possible (single pixels in this study) to minimize the distraction of the viewers. Because the red/green-colored chromatic visual stimulus was reported to cause less eye fatigue than the conventional white/black-colored stimulus (
Procedure to generate the experimental videos employed in this study.
The participants sat on a comfortable armchair and watched the clips presented on a 24″ LCD monitor screen 70 cm away from them. The EEG data were recorded while the participants watched the clips, and they reported the two most emotionally impressive (e.g., most fearful or most nervous) scenes after watching each video clip. The participants were instructed to select two scenes in each video clip because the participants could have difficulty selecting more than two impressive scenes in such a short video. The order of movie screening was randomized across participants. Additional 31 participants of similar age and gender were also presented with the two video clips without EEG recording. The weblinks to the two video clips were provided to them; then, each participant responded to the two most emotionally impressive scenes in each movie clip through an online poll.
The EEG data were recorded at a 2,048 Hz sampling rate from three scalp electrodes (O1, Oz, and O2) using a commercial biosignal recording system (BioSemi ActiveTwo, Biosemi, Amsterdam, Netherlands). The ground and reference electrodes were replaced with a common mode sense active electrode and a driven right leg passive electrode, both of which were located in the posterior region. The recorded EEG data were down-sampled to 512 Hz and re-referenced to Cz. Since we only analyzed the changes of the SSVEP response at the stimulation frequency, no other artifact elimination techniques were applied. The temporal changes in the spectral power at 6 Hz were evaluated using fast Fourier transform (FFT) with a sliding 10-s moving window and a 90% overlap. Then, the spectral power series was z-score normalized and grand averaged over the O1, Oz, and O2 channels and all experimental participants.
Polynomial regression was employed to estimate the overall trend of temporal changes in spectral power from small-sized samples. The order of the polynomial regression was determined by finding the lowest order in which the root-mean-square error (RMSE) of the polynomial regression reduced to less than 0.9 α, when α indicates the RMSE of 1st-order polynomial regression.
The RMSE changes as a function of polynomial order for approximating SSVEP responses for
Lastly, the regression results were qualitatively and quantitatively compared with the survey results; the survey consisted of questions regarding the most impressive scenes. To compare the regression results with the survey results, a moving average with a window size of three samples was applied to the survey results. Then, to evaluate the correlations between the regression results and survey results, we computed Pearson’s correlation coefficient. Because SSVEP regression results had a total of 270 data points while the survey results had only 27 data points, we averaged every 10 SSVEP regression results to match the number of data points in both data. Please note that moving average was not applied to the regression results of SSVEP response.
To validate whether the SSVEP was successfully evoked by the newly proposed stimuli, we calculated the grand averaged spectral power of the recorded EEG data during “fear” and “nervous” movie screening (
Grand averaged frequency spectra and spectrograms evaluated with the EEG data recorded during
Grand averaged polynomial regression result of SSVEP response at 6 Hz during “fear” movie screening. The minima corresponded with relatively ordinary scenes, while the maxima corresponded to the scenes with appearance of specter. The scenes corresponding to each local extrema can be found using the time stamps in the boxes. The time stamps correspond to the time points in the following movie clip:
Grand averaged polynomial regression result of SSVEP response at 6 Hz during “nervous” movie screening. The minima corresponded to the scenes that play a role in connecting the events to proceed the story, while the maxima corresponded to the scenes in which a boy was loading or triggering a gun. The scenes corresponding to each local extrema can be found using the time stamps in the boxes. The time stamps correspond to the time points in the following movie clip:
Then, the regression results were compared with the survey results of most impressive scenes with respect to each movie to evaluate whether the response of the larger group can be predicted using the proposed method (
Comparison of the polynomial regression results of SSVEP response to the survey results of the most emotionally arousing scenes in the clips, for the
In this study, we proposed a new method for estimating emotional arousal changes in a group of individuals during video screening by employing a novel visual stimulation method that overlays a background SSVEP stimulus on a video clip. We first confirmed that the SSVEP response was stably evoked by the proposed visual stimulus throughout the screening of the movie clip (
In the previous studies on the estimation of emotional state changes using physiological signals had some limitations. For example,
In previous studies on SSVEP changes in response to brain state changes, different methods were implemented to evoke SSVEP responses. First,
In the film editing stage, filmmakers must know whether the target emotion was successfully induced to the viewers as they had intended, because simply changing the sequence of some scenes of a film can significantly influence viewers’ emotional responses (
Recently, the decoding of emotional states from EEG has attracted increased attention because of the popularization of low-cost wearable EEG systems (
However, some potential issues exist that must be addressed before the proposed method can be used in practical applications. First, there are people who do not generate sufficiently large SSVEP responses even when identical flickering or reversing visual stimuli are presented (
In this study, all subjects who participated in the EEG recording experiment were males and the number of the participants was relatively small (only six in this study), because some participants that we had recruited were excluded as they did not exhibit clear SSVEP response peak at 6 Hz. In our experiments, there were more than six participants (eight participants) whose EEG signals did not exhibit clear SSVEP peaks particularly because our background visual stimuli were too weak to evoke SSVEP responses in some participants. Nevertheless, we did not recruit more numbers of participants because it was thought that estimating the grand trend in the emotional arousal changes of much larger numbers of participants (37 viewers in this study) with such a small number of participants would rather strengthen the practical usability of our method. That is, our study might suggest that the proposed method can effectively reduce the number of participants required for an evaluation of movies. Furthermore, given that the females are more sensitive to emotionally unpleasant stimuli than males (
In addition, we applied z-score normalization to individual SSVEP responses before grand averaging, to prevent the averaged SSVEP results from being biased by data of certain participants. In
Comparison between grand averaged and individual polynomial regression results of SSVEP response at 6 Hz.
In summary, we proposed a new method for estimating emotional arousal changes in a group of individuals during movie screening by overlaying SSVEP stimuli on the original video clips. Our experimental results agreed well with previous SSVEP studies that reported a positive correlation between the SSVEP amplitude and emotional arousal. This method might be used to track continuous emotional arousal changes in viewers during movie screening and to evaluate whether the movie has been edited well as intended by filmmakers.
The raw EEG datasets and survey results are available at the following link:
The studies involving human participants were reviewed and approved by the institutional review board of Hanyang University. The patients/participants provided their written informed consent to participate in this study.
D-WK and C-HI designed the study. SP and D-WK conducted the experiment and data analyses. C-HH helped to perform the analyses through the constructive discussions. SP and C-HI wrote the manuscript. All authors reviewed and approved the final manuscript.
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.