Skip to main content

ORIGINAL RESEARCH article

Front. Physiol., 10 October 2018
Sec. Autonomic Neuroscience

Infrared Camera-Based Non-contact Measurement of Brain Activity From Pupillary Rhythms

  • 1Industry-Academy Cooperation Team, Sangmyung University, Seoul, South Korea
  • 2Department of Intelligent Engineering Informatics for Human, Sangmyung University, Seoul, South Korea

Pupillary responses are associated with affective processing, cognitive function, perception, memory, attention, and other brain activities involving neural pathways. The present study aimed to develop a noncontact system to measure brain activity based on pupillary rhythms using an infra-red web camera. Electroencephalogram (EEG) signals and pupil imaging of 70 undergraduate volunteers (35 female, 35 male) were measured in response to sound stimuli designed to evoke arousal, relaxation, happiness, sadness, or neutral responses. This study successfully developed a real-time system that could detect an EEG spectral index (relative power: low beta in FP1; mid beta in FP1; SMR in FP1; beta in F3; high beta in F8; gamma P4; mu in C4) from pupillary rhythms using the synchronization phenomenon in harmonic frequency (1/100 f) between the pupil and brain oscillations. This method was effective in measuring and evaluating brain activity using a simple, low-cost, noncontact system, and may be an alternative to previous methods used to evaluate brain activity.

Introduction

Human vital signs have been used to assess human behavior in various fields, including ubiquitous health care, emotional information and communication technology, emotional recognition and engineering, human computer interaction, and security (Park et al., 2018). However, sensor-based measurement methods have a significant disadvantage because attachment of sensor devices to the human body is required. Subjects may experience pressure and discomfort when sensors are attached to their body or skin, and these limitations restrict their direct application in industrial and medical fields. Moreover, these sensors are expensive and bulky.

Vital sign monitoring technology has recently been incorporated into wearable wireless devices and have advanced the development of portable measuring equipment. These wearable devices should be manufactured in the form of accessory items, such as watches, bracelets, or glasses that measure heart rate (HR), respiration, skin temperature, and galvanic skin responses. These devices have the advantages of convenience and less burden on users for sensor attachment compared with previous wire-based sensors. However, these sensors are expensive and are susceptible to noise caused by movement. Thus, noncontact, low-cost measurement methods to detect physiological signals unaffected by movement for various applications in industrial and medical fields are required (Boric-Lubeke and Lubecke, 2002; Suzuki et al., 2009; Poh et al., 2011; Scully et al., 2012; Balakrishnan et al., 2013; Tarassenko et al., 2014; Park et al., 2018).

Considering these limitations and requirements, noncontact measurement methods have recently been proposed and developed by many researchers. Previous studies have measured cardiac activity and respiration using radar, thermal imaging, web-camera (web-cam), and infrared (IR) camera. Microwave and Doppler radars measure HR and respiration rate by sensing movement of the chest during inspiration and expiration (Droitcour et al., 2001; Lohman et al., 2002; Uenoyama et al., 2006; Fletcher and Han, 2009; Li et al., 2009; Suzuki et al., 2009; Gu et al., 2010; Kao et al., 2013; Lim et al., 2015). Thermal imaging uses temperature variations in the wrist, face (nasal), and neck (carotid artery and jugular vein) caused by blood flow moving from the heart to the brain to monitor HR (Chekmenev et al., 2005, 2009; Sun et al., 2006; Garbey et al., 2007; Gault et al., 2010) and respiration rate (Chekmenev et al., 2005; Murthy and Pavlidis, 2005; Murthy et al., 2009; Fei and Pavlidis, 2010; Abbas et al., 2011). Web-cams detect color variations in the face, and micro-movements in the head and chest caused by blood flow and expiration-inspiration, and extract information related to cardiac activity and respiration from these factors (Poh et al., 2011; Balakrishnan et al., 2013; Holton et al., 2013; Janssen et al., 2015). Pupillary rhythm measured using IR cameras is a reflection of autonomic nervous system (ANS) activity, and related to cardiac activity and blood pressure (Loewenfeld and Lowenstein, 1993; Steinhauer et al., 2000; Verney et al., 2001; Siegle et al., 2004; Park et al., 2018). Many previous studies have reported that pupillary rhythm is synchronized with HR variability (Calcagnini et al., 1997; Hung and Zhang, 2006; Onorati et al., 2013; Parnandi and Gutierrez-Osuna, 2013; Park et al., 2018), blood pressure variation (Hung and Zhang, 2006; Bär et al., 2009), and respiration rhythm (Calcagnini et al., 1997). Previous studies have developed methods for non-contact measurement of physiological signals using each of the above modalities. These technologies can accurately measure multiple physiological cardiac-, respiratory-, and blood pressure-related parameters through a simple, low cost, non-contact measurement system, which have applications in various fields of industry and medicine. However, previous studies have not considered brain activity, such as electroencephalogram (EEG) spectra, for noncontact measurement. The EEG spectrum is also a human physiological response that is as highly used and extensively studied as other physiological signals. However, measuring EEG signals is an inconvenient, complex and, at times, onerous process, and requires the user to wear specialized sensors. Therefore, noncontact technology in this context requires significant research effort.

The behavior of the pupils reflects physiological and neurological mechanisms in humans, and responses in the pupil can be measured using cameras (i.e., noncontact). Additionally, the pupillary response is strongly related to central nervous system activity (i.e., functional brain processing) and based on afferent and efferent neural-pathways (Kahneman and Beatty, 1966; Beatty and Lucero-Wagoner, 2000; Fotiou et al., 2000; Partala and Surakka, 2003; Kojima et al., 2004; Kozicz et al., 2011; Prehn et al., 2011; Júnior et al., 2015). In the afferent pathway (i.e., CN II) from retinal receptors, when the pupil is stimulated by light input, the retina (rod and cone cells) synapse with their respective bipolar cells (the primary neuron), and this signal transmits to ganglion cells (the secondary neuron). And then the ganglion cell axons project an electrical signal through the optic nerve to the ipsilateral pretectal nucleus of the mid-brain. In the efferent pathway (i.e., CN III) from the mid brain, the pretectal nucleus is connected to the Edinger-Westphal nucleus (Loewenfeld and Lowenstein, 1993; Steinhauer et al., 2000; Verney et al., 2001; Siegle et al., 2004; Prehn et al., 2011). For example, the pupillary light reflex is known to be a clinical indicator for assessing mid-brain function such as the connectivity between the Edinger-Westphal nucleus and sphincter muscle in the pupil (Fotiou et al., 2000; Kojima et al., 2004; Kozicz et al., 2011; Júnior et al., 2015), and it is related to the stimulation from the pupillary smooth muscles of the sphincter muscle to cause pupil contraction (Maggs et al., 2012). In addition, previous studies have reported that pupillary rhythm (pupil diameter) is related to affective processing (Surakka et al., 1998; Partala and Surakka, 2003), cognitive function (Hess and Polt, 1964; Kahneman and Beatty, 1966; Ahern and Beatty, 1979; Just and Carpenter, 1993; Just et al., 1996; Klingner et al., 2008), perception (Hakerem and Sutton, 1966; Norman and Bobrow, 1975), memory (Beatty and Kahneman, 1966; Kahneman and Beatty, 1966; Just and Carpenter, 1993; Just et al., 1996), attention (Hink et al., 1977; Beatty, 1988), and brain activity (Just and Carpenter, 1993; Just et al., 1996).

The EEG spectrum is known to be an indicator in assessing brain processing in the above-mentioned areas (van Beijsterveldt and Boomsma, 1994; Dietrich and Kanso, 2010), and these brain functions have been shown to be reflected in pupillary rhythms in previous studies. We hypothesized that pupillary rhythm is synchronized and connected with brain activity. The aim of this study, therefore, was to measure brain activity (reflected by EEG spectra) by examining pupillary rhythm using a vision-based, noncontact measurement system. For this purpose, this study analyzed synchronization in each frequency range (e.g., delta, theta, alpha, and beta) between the pupil and EEG rhythms based on harmonic frequency. The technique represents a new and more advanced non-contact method of measuring brain activity that uses an IR web-cam to image the pupils.

Methods

Participants

Seventy undergraduate volunteers (35 male, 35 female), ranging in age from 22 to 28 years (mean age, 23.46 ± 0.82 years), participated in the present study. All participants were right-handed, and had normal or corrected-to-normal vision (i.e., participant's vision >0.8), and no family or medical history of disease affecting their visual functionality or central nervous system. In addition, participants were required to abstain from alcohol, smoking, and caffeine for at least 12 h before the experiment, and to sleep according to their normal schedule. The participants were notified of the above restrictions and requirements, and provided informed written consent before being tested. This study was carried out in accordance with the recommendations of Sangmyung University Institutional Bioethics Review Board (SMUIBRB), Seoul, South Korea (No. BE2015-7-1). The protocol was approved by the SMUIBRB.

Stimuli, Experimental Protocol, and Data Acquisition

The experiment involved reference and main-task stimuli. The reference was presented for 3 min before the main task to stabilize physiological state without stimulus. In the main task, sound stimuli invoking arousal, relaxation, positive, negative, and neutral responses were presented in random order to participants for 5 min, with each stimulus lasting 1 min. The stimulus was selected to be sound instead of video to minimize the effect of illumination on the pupil. The participants gazed at a black wall, placed at a distance of 1.5 m, while sitting in a comfortable chair listening to the sound stimuli. Pupil imaging and EEG signals were measured simultaneously during the experiment, as shown in Figure 1. The black wall and sound stimuli were selected to minimize the influence of illumination changes on pupil size variation (PSV) because pupil size is strongly affected by these variations.

FIGURE 1
www.frontiersin.org

Figure 1. Experimental protocol and configuration. EEG, electroencephalogram; IR, infra-red.

This study considered changes in physiological state in the experimental design because the EEG spectrum is well known to be strongly influenced by physiological conditions. Stimuli comprised five components including arousal, relaxation, and positive, negative, and neutral based on the cir-complex model proposed by Russell (1979) to provoke physiological variation in the participants. Stimuli causing physiological state variation were similar to those applied in previous studies with the following specifications: the stimuli sources were recorded for each emotion; the stimuli were categorized into five groups (arousal, relaxation, positive, negative, and neutral); a chi-squared test to assess goodness-of-fit, based on subjective ratings from 150 subjects; principal component analysis was based on varimax (orthogonal) rotation from 150 subjects (seven-point scale); and representative stimuli for each emotion were selected (sound files in Supplementary Material; Park et al., 2018). The EEG signals were recorded at a 500 Hz sampling rate from 19 channels (FP1, FP2, F3, Fz, F4, F7, F8, C3, Cz, C4, T7 [T3], T8 [T4], P7 [T5], P8 [T6], P3, Pz, P4, O1, and O2 regions) based on the international 10–20 system (ground: FAz; reference: average between electrodes on the two ears; and DC level: 0–150 Hz) using an EEG device (202, Mitsar Inc., Russia). Pupil images were recorded at 125 fps with a resolution of 960 × 400 using an IR camera (GS3-U3-23S6M-C, Point Gray Research Inc., Canada).

Assessment of Brain Activity From Pupillary Response

Detecting Brain Activity From EEG Signals (Ground Truth)

EEG signals were processed using a band pass filter (BPF, Butterworth type of order 6) of 1–50 Hz, and the EEG spectrum was analyzed using a fast Fourier transform (FFT) method. The EEG spectrum was divided according to frequency band into the following ranges: delta 1–4 Hz; theta 4–8 Hz; alpha 8–13 Hz; beta 13–30 Hz; gamma 30–50 Hz; slow alpha 8–11 Hz; fast alpha 11–13 Hz; low beta 12–15 Hz; mid beta 15–20 Hz, high beta 20–30 Hz, mu 9–11 Hz; the SMR 12.5–15.5 Hz; and total power 1–50 Hz (Ramaekers et al., 1992; Deuschl and Eisen, 1999; Berta et al., 2013; Shin et al., 2014; Novais and Konomi, 2016). Each band power was extracted. The relative powers from each of the frequency bands were then calculated using the ratio between the total band power and each band power. This procedure was executed using the sliding moving average with a window size of 30 s and a shift of 1 s, as shown in Figure 2.

FIGURE 2
www.frontiersin.org

Figure 2. Signal processing for detecting the electroencephalogram (EEG) spectral index from pupillary response and electrocardiograph signals. (A) Pupil diameter at 125 fps. (B) Resampled pupil diameter at 1 Hz based on sliding moving average (window size: 125 fps and shift: 125 fps). (C) Band pass filtered signals of each frequency band from the delta to SMR band with harmonic frequency of 1/100f. (D) Spectral power for each band from pupillary rhythm using fast Fourier transform analysis. (E) Relative powers calculated using the ratio between total and each band power. (F) Raw EEG signals. (G) Band pass filtered signals of 1–50 Hz. (H) EEG spectrum analysis and extraction of each band power (from delta to SMR). (I) Relative power of each band from EEG signals (ground truth).

Detecting Brain Activity From Pupil Signals

The pupil area was calculated using the method described in previous studies (Daugman, 2004; Lee et al., 2009; Park et al., 2018). Gray-scale images captured from an IR camera were processed using binarization with a threshold value, as shown in Equation (1). This equation for binarization was extracted from a linear regression model between mean and maximum brightness value from the entire image in a previous study (Park et al., 2018). The pupil position from the binarized image was then calculated using a circular edge detection algorithm (Daugman, 2004; Lee et al., 2009), as shown in Equation (2).

Threshold for binarization=(-0.418×Bmean)                                                                 +(1.051×Bmax)+7.973    (1)
Max(r,  x0,  y0)|Gσ(r)*rr,  x0,  y0I(x,y)2πrds|    (2)

In Equation (1), Bmean and Bmax denote the mean and maximum brightness value from the entire image in gray scale, respectively. In Equation (2), I(x, y) indicates a gray level at the (x, y) position, and (x0, y0) and r represent the center position and radius of the pupil, respectively. (r) represents a smoothing function to detect a circular edge blurred at a scale set by σ (Gaussian type). An accurate pupil position was identified by the reflected light caused by the IR lamp when multiple pupil positions were selected, as shown in Figure 3.

FIGURE 3
www.frontiersin.org

Figure 3. Protocol for measuring the pupil area. (A) A gray-scale image from the infrared web-cam. (B) Binarization image based on auto threshold. (C) Detecting the reflected light caused by the infrared lamp. (D) Calculating pupil position using a circular edge detection algorithm.

The process of assessing brain activity (i.e., EEG spectral power for each band) was as follows. Pupil diameter was processed using the sliding moving average (i.e., window size of 1 s and shift of 1 s) from 125 to 1 fps (1 Hz resampling). For example, a pupil diameter of 125 fps was calculated by mean value (1 fps). The moving average procedure was applied in the proposed method because pupil area was not detected during eye closure. Pupil diameter can be acquired using this method if the time the eyes are closed is < 1 s. However, non-tracked pupil diameters caused by the eye closing >1 s were not involved in the moving average procedure. The resampled pupil diameter was processed using the BPF (Butterworth type of order 6) into separate frequency bands of delta (0.01–0.04 Hz), theta (0.04–0.08 Hz), alpha (0.08–0.13 Hz), beta (0.13–0.30 Hz), gamma (0.30–0.50 Hz), slow alpha (0.08–0.11 Hz), fast alpha (0.11–0.13 Hz), low beta (0.12–0.15 Hz), mid beta (0.15–0.20 Hz), high beta (0.20–0.30 Hz), mu (0.09–0.11 Hz), SMR (0.125–0.155 Hz), and total band (0.01–0.50 Hz). These frequency bands in pupillary rhythm were applied to the harmonic frequency band with a 1/100 ratio from the EEG spectrum band, assuming that the pupillary and brain rhythms were synchronized. Filtered signals for each frequency band (from delta to total band) were proceeded using FFT analysis to change into the spectrum from time serial data. Each band power was defined by the summation of all frequency power value from each spectrum data. The relative powers of each frequency band, from delta to SMR, were calculated using the ratio between the total and each band power, as shown in Equation (3). This procedure was performed using the sliding moving average with a window size of 30 s and a shift of 1 s, as shown in Figure 2.

Relative power (%)= Each band powerTotal band power×100    (3)

Statistical Analysis

This study measured the EEG spectral index from pupillary rhythm, represented by delta, theta, alpha, beta, gamma, slow alpha, fast alpha, low beta, mid beta, high beta, mu, and SMR relative power for the 19 channel brain regions, based on the synchronization between the brain and pupillary rhythms. Relative powers for EEG spectral index extracted from pupillary rhythm were compared with the EEG spectral index from brain activity (ground truth) based on the correlation analysis (Pearson correlation coefficient, r), mean error value (ME), and Bland-Altman plot (difference plot). The correlation coefficient ranging in value from −1 to 1 was defined by the linear relationship between two or more variables. A correlation coefficient approaching a value of −1 indicates a strong negative correlation, and that approaching a value of 1 indicates a strong positive correlation (Stigler, 1989). Mean error was defined as the difference in value of the relative powers of each band between pupil and brain. A Bland-Altman plot (mean-difference plot) is a data plotting method for assessing agreement between two methods. Bland and Altman make the point that any two methods that are designed to measure the same parameter (or property) should have good correlation when a set of samples are chosen such that the property to be determined varies considerably. This plot consists of the x and y axes, and correspond to the mean and difference values between two measures, respectively. If most of the measured values from two variables are within d¯±2s (more precisely, between d¯+1.96s and d¯-1.96 s: a 95% confidence level) in a Bland-Altman plot, the two measures have agreement, where d¯ and s are mean difference and standard deviation, respectively (Bland and Altman, 1986). All statistical analyses were performed using SPSS Statistics version 19.0 (IBM Corporation. Armonk, NY, USA).

Results

Representative examples of an extracted EEG spectral index from the pupillary response, and the EEG signals for the test participants (participants 6 and 42) are shown in Figure 4. When comparing the results for the ground truth, the EEG spectral index from the pupillary response demonstrated a strong positive correlation with EEG signals for seven parameters (participants 6 and 42), in which r = 0.863 (p < 0.001) and 0.862 (p < 0.001) for low beta power in the FP1 region; r = 0.853 (p < 0.001) and 0.803 (p < 0.001) for mid beta power in the FP1 region; r = 0.800 (p < 0.001) and 0.840 (p < 0.001) for SMR power in the FP1 region; r = 0.857 (p < 0.001) and 0.882 (p < 0.001) for beta power in the F3 region; r = 0.826 (p < 0.001) and 0.838 (p < 0.001) for high beta power in the F8 region; r = 0.882 (p < 0.001) and 0.866 (p < 0.001) for gamma power in the P4 region; r = .882 (p < 0.001) and 0.868 (p < 0.001) for mu power in the C4 region. This result processed using the sliding moving average with a window size of 30 s and a shift of 1 s measurement over a 300 s interval.

FIGURE 4
www.frontiersin.org

Figure 4. Representative examples (participant 6 [left panel]; participant 42 [right panel]) of the electroencephalogram (EEG) spectral index from the pupil and EEG signals (low beta power in the FP1 region; mid beta power in the FP1 region; SMR power in the FP1 region; beta power in the F3 region; high beta power in the F8 region; gamma power in the P4 region; mu power in the C4 region).

A comparison of the results for the 70 participants in reference state (i.e., neutral) is shown in Figure 5. When comparing the results of the ground truth, the EEG spectral index from the pupillary response demonstrated a strong positive correlation (0.970–0.993) and low mean error (0.118–0.325) with EEG signals for seven parameters, in which r = 0.974 (p < 0.001) and ME = 0.118 ± 0.085 for low beta power in the FP1 region; r = 0.975 (p < 0.001) and ME = 0.148 ± 0.089 for mid beta power in the FP1 region; r = 0.970 (p < 0.001) and ME = 0.146 ± 0.095 for SMR power in the FP1 region; r = 0.993 (p < 0.001) and ME = 0.325 ± 0.233 for beta power in the F8 region; r = 0.976 (p < 0.001) and ME = 0.212 ± 0.159 for high beta power in the F8 region; r = 0.987 (p < 0.001) and ME = 0.217 ± 0.167 for gamma power in the P4 region; r = 0.977 (p < 0.001) and ME = 0.204 ± 0.161 for mu power in the C4 region.

FIGURE 5
www.frontiersin.org

Figure 5. Results of correlation analysis in reference state for coefficient (r) and mean error (ME) of electroencephalogram (EEG) spectral index between the pupil and EEG signals (p < 0.001 [n = 70]).

A comparison of the results for the 70 participants in variation of physiological state (i.e., positive, negative, arousal, and relaxation) is shown in Figure 6. When comparing the results of the ground truth, the EEG spectral index from the pupillary response demonstrated a strong positive correlation (0.951–0.991) and low mean error (0.148–0.345) with EEG signals for seven parameters, in which r = 0.965 (p < 0.001) and ME = 0.149 ± 0.121 for low beta power in the FP1 region; r = 0.964 (p < 0.001) and ME = 0.177 ± 0.113 for mid beta power in the FP1 region; r = 0.951 (p < 0.001) and ME = 0.148 ± 0.144 for SMR power in the FP1 region; r = 0.991 (p < 0.001) and ME = 0.345 ± 0.243 for beta power in the F8 region; r = 0.976 (p < 0.001) and ME = 0.263 ± 0.173 for high beta power in the F8 region; r = 0.986 (p < 0.001) and ME = 0.312 ± 0.222 for gamma power in the P4 region; r = 0.962 (p < 0.001) and ME = 0.315 ± 0.243 for mu power in the C4 region. Result for the reference condition of correlation coefficient and mean error is slightly higher and lower than the condition of physiological variation. However, these results in both conditions of reference and physiological variation represented the brain region showing the highest correlation and lowest mean error for each frequency band with ground truth. The aforementioned results revealed the highest correlation and the lowest error among analysis results for all EEG frequency bands in each brain region, and the overall results are shown in Table 1. Results for the other frequency bands including delta, theta, alpha, slow and fast alpha were not statistically significant.

FIGURE 6
www.frontiersin.org

Figure 6. Results of correlation analysis in variation of physiological state for coefficient (r) and mean error (ME) of electroencephalogram (EEG) spectral index between the pupil and EEG signals (p < 0.001 [n = 70]).

TABLE 1
www.frontiersin.org

Table 1. Results of Pearson correlation coefficient and mean error in reference (physiological variation) for electroencephalogram (EEG) spectral index between the pupil and EEG signals (N = 70, p < 0.05).

In addition, results of the Bland-Altman analysis in reference and physiological variation conditions, shown in Figures 7, 8, confirmed that the measured values for the EEG spectral index were within the 95% limit of agreement (± 2 SD). The Bland-Altman plots demonstrated good agreement between EEG spectral indexes from the pupil and EEG signals in both conditions.

FIGURE 7
www.frontiersin.org

Figure 7. Representative Bland-Altman plots in reference condition for electroencephalogram (EEG) spectral indexes (low beta in FP1; mid beta in FP1 region; SMR in FP1; beta in F3; high beta in F8; gamma in P4; mu in C4) between pupil and EEG signals. The centre solid line represents the mean difference between two measures, with the upper and lower dotted lines representing the 95% limits (±2 SD) of agreement (n = 70).

FIGURE 8
www.frontiersin.org

Figure 8. Representative Bland-Altman plots in physiological variation condition for electroencephalogram (EEG) spectral indexes (low beta in FP1; mid beta in FP1 region; SMR in FP1; beta in F3; high beta in F8; gamma in P4; mu in C4) between pupil and EEG signals. The center solid line represents the mean difference between two measures, with the upper and lower dotted lines representing the 95% limits (±2 SD) of agreement (n = 70).

A real-time system for noncontact measurement the EEG spectral index was developed with the IR web-cam (Figure 9). This system comprised an IR webcam, a near-IR illuminator (IR lamp), and a personal computer for analysis. The IR web-cam was developed by reconfiguring a high-definition web-cam (HD-5000, Microsoft Inc., USA) by replacing the IR filter with an IR pass filter (Kodac Inc., USA) inside the webcam. A real-time system can be non-contact measure the relative power for low beta in FP1, mid beta in FP1 region, SMR in FP1, beta in F3, high beta in F8, gamma in P4, and mu in C4 using the IR web-cam. This system was developed using Visual C++ 2010 and OpenCV 2.4.3, and the signal processing used LabVIEW 2010 (National Instruments Inc., Austin, TX, USA).

FIGURE 9
www.frontiersin.org

Figure 9. The real-time system for measuring brain activity using an infrared (IR) web-cam. (A) Overview the real-time system. (B) Configuration of the measuring device including the web-cam. (C) Introduction to the measuring software: (a) Protocol for detecting the pupil area; (b) Pupillary response including raw, re-sampled, and filtered pupil diameter data; (c and c′) Output of the relative power for the electroencephalogram spectral index (low beta power in the FP1 region; mid beta power in the FP1 region; SMR power in the FP1 region; beta power in the F3 region; high beta power in the F8 region; gamma power in the P4 region; mu power in the C4 region).

Discussion

The aim of the present study was to develop a noncontact system to measure the relative power of the spectral index based on pupillary rhythms. Pupillary rhythm is closely connected to brain activities based on neural pathways; this study founded that the pupillary rhythms was strongly correlated with the seven parameters of brain activity using the synchronization phenomenon in harmonic frequency (1/100 f) during varying physiological states. The correlation coefficient was very high (>0.951 [0.951–0.991]), and the mean error was very low (< 0.345% [0.148–0.345%]) in both the EEG spectral index. Regarding concurrent validity, examination of Bland-Altman plots revealed small differences in EEG spectral index between the two measurements.

Many previous studies have been reported that pupil size variation (PSV), also known as task-evoked pupillary response (TEPR), is correlated with functional brain processing such as attention, cognitive load, and memory. Previous researches demonstrated that changes in pupil diameter correlated with cortical activity in dorsal attention network (DAN) (i.e., activity in the superior colliculus and the right thalamus) (Hafed et al., 2013; Alnæs et al., 2014; Wang and Munoz, 2015; Joshi et al., 2016), locus coeruleus–norepinephrine (LC–NE) system (Gabay et al., 2011; Geva et al., 2013; Hong et al., 2014; Murphy et al., 2014; Joshi et al., 2016), and cingulate cortex (Ebitz and Platt, 2015; Joshi et al., 2016) related to attention and cognitive function. These connectivity between changes in pupil diameter and neural network from the cortical area to brainstem is interpreted by a top-down control executive network related to attention. Changes in pupil diameter results from cortical modulation between the LC-NE system in the brainstem and the neo-cortex involving the medial–ventral prefrontal cortex (MVPC), anterior cingulate cortex (ACC), and lateral prefrontal cortex (LPC) (Bush et al., 2000; MacDonald et al., 2000). The pupil diameter response has been shown with a long delay after LC-NE system is activated by the stimulus. The delay is to modulate the autonomic arousal via a top-down pathway among brainstem and cortical areas (i.e., MVPC, ACC, and LPC), and this process leads to the change in pupil size (Christ et al., 2008; Laeng et al., 2011; Niendam et al., 2012; Buckner, 2013; Geva et al., 2013; van Steenbergen and Band, 2013). Also, some study reported that changes in pupil includes a three attention networks (i.e., top-down control) related to an early component (Pa) and a prominent late component (Pe): (1) Alerting by Pa, (2) Orienting by acceleration of Pa, and (3) Executive control by Pe (Fan et al., 2002; Isaacowitz et al., 2009; Geva et al., 2013). The Pa response in pupil is associated with the recruitment of autonomic resources through alerting and attention shifting by activation of the posterior attention system. Pe's delayed activation is related to recruiting the mental resources required for executive monitoring from MVPC, ACC, and LPC (Lorist et al., 2000; Kennerley and Walton, 2011; Geva et al., 2013), and reflects the activation in LC-NE system by ACC top-down regulation (Botvinick, 2007).

In our study, the pupillary rhythms is correlated with the brain oscillations related to low beta, mid beta, high beta, mu, beta, SMR, and gamma bands, and these parameters also are known as the indicators for attention and cognitive function supported by many previous studies. For example, the SMR, mu, low beta, mid beta, and high beta waves are correlated with a high level of attention (Tansey, 1984; Egner and Gruzelier, 2001, 2004; Lee and Han, 2011; Gruzelier, 2014; Ziółkowski et al., 2014; Kim et al., 2015), and gamma waves are connected to selective attention (Brovelli et al., 2005; Jensen et al., 2007) and memory function (Pesaran et al., 2002; Howard et al., 2003; Jensen et al., 2007). The mu and beta rhythms are continuous with cognitive load or mental workload (Pfurtscheller and Klimesch, 1992; Murata, 2005; Krause et al., 2007; Tanaka et al., 2014). Above mentioned, previous studies reported that the PSV is closely connected with the neural network and the brain activity related to attention and cognition. Pupillary rhythm is involved in brain rhythm as a result of brain processing, and our findings is supported by these connectivity and previous results.

In addition, other previous studies also have been reported the directly relationship between the pupillary rhythms and the brain oscillation. Keegan and Merritt (1995) reported a significant positive correlation among beta band power and pupil diameter, and there is a negative correlation between pupil diameter and delta, theta, and alpha band powers. Other study showed a negative correlation between slow brain activities such (theta and alpha band powers) (Wang, 2013). Hong et al. (2014) showed that the pupil diameter is significant correlated with the EEG alpha rhythms based on tight coupling between attentional state and evoked neural activity. In another study reported that the alpha frequency band power in EEG is negatively correlated with the pupil dilation (Scharinger et al., 2015). A study of Vinck et al. (2015) revealed that gamma band power is strongly correlated (i.e., positive relationship) with pupil dynamics. Our results were consistent to the previous findings, showing a strong correlation between the pupillary rhythm and frequency band powers (mu, low beta, mid beta, high beta, and gamma) of similar EEG band ranges that were mentioned in the studies above. Parameters that had no correlation (delta and theta) in our current study may find a significant relationship between pupil rhythm and these band oscillation in other harmonic frequency (i.e., 1/10 f, 1/20 f, etc.), and research on this phenomena is needed through further study.

In addition, the effect of the variation of the human physiological states on the pupil response was considered in this study. A strong correlation between the pupil and the EEG oscillations was also found in the condition of the physiological change rather the neutral state. Other non-contact measurement studies reported a correlation between behavior factor and physiological response only in neutral or reference states (Loewenfeld and Lowenstein, 1993; Steinhauer et al., 2000; Verney et al., 2001; Siegle et al., 2004; Poh et al., 2011; Balakrishnan et al., 2013; Holton et al., 2013; Janssen et al., 2015). However, because non-contact measurement technology needs to be verified under physiological change in order to apply the various environments or industries. Many previous studies reported that autonomic nervous system (ANS) has been found to be closely connected with the pupillary rhythms, exhibiting repeated contraction and expansion via the sphincter and dilator muscles (Bonvallet and Zbrozyna, 1963; Loewenfeld and Lowenstein, 1993; Steinhauer et al., 2000; Verney et al., 2001; Siegle et al., 2004), and the emotional stimuli presented in this study such as arousal, relaxation, positive, and negative are closely related to the ANS (Yang et al., 2007; Levenson, 2014; Dudas et al., 2017). Also, changes in pupillary rhythms have been related to neural activity in the dorsolateral prefrontal cortex (DLPFC) (Siegle et al., 2011), and this region is known to associate with a more cognitive effort to regulate emotions for affective event (Ochsner et al., 2004; Vanderhasselt et al., 2014; Dudas et al., 2017). Our study found a strong correlation between brain and pupillary rhythms not only in the neutral state but also in the condition of physiological changes. According to previous studies, it can be interpreted by the connectivity among changes in pupillary rhythms, autonomic balance in ANS, and activation in DLPFC. Relationship between changes in pupil diameter and in emotional states (i.e., negative, positive, arousal, and neutral) has been reported by many previous researches (Pesaran et al., 2002; Partala and Surakka, 2003; Bradley et al., 2008; Geangu et al., 2011; Henderson et al., 2014; Vanderhasselt et al., 2014).

To wear the EEG electrode cap has required to the burden and the time constraints for attaching the sensor to user, and expensive sensor, and it can be restrict application in industrial fields. Our findings can be apply to the various fields related to non-contact evaluation and monitoring of the attention and the cognitive function using an inexpensive infrared web-cam without the burden for sensor attachment. For examples, our results may be applied to industrial domain such as education, neuro-feedback, ergonomics, brain computer interface (BCI), and emotional engineering as follows: (1) Evaluating immersion or concentration level of users in off- and on-line education domains. (2) Neuro-feedback training system to enhance the attention level. (3) Human assistance system to improve the usability based on monitoring for physical and mental fatigue, and drowsiness states. (4) Interface technology to control the external device using event-related (de)synchronization (ERD/ERS) and steady-state visual evoked potential (SSVEP). (5) Measuring the emotional states such as arousal, relaxation, positive, and negative. If the following limitations are complemented, there is a possibility of applying to the various fields besides the above-mentioned examples.

This study, however, had several limitations. Pupillary response is strongly influenced by blinking and ambient light. Because the pupil area is not detected during blinking, the method proposed in this study cannot directly measure brain activity; consequently, we applied a 1 Hz re-sampling protocol. If the blinking period is < 1 s, the proposed method can obtain the mean pupil diameter by calculating the mean pupil diameter during the period when the eye is open. However, if the duration of blinking is >1 s, brain activity during that period cannot be measured. To resolve this problem completely, an algorithm for restoring pupil data from an unmeasured period is required, and the protocol for measuring brain activity using reconstructed pupil data will need verification. Additionally, pupil size is very sensitive to changes in illumination. Pupil size is controlled by the occulomotor nerve innervated via the sphincter (i.e., parasympathetic nerve) and the ophthalmic nerve is innervated via dilator muscles (i.e., sympathetic nerve) according to changes in illumination (Bonvallet and Zbrozyna, 1963; Usui and Shirakashi, 1982; Loewenfeld and Lowenstein, 1993; Steinhauer et al., 2000; Verney et al., 2001; Siegle et al., 2004; Kozicz et al., 2011; Andreassi, 2013; Júnior et al., 2015). Because measurement of brain activity was based on pupillary rhythm (i.e., pupil size variation) in this study, the proposed method may be vulnerable to errors in rapid light alterations. In a previous study, the effect of rapid light changes (550, 350, 150, 40, and 2 lux) on pupil size was tested, and respective pupil sizes were 3.5, 4.2, 5.2, 5.03, and 5.4 mm. Pupil size has not been shown to be significantly affected by illumination conditions < 150 lux (Maqsood, 2017). The ambient light in this experiment was measured to be in the in range of 135–160 lux (mean 147.42 ± 6.02 lux) using a commercially available device (Visible Light SD Card Logger, Sper Scientific Meters Ltd., Scottsdale, AZ, USA), which did not affect the measurement of brain activity using the proposed method. However, in experimental design of this study, condition for the ambient light was not considered with presenting the sound instead of visual stimuli. Thus, proposed method in this study cannot be applied to the condition what the ambient light changes rapidly. In order to resolve this issue, further research investigating the relationship between changes in pupil diameter and a rapid change in ambient light intensity is required before rigorous application in various fields.

In this study brain activity was measured using a low-cost IR webcam. Pupillary rhythms with harmonic frequency (1/100 f) revealed a high correlation in our protocol with seven parameters (relative powers) of the EEG spectral index [i.e., low beta (FP1), mid beta (FP1), SMR (FP1), beta (F3), high beta (F8), gamma (P4), and mu (C4)] during the changes of the physiological states (arousal, relaxation, positive, negative, and neutral states). However, in order for the proposed method to be applied in various fields, the method needs to be investigated under different conditions (rapid illumination changes, other physiologic conditions, or other stimuli for causing physiological state) that were not included in the current protocol. To overcome the above limitations in further studies, the proposed method may measure and evaluate brain activity in various application fields using a simple, low-cost, non-contact system. Because measurement of brain activity using sensors has traditionally required complex and expensive equipment, and involved the inconveniences and burden of sensor attachment.

Author Contributions

SP and MW designed the study with investigation of previous studies and performed the experiments. SP performed the data analysis and verification for statistical significance, and wrote the manuscript with support from MW. SP and MW discussed the results and contributed to the final manuscript. MW conceived the study and was in charge of overall direction and planning.

Funding

This work was partly supported by the Global Frontier R&D Program on Human-centered Interaction for Coexistence, funded by the National Research Foundation of Korea, grant funded by the Korean Government (MEST) [grant number 2015-0029756]; and by the National Research Foundation of Korea (NRF) grant funded by the Korea government (Ministry of Science, ICT & Future Planning) [No. 2017R1C1B2010488].

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fphys.2018.01400/full#supplementary-material

References

Abbas, A. K., Heimann, K., Jergus, K., Orlikowsky, T., and Leonhardt, S. (2011). Neonatal non-contact respiratory monitoring based on real-time infrared thermography. Biomed. Eng. Online 10, 93–110. doi: 10.1186/1475-925X-10-93

PubMed Abstract | CrossRef Full Text | Google Scholar

Ahern, S., and Beatty, J. (1979). Pupillary responses during information processing vary with Scholastic Aptitude Test scores. Science 205, 1289–1292. doi: 10.1126/science.472746

PubMed Abstract | CrossRef Full Text | Google Scholar

Alnæs, D., Sneve, M. H., Espeseth, T., Endestad, T., van de Pavert, S. H., and Laeng, B. (2014). Pupil size signals mental effort deployed during multiple object tracking and predicts brain activity in the dorsal attention network and the locus coeruleus. J. Vis. 14, 1–20. doi: 10.1167/14.4.1

PubMed Abstract | CrossRef Full Text | Google Scholar

Andreassi, J. L. (2013). Psychophysiology: Human Behavior & Physiological Response. Mahwah, NJ: Psychology Press.

Google Scholar

Balakrishnan, G., Durand, F., and Guttag, J. (2013). “Detecting pulse from head motions in video,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Portland, OR: IEEE), 3430–3437.

Google Scholar

Bär, K. J., Schulz, S., Koschke, M., Harzendorf, C., Gayde, S., Berg, W., et al. (2009). Correlations between the autonomic modulation of heart rate, blood pressure and the pupillary light reflex in healthy subjects. J. Neurol. Sci. 279, 9–13. doi: 10.1016/j.jns.2009.01.010

PubMed Abstract | CrossRef Full Text | Google Scholar

Beatty, J. (1988). Pupillometric signs of selective attention in man,” in Neurophysiology and Psychophysiology: Experimental and Clinical Applications, eds G. C. Galbraith, M. L. Kietzman, and E. Donchin (Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.), 138–143.

Beatty, J., and Kahneman, D. (1966). Pupillary changes in two memory tasks. Psychon. Sci. 5, 371–372. doi: 10.3758/BF03328444

CrossRef Full Text | Google Scholar

Beatty, J., and Lucero-Wagoner, B. (2000). “The pupillary system,” in Handbook of Psychophysiology, eds J. T. Cacioppo, L. G. Tassinary, and G. G. Berntson (New York, NY: Cambridge University Press), 142–162.

Google Scholar

Berta, R., Bellotti, F., De Gloria, A., Pranantha, D., and Schatten, C. (2013). Electroencephalogram and physiological signal analysis for assessing flow in games. IEEE Trans. Comput. Intell. AI Games 5, 164–175. doi: 10.1109/TCIAIG.2013.2260340

CrossRef Full Text | Google Scholar

Bland, J. M., and Altman, D. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 327, 307–310. doi: 10.1016/S0140-6736(86)90837-8

CrossRef Full Text | Google Scholar

Bonvallet, M., and Zbrozyna, A. (1963). Les commandes réticulaires du système autonome et en particulier de l'innervation sympathique et parasympathique de la pupille. Arch. Ital. Biol. 101, 174–207.

Google Scholar

Boric-Lubeke, O., and Lubecke, V. M. (2002). Wireless house calls: using communications technology for health care and monitoring. IEEE Microw. Mag. 3, 43–48. doi: 10.1109/MMW.2002.1028361

CrossRef Full Text | Google Scholar

Botvinick, M. M. (2007). Conflict monitoring and decision making: reconciling two perspectives on anterior cingulate function. Cogn. Affect. Behav. Neurosci. 7, 356–366. doi: 10.3758/CABN.7.4.356

PubMed Abstract | CrossRef Full Text | Google Scholar

Bradley, M. M., Miccoli, L., Escrig, M. A., and Lang, P. J. (2008). The pupil as a measure of emotional arousal and autonomic activation. Psychophysiology 45, 602–607. doi: 10.1111/j.1469-8986.2008.00654.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Brovelli, A., Lachaux, J. P., Kahane, P., and Boussaoud, D. (2005). High gamma frequency oscillatory activity dissociates attention from intention in the human premotor cortex. Neuroimage 28, 154–164. doi: 10.1016/j.neuroimage.2005.05.045

PubMed Abstract | CrossRef Full Text | Google Scholar

Buckner, R. L. (2013). The brain's default network: origins and implications for the study of psychosis. Dialogues Clin. Neurosci. 15, 351–358.

PubMed Abstract | Google Scholar

Bush, G., Luu, P., and Posner, M. I. (2000). Cognitive and emotional influences in the anterior cingulate cortex. Trends Cogn. Sci. 4, 215–222. doi: 10.1016/S1364-6613(00)01483-2

PubMed Abstract | CrossRef Full Text | Google Scholar

Calcagnini, G., Lino, S., Censi, F., and Cerutti, S. (1997). Cardiovascular autonomic rhythms in spontaneous pupil fluctuations. Comput. Cardiol. 24, 133–136.

Google Scholar

Chekmenev, S. Y., Farag, A. A., Miller, W. M., Essock, E. A., and Bhatnagar, A. (2009). “Multiresolution approach for noncontact measurements of arterial pulse using thermal imaging,” in Augmented Vision Perception in Infrared (London: Springer), 87–112.

Google Scholar

Chekmenev, S. Y., Rara, H., and Farag, A. A. (2005). “Non-contact, wavelet-based measurement of vital signs using thermal imaging,” in The First International Conference on Graphics, Vision, and Image Processing (GVIP) (Cairo), 107–112.

Google Scholar

Christ, S. E., Van Essen, D. C., Watson, J. M., Brubaker, L. E., and McDermott, K. B. (2008). The contributions of prefrontal cortex and executive control to deception: evidence from activation likelihood estimate meta-analyses. Cereb. Cortex 19, 1557–1566. doi: 10.1093/cercor/bhn189

PubMed Abstract | CrossRef Full Text | Google Scholar

Daugman, J. (2004). How iris recognition works. IEEE Trans. Circuits Syst. Video Technol. 14, 21–30. doi: 10.1109/TCSVT.2003.818350

CrossRef Full Text | Google Scholar

Deuschl, G., and Eisen, A. (1999). International Federation of Clinical Neurophysiology, and International Federation of Societies for Electroencephalography and Clinical Neurophysiology. Recommendations for the Practice of Clinical Neurophysiology : Guidelines of the International Federation of Clinical Neurophysiology. Amsterdam, NY: Elsevier.

Dietrich, A., and Kanso, R. (2010). A review of EEG, ERP, and neuroimaging studies of creativity and insight. Psychol. Bull. 136, 822–848. doi: 10.1037/a0019749

PubMed Abstract | CrossRef Full Text | Google Scholar

Droitcour, A., Lubecke, V., Lin, J., and Boric-Lubecke, O. (2001). “A microwave radio for Doppler radar sensing of vital signs,” in IEEE MTT-S International Microwave Symposium (Phoenix, AZ), 175–178.

Google Scholar

Dudas, R. B., Mole, T. B., Morris, L. S., Denman, C., Hill, E., Szalma, B., et al. (2017). Amygdala and dlPFC abnormalities, with aberrant connectivity and habituation in response to emotional stimuli in females with BPD. J. Affect. Disord. 208, 460–466. doi: 10.1016/j.jad.2016.10.043

PubMed Abstract | CrossRef Full Text | Google Scholar

Ebitz, R. B., and Platt, M. L. (2015). Neuronal activity in primate dorsal anterior cingulate cortex signals task conflict and predicts adjustments in pupil-linked arousal. Neuron 85, 628–640. doi: 10.1016/j.neuron.2014.12.053

PubMed Abstract | CrossRef Full Text | Google Scholar

Egner, T., and Gruzelier, J. H. (2001). Learned self-regulation of EEG frequency components affects attention and event-related brain potentials in humans. Neuroreport 12, 4155–4159. doi: 10.1097/00001756-200112210-00058

PubMed Abstract | CrossRef Full Text | Google Scholar

Egner, T., and Gruzelier, J. H. (2004). EEG biofeedback of low beta band components: frequency-specific effects on variables of attention and event-related brain potentials. Clin. Neurophysiol. 115, 131–139. doi: 10.1016/S1388-2457(03)00353-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Fan, J., McCandliss, B. D., Sommer, T., Raz, A., and Posner, M. I. (2002). Testing the efficiency and independence of attentional networks. J. Cogn. Neurosci. 14, 340–347. doi: 10.1162/089892902317361886

PubMed Abstract | CrossRef Full Text | Google Scholar

Fei, J., and Pavlidis, I. (2010). Thermistor at a distance: unobtrusive measurement of breathing. IEEE Trans. Biomed. Eng. 57, 988–998. doi: 10.1109/TBME.2009.2032415

PubMed Abstract | CrossRef Full Text | Google Scholar

Fletcher, R., and Han, J. (2009). “Low-cost differential front-end for Doppler radar vital sign monitoring,” in IEEE MTT-S International Microwave Symposium (Boston, MA), 1325–1328.

Google Scholar

Fotiou, F., Fountoulakis, K. N., Tsolaki, M., Goulas, A., and Palikaras, A. (2000). Changes in pupil reaction to light in Alzheimer's disease patients: a preliminary report. Int. J. Psychophysiol. 37, 111–120. doi: 10.1016/S0167-8760(00)00099-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Gabay, S., Pertzov, Y., and Henik, A. (2011). Orienting of attention, pupil size, and the norepinephrine system. Atten. Percept. Psychophys. 73, 123–129. doi: 10.3758/s13414-010-0015-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Garbey, M., Sun, N., Merla, A., and Pavlidis, I. (2007). Contact-free measurement of cardiac pulse based on the analysis of thermal imagery. IEEE Trans. Biomed. Eng. 54, 1418–1426. doi: 10.1109/TBME.2007.891930

PubMed Abstract | CrossRef Full Text | Google Scholar

Gault, T. R., Blumenthal, N., Farag, A. A., and Starr, T. (2010). “Extraction of the superficial facial vasculature, vital signs waveforms and rates using thermal imaging,” in 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (San Francisco, CA), 1–8.

Google Scholar

Geangu, E., Hauf, P., Bhardwaj, R., and Bentz, W. (2011). Infant pupil diameter changes in response to others' positive and negative emotions. PLoS ONE 6:e27132. doi: 10.1371/journal.pone.0027132

PubMed Abstract | CrossRef Full Text | Google Scholar

Geva, R., Zivan, M., Warsha, A., and Olchik, D. (2013). Alerting, orienting or executive attention networks: differential patters of pupil dilations. Front. Behav. Neurosci. 7:145. doi: 10.3389/fnbeh.2013.00145

PubMed Abstract | CrossRef Full Text | Google Scholar

Gruzelier, J. H. (2014). Differential effects on mood of 12–15 (SMR) and 15–18 (beta1) Hz neurofeedback. Int. J. Psychophysiol. 93, 112–115. doi: 10.1016/j.ijpsycho.2012.11.007

PubMed Abstract | CrossRef Full Text | Google Scholar

Gu, C., Li, C., Lin, J., Long, J., Huangfu, J., and Ran, L. (2010). Instrument-based noncontact Doppler radar vital sign detection system using heterodyne digital quadrature demodulation architecture. IEEE Trans. Instrum. Meas. 59, 1580–1588. doi: 10.1109/TIM.2009.2028208

CrossRef Full Text | Google Scholar

Hafed, Z. M., Lovejoy, L. P., and Krauzlis, R. J. (2013). Superior colliculus inactivation alters the relationship between covert visual attention and microsaccades. Eur. J. Neurosci. 37:1169–1181. doi: 10.1111/ejn.12127

PubMed Abstract | CrossRef Full Text | Google Scholar

Hakerem, G., and Sutton, S. (1966). Pupillary response at visual threshold. Nature 212, 485–486. doi: 10.1038/212485a0

PubMed Abstract | CrossRef Full Text | Google Scholar

Henderson, R. R., Bradley, M. M., and Lang, P. J. (2014). Modulation of the initial light reflex during affective picture viewing. Psychophysiology 51, 815–818. doi: 10.1111/psyp.12236

PubMed Abstract | CrossRef Full Text | Google Scholar

Hess, E. H., and Polt, J. M. (1964). Pupil size in relation to mental activity during simple problem-solving. Science 143, 1190–1192. doi: 10.1126/science.143.3611.1190

PubMed Abstract | CrossRef Full Text | Google Scholar

Hink, R. F., Van Voorhis, S. T., Hillyard, S. A., and Smith, T. S. (1977). The division of attention and the human auditory evoked potential. Neuropsychologia 15, 597–605. doi: 10.1016/0028-3932(77)90065-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Holton, B. D., Mannapperuma, K., Lesniewski, P. J., and Thomas, J. C. (2013). Signal recovery in imaging photoplethysmography. Physiol. Meas. 34, 1499–1511. doi: 10.1088/0967-3334/34/11/1499

PubMed Abstract | CrossRef Full Text | Google Scholar

Hong, L., Walz, J. M., and Sajda, P. (2014). Your eyes give you away: prestimulus changes in pupil diameter correlate with poststimulus task-related EEG dynamics. PLoS ONE 9:e91321. doi: 10.1371/journal.pone.0091321

PubMed Abstract | CrossRef Full Text | Google Scholar

Howard, M. W., Rizzuto, D. S., Caplan, J. B., Madsen, J. R., Lisman, J., Aschenbrenner-Scheibe, R., et al. (2003). Gamma oscillations correlate with working memory load in humans. Cereb. Cortex 13, 1369–1374. doi: 10.1093/cercor/bhg084

PubMed Abstract | CrossRef Full Text | Google Scholar

Hung, K., and Zhang, Y. T. (2006). “Preliminary investigation of pupil size variability: toward non-contact assessment of cardiovascular variability,” in 3rd IEEE/EMBS International Summer School on Medical Devices and Biosensors (Cambridge), 137–140.

Google Scholar

Isaacowitz, D. M., Toner, K., and Neupert, S. D. (2009). Use of gaze for real-time mood regulation: effects of age and attentional functioning. Psychol. Aging 24, 989–994. doi: 10.1037/a0017706

PubMed Abstract | CrossRef Full Text | Google Scholar

Janssen, R., Wang, W., Moço, A., and de Haan, G. (2015). Video-based respiration monitoring with automatic region of interest detection. Physiol. Meas. 37, 100–114. doi: 10.1088/0967-3334/37/1/100

PubMed Abstract | CrossRef Full Text | Google Scholar

Jensen, O., Kaiser, J., and Lachaux, J. P. (2007). Human gamma-frequency oscillations associated with attention and memory. Trends Neurosci. 30, 317–324. doi: 10.1016/j.tins.2007.05.001

PubMed Abstract | CrossRef Full Text | Google Scholar

Joshi, S., Li, Y., Kalwani, R. M., and Gold, J. I. (2016). Relationships between pupil diameter and neuronal activity in the locus coeruleus, colliculi, and cingulate cortex. Neuron 89, 221–234. doi: 10.1016/j.neuron.2015.11.028

PubMed Abstract | CrossRef Full Text | Google Scholar

Júnior, E. D. D. S., Da Silva, A. V., Da Silva, K. R., Haemmerle, C. A., Batagello, D. S., Da Silva, J. M., et al. (2015). The centrally projecting Edinger–Westphal nucleus—I: efferents in the rat brain. J. Chem. Neuroanat. 68, 22–38. doi: 10.1016/j.jchemneu.2015.07.002

CrossRef Full Text | Google Scholar

Just, M. A., and Carpenter, P. A. (1993). The intensity dimension of thought: pupillometric indices of sentence processing. Can. J. Exp. Psychol. 47:310. doi: 10.1037/h0078820

PubMed Abstract | CrossRef Full Text | Google Scholar

Just, M. A., Carpenter, P. A., Keller, T. A., Eddy, W. F., and Thulborn, K. R. (1996). Brain activation modulated by sentence comprehension. Science 274, 114–116. doi: 10.1126/science.274.5284.114

PubMed Abstract | CrossRef Full Text | Google Scholar

Kahneman, D., and Beatty, J. (1966). Pupil diameter and load on memory. Science 154, 1583–1585. doi: 10.1126/science.154.3756.1583

PubMed Abstract | CrossRef Full Text | Google Scholar

Kao, T. Y. J., Yan, Y., Shen, T. M., Chen, A. Y. K., and Lin, J. (2013). Design and analysis of a 60-GHz CMOS Doppler micro-radar system-in-package for vital-sign and vibration detection. IEEE Trans. Microw. Theory Technol. 61, 1649–1659. doi: 10.1109/TMTT.2013.2247620

CrossRef Full Text | Google Scholar

Keegan, A., and Merritt, S. L. (1995). “A discrete-time model to test links between EEG power and pupil diameter measured by infrared cameras,” in Proc. Opt. Eng. Midwest, 728–735.

Google Scholar

Kennerley, S. W., and Walton, M. E. (2011). Decision making and reward in frontal cortex: complementary evidence from neurophysiological and neuropsychological studies. Behav. Neurosci. 125, 297–317. doi: 10.1037/a0023575

PubMed Abstract | CrossRef Full Text | Google Scholar

Kim, H. S., Song, G. S., Ahn, H. B., Lee, C. Y., and Kim, J. H. (2015). “A study on the effects of LED light's pulse width modulation on work concentration,” in International Conference on Knowledge Management in Organizations (Maribor), 550–560.

Google Scholar

Klingner, J., Kumar, R., and Hanrahan, P. (2008). “Measuring the task-evoked pupillary response with a remote eye tracker,” in Proceedings of the 2008 Symposium on Eye Tracking Research & Applications (Savannah, GA), 69–72.

Google Scholar

Kojima, M., Shioiri, T., Hosoki, T., Kitamura, H., Bando, T., and Someya, T. (2004). Pupillary light reflex in panic disorder. Eur. Arch. Psychiatry Clin. Neurosci. 254, 242–244. doi: 10.1007/s00406-004-0487-0

PubMed Abstract | CrossRef Full Text | Google Scholar

Kozicz, T., Bittencourt, J. C., May, P. J., Reiner, A., Gamlin, P. D., Palkovits, M., et al. (2011). The Edinger-Westphal nucleus: a historical, structural, and functional perspective on a dichotomous terminology. J. Comp. Neurol. 519, 1413–1434. doi: 10.1002/cne.22580

PubMed Abstract | CrossRef Full Text | Google Scholar

Krause, C. M., Pesonen, M., Haarala Björnberg, C., and Hämäläinen, H. (2007). Effects of pulsed and continuous wave 902 MHz mobile phone exposure on brain oscillatory activity during cognitive processing. Bioelectromagnetics 28, 296–308. doi: 10.1002/bem.20300

PubMed Abstract | CrossRef Full Text | Google Scholar

Laeng, B., Ørbo, M., Holmlund, T., and Miozzo, M. (2011). Pupillary stroop effects. Cogn. Process. 12, 13–21. doi: 10.1007/s10339-010-0370-z

PubMed Abstract | CrossRef Full Text | Google Scholar

Lee, B., and Han, D. (2011). Comparative analysis of brain activation between healthy elderly women and healthy adult women. J. Phys. Ther. Sci. 23, 875–878. doi: 10.1589/jpts.23.875

CrossRef Full Text | Google Scholar

Lee, E. C., Park, K. R., Whang, M., and Min, K. (2009). Measuring the degree of eyestrain caused by watching LCD and PDP devices. Int. J. Ind. Ergon. 39, 798–806. doi: 10.1016/j.ergon.2009.02.008

CrossRef Full Text | Google Scholar

Levenson, R. W. (2014). The autonomic nervous system and emotion. Emot. Rev. 6, 100–112. doi: 10.1177/1754073913512003

CrossRef Full Text | Google Scholar

Li, C., Cummings, J., Lam, J., Graves, E., and Wu, W. (2009). Radar remote monitoring of vital signs. IEEE Microw. Mag. 10, 47–56. doi: 10.1109/MMM.2008.930675

CrossRef Full Text | Google Scholar

Lim, E., Lee, H. K., Myoung, H. S., and Lee, K. J. (2015). Development of a noncontact heart rate monitoring system for sedentary behavior based on an accelerometer attached to a chair. Physiol. Meas. 36:N61. doi: 10.1088/0967-3334/36/3/N61

PubMed Abstract | CrossRef Full Text | Google Scholar

Loewenfeld, I. E., and Lowenstein, O. (1993). The Pupil: Anatomy, Physiology, and Clinical Applications, Vol. 2. Detroit, MI: Wayne State University Press; Wiley-Blackwell.

Google Scholar

Lohman, B., Boric-Lubecke, O., Lubecke, V. M., Ong, P. W., and Sondhi, M. M. (2002). A digital signal processor for Doppler radar sensing of vital signs. IEEE Eng. Med. Biol. Mag. 21, 161–164. doi: 10.1109/MEMB.2002.1044188

PubMed Abstract | CrossRef Full Text | Google Scholar

Lorist, M. M., Klein, M., Nieuwenhuis, S., de Jong, R., Mulder, G., and Meijman, T. F. (2000). Mental fatigue and task control: planning and preparation. Psychophysiology 37, 614–625. doi: 10.1111/1469-8986.3750614

PubMed Abstract | CrossRef Full Text | Google Scholar

MacDonald, A. W., Cohen, J. D., Stenger, V. A., and Carter, C. S. (2000). Dissociating the role of the dorsolateral prefrontal and anterior cingulate cortex in cognitive control. Science 288, 1835–1838. doi: 10.1126/science.288.5472.1835

PubMed Abstract | CrossRef Full Text | Google Scholar

Maggs, D. J., Miller, P., and Ofri, R. (2012). Slatter's Fundamentals of Veterinary Ophthalmology. London: Elsevier Saunders.

Google Scholar

Maqsood, F. (2017). Effects of varying light conditions and refractive error on pupil size. Cogent Med. 4, 1–7. doi: 10.1080/2331205X.2017.1338824

CrossRef Full Text | Google Scholar

Murata, A. (2005). An attempt to evaluate mental workload using wavelet transform of EEG. Hum. Factors 47, 498–508. doi: 10.1518/001872005774860096

PubMed Abstract | CrossRef Full Text | Google Scholar

Murphy, P. R., O'connell, R. G., O'sullivan, M., Robertson, I. H., and Balsters, J. H. (2014). Pupil diameter covaries with BOLD activity in human locus coeruleus. Hum. Brain. Mapp. 35, 4140–4154. doi: 10.1002/hbm.22466

PubMed Abstract | CrossRef Full Text | Google Scholar

Murthy, J. N., van Jaarsveld, J., Fei, J., Pavlidis, I., Harrykissoon, R. I., Lucke, J. F., et al. (2009). Thermal infrared imaging: a novel method to monitor airflow during polysomnography. Sleep 32, 1521–1527. doi: 10.1093/sleep/32.11.1521

PubMed Abstract | CrossRef Full Text | Google Scholar

Murthy, R., and Pavlidis, I. (2005). Non-Contact Monitoring of Breathing Function Using Infrared Imaging. Technical Report Number UH-CS-05-09, 9 April. Department of Computer Science, University of Houston, Houston, TX.

Google Scholar

Niendam, T. A., Laird, A. R., Ray, K. L., Dean, Y. M., Glahn, D. C., and Carter, C. S. (2012). Meta-analytic evidence for a superordinate cognitive control network subserving diverse executive functions. Cogn. Affect. Behav. Neurosci. 12, 241–268. doi: 10.3758/s13415-011-0083-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Norman, D. A., and Bobrow, D. G. (1975). On data-limited and resource-limited processes. Cogn. Psychol. 7, 44–64. doi: 10.1016/0010-0285(75)90004-3

CrossRef Full Text | Google Scholar

Novais, P., and Konomi, S. (2016). “Sensing the music: an audiovisual environment for ASD therapy,” in Intelligent Environments 2016: Workshop Proceedings of the 12th International Conference on Intelligent Environments (London), 338–346.

Google Scholar

Ochsner, K. N., Knierim, K., Ludlow, D. H., Hanelin, J., Ramachandran, T., Glover, G., et al. (2004). Reflecting upon feelings: an fMRI study of neural systems supporting the attribution of emotion to self and other. J. Cogn. Neurosci. 16, 1746–1772. doi: 10.1162/0898929042947829

PubMed Abstract | CrossRef Full Text | Google Scholar

Onorati, F., Barbieri, R., Mauri, M., Russo, V., and Mainardi, L. (2013). Characterization of affective states by pupillary dynamics and autonomic correlates. Front. Neuroeng. 6:9. doi: 10.3389/fneng.2013.00009

PubMed Abstract | CrossRef Full Text | Google Scholar

Park, S., Won, M. J., Lee, D. W., and Whang, M. (2018). Non-contact measurement of heart response reflected in human eye. Int. J. Psychophysiol. 123, 179–198. doi: 10.1016/j.ijpsycho.2017.07.014

PubMed Abstract | CrossRef Full Text | Google Scholar

Parnandi, A., and Gutierrez-Osuna, R. (2013). “Contactless measurement of heart rate variability from pupillary fluctuations,” in 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII) (Washington, DC), 191–196.

Google Scholar

Partala, T., and Surakka, V. (2003). Pupil size variation as an indication of affective processing. Int. J. Hum. Comput. Stud. 59, 185–198. doi: 10.1016/S1071-5819(03)00017-X

CrossRef Full Text | Google Scholar

Pesaran, B., Pezaris, J. S., Sahani, M., Mitra, P. P., and Andersen, R. A. (2002). Temporal structure in neuronal activity during working memory in macaque parietal cortex. Nat. Neurosci. 5, 805–811. doi: 10.1038/nn890

PubMed Abstract | CrossRef Full Text | Google Scholar

Pfurtscheller, G., and Klimesch, W. (1992). “Event-related synchronization and desynchronization of alpha and beta waves in a cognitive task,” in Induced Rhythms in the Brain (Boston, MA: Birkhäuser), 117–128. doi: 10.1007/978-1-4757-1281-0_6

CrossRef Full Text | Google Scholar

Poh, M. Z., McDuff, D. J., and Picard, R. W. (2011). Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Trans. Biomed. Eng. 58, 7–11. doi: 10.1109/TBME.2010.2086456

PubMed Abstract | CrossRef Full Text | Google Scholar

Prehn, K., Heekeren, H. R., and van der Meer, E. (2011). Influence of affective significance on different levels of processing using pupil dilation in an analogical reasoning task. Int. J. Psychophysiol. 79, 236–243. doi: 10.1016/j.ijpsycho.2010.10.014

PubMed Abstract | CrossRef Full Text | Google Scholar

Ramaekers, J. G., Uiterwijk, M. M., and O'hanlon, J. F. (1992). Effects of loratadine and cetirizine on actual driving and psychometric test performance, and EEG during driving. Eur. J. Clin. Pharmacol. 42, 363–369.

PubMed Abstract | Google Scholar

Russell, J. A. (1979). Affective space is bipolar. J. Pers. Soc. Psychol. 37, 345–356. doi: 10.1037/0022-3514.37.3.345

CrossRef Full Text | Google Scholar

Scharinger, C., Kammerer, Y., and Gerjets, P. (2015). Pupil dilation and EEG alpha frequency band power reveal load on executive functions for link-selection processes during text reading. PLoS ONE 10:e0130608. doi: 10.1371/journal.pone.0130608

PubMed Abstract | CrossRef Full Text | Google Scholar

Scully, C. G., Lee, J., Meyer, J., Gorbach, A. M., Granquist-Fraser, D., Mendelson, Y., et al. (2012). Physiological parameter monitoring from optical recordings with a mobile phone. IEEE Trans. Biomed. Eng. 59, 303–306. doi: 10.1109/TBME.2011.2163157

PubMed Abstract | CrossRef Full Text | Google Scholar

Shin, D., Lee, G., Shin, D., and Shin, D. (2014). “Mental state measurement system using EEG analysis,” in Future Information Technology (Berlin; Heidelber: Springer-Verlag), 451–456.

Google Scholar

Siegle, G. J., Steinhauer, S. R., Friedman, E. S., Thompson, W. S., and Thase, M. E. (2011). Remission prognosis for cognitive therapy for recurrent depression using the pupil: utility and neural correlates. Biol. Psychiatry 69, 726–733. doi: 10.1016/j.biopsych.2010.12.041

PubMed Abstract | CrossRef Full Text | Google Scholar

Siegle, G. J., Steinhauer, S. R., and Thase, M. E. (2004). Pupillary assessment and computational modeling of the Stroop task in depression. Int. J. Psychophysiol. 52, 63–76. doi: 10.1016/j.ijpsycho.2003.12.010

PubMed Abstract | CrossRef Full Text | Google Scholar

Steinhauer, S. R., Condray, R., and Kasparek, A. (2000). Cognitive modulation of midbrain function: task-induced reduction of the pupillary light reflex. Int. J. Psychophysiol. 39, 21–30. doi: 10.1016/S0167-8760(00)00119-7

PubMed Abstract | CrossRef Full Text | Google Scholar

Stigler, S. M. (1989). Francis Galton's account of the invention of correlation. Stat. Sci. 4, 73–79.

Google Scholar

Sun, N., Pavlidis, I., Garbey, M., and Fei, J. (2006). “Harvesting the thermal cardiac pulse signal,” in Med. Image Comput. Comput. Assist. Interv, eds R. Larsen, M. Nielsen, and J. Sporring (Berlin; Heidelberg: Springer-Verlag), 569–576.

Google Scholar

Surakka, V., Tenhunen-Eskelinen, M., Hietanen, J. K., and Sams, M. (1998). Modulation of human auditory information processing by emotional visual stimuli. Brain Res. Cogn. Brain Res. 7, 159–163. doi: 10.1016/S0926-6410(98)00021-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Suzuki, S., Matsui, T., Kawahara, H., Ichiki, H., Shimizu, J., Kondo, Y., et al. (2009). A non-contact vital sign monitoring system for ambulances using dual-frequency microwave radars. Med. Biol. Eng. Comput. 47, 101–105. doi: 10.1007/s11517-008-0408-x

PubMed Abstract | CrossRef Full Text | Google Scholar

Tanaka, M., Ishii, A., and Watanabe, Y. (2014). Neural effects of mental fatigue caused by continuous attention load: a magnetoencephalography study. Brain Res. 1561, 60–66. doi: 10.1016/j.brainres.2014.03.009

PubMed Abstract | CrossRef Full Text | Google Scholar

Tansey, M. A. (1984). EEG sensorimotor rhythm biofeedback training: some effects on the neurologic precursors of learning disabilities. Int. J. Psychophysiol. 1, 163–177. doi: 10.1016/0167-8760(84)90036-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Tarassenko, L., Villarroel, M., Guazzi, A., Jorge, J., Clifton, D. A., and Pugh, C. (2014). Non-contact video-based vital sign monitoring using ambient light and auto-regressive models. Physiol. Meas. 35, 807–831. doi: 10.1088/0967-3334/35/5/807

PubMed Abstract | CrossRef Full Text | Google Scholar

Uenoyama, M., Matsui, T., Yamada, K., Suzuki, S., Takase, B., Suzuki, S., et al. (2006). Non-contact respiratory monitoring system using a ceiling-attached microwave antenna. Med. Biol. Eng. Comput. 44, 835–840. doi: 10.1007/s11517-006-0091-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Usui, E., and Shirakashi, T. (1982). “Mechanics of machining-from descriptive topredictive theory,” in On the Art of CuttingMetals? 75 Years Later, Vol. 7. ASME Publisher, PED, 13–35.

van Beijsterveldt, C. E., and Boomsma, D. I. (1994). Genetics of the human electroencephalogram (EEG) and event-related brain potentials (ERPs): a review. Hum. Genet. 94, 319–330. doi: 10.1007/BF00201587

PubMed Abstract | CrossRef Full Text | Google Scholar

van Steenbergen, H., and Band, G. P. (2013). Pupil dilation in the simon task as a marker of conflict processing. Front. Hum. Neurosci. 7:215. doi: 10.3389/fnhum.2013.00215

PubMed Abstract | CrossRef Full Text | Google Scholar

Vanderhasselt, M. A., Remue, J., Ng, K. K., and De Raedt, R. (2014). The interplay between the anticipation and subsequent online processing of emotional stimuli as measured by pupillary dilatation: the role of cognitive reappraisal. Front. Psychol. 5:207. doi: 10.3389/fpsyg.2014.00207

PubMed Abstract | CrossRef Full Text | Google Scholar

Verney, S. P., Granholm, E., and Dionisio, D. P. (2001). Pupillary responses and processing resources on the visual backward masking task. Psychophysiology 38, 76–83. doi: 10.1111/1469-8986.3810076

PubMed Abstract | CrossRef Full Text | Google Scholar

Vinck, M., Batista-Brito, R., Knoblich, U., and Cardin, J. A. (2015). Arousal and locomotion make distinct contributions to cortical activity patterns and visual encoding. Neuron 86, 740–754. doi: 10.1016/j.neuron.2015.03.028

PubMed Abstract | CrossRef Full Text | Google Scholar

Wang, C. A., and Munoz, D. P. (2015). A circuit for pupil orienting responses: implications for cognitive modulation of pupil size. Curr. Opin. Neurobiol. 33, 134–140. doi: 10.1016/j.conb.2015.03.018

PubMed Abstract | CrossRef Full Text | Google Scholar

Wang, Y. (2013). An Investigation of Visual Field Test Parameters in Glaucoma, Patterns of Visual Field Loss in Diabetics and Multispectral Imaging of the Optic Nerve Head in Glaucoma. Doctoral dissertation, The University of Manchester.

Google Scholar

Yang, T. T., Simmons, A. N., Matthews, S. C., Tapert, S. F., Bischoff-Grethe, A., Frank, G. K., et al. (2007). Increased amygdala activation is related to heart rate during emotion processing in adolescent subjects. Neurosci. Lett. 428, 109–114. doi: 10.1016/j.neulet.2007.09.039

PubMed Abstract | CrossRef Full Text | Google Scholar

Ziółkowski, A., Gorkovenko, A., Pasek, M., Włodarczyk, P., Zaranska, B., Dornowski, M., et al. (2014). EEG correlates of attention concentration in successful amateur boxers. Neurophysiology 46, 422–427. doi: 10.1007/s11062-015-9468-3

CrossRef Full Text | Google Scholar

Keywords: vital sign monitoring, brain activity, pupil size variation, harmonic frequency, non-contact measurement

Citation: Park S and Whang M (2018) Infrared Camera-Based Non-contact Measurement of Brain Activity From Pupillary Rhythms. Front. Physiol. 9:1400. doi: 10.3389/fphys.2018.01400

Received: 05 May 2018; Accepted: 14 September 2018;
Published: 10 October 2018.

Edited by:

Bernhard Schaller, Universität Zürich, Switzerland

Reviewed by:

Herbert Jelinek, Charles Sturt University, Australia
Andreas Voss, Institut für Innovative Gesundheitstechnologien (IGHT), Germany

Copyright © 2018 Park and Whang. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Mincheol Whang, d2hhbmdAc211LmFjLmty

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.