A Light Spot Humanoid Motion Paradigm Modulated by the Change of Brightness to Recognize the Stride Motion Frequency

Introduction

The steady-state visual evoked potential (SSVEP) is an electrophysiological signal evoked by periodic visual stimulation, with a stationary distinctive spectrum showing characteristic SSVEP peaks, stable over time (Vialatte et al., 2010). It has the advantages of high information transfer rate (ITR) and no need for training (Chen et al., 2015). Thus, it has attracted more and more attention. Since the cones cells concentrate at the center of the retina (McFarland and Wolpaw, 2011), the eye’s sensitivity to visual stimuli is the highest at the center of the visual field. In this way, the visual evoked potential signal carries some of the properties of the stimuli on which the user concentrates, such as the frequency and phase of the signal. Accordingly, it is possible to tell which stimuli the user is gazing at by comparing the extracted signal with the stimuli and then convert such information into computer instructions (Zhao et al., 2017).

According to a previous report (Regan, 1989), the low (4–12 Hz), medium (12–30 Hz) and high frequency range (−30 Hz) are the three frequency ranges to elicit an SSVEP. In general, the low frequency range could elicit larger amplitude SSVEP responses than the medium and high frequency ranges. Moreover, the larger the amplitude of the SSVEP, the easier its detection. Thus, many studies used low and medium frequency ranges to elicit SSVEP (No-Sang et al., 2015). However, low and medium frequency SSVEP ranges interfere with the alpha rhythm, and could cause an epileptic seizure (Fisher et al., 2005). The weakest SSVEP is found in the high frequency range (Diez et al., 2013). However, high frequency stimulation has the advantage of greatly decreasing the visual fatigue, caused by flickering, so that these stimuli can be used to develop a more comfortable SSVEP-based brain–computer interface (BCI) (Molina and Mihajlovic, 2010; Volosyak et al., 2011; Diez et al., 2013, 2014). However, the frequencies lower than 4 Hz seemed to be abandoned in eliciting SSVEP.

Usually, the traditional stimulus mode of SSVEP is light flashing or graph flipping (Lin et al., 2006), using a flickering light (the change of brightness) as a stimulus paradigm. The paradigm based on a flickering light is likely to cause visual fatigue and discomfort with a consequent decrease of the recognition accuracy. In recent years, BCI paradigms based on motion perception have been proposed (Snowden and Freeman, 2004). Xie et al. (2012) designed a BCI paradigm using Newton’s rings based on steady-state motion visual evoked potential (SSMVEP) that increased the recognition accuracy to a favorable level (Xie et al., 2014). Yan et al. (2018) designed four novel stimulus paradigms based on basic motion modes and demonstrated that any stimulus paradigms with periodic motion can induce SSMVEPs. However, all these research studies on the SSMVEP used the low and medium frequency bands. In fact, many human-related motion frequencies in our daily lives are lower than 1 Hz, such as the stride motion frequency, wave frequency, etc. In particular, the stride motion frequency refers to the number of times the legs alternate within 1 s. Human’s normal walking speed ranges from 10 to 80 steps per minute, which means that the stride motion frequency ranges from 0.08 to 0.67 Hz. The stride motion frequencies are lower than 4 Hz. Accordingly, few studies have reported the recognition of stride motion frequencies based on SSVEP. Moreover, all the SSVEP paradigms are based on flicker or motion separately. There is no paradigm combining flickering light and motion. Also, the characteristics of the brain response in the motion paradigm modulated by the change of brightness are unclear.

Besides, many researchers used SSVEP to control external moving devices, such as a wheelchair (Jzau-Sheng et al., 2014), a lower limb exoskeleton (No-Sang et al., 2015), a hand orthosis (Ortner et al., 2011), etc. However, all the stimulations were flickering LEDs or squares in the screen. The meanings of the targets were designed by the experimenters. Subjects would not know the meaning of the stimulation if the experimenter did not tell them the meaning. Consequently, it could be better for subjects to understand the meaning of the paradigms if the paradigms had similarity with the controlled devices.

In this study, to classify the different stride motion frequencies from the visual area in brain and to explore the characteristics of the brain response to the motion paradigm modulated by the change of brightness, the light spot humanoid motion paradigm is proposed for the first time. The paradigm was modulated with high-frequency flicker and low-frequency motion to produce humanoid walking. The flicker frequency was higher than 30 Hz so that the human eyes felt almost no flicker. Additionally, the motion frequency was the human stride frequency. The characteristics of the brain response to this paradigm were examined. In addition, canonical correlation analysis (CCA) was used to distinguish the different stride frequencies online. Then, the accuracies of distinguishing the different stride frequencies with the same and different high frequencies were compared. Finally, based on our previous rehabilitation training robot (Zhang et al., 2015), we employed the designed paradigm to control a lower limb robot with the same frequency stimulation. The robot provided motor feedback according to the identification of the electroencephalograph (EEG) signals when subjects stared at the humanoid motion paradigms with different stride frequencies. Moreover, the angles of the knee joint of the robot were obtained to test whether the feedback motion frequency was the same as the visual stimulation frequency.

Materials and Methods

Ethics Statement

Ten healthy male subjects (ages 23–27) participated in the experiments. They were all volunteers from Xi’an Jiaotong University. Subjects were studied after giving informed written consent in accordance with a protocol approved by the institutional review board of Xi’an Jiaotong University.

Design of the Light Spot Humanoid Motion Paradigm Modulated by the Change of Brightness

In this study, 20 green straw-hat LEDs were chosen as the visual stimulation which is shown in Figure 1. The 20 LEDs were placed in a human form based on the main points of the human skeleton, which included head, shoulder, elbow, spine, hip, knee, and ankle. Those LEDs were divided into three groups. Group 1 included the points of the human skeleton that almost immovable in human walking and comprised LED 1, LED 2, LED 3, LED 4, LED 8, LED 10, LED 11, and LED 12, which showed head, shoulder, spine, and hip. Thus, LEDs in group 1 were designed for the high-frequency flicker. Group 2 comprised LED 5, LED 9, LED 14, LED 15, LED 18, and LED 20, which showed arms and legs in stance phase. Group 3 comprised LED 6, LED 7, LED 13, LED 16, LED 17, and LED 19, which showed arms, and legs in swing phase. The LEDs in group 2 and group 3 flickered alternately at low frequency, which can form the walking motion as shown in Figure 1B. The motion was similar to the motion when the human traffic light turned green.

FIGURE 1

FIGURE 1. The light spot humanoid motion stimulus paradigm modulated by the change of brightness. (A) Are 20 green straw-hat LEDs and a hemispherical lampshade, and (B) is the moving process of the light spot human.

In addition, to create a more effective sense of motion, a lampshade was used to cover the LEDs. The hemispherical lampshade was made of light diffusing material with 85% light transmittance and each LED projected its light to the lampshade. The point light source became the surface light source. The light spot humanoid motion with a lampshade was more like the motion displayed on a screen. On the other hand, the lampshade made the paradigm more pedestrian and avoided the visual discomfort of the subjects due to the stimulation of the LEDs. Also, subjects were not able to see those LEDs that had been turned off in the designed paradigm with a lampshade. And those LEDs that had been turned off might confuse the subjects to recognize the shape of the paradigm.

Additionally, the STM32F103RB (ST-Microelectronics company, Geneva, Switzerland) was chosen as the microcontroller to control the flickering of the LEDs. By using the timer of STM32, a pulse width modulation (PWM) signal can be produced. Changing the duty ratio of the PWM signal could change the brightness of the LEDs. The duty ratios of the PWM to light up LEDs in group 1, group 2, and group 3 were changed as DR1, DR2, and DR3, respectively. The sum of DR1, DR2, and DR3 was the whole brightness of the paradigm. And the overall brightness was changed at frequency F. At the same time, DR2 and DR3 had a phase difference of π, and the motion was achieved by this phase difference. In addition, another timer was used to guarantee the accuracy of the cycle. The time interval was 1 ms.

$$\mathit{DR}1=(300+300\times \sin (2\times \mathrm{\pi }\times F\times t)/900)\mathrm{ }\left(1\right)$$

$$\mathit{DR}2=(250+200\times \mathrm{Sin}(2\times \mathrm{\pi }\times F\times t)+200\times \mathrm{Sin}(2\times \mathrm{\pi }\times f\times t))/900\mathrm{ }\left(2\right)$$

$$\mathit{DR}3=(250-200\times \sin (2\times \mathrm{\pi }\times F\times t)-200\times \sin (2\times \mathrm{\pi }\times f\times t))/900\mathrm{ }\left(3\right)$$

where F denotes the flicker frequency, f represents the motion frequency, and t is the time interval.

EEG Signal Measurement

The EEG signals were recorded from six EEG electrodes connected to a g.GAMMAbox and a g.USBamp EEG amplifier (g.tec Guger Technologies OG., Graz, Austria). The brain electrophysiological responses induced by visual stimulation were mainly distributed in the occipital region. So the six EEG electrodes were placed at PO3, PO4, POz, O1, O2, and Oz based on the international 10–20 system. The unilateral (left or right) earlobe was used as the recording reference and the Fpz was used as ground. All electrodes’ impedances were kept below 5 k Ohm during the experiments. The sampling frequency was 1200 Hz.

Lower Limb Robot

The XYKXZFK-9 lower limb robot (Xiangyu Medical Equipment Co. Ltd., China) was chosen to obtain motor feedback. This robot had been described in our previous research (Zhang et al., 2015). The robot was able to drive the lower extremity of users with reciprocating exercise similar to treadmill exercise. The knee joint was driven by the motor and other joints were follow-up. The control instructions of the robot can be sent through serial communication with computer. The control instructions included the speed of the pace, amplitude of the pace, etc. In addition, the angles of the knee joint during the movement can be obtained from the encoder.

Experimental Design

Two offline experiments described in Table 1 were designed to explore the characteristics of the brain response to the light spot humanoid motion paradigm modulated by the change of brightness. In addition, one online experiment described in Table 1 was designed to demonstrate the feasibility of recognizing stride motion frequencies and the EEG-driven lower limb robot with the same frequency stimulation.

TABLE 1

TABLE 1. Experimental design.

In the offline experiments (E1 and E2), the subjects were asked to sit on a comfortable chair in a quiet and ordinarily lit office room, and the designed paradigm was placed 40 cm in front of them. The LEDs were flickered in the way described in Section “Design of the Light Spot Humanoid Motion Paradigm Modulated by the Change of Brightness” for 5 s and turned off for 5 s automatically. All the subjects were required to stare at the paradigm for 5 s per trial with an interval of 5 s. There were 10 trials for each high frequency in E1, the high frequencies were 35, 45, 55, 65, 75, and 85 Hz, successively. Only the LEDs in group1 were lightened during the experiment E1. In addition, there were 20 trials for each stimulus frequency in E2. Specifically, 0.2, 0.4, and 0.6 Hz were chosen as the motion frequencies, while 45, 43, and 41 Hz were chosen as the flicker frequencies. Thus, the stimulus frequencies were: (45 Hz, 0.2 Hz), (45 Hz, 0.4 Hz), (45 Hz, 0.6 Hz), (43 Hz, 0.2 Hz), (43 Hz, 0.4 Hz), (43 Hz, 0.6 Hz), (41 Hz, 0.2 Hz), (41 Hz, 0.4 Hz), and (41 Hz, 0.6 Hz). As a result, the offline experiments E2 consisted of nine rounds. Throughout all the experiments, the EEG signals and the time were automatically recorded in the computer hard disk. After the experiments, the subjects were asked about the intuitive perception of the designed paradigms and the meaning of the paradigms.

Based on the offline experimental results, we explored the characteristics of the brain response to the light spot humanoid motion paradigm. Then, we chose the optimal frequency as the high-flicker frequency to perform the online experiments.

In the online experiments (E3), the subjects were fixed on the lower limb robot (XYKXZFK-9). And the standing angle of the robot was 70°. The three designed paradigms were placed 80 cm in front of the subjects. The motion frequencies were 0.2, 0.4, and 0.6 Hz, respectively. The LEDs flickered in the same way described in Section “Design of the Light Spot Humanoid Motion Paradigm Modulated by the Change of Brightness” for 5 s. Then, the LEDs were turned off for 12 s while the lower limb robot drove the subjects to perform lower limb movements with different stride frequencies according to the results of the identification. Additionally, the focused target was identified by CCA. Subjects were required to stare at the three targets one by one, there were 20 trials for each target. The stimulation duration was 4, 3, 2, and 1 s in succession. The process of the experiments was the same as described above. Notably, two different methods were used to present the humanoid motion with three different motion frequencies. One was that the high flicker frequency was the same (HFS). For example, the stimulus frequencies were (41 Hz, 0.2 Hz), (41 Hz, 0.4 Hz), and (41 Hz, 0.6 Hz). The other one was that the high flicker frequency was different (HFD). For example, the stimulus frequencies were (41 Hz, 0.2 Hz), (45 Hz, 0.4 Hz), and (43 Hz, 0.6 Hz). During the experiments, the EEG signals and the angles of the knee joint of the lower limb robot were automatically recorded in the computer hard disk.

Pre-processing of the EEG Data

The data collected from each subject were analyzed independently. The corresponding EEG data segments were extracted according to the starting time and ending time of the stimulus. The notch filtering from 48 to 52 Hz was conducted to remove the power line interface and the band pass filtering from 30 to 95 Hz was performed to remove the low frequency drift.

Common Average Reference

Common average reference (CAR) is commonly used in EEG, where it is necessary to identify small signal sources in very noisy recordings. The CAR is calculated as described in Equation (4). The mean value of all the electrodes is removed for a certain electrode (Alhaddad, 2012).

$$O_z=O_Z-(O_1+O_2+{\mathit{PO}}_3+{\mathit{PO}}_4+{\mathit{PO}}_z+O_z)/6\mathrm{ }\left(4\right)$$

Signal-to-Noise Ratio

To evaluate the significance of the amplitude frequency response in the EEG data, the signal-to-noise ratio (SNR) of the SSVEP is computed. The SNR of the SSVEP is calculated as the ratio between the power of a given frequency and the average power of its 20 surrounding neighbors.

$$\mathit{SNR}(f_n)=P(f_n)/\left(\frac{1}{20}\times {\displaystyle \sum _{\begin{array}{l}\text{q=-10}\\ \text{ q}\ne \text{0}\end{array}}^{10}P(f_{n+q})}\right)$$

where P(f_n) represents the Fourier power of a given frequency f_n.

Additionally, the EEG data epochs were extracted every 5 s. So the spectral resolution was 0.2 Hz.

Canonical Correlation Analysis

Canonical correlation analysis is widely used in SSVEP target recognition, where it is used to calculate the correlations between reference signals and multi-channel EEG data (Lin et al., 2006). The formula to calculate the correlation coefficient is as described in Equation (6).

$$\mathrm{\rho }=\underset{w_x,w_y}{\max }\frac{E[w_x^T{\mathit{XY}}^T{w}_y]}{\sqrt{E[w_x^T{\mathit{XX}}^T{w}_y]\mathrm{E}[w_x^T{\mathit{YY}}^T{w}_y]}}\mathrm{ }(6)$$

where X is the reference signals and Y is the EEG data.

In this study, the reference signals X were composed of eight groups of sine and cosine signals which were described as follows.

$$X=\left\{\begin{array}{l}\sin (2\times \mathrm{\pi }\times (F-f)\times t)\hfill \\ \cos (2\times \mathrm{\pi }\times (F-f)\times t)\hfill \\ \sin (2\times \mathrm{\pi }\times F\times t)\hfill \\ \cos (2\times \mathrm{\pi }\times F\times t)\hfill \\ \sin (4\times \mathrm{\pi }\times F\times t)\hfill \\ \cos (4\times \mathrm{\pi }\times F\times t)\hfill \\ \sin (2\times \mathrm{\pi }\times (F+f)\times t)\hfill \\ \cos (2\times \mathrm{\pi }\times (F+f)\times t)\hfill \end{array}\right\},t=\frac{1}{\mathit{Fs}},\frac{2}{\mathit{Fs}},\dots ,\frac{T}{\mathit{Fs}}\mathrm{ }(7)$$

Where F is the high flicker frequency and f is the low motion frequency.

After CCA is performed separately on each stimulus, the target can be determined by the maximum canonical coefficient.

Statistical Analysis

Statistical analysis was conducted using Wilcoxon signed-rank test which is a non-parametric statistical hypothesis test. Statistical significance is defined as p < 0.05.

Results

Offline Experimental Results of the Brain Response to High Frequency Stimulation

The experiments on the brain response to high frequency stimulation were designed to examine the characteristics of the brain response to high frequency stimulation and to determine the appropriate frequency for use as the high flicker frequency in our designed paradigm. The EEG spectra of one subject staring at the high-frequency flicker stimulations are shown in Figure 2. The high frequencies were 35, 45, 55, 65, 75, and 85 Hz, respectively. Before performing the fast Fourier transform, the EEG data of ten cycles corresponding to each frequency were superimposed and averaged on the time domain. Additionally, six channel signals were fused to a single channel data according to CAR. The data shown in Figure 2 reveals that when the stimulation frequency was less than 65 Hz the spectrum had a significant peak at the stimulation frequency, but when the stimulation frequency was 75 or 85 Hz, the amplitude at the stimulation frequency in the spectrum was not significant.

FIGURE 2

FIGURE 2. The EEG spectra when one subject stared at the high frequency flicker stimulations. (A) The high flicker frequency is 35 Hz, (B) the high flicker frequency is 45 Hz, (C) the high flicker frequency is 55 Hz, (D) the high flicker frequency is 65 Hz, (E) the high flicker frequency is 75 Hz, and (F) the high flicker frequency is 85 Hz.

Interestingly, the amplitude at the frequency in the spectrum was significant even when the second harmonic frequency was higher than 65 Hz. Taking the stimulation frequency of 45 Hz as an example, the second harmonic frequency was 90 Hz. The data shown in Figure 2B reveals that the peak in the spectrum at 90 Hz was significant. Even though the mechanism was not clear, we can add the second harmonic frequency as another feature to perform online identification.

Furthermore, in order to select the appropriate high flicker frequency for our designed paradigm, the SNRs were calculated. The average SNRs of the EEG data when subjects stared at the high frequency stimulations are shown in Figure 3. These results illustrate that the SNRs decreased as the stimulus frequencies increased. According to the feedback from the subjects, all the subjects felt the flickering of the LEDs when the frequency was 35 Hz and felt almost no flickering of the LEDs at other frequencies. Accordingly, the motion paradigm modulated by the change of brightness was designed without the sense of flicker. So the frequencies around 45 Hz were chosen as the high flicker frequencies in our designed paradigm.

FIGURE 3

FIGURE 3. The average SNRs of EEG data when subjects stared at the high frequency stimulations (35, 45, 55, 65, 75, and 85 Hz, respectively).

Offline Experimental Results on the Brain Response to the Light Spot Humanoid Motion Stimulation

The experiment on the brain response to the light spot humanoid motion stimulation was designed to explore the characteristics of the brain response to the motion modulated by the change of brightness. The EEG spectra of one subject staring at the various stimulations are shown in Figure 4. The flicker frequencies and motion frequencies (F, f) in the stimulations were (45 Hz, 0.2 Hz), (45 Hz, 0.4 Hz), (45 Hz, 0.6 Hz), (43 Hz, 0.2 Hz), (43 Hz, 0.4 Hz), (43 Hz, 0.6 Hz), (41 Hz, 0.2 Hz), (41 Hz, 0.4 Hz), and (41 Hz, 0.6 Hz) respectively.

FIGURE 4

FIGURE 4. The EEG spectra when one subject stared at the light spot humanoid motion stimulations. (A) The flicker frequency F was 45 Hz and the motion frequency f was 0.2 Hz, (B) the flicker frequency F was 45 Hz and the motion frequency f was 0.4 Hz, (C) the flicker frequency F was 45 Hz and the motion frequency f was 0.6 Hz, (D) the flicker frequency F was 43 Hz and the motion frequency f was 0.2 Hz, (E) the flicker frequency F was 43 Hz and the motion frequency f was 0.4 Hz, (F) the flicker frequency F was 43 Hz and the motion frequency f was 0.6 Hz, (G) the flicker frequency F was 41 Hz and the motion frequency f was 0.2 Hz, (H) the flicker frequency F was 41 Hz and the motion frequency f was 0.4 Hz, (I) the flicker frequency F was 41 Hz and the motion frequency f was 0.6 Hz.

The process of the EEG data was the same as the experiment on the brain response to high frequency stimulation (see details in Section “Pre-processing of the EEG Data”). Then the EEG data were filtered with a CCA-based filter. The duration of the stimulation was 5 s. So the spectral resolution was 0.2 Hz. As shown in Figure 4, peaks in amplitude can be precisely identified at the high flicker frequency, the sums and differences of the flickering frequency and motion frequency. In other words, the peaks were mainly evoked at the frequencies F, F + f, and F – f. Therefore, even the frequency which is lower than 1 Hz does not easily induce SSVEP, humanoid motion modulated with high-frequency light intensity scintillation can induce regular features. Moreover, from the inquiries after the experiments, we determined that all the subjects could hardly aware of the modulation with high-frequency flicker.

Then, based on the above described results, Equation (7) was chosen as the reference signal to perform the CCA with the EEG data. Additionally, the offline classification accuracies were calculated among all the ten subjects. In order to compare the relative classification performance of each target, Figure 5 shows confusion matrices for the stimulations for 5 s among all the subjects. The label ‘1’ to ‘9’ referred to the stimulations with the following stimulus frequencies (45 Hz, 0.2 Hz), (45 Hz, 0.4 Hz), (45 Hz, 0.6 Hz), (43 Hz, 0.2 Hz), (43 Hz, 0.4 Hz), (43 Hz, 0.6 Hz), (41 Hz, 0.2 Hz), (41 Hz, 0.4 Hz), and (41 Hz, 0.6 Hz), respectively. It was observed that the main misclassification occurred at those stimulations with the same flickering frequency. Also, the lower the flicker frequency of the stimulation, the higher the classification accuracy.

FIGURE 5

FIGURE 5. Confusion matrices for the stimulations at 5 s stimulus time among all the subjects. The color scale reveals the classification accuracies, the diagonals labeled with the correct classification accuracies.

Online Experimental Results

In the online experiments, CCA was used to calculate the accuracies of classifying three different motion frequencies. The online classification accuracies of the three stimulations by two different methods to depict the humanoid motion are shown in Table 2. The stimulus duration was 5 s, and the average accuracies by the two methods reached 87 and 95%, respectively. Remarkably, the classification accuracies of six subjects reached 100% with the designed paradigm with the different high flicker frequencies. In addition, the accuracies of most subjects were high, but subject 6 and 9 exhibited low classification accuracies. The low accuracies implied that the value of the canonical coefficient was low. The designed paradigm did not induce very strong SSVEP at the desired frequencies in these two subjects.

TABLE 2

TABLE 2. Online classification accuracies with 5 s stimulus duration.

Subsequently, the accuracies were averaged among all the subjects. The online accuracies with different stimulus durations are shown in Figure 6. We test normality and homogeneity of variances between the accuracies of distinguishing the different stride frequencies with the same and different high frequencies. We found that the distributions of the accuracies in 5 s duration using HFD was not normal (p = 0.002 < 0.05, Kolmogorov–Smirnov test). The distributions of accuracies in other groups were normal distribution. Thus, we chose Wilcoxon signed-rank test, which had no assumption on normality, to compare the accuracies of distinguishing the different stride frequencies with the same and different high frequencies. Statistical significance is defined as p < 0.05. The results showed that the classification accuracies of the designed paradigm using HFS were significantly lower than the accuracies of the designed paradigm using HFD when the durations were 5, 4, 3, 2, and 1 s [(z = −2.807, p = 0.005), (z = 2.710, p = 0.007), (z = −2.805, p = 0.005), (z = −2.805, p = 0.005), (z = −2.807, p = 0.005) respectively]. In addition, the classification accuracies of both methods were decreasing as the duration decreased. Specifically, when the stimulation duration was less than 3 s, there was a tendency for a sharp decrease in the classification accuracies.

FIGURE 6

FIGURE 6. The average classification accuracies among all the subjects with different stimulation durations.

Finally, the angles of the knee joint of the lower limb robot were acquired at the same time during the acquisition of the EEG signals during the online experiments. The time domain of the duty ratio of the PWM, time-frequency map of the EEG data, and angles of the knee joint of the lower limb robot are shown in Figure 7 for one subject who successively stared at three targets (M2: (41 Hz, 0.2 Hz), (45 Hz, 0.4 Hz), (43 Hz, 0.6 Hz)).

FIGURE 7

FIGURE 7. Time domain of duty ratio of the PWM, time-frequency map of EEG data, and angles of knee joint of the lower limb robot. (A) The time domain of duty ratio of the PWM of LEDs in group 2 in three targets. (B) The time domain of duty ratio of the PWM of LEDs in group 3 in three targets. (C) The time-frequency map of EEG data during the online experiment and the color scale reveals the values of power spectrum. (D) The angles of left knee joint of the lower limb robot during the online experiment. (E) The angles of right knee joint of the lower limb robot during the experiment.

The paradigms would light on for 5 s in the way described in Section “Design of the Light Spot Humanoid Motion Paradigm Modulated by the Change of Brightness,” and light out for 12 s, as shown in Figures 7A,B. Thus, the EEG data for the 0–5 s, 17–22 s, and 34–39 s periods were the data processed online. The EEG data were filtered with a CCA-based filter. Then, the time-frequency map was generated as shown in Figure 7C. The mapre 7C. The map revealed that the stimulations induced corresponding frequencies in the brain. Additionally, the system successfully recognized the intentions of the subjects as they stared at the three targets successively. It should be noted that our robot moved the left leg first in every operation and the speed of the first step is a fixed value in Figure 7D. Furthermore, when the subject stared at target 1 when the motion frequency of the stimulus was 0.2 Hz, as shown in Figures 7A,B, from 0 to 5 s, the robot’s movement frequency was also 0.2 Hz, as shown in Figures 7D,E from 10 to 15 s. Thus, these results demonstrated that a brain-controlled lower-limb robot with the same stimulus frequency and actual movement frequency was feasible.

Discussion and Conclusion

In this study, a light spot humanoid motion paradigm modulated by the change of brightness was proposed. In addition, the characteristics of the brain response to the motion paradigm modulated by the change of brightness were determined for the first time. By using these characteristics, we realized the recognition of the stride motion frequencies which were not in the regular frequency bands of the SSVEP. Moreover, we used the designed paradigm to control a lower limb robot by EEG with the same frequency stimulation.

The designed paradigm was created by the modulation of the motion and the change of brightness. It was an imitation of the liquid cry crystal display (LCD) monitor which is a usual way to elicit an SSVEP. The refresh rate of the designed paradigm was the high flicker frequencies (45, 43, and 41 Hz), while the screen refresh rate of the LCD was 60 Hz or larger. However, 60 Hz was too high to evoke available SSVEP. Thus, it can be speculated that if the stride motion was displayed on an LCD monitor with a lower screen refresh rate (lower than 45 Hz), the modulation phenomena in the EEG spectra also could be occurred. Furthermore, one group’s LED in the designed paradigm flickered in the traditional way. Subjects did not feel the flicker of the paradigm using the high flicker frequency, while the other two groups’ LEDs had a phase difference of π to produce alternate changes. In addition, these LEDs were covered with a lampshade which made the alternate changes generate a motion. And that was just like a motion displayed on a screen instead of an individual flicker. The total brightness of the paradigm was DR1 + DR2 + DR3 = DR1 + 500, and there was no motion frequency in DR1. Accordingly, the motion frequencies did not reflect in the total brightness.

Moreover, only stimulation frequencies within a specific frequency range can evoke strong SSVEP (Vialatte et al., 2010). Thus, some researchers used multiple frequencies to encode more targets (Teng et al., 2010; Zhang et al., 2012). Earlier, Mukesh et al. (2006) proposed a novel dual-frequency stimulation method to increase the number of selections in BCI. The peaks in the spectrum mainly occurred at F1, F2, F1 + F2, 2 × F2, F1 + 2 × F2, 2 × F1 + 2 × F2, 3 × F2, F1 + 3 × F2, 2 × F1 + 3 × F2. Also, Shyu et al. (2010) obtained a dual-frequency stimulation with two flickering LEDs. The peaks mainly occurred at F1, F2, 2 × F2 – F1, 2 × F1 – F2. More recently, Chang et al. (2014) proposed an amplitude-modulated visual stimulation, and the peaks occurred at 2 × f_c, 2 × f_m, f_c ± f_m, f_c ± 3 × f_m, 2 × f_c ± 4 × f_m. All the above-mentioned studies modulated the frequencies based on brightness and the results were not exactly the same. However, to the best of our knowledge, no study has reported the characteristics of the brain response to the motion paradigm modulated by the change of brightness. In this study, the high-frequency flicker and stride motion were modulated by the visual stimulation. The results of this study showed that the designed paradigm induced the main frequencies at F, F - f, and F + f (F: flicker frequency, f: motion frequency) in the brain. The motion frequency was modulated onto the side band of the high flicker frequency. However, there were no peaks at other modulation frequencies (such as F ± 2 × f) in the spectrum, which was different from the existing research. The reasons why those combined frequencies occurred are still unknown and need to be explored in the future. Besides, the frequency to elicit SSVEP was larger than 4 Hz according to (Regan, 1989). All these studies used the frequencies in the regular frequency regions (larger than 4 Hz), and no study considered how frequencies lower than 4 Hz could be used. In this study, we used the motion frequencies lower than 1 Hz, and demonstrated that such frequencies (lower than 1 Hz) can be used by modulated them with high flicker frequencies.

In addition, based on the online experimental results, the classification accuracies of the designed paradigm using HFD were significantly higher than the accuracies of the designed paradigm using HFS. The reason was that the paradigm using HFD had one more feature, namely the high flicker frequencies, than the paradigm using HFS. This finding indicated that not only the motion frequency f was different, but also the flicker frequency F in the reference signal was different when CCA was used to perform the classification. Regarding the visual perception, the subjects hardly felt the difference between the two methods. However, the difference can be detected through the EEG data. Moreover, the average accuracy reached 95% with the designed paradigm using HFD, and the accuracy was more than enough for the application in the BCI systems.

Finally, based on the EEG-driven lower limb rehabilitation training system, which we proposed in our previous study (Zhang et al., 2015), we applied the designed paradigm to control a robot by EEG with the same frequency stimulation. Even though in our previous study (Zhang et al., 2015) the virtual reality was designed to provide visual feedback, the stimulus was the normal SSMVEP paradigm. Accordingly, subjects could not know the meaning of the paradigm if the experimenter did not tell them. In addition, other SSVEP-based exoskeleton robots (Kwak et al., 2014) had used normal light scintillation stimulation. Also, there was no similarity between the visual stimulus and the motor feedback. In this study the humanoid motion paradigm was designed to induce SSVEP to control a robot. Additionally, all the subjects were asked what the paradigm was. And they responded that the paradigm was a walking human and the walking speeds were different. Thus, the designed paradigm has similarity with the motor feedback.

Overall, this study mainly determined the characteristics of the brain response to the motion paradigm modulated by the change of brightness and distinguished different stride motion frequencies from visual areas of the brain by designing a novel SSVEP paradigm. We found that the peaks were mainly evoked at the frequencies F, F + f, and F – f in the spectrum of the EEG data as the brain response to the paradigm. In addition, based on the findings, we achieved the recognition of the different stride motion frequencies by modulating the change of brightness on motion. Finally, the online experiment demonstrated the feasibility of the same frequency stimulation for an EEG-driven robot. In our future studies, we plan to increase the number of LEDs and divide more groups in the paradigm to make the paradigm more like a human with more realistic movement. And we will implement other algorithms to process the EEG data to improve the performance. Besides, all the subjects in the experiments are males. That may limit the extensibility of the results to general population. Thus, we may try to recruit female subjects to explore the sex effects. More importantly, visual stimulation can stimulate the motor cortex through the mirror neurons (Rizzolatti and Craighero, 2004) in the human brain. Nojima et al. (2012) also found that the mirror visual feedback has an important role in brain plasticity in the motor cortex. Thus an important question is whether it is feasible to use the SSVEP in rehabilitation training if the humanoid motion paradigm was chosen as the stimulation. Finding such an answer will be an interesting follow-up study.

Data Availability Statement

The datasets [the main data about the paradigm modulated by the change of brightness and motion] for this study can be found in the [FigShare] https://figshare.com/s/094b2fa683acca657991.

Author Contributions

XinZ: study design, manuscript preparation, data acquisition, and analysis. GX: study design and study supervision. XunZ: helped with the experiments. QW: helped in polishing the paper.

Funding

This research was supported by National Natural Science Foundation of China (Nos. 51475360 and 61503298) and National High Technology Research and Development Program of China (Approval No. 2015AA042301).

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We want to thank the subjects for participating in these experiments and reviewers for their helpful comments.

References

Alhaddad, M. J. (2012). Common average reference (CAR) improves P300 speller. Inter. J. Eng. Technol. 2:31.

Google Scholar

Chang, M. H., Baek, H. J., Lee, S. M., and Park, K. S. (2014). An amplitude-modulated visual stimulation for reducing eye fatigue in SSVEP-based brain-computer interfaces. Clin. Neurophysiol. 125, 1380–1391. doi: 10.1016/j.clinph.2013.11.016

PubMed Abstract | CrossRef Full Text | Google Scholar

Chen, X., Wang, Y., Nakanishi, M., Gao, X., Jung, T.-P., and Gao, S. (2015). High-speed spelling with a noninvasive brain–computer interface. Proc. Natl. Acad. Sci. U.S.A. 112, E6058–E6067. doi: 10.1073/pnas.1508080112

PubMed Abstract | CrossRef Full Text | Google Scholar

Diez, P. F., Mut, V. A., Laciar, E., and Perona, E. M. A. (2014). Mobile robot navigation with a self-paced brain-computer interface based on high-frequency SSVEP. Robotica 32, 695–709. doi: 10.1017/S0263574713001021

CrossRef Full Text | Google Scholar

Diez, P. F., Torres Müller, S. M., Mut, V. A., Laciar, E., Avila, E., Bastos-Filho, T. F., et al. (2013). Commanding a robotic wheelchair with a high-frequency steady-state visual evoked potential based brain-computer interface. Med. Eng. Phys. 35, 1155–1164. doi: 10.1016/j.medengphy.2012.12.005

PubMed Abstract | CrossRef Full Text | Google Scholar

Fisher, R. S., Harding, G., Erba, G., Barkley, G. L., and Wilkins, A. (2005). Photic- and pattern-induced seizures: a review for the epilepsy foundation of america working group. Epilepsia 46, 1426–1441. doi: 10.1111/j.1528-1167.2005.31405.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Jzau-Sheng, L., Wang, M., and Cheng-Hung, H. (2014). An SSVEP-based BCI system with SOPC platform for electric wheelchairs. Trans. Comput. Sci. Technol. 3, 35–40.

Google Scholar

Kwak, N. S., Müller, K. R., and Lee, S. W. (2014). “Toward exoskeleton control based on steady state visual evoked potentials,” in Proceedings of the International Winter Workshop on Brain-Computer Interface (BCI), New Delhi, 1–2. doi: 10.1109/iww-BCI.2014.6782571

CrossRef Full Text | Google Scholar

Lin, Z., Zhang, C., Wu, W., and Gao, X. (2006). Frequency recognition based on canonical correlation analysis for SSVEP-Based BCIs. IEEE Trans. Biomed. Eng. 54, 1172–1176. doi: 10.1109/TBME.2006.886577

PubMed Abstract | CrossRef Full Text | Google Scholar

McFarland, D. J., and Wolpaw, J. R. (2011). Brain-computer interfaces for communication and control. Commun. ACM 54, 60–66. doi: 10.1145/1941487.1941506

PubMed Abstract | CrossRef Full Text | Google Scholar

Molina, G. G., and Mihajlovic, V. (2010). Spatial filters to detect steady-state visual evoked potentials elicited by high frequency stimulation: BCI application. Biomed. Tech. 55, 173–182. doi: 10.1515/bmt.2010.013

PubMed Abstract | CrossRef Full Text | Google Scholar

Mukesh, T. M. S., Jaganathan, V., and Reddy, M. R. (2006). A novel multiple frequency stimulation method for steady state VEP based brain computer interfaces. Physiol. Meas. 27, 61–71. doi: 10.1088/0967-3334/27/1/006

PubMed Abstract | CrossRef Full Text | Google Scholar

No-Sang, K., Klaus-Robert, M., and Seong-Whan, L. (2015). A lower limb exoskeleton control system based on steady state visual evoked potentials. J. Neural Eng. 12:056009. doi: 10.1088/1741-2560/12/5/056009

PubMed Abstract | CrossRef Full Text | Google Scholar

Nojima, I., Mima, T., Koganemaru, S., Thabit, M. N., Fukuyama, H., and Kawamata, T. (2012). Human motor plasticity induced by mirror visual feedback. J. Neurosci. 32, 1293–1300. doi: 10.1523/JNEUROSCI.5364-11.2012

CrossRef Full Text | Google Scholar

Ortner, R., Allison, B. Z., Korisek, G., Gaggl, H., and Pfurtscheller, G. (2011). An SSVEP BCI to control a hand orthosis for persons with tetraplegia. IEEE Trans. Neural. Syst. Rehabil. Eng. 19, 1–5. doi: 10.1109/TNSRE.2010.2076364

PubMed Abstract | CrossRef Full Text | Google Scholar

Regan, D. (1989). Human Brain Electrophysiology: Evoked Potentials And Evoked Magnetic Fields In Science And Medicine. New York, NY: Elsevier.

Google Scholar

Rizzolatti, G., and Craighero, L. (2004). The mirror-neuron system. Annu. Rev. Neurosci. 27, 169–192. doi: 10.1146/annurev.neuro.27.070203.144230

CrossRef Full Text | Google Scholar

Shyu, K. K., Lee, P. L., Liu, Y. J., and Sie, J. J. (2010). Dual-frequency steady-state visual evoked potential for brain computer interface. Neurosci. Lett. 483, 28–31. doi: 10.1016/j.neulet.2010.07.043

PubMed Abstract | CrossRef Full Text | Google Scholar

Snowden, R. J., and Freeman, T. C. A. (2004). The visual perception of motion. Curr. Biol. 14, R828–R831. doi: 10.1016/j.cub.2004.09.033

PubMed Abstract | CrossRef Full Text | Google Scholar

Teng, F., Choong, A.M., Gustafson, S., Waddell, D., Lawhead, P., and Chen, Y. (2010). “Steady state visual evoked potentials by dual sine waves,” in Proceedings of the 48th Annual Southeast Regional Conference, Oxford. doi: 10.1145/1900008.1900077

CrossRef Full Text | Google Scholar

Vialatte, F.-B., Maurice, M., Dauwels, J., and Cichocki, A. (2010). Steady-state visually evoked potentials: focus on essential paradigms and future perspectives. Prog. Neurobiol. 90, 418–438. doi: 10.1016/j.pneurobio.2009.11.005

PubMed Abstract | CrossRef Full Text | Google Scholar

Volosyak, I., Valbuena, D., Luth, T., Malechka, T., and Graser, A. (2011). BCI demographics II: how many (and what kinds of) people can use a high-frequency SSVEP BCI? IEEE Trans. Neural Syst. Rehabil. Eng. 19, 232–239. doi: 10.1109/TNSRE.2011.2121919

PubMed Abstract | CrossRef Full Text | Google Scholar

Xie, J., Xu, G., Wang, J., Zhang, F., and Zhang, Y. (2012). Steady-state motion visual evoked potentials produced by oscillating newton’s rings: implications for brain-computer interfaces. PLoS One 7:e39707. doi: 10.1371/journal.pone.0039707

PubMed Abstract | CrossRef Full Text | Google Scholar

Xie, J., Xu, G., Wang, J., Zhang, S., Zhang, F., Li, Y., et al. (2014). Addition of visual noise boosts evoked potential-based brain-computer interface. Sci. Rep. 4:4953. doi: 10.1038/srep04953

PubMed Abstract | CrossRef Full Text | Google Scholar

Yan, W., Xu, G., Xie, J., Li, M., and Dan, Z. (2018). Four novel motion paradigms based on steady-state motion visual evoked potential. IEEE Trans. Biomed. Eng. 65, 1696–1704. doi: 10.1109/TBME.2017.2762690

PubMed Abstract | CrossRef Full Text | Google Scholar

Zhang, X., Xu, G., Xie, J., Li, M., Pei, W., and Zhang, J. (2015). “An EEG-driven lower limb rehabilitation training system for active and passive co-stimulation,” in Proceedings 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Honolulu, HI, 4582–4585.

Google Scholar

Zhang, Y., Xu, P., Liu, T., Hu, J., Zhang, R., and Yao, D. (2012). Multiple frequencies sequential coding for SSVEP-based brain-computer interface. PLoS One 7:e29519. doi: 10.1371/journal.pone.0029519

PubMed Abstract | CrossRef Full Text | Google Scholar

Zhao, X., Zhao, D., Wang, X., and Hou, X. (2017). A SSVEP stimuli encoding method using trinary frequency-shift keying encoded SSVEP (TFSK-SSVEP). Front. Hum. Neurosci. 11:278. doi: 10.3389/fnhum.2017.00278

PubMed Abstract | CrossRef Full Text | Google Scholar