Effect of Different Movement Speed Modes on Human Action Observation: An EEG Study

Action observation (AO) generates event-related desynchronization (ERD) suppressions in the human brain by activating partial regions of the human mirror neuron system (hMNS). The activation of the hMNS response to AO remains controversial for several reasons. Therefore, this study investigated the activation of the hMNS response to a speed factor of AO by controlling the movement speed modes of a humanoid robot's arm movements. Since hMNS activation is reflected by ERD suppressions, electroencephalography (EEG) with BCI analysis methods for ERD suppressions were used as the recording and analysis modalities. Six healthy individuals were asked to participate in experiments comprising five different conditions. Four incremental-speed AO tasks and a motor imagery (MI) task involving imaging of the same movement were presented to the individuals. Occipital and sensorimotor regions were selected for BCI analyses. The experimental results showed that hMNS activation was higher in the occipital region but more robust in the sensorimotor region. Since the attended information impacts the activations of the hMNS during AO, the pattern of hMNS activations first rises and subsequently falls to a stable level during incremental-speed modes of AO. The discipline curves suggested that a moderate speed within a decent inter-stimulus interval (ISI) range produced the highest hMNS activations. Since a brain computer/machine interface (BCI) builds a path-way between human and computer/mahcine, the discipline curves will help to construct BCIs made by patterns of action observation (AO-BCI). Furthermore, a new method for constructing non-invasive brain machine brain interfaces (BMBIs) with moderate AO-BCI and motor imagery BCI (MI-BCI) was inspired by this paper.

1. Introduction

Human mirror neuron system (hMNS) components activate the sensorimotor, parietal, and occipital cortices when a human performs, observes, or imitates an action (Rizzolatti, 2005; Tanji et al., 2015). Initial studies of the hMNS focused on motor imitation and learning, as well as action understanding and observation (Rizzolatti and Sinigaglia, 2010; Pineda et al., 2013). Furthermore, the hMNS has gradually come to be regarded as crucial for social skills such as understanding the intensions and emotional states of others (Blakemore and Decety, 2001; Schulte-Rüther et al., 2007). Action observation (AO) was considered one of the primary ways of inducing hMNS activations in early studies (Caspers et al., 2010; Mukamel et al., 2010; Rozzi and Fogassi, 2017). The modulation of hMNS recruitments by AO has positive effects on cognitive psychology and sport rehabilitation for creating scientific recovery strategies (Franceschini et al., 2012). hMNS activations by AO also provide another way to construct a brain computer/machine interface (BCI) for mechanical control and medical auxiliary purposes (Neuper et al., 2009). A BCI constructs an alternative method for communicating between human and computers/machines. In BCIs, the motor related BCI was constructed by the variation patterns in mu (8–12 Hz) and beta (18–25 Hz) bands power extracted by electroencephalography (EEG) when a human image, observer, or execute limb/leg movements. Variations in band power in the two rhythms are referred to as event-related desynchronization (ERD) suppression. Hence, the ERD suppression will be caused by motor imagery (MI) and AO. In fact, the definitions of AO tasks influence the degree of ERD suppression. However, the relationships between action definitions for AO and hMNS activations are still unknown from a scientific view.

In general, in the movements designing of an AO therapy for stroke patients or a BCI system, the parameters and affecting factors will influence the performance of therapy or BCI accuracy. In fact, the affecting factors will be controlled if we explore the disciplines of the relationships between movements for AO and the hMNS activations. Hence, the the research of the affecting factors of movements for action observation will do favor for designing a AO therapy scheme or an AO-BCI system. Most prior studies offer different hypotheses regarding the relationships between movements for AO and the hMNS activations (Iacoboni et al., 2005; Filimon et al., 2007; Newman-Norlund et al., 2007). Since many AO studies are based in cognitive psychology and sports psychology, while few are based in rehabilitation and BCI research, the distribution and degree of hMNS activations remain controversial with respect to several features of AO (Perry et al., 2010; Vogt et al., 2013). The mode of AO movements is the main factor that limits hMNS and motor system activation. Several neuro-physiological studies indicate that ongoing movement plays an important role in triggering hMNS responses when activations are induced by AO (Maranesi et al., 2013). Aside from the condition of ongoing movements, other studies hypothesize that the manifestations of actions in AO tasks influence hMNS activations. In a study of action meaning (Agnew et al., 2012), compared with meaningless movements, meaningful movements caused higher hMNS activations. For example, lipreading of speech is one of the meaningful movements. Researches on lipreading showed lipreading will enhance speech perception and speech recognition (Summerfield, 1992; Silsbee and Bovik, 1996). Lipreading enhancement in young deaf children will help auditory speech perception (Geers, 1994). Lipreading and audiovisual speech integration are used in therapy of autism (Smith and Bennetto, 2007). Hence, the researches on action meaning will be useful for the therapy, rehabilitation, and neuroscience. In addition, the degree of movement complexity in an mode of AO movement has also been found to affecting hMNS activations (Biagi et al., 2010; Gatti et al., 2017). Moreover, additional variables that modulate hMNS activation and ERD suppression, such as experience and familiarity, have also been studied (Calvo-Merino et al., 2006; Bello et al., 2014).

Although the factors affecting the relationships between movements for AO and hMNS activation have been explored by the above studies, hMNS specificity and its full significance remain uncertain. The definitions of AO tasks clearly influence the degree of ERD suppression (Orgs et al., 2008). Furthermore, the manifestations of AO tasks and their stimuli affect attentional resources and hMNS activation (Oberman et al., 2007). Since the majority of affecting factors are related to the modes of AO movements, explorations of the relationships between movements for AO and hMNS activation are translated into the relationships between movement modes and the degree of ERD suppressions. Speed, complexity, magnitude, and individual proficiency are well-characterized factors known to affect movement modes, and have been explored in previous studies (Abdi and Williams, 2010; Biagi et al., 2010; Agnew et al., 2012; Maranesi et al., 2013; Bello et al., 2014). However, to the best of our knowledge, the present study is the first to investigate the effect of speed during AO. If the speed affecting factor is well-explored for AO among individuals, the designing for therapy scheme and AO-BCI will be more accurate for individuals. In addition, since the parameters and affecting factors will be well controlled in designing AO-BCI, but not in designing MI-BCI, the performance of AO-BCI will be better than MI-BCI. Furthermore, based on BCI system, the brain computer/machine brain interfaces (BMBIs) are capable of bidirectional communication with the brain. The interfaces consist of efferent and afferent modules. The efferent modules decode motor intentions by the variation patterns in mu and beta bands power, like conventional BCIs. The afferent modules encode feedback about the interactions of the machine through patterns of intracortical microstimulation. In general, the efferent modules are always designed by patterns of motor imagery. Meanwhile, the afferent modules can't be designed by patterns of MI. However, the afferent modules will be designed by patterns of action observation. When the efferent modules by MI drive a machine performing movements, the feedback by observation to activate hMNS will cause ERD suppression to design afferent modules. Today's BMBIs are constructed without considering the affecting factors of movements, so the performance is bad. However, if the speed and other affecting factors are considered for designing BMBIs, the performance will be better.

EEG (Lei et al., 2015; Abbott, 2016) has been widely used to construct BCI systems for exploring hMNS activations due to its high temporal resolution. In our study, the experimental movements were made by a humanoid robot at four different speed modes for AO tasks. A humanoid robot platform provides identical appearance, background and complexity to eliminate other effects. All four speed modes, slow (0.25 Hz), moderate (1.25 Hz), fast (2.75 Hz), and finalistic (4 Hz), were defined by questionnaires administered to 26 participants. Another six subjects were asked to attend AO experiments and an MI experiment for comparison. All eight stimuli (left movement and right movement at four speed modes) were randomly presented to all subjects in AO experiments, and another two stimuli were randomly presented to the same subjects in MI experiments. Both experiments are designed by a BCI form to validate the relationships between speed mode of AO and the hMNS activations. Due to the high temporal resolution, quantitative EEG (Lei et al., 2015; Abbott, 2016) is used as a convenient and cheap method to explore hMNS activations by measuring ERD suppressions (Oberman et al., 2007; Orgs et al., 2008). The ERD suppressions in EEG data are analyzed using a series of BCI methods and spatio-spectral-temporal characteristics. The following three main points are analyzed and discussed in this study:

(1) The brain regions involved in AO, such as the sensorimotor and occipital regions, are compared since studies on activation regions for AO have paradoxical conclusions (Perry et al., 2010; Frenkel-Toledo et al., 2013).

(2) The patterns of the relationships between AO speed modes and hMNS activations. If the patterns are clearly explored by experiments, individual rehabilitation strategies with specific optimal speeds will be precisely designed for individuals.

(3) Application to brain machine brain interface (BMBI). The average pattern of all individuals will show optimal speed ranges that are best suited to constructing AO-BCIs. Moreover, a well-established non-invasive AO-BCI with a high rate of accuracy and information transfer rate (ITR) will help to build BMBIs.

2. Materials and Methods

2.1. Ethics Statement

This study was carried out in accordance with the recommendations of the guideline of EEG experiment, Xiamen University ethics committee with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the Xiamen University ethics committee.

2.2. Participants

Six right-handed healthy human subjects (five male and one female, mean age: 24.6 years, SD = 1.46) were invited to participate in AO and MI experiments with payment. All subjects had no history of neurological disease and either normal or corrected-to-normal vision. Subjects also did not have any professional experience with AO or MI experiments. In accordance with the principles of the Helsinki Declaration, all subjects provided written informed consent approved by the Xiamen University.

2.3. Stimuli Design

As shown in Figure 1, stimuli for AO in different speed modes were performed by a humanoid robot. AO stimuli were presented to all subjects in 12 sessions as experiment 1. Afterwards, an MI comparison experiment in which two guidance arrows were presented to the same subjects in three sessions was performed as experiment 2.

FIGURE 1

Figure 1. Stimuli design of AO in different speed modes. The Step 1 contains one loop of periodic-swinging arm movement includes 100 frames over four seconds because the platform has a fixed frame rate of 25 fps. The Step 2 summarizes the results of the correct maximum loop number counted by each participant. The Step 3 defines the role of speed modes in the pattern of relationships between AO tasks and hMNS activations. As well as for simplicity and convenience, four different loops with a constant interval were constructed. Step 1: One loop for AO at standard speed. Step 2: Twenty-six participants count loops at higher speed modes to determine the max (16 loops). Step 3: Select four speed modes for experiments by equi-partition.

2.3.1. Experiment 1: AO Experiments in Different Speed Modes

In previous studies, AO experimental paradigms were designed with sequential video clips of human arm, hand, or finger movements presented to subjects (Frenkel-Toledo et al., 2014; Gatti et al., 2017). However, our experiments were designed to explore speed factor during AO. Because human movements feature unconstant speed, angle, and magnitude across repetitive experiments, the inconsistency of the variables will introduce confounding factors that disturb the analyses of hMNS activations. To eliminate such confounding factors, a fine-tuning parameter-setting paradigm must be incorporated into the design. Based on the anthropomorphic and relevant AO studies (Press, 2011), the hMNS is strongly activated by observations of both human and non-human agents (e.g., humanoid robots). Human or non-human agents do not differ significantly in their ability to activate the hMNS in AO studies involving humans and humanoid robots. Since the robot platform features an operating system for controlling all movement parameters, a humanoid robot platform was used in the design of our AO experimental paradigms.

Based on the “NAO Choregraphe 1.14 platform” (Pot et al., 2009), periodic-swinging movements of the robot's left/right arm were utilized as AO tasks in experiment 1. As shown in Figure 1 (Step 1), one loop of periodic-swinging arm movement includes 100 frames over 4 s because the platform has a fixed frame rate of 25 fps. Five key frames for both the left and right arms show the same procedure of the periodic movements. The robot stretches its arm at the beginning of the loop and waves its arm up to the head during half of the loop. The remainder of the loop consists of the same procedure in the opposite direction. The platform ensures that the robot's left/right-arm movements maintain a constant speed, angle, and magnitude. The entity body of the robot was presented to subjects in order to distinguish left-arm or right-arm movements during AO using EEG-based BCI analysis methods (Bauer and Gharabaghi, 2015). The robot's arms are dynamic, while the other parts of the robot remain static to keep the speed variable clear and unique.

Theoretically, more speed modes will be better for the results. However, due to the limitation of participants and the experimental platform, we only explore the rough pattern of the factor affections for simplicity and convenience. Four different loops with a constant interval were constructed as shown in Figure 1 (Step 3). For simplicity, the four incremental speed movements are defined as “slow movement,” “moderate movement,” “fast movement,” and “finalistic movement.” In Figure 1 (Step 1), one loop of 100 frames over 4 s is defined as “slow movement.” Due to time duration in AO experiments (4 s per trial), we obtain higher movement speed modes by adding number of loops, and compressing multi-loops within 4 s. To identify a robust and suitable number of multi-loops for “finalistic movement,” 26 participants were invited to watch 10 different looped (11–20) video clips that were compressed into 4 s. Participants were asked to count the number of loop in each video clip. The results from all participants were then compared with ground truth to identify the maximum loop number for each participant. Figure 1 (Step 2) summarizes the results of the correct maximum loop number counted by each participant. The results show that 16 loops was the maximum loop number for all of the participants. Therefore, “finalistic movement” was defined as 16 loops compressed into 4 s. “Moderate movement” was set to six loops, and “fast movement” was set to 11 loops, by the principle of equipartition. In all, experiment 1 gives four different speed modes of AO to explore the role of speed. All four speed modes were presented to subjects during AO in experiment 1.

2.3.2. Experiment 2: Relevant MI Experiment

After experiment 1, all six healthy human subjects were asked to attend experiment 2. The MI experiment was compared with the AO experiment. Following the design criteria of MI experimental paradigms (Tangermann et al., 2012), two guidance arrows were defined in experiment 2 to guide the imagined movement direction. In experiment 2, the subjects were asked to imagine his/her-self as performing the same movement as the humanoid robot in experiment 1. Both MI guidance arrows used in experiment 2 were presented to all subjects after experiment 1.

2.4. Experimental Procedures

As shown in Figure 2, the experimental procedures for AO in different speed modes and MI were divided into trials and sessions. EEG recordings were set for both AO and MI experiments.

FIGURE 2

Figure 2. Experimental procedures. In each trial, initial instructions guided subjects to simply observe the robot's movements (experiment 1) or guidance arrows (experiment 2) on the screen and a manually respond as rapidly as possible whenever the catch trial static robot (experiment 1) or blank (experiment 2) is presented. Each trial started with a fixation cross presented in the middle of a gray screen. The duration of the fixation cross was 2,000 ms. After presentation of the fixation cross, stimuli were presented in the middle of the gray screen for 4,000 ms to investigate participants' ERD suppressions via EEG recordings and were followed by a 2,000-ms pause as a break. Each of the eight AO stimuli (left and right in four speed modes) was randomly presented four times, and the catch trial stimulus was randomly presented eight times within one session in Experiment 1. Each of the two MI stimuli (left and right) was randomly presented four times, and the catch trial stimulus was randomly presented eight times within one session in Experiment 2. (A) Example trial sequence for experiment 1 and experiment 2. (B) Task descriptions for experiment 1. (C) Task descriptions for experiment 2. (D) Example session sequence for experiment 1 and experiment 2.

2.4.1. Trial Procedures

Influenced by conventional AO/MI experimental trial setups (Calvo-Merino et al., 2004; Wang et al., 2012), the presentation of a trial is shown in Figure 2A. In each trial, initial instructions guided subjects to simply observe the robot's movements (experiment 1) or guidance arrows (experiment 2) on the screen and a manually respond as rapidly as possible whenever the catch trial static robot (experiment 1) or blank (experiment 2) is presented. Subjects were also asked to minimize eye movements and keep their eyes fixed on the monitor when observing the stimuli. Manual responses to the catch trial were input by pressing the “SPACE” button. Each trial started with a fixation cross presented in the middle of a gray screen. The duration of the fixation cross was 2,000 ms. After presentation of the fixation cross, stimuli were presented in the middle of the gray screen for 4,000 ms to investigate participants' ERD suppressions via EEG recordings and were followed by a 2,000-ms pause as a break. If the presentation was an AO stimulus (experiment 1), subjects were asked to focus on the moving arm of the robot; otherwise, when the presentation was an MI stimulus (experiment 2), subjects were asked to imagine him/her-self moving like the robot by following the guidance arrows. Subsequent trials followed the same procedure for both experiment 1 and experiment 2.

2.4.2. Session Procedures

In experiment 1, each subject completed 12 AO sessions within 6 days, whereas in experiment 2, each subject completed three MI sessions within 2 days. Data recordings for one session occurred in the morning, with another session taking place later in the afternoon the same day. As shown in Figure 2D, each session comprised two tasks, and each task was divided into 40 trials. A relaxation time to prevent fatigue was determined by subjects between the two tasks. Experiment 1 comprised eight AO stimuli and one catch trial stimulus, which are described in Figure 2B. Each of the eight AO stimuli (left and right in four speed modes) was randomly presented four times, and the catch trial stimulus was randomly presented eight times within one session (See Figure 2D). Experiment 2 comprised two MI stimulus arrows and one blank catch trial stimulus, which are described in Figure 2C. Each of the two MI stimuli (left and right) was randomly presented four times, and the catch trial stimulus was randomly presented eight times within one session (See Figure 2D). Response times to the catch trial stimulus were recorded for both experiments 1 and 2 because the response times help confirm attentional states during AO and MI experiments.

2.4.3. EEG Recordings

Each subject was seated individually in a comfortable arm chair in front of a computer monitor in a dimly lit, sound-attenuated room and asked to read the instructions carefully on the screen. AO stimuli (experiment 1) and MI stimuli (experiment 2) were displayed on a DELL P2314H LCD monitor with a refresh rate of 60 frames/s. Displays were controlled by the psychology software tool “E-Prime 2.0.” The distances between monitor and the subjects were set at 100 cm by following the BCI experimental criterion to prevent other confounding factors in the hMNS activation (Onishi et al., 2017). During the entire run of both experiments, a spatially homogeneous gray background with a luminance of 38 cd/m2 was enabled. EEG recordings for all subjects were performed on a “NeuroScan SynAmps2” device with a “Neuroscan QuikCap international 10–20 system.” EEG signals were referenced to the nose, grounded at the frontal position (Fpz), and sampled at 250 Hz. After data acquisition, the preprocessing operations on the signals for notch filtering and bandpass filtering were 48–52 and 0.1–100 Hz, respectively. All 64 electrode impedances in QuikCap were kept below 5kω during the experiment. Moreover, the horizontal and vertical EOG signals were recorded. The EOG signals were used to correct for the influence of blinking and eyeball movements.

2.5. Analysis Methods

As shown in Figure 3, three main EEG data analyses procedures were utilized in this study. Figure 3 (Step 1) illustrates the preprocessing procedure. Raw EEGs from six subjects were first corrected for EOG artifacts via blind component separation. Then, a band filter ranging from 0.15 to 30 Hz was adopted for EEG data, since the effective parts are above 0.15 Hz and under 30 Hz. By using AO (experiment 1) and MI (experiment 2) labels recorded in “NeuroScan,” fragments of each experimental condition were extracted for all subjects. After fragment extraction, a baseline calibration method to prevent deviation and a manual removal method to reducing artifacts were performed on all fragments.

FIGURE 3

Figure 3. EEG data analysis procedures. Step 1: Pre-processing. Step 2: BCI analyses. Step 3: Extracting spatio-spectral-temporal characteristics.

Recent studies first found that hMNS activations initially occurs in the sensorimotor region of the brain during AO (Perry et al., 2010). The ERD suppressions in the sensorimotor region then extend to attentional resources, which cause more intense hMNS activations in the occipital region (Frenkel-Toledo et al., 2013). Likewise, ERD suppressions caused by MI are first found in the sensorimotor region and then extend to global cerebral activity. To explore the affected regions of the brain during AO, the patterns of relationships between AO speed modes and hMNS activations, as well as to compare AO and MI to construct a BMBI, the sensorimotor, occipital and “sensorimotor + occipital” regions were adopted for BCI analyses of both AO and MI experiments. Potentials covering central-parietal regions C3, C1, Cz, C2, C4, P3, P1, Pz, P2, and P4 were chosen for the sensorimotor region, while potentials covering parietal-occipital regions PO5, PO3, POz, PO4, PO6, O1, Oz, and O2 were chosen for the occipital region. The “sensorimotor + occipital” region included both sets of potentials.

As illustrated in Figure 3 (Step 2), the procedure for BCI analyses comprised four steps. First, left fragments and right fragments were grouped by identical AO speed (experiment 1). Second, all groups of fragments were presented to extract filter bank common spatial pattern (FB-CSP) features (Ang et al., 2012). Since ERD suppressions are modulated by the power variations of the left and right cerebral hemispheres, an FB-CSP algorithm was adopted to extract power variations using a bank of filters and an optimization method. The optimal CSP eigen-vectors were set to S = 4 based on experience. Third, the principal component analysis (PCA) algorithm of a linear dimensionality reduction technique was adopted to reduce feature dimensions (Abdi and Williams, 2010). Because of the high sampling rate in EEG recordings, FB-CSP features have a high number of dimensions that are hard to classify by machine-learning models. The PCA algorithm looks for the highest variability in FB-CSP features by projecting original features to lower dimensions. The optimal PCA was set to M = 10 for the regulation of “99%” of the variability. Finally, the dimension-reduced FB-CSP features were incorporated into the support vector machine (SVM) classifier for BCI classification (Sardouie and Shamsollahi, 2012). The SVM classifier is a common BCI classifier with excellent generalizability by searching for the largest margin of a decision hyperplane that will strictly classify the unseen test data. The reductive features were incorporated into SVM classifiers with a polynomial kernel. A 6*6 cross-validation strategy was applied to train the SVM classifiers for all five groups. Since each group has 392 trials of effective EEG fragments with a random order for left and right, 320 trials were provided as training data, and the remaining 64 trials as evaluation data in one loop of evaluation.

Figure 3 (Step 3) displays the procedure for extracting spatio-spectral-temporal characteristics. Subject 4, who had the best BCI accuracies, was chosen for the analyses of these characteristics. The temporal characteristics were first extracted by calculating mean voltages of EEG fragments among four different speed AO modes (both for left and right). Then, a power spectral density (PSD) (Demandt et al., 2012) algorithm with parameters of specific rhythm ranges was used for each average fragment to obtain spectral characteristics. The spectral characteristics were drawn by the “eeglab 14.1.1” tool as “ERD images” (Delorme and Makeig, 2004). Brain electrical activity maps (BEAMs) (Duffy et al., 1979) were imported to obtain the positions of the potentials, and the spatio-spectral characteristics were drawn by the “eeglab 14.1.1” tool.

In addition, the response times of catch trial stimuli as feedback behavioral data were analyzed by statistical analyses. Mean time values for all 15 sessions (Sessions 1–12 for experiment 1 and sessions 13–15 for experiment 2) were calculated for all subjects to obtain the attentional conditions of all sessions by calculating mean values and standard deviations among all sessions.

3. Results

3.1. Feedback Behavioral Data

Catch trial stimuli appear 16 times in each session, and we have recorded the response times for all catch trial stimuli in each session. Tables 1, 2 illustrate the mean response times within one session for experiment 1 and experiment 2, respectively. The mean time and standard deviations among all sessions in both experiments are also presented.

TABLE 1

Table 1. Statistical analyses of response times for Experiment 1 (ms).

TABLE 2

Table 2. Statistical analyses of response times for Experiment 2 (ms).

As shown in Table 1, the average response times among all sessions of experiment 1 were within 2,000 ms, and the corresponding standard deviations were within 100–200 ms. These results demonstrate that all subjects exhibited focused attention during AO experiments and EEG recordings. Therefore, EEG recordings in experiment 1 were free from attention artifacts. The standard deviations of all subjects were within a similar range, indicating that the responses to stimuli were stable. The response times of Subjects 2 and 3 had lower averages and standard deviations, which may be related to their younger age, as this tends to be related to faster response times. Since our behavioral data conform to experimental principles, the EEG data of experiment 1 were considered objective and appropriate for analyses. Similarly, as illustrated in Table 2, the average response time among all sessions of experiment 2 were within 300 ms, and the corresponding standard deviations were within 10 and 20 ms. These results conform with the response times of MI experiments, indicating that the EEG recordings in experiment 2 were objective and appropriate for analyses. Compared with Tables 1, 2, the blank catch trial stimuli was easier to recognize than a static robot, and the individual differences in the response times of younger subjects has less influence on the MI experiments.

3.2. BCI Analyses of AO and MI Experiments

After preprocessing, the EEG fragments (left and right) of four speed AO modes and MI modes were used for BCI analyses. In FB-CSP feature extraction, the bank of bank filters following a previous study are shown in Table 3. Table 3 illustrates the BCI analysis results for three regions in four AO speed modes and MI mode. The average accuracies and standard deviations of BCI classification and ITR were calculated. To display the comparison of the results, Figure 4 shows the average accuracies of all modes in all regions.

TABLE 3

Table 3. The subbands of band-pass filters in FB-CSP feature extraction.

FIGURE 4

Figure 4. The average accuracies of all modes in all regions. Potentials covering central-parietal regions C3, C1, Cz, C2, C4, P3, P1, Pz, P2, and P4 were chosen for the sensorimotor region, while potentials covering parietal-occipital regions PO5, PO3, POz, PO4, PO6, O1, Oz, and O2 were chosen for the occipital region. The “sensorimotor + occipital” region included both sets of potentials.

From the BCI analysis results in Table 4 and Figure 4, we see four findings:

(1) For the accuracies in the occipital region, the results were AO-0.25 Hz (71.44%), AO-1.5 Hz (74.78%), AO-2.75 Hz (58.42%), and AO-4 Hz (59.29%). However, the results were AO-0.25 Hz (61.76%), AO-1.5 Hz (69.27%), AO-2.75 Hz (56.29%), and AO-4 Hz (55.99%) in the sensorimotor region. Similarity, the accuracies of the “sensorimotor+occipital” were AO-0.25 Hz (62.07%), AO-1.5 Hz (68.92%), AO-2.75 Hz (56.81%), and AO-4 Hz (56.33%). Considering the results across all regions, the BCI accuracies from increasing speed modes of AO presented a pattern of initially increasing and then decreasing to a stable level.

(2) Comparing AO-BCI accuracies between occipital and sensorimotor regions, the results were 71.44 and 61.76% at AO-0.25 Hz, 74.78 and 69.27% at AO-1.5 Hz, 58.42 and 56.29% at AO-2.75 Hz, and 59.29 and 55.99% at AO-4 Hz. AO in any speed mode had a more significant BCI accuracy in the occipital region than in the sensorimotor region. However, the accuracies of MI-BCI were 57.72% in the occipital region and 61.35% in the sensorimotor region. MI mode had a more significant BCI accuracy in the sensorimotor region than in the occipital region.

(3) The BCI accuracies in the “sensorimotor+occipital” region were at a similar level to those in the sensorimotor region. The results were 62.07 and 61.76% at AO-0.25 Hz, 68.92 and 69.27% at AO-1.5 Hz, 56.81 and 56.29% at AO-2.75 Hz, and 56.33 and 55.99% at AO-4 Hz. However, a large difference was detected between the “sensorimotor+occipital” and occipital regions. The results were 62.07 and 71.44% at AO-0.25 Hz, 68.92 and 74.78% at AO-1.5 Hz, 56.81 and 58.42% at AO-2.75 Hz, and 56.33 and 55.99% at AO-4 Hz. These results suggested the BCI accuracy was more robust in the sensorimotor region and was resistant the influence of other regions. Moreover, the higher BCI accuracies in the occipital region may be influenced by factors other than hMNS activations.

(4) Because the standard deviations of all experimental modes in all regions were within the range of [0, 10], AO in different speed modes and MI mode have similar robustness. The average standard deviations in all regions between AO modes showed that AO-0.25 Hz (5.69) and AO-1.5 Hz (5.80) have lower robustness than AO-2.75 Hz (2.98) and AO-4 Hz (2.13). Since ITR presents the efficiency of BCI, the average results of ITR in all regions showed that AO-1.5 Hz (2.01) had a significantly higher efficiency than AO-0.25 Hz (1.10), AO-2.75 Hz (0.23), AO-4 Hz (0.24), and MI (0.43). AO in moderate-speed mode was more suitable for the design of BCIs.

(5) The average accuracies have the same pattern of initially increasing and then decreasing to a stable level. In the results between subjects, subjects “S1, S3, S4” correspond to the average pattern, but subjects “S2 and S5” not correspond to the average pattern from AO-0.25 Hz to AO-4 Hz. However, subjects “S2 and S5” have approximate accuracies in AO-0.25 Hz and AO-4 Hz. This because different individuals have different patterns of speed modes, and our study was researched on rough speeds due to limitations of device and platform.

TABLE 4

Table 4. The BCI analysis results for three regions for four AO modes and MI modes.

These five findings will help us understand affected regions in the brain during AO and the patterns of relationships between AO speed modes and hMNS activations. By selecting the optimal speed mode of AO, we also solved the problem of BMBI. These findings and conclusions are discussed below.

3.3. Spatio-Spectral-Temporal Characteristics of AO Experiments

To further analyse the detailed characteristics, the best-performing data, S4, was chosen to extract the spatio-spectral-temporal characteristics of AO modes.

3.3.1. Temporal Characteristics

The average microvolt of C3 and C4 are drawn on left movements and right movements, respectively. The difference of “C3-C4” in left movements and “C4-C3” in right movements are drawn for temporal characteristics in Figure 5. Two key sites, C3 and C4 from the sensorimotor region, were chosen to analyse temporal characteristics.

FIGURE 5

Figure 5. The temporal characteristics of AO in four different speed modes. Two key sites, C3 and C4 from the sensorimotor region, were chosen to analyse temporal characteristics. The average microvolt of C3 and C4 are drawn on left movements and right movements, respectively. The difference of “C3–C4” in left movements and “C4–C3” in right movements are also drawn for temporal characteristics.

Three key findings from the results in Figure 5 are as follows:

(1) Look from the overall, the average microvolt variations of C4 sites are higher than C3 sites on left movements, while the average microvolt variations of C3 sites are higher than C4 sites on right movements. These findings demonstrate the brain's hemisphere lateralization effect during AO.

(2) Look from the difference average microvolt variations of (C3-C4) in left movements and (C4-C3) in right movements, the differences of AO-0.25 Hz and AO-1.5 Hz are significantly higher than AO-2.75 Hz and AO-4 Hz for both left movements and right movements. These results confirm that variations during AO captured a pattern of increases with slow-speed movements and decreases with fast-speed movements.

(3) In fact, we have calculated the average of difference variations from 1,000 samples for both four different speed modes on left movements and right movements. The results are 0.958 (AO-0.25 Hz), 0.974 (AO-1.5 Hz), 0.048 (AO-2.75 Hz), and 0.112 (AO-4 Hz) on left movements, and 0.907 (AO-0.25 Hz), 0.988 (AO-1.5 Hz), 0.821 (AO-2.75 Hz), and 0.5865 (AO-4 Hz) on right movements. These results suggest some relationships between speed and stimulus direction.

3.3.2. Spectral Characteristics

Figure 6 shows the spectral characteristics of AO in four different speed modes. Four key potentials, C3 and C4 for the sensorimotor region and O1 and O2 for the occipital region, were chosen to analyse the spectral characteristics.

FIGURE 6

Figure 6. The spectral characteristics of AO in four different speed modes. Four key potentials, C3 and C4 for the sensorimotor region and O1 and O2 for the occipital region, were chosen to analyse the spectral characteristics. The curves for all four AO modes (left movements and right movements) are drawn by “MATLAB R2012a.” (A) Left movements of AO. (B) Right movements of AO.

Three important findings from Figure 6 are as follows:

(1) In all of the AO modes, the power varied significantly in the mu range (8–12 Hz) at all C3/C4/O1/O2 potentials, but the variation in the beta range (18–25 Hz) was lower. These results show that the mu range is more sensitive in hMNS during AO.

(2) Regardless of whether left or right movements were examined, the power captured variation in the sensorimotor region (C3/C4) with a pattern where the largest was in AO-1.5 Hz, moderate was in AO-2.75 Hz and AO-4 Hz, and the smallest as in AO-0.25 Hz. However, the power captured variation in the occipital region (O1/O2) had an entirely different pattern in which the largest was in AO-4 Hz and the smallest was AO-1.5 Hz, AO-2.75 Hz, and AO-0.25 Hz with no obvious differences. These results suggest that the sensorimotor region was significantly activated at AO-1.5 Hz, whereas the occipital region was significantly activated at AO-4 Hz.

(3) Regardless of whether left or right movements were examined, the hemisphere effect of the power difference was prominent in the occipital region (O1/O2) but was not obvious in the sensorimotor region (C3/C4). These results demonstrate that the occipital region is more activated than the sensorimotor region.

3.3.3. Spatio-Spectral Characteristics

Figure 7 shows the spatio-spectral characteristics of AO in four different speed modes. The key rhythms, 8, 12, 18, and 25 Hz, were chosen to extract spatio-spectral characteristics from 64 potentials, since the ERD suppressions were induced in the mu (8–12 Hz) and beta (18–25 Hz) ranges.

FIGURE 7

Figure 7. The spatio-spectral characteristics of AO in four different speed modes. The key rhythms, 8, 12, 18, and 25 Hz, were chosen to extract spatio-spectral characteristics from 64 potentials, since the ERD suppressions were induced in the mu (8–12 Hz) and beta (18–25 Hz) ranges. The “BEAM” for all four AO modes(left and right movements) are drawn by “eeglab 14.1.1.” (A) Left movements of AO. (B) Right movements of AO.

From the results in Figure 7, we find that:

(1) In all eight AO modes, the BEAMs showed that the power in the mu range (8–12 Hz) was higher than that in the beta range (18–25 Hz) in the occipital region. These results demonstrate that the mu range plays a predominant role in hMNS activations.

(2) In contrast to the ERD suppressions during the four different AO speed modes, the ERD suppressions were stronger at AO-0.25 Hz and AO-1.5 Hz than at AO-2.75 Hz and AO-4 Hz in the sensorimotor region. These findings demonstrate that in the sensorimotor region, slow-speed movements cause greater hMNS activation than fast-speed movements during AO.

(3) Comparing left and right movements in one AO mode, the left movements produced more obvious ERD suppressions in the right hemisphere of the brain, whereas the right movements produced more obvious ERD suppressions in the left hemisphere of the brain. These results are due to the mirror effect from the hMNS. However, the movements that subjects observed were also mirror movements. Therefore, the ERD suppressions occurred in the ipsilateral region of the subject's brain relative to the movements of the robot.

4. Discussion

This study further explored the brain regions affected during AO, the patterns of relationships between AO speed modes and hMNS activations, and the comparison of AO and MI for constructing BMBIs. AO experiments with different speed modes (experiment 1) and an MI mode (experiment) were explored at the central nervous system level since hMNS activation caused by AO are reflected by ERD suppressions, which are measured via EEG recording. BCI analyses and spatio-spectral-temporal characteristics were used to explore the variations in ERD suppressions. Based on our results, we were able to draw three major conclusions, which are discussed below:

(1) The occipital region exhibits higher hMNS activation than the sensorimotor region during AO, but the sensorimotor region shows more robust hMNS activations than the occipital region.

(2) hMNS activations initially rise and then fall to a stable level during incremental AO speed modes. A moderate-speed AO produced the best hMNS activations in our results.

(3) The BCI accuracy and ITR of moderate-speed AO-BCI produced the same level of MI-BCI as training experience, suggesting a new method for constructing non-invasive BMBIs in healthy individuals.

4.1. The Affected Regions of the Brain During AO

According to a statistical study (Molenberghs et al., 2012), the support of mirror neurons, the connections across regions, and the strong mirror properties in the hMNS produce mirror properties in the sensorimotor and occipital regions during AO. In fact, the sensorimotor region is related to action imitation and action presentation. Processing of the subjects' action preparation and spatial orientation for a specific action depend on the sensorimotor region. The occipital region is part of the lateral occipitotemporal cortex (LOTC) (Barton et al., 1996; Kable and Chatterjee, 2006). The LOTC is a set of regions that are thought to function in integrating information and determining the purpose of an action. Several previous studies have found that the LOTC region forms direct and indirect connections with hMNS regions (Weiner and Grill-Spector, 2011; Lingnau and Downing, 2015). Therefore, the occipital, sensorimotor, and “sensorimotor+occipital” regions were adopted for BCI analyses and the extraction of spatio-spectral-temporal characteristic from EEG data during AO in this study.

The accuracies of BCI show that the occipital region exhibits higher ERD suppressions than the sensorimotor region (see Figure 4). Similar results were also found in the BEAMs (see Figure 7). The differences in affected regions during AO found in our study support a previous model of hMNS activation affecting multiple regions (Perry et al., 2010; Frenkel-Toledo et al., 2013). Two main reasons may account for the differences in hMNS activations between the two regions: First, AO paradigms are visually evoked stimuli (Wieser et al., 2016; Park, 2017). Therefore, the visually relevant occipital region first generates power variations (Pegado et al., 2014). After recognition of stimuli, hMNS activations generate power variations in the sensorimotor region (Oberman et al., 2005, 2008). Since the occipital region undergoes rapid and prolonged activation in response to visual stimuli, the degree of activation in the occipital region must be higher than that in the sensorimotor region. Second, the arm movements of the robot turn into attended information during the presentation of AO stimuli (Hwang et al., 2018). Recent studies suggest that attended information increases the activation of mu rhythms (8–12 Hz) in the occipital region during the presentation of visually evoked stimuli (Gray et al., 2015; Sprague et al., 2015). However, distracting information will suppress the activation of the alpha rhythm in the occipital region (Händel et al., 2011; Klimesch, 2012; Zumer et al., 2014). In addition, spectral characteristics and spatio-spatial characteristics suggest that the mu range plays a prominent role in hMNS activations (See Figures 5, 6). Therefore, the attended information provided by the left/right-arm movements of the robot cause more intense power differences in the occipital region than in the sensorimotor region.

Comparing the accuracies of BCI in the three explored regions, the results reflect the robustness of hMNS activation in these regions. The accuracies of BCI in the “sensorimotor+occipital” region remained the same as the sensorimotor region but were significantly different in the occipital region (See Figure 4). Since the activation of the sensorimotor region is caused by ERD suppressions, whereas the activation of the occipital region is caused by both ERD suppressions and attended information, ERD suppressions play a dominant role in the activation of the “sensorimotor+occipital” region. In fact, a large number of sessions must be presented over several days. Therefore, the attended information will change during all sessions. As time passes, the activation of the mu range caused by attended information will gradually reduce due to visual fatigue (Lambooij et al., 2009). However, ERD suppressions remain steady during the presentation of stimuli without influence from visual fatigue (Nam et al., 2011). Therefore, the sensorimotor region is more robust for hMNS activations than the occipital region during AO. The findings in our study of hMNS activations conform to the affected regions and formation principles during AO and support further studies on the hMNS.

4.2. The Patterns of hMNS Activations Affected by AO Speed Modes

We found that the patterns of hMNS activations first rise and then fall to a stable level during incremental-speed modes of AO (See Figure 4). Relationships between speed modes and left/right movements were also found in spatio-spatial characteristics (See Figure 7). The patterns were appropriate for both the sensorimotor region and the occipital region. The reasons why the curves reveal these patterns are as follows. Since AO stimuli in different speed modes were presented in a periodic way in experiment 1, the refractory period of visual stimuli must be considered during the presentation (Huettel and McCarthy, 2000). Previous studies suggest that the inter-stimulus intervals (ISIs) in the refractory period are within 360–2,000 ms for auditory and visual senses (Coch et al., 2005; Brisson and Jolicœur, 2007). Other studies demonstrate that the ISIs of repetitive contents are within 500 ms (Davis et al., 1972; Johannsen and Röder, 2014). In fact, the ISIs in our experiments were 4,000 ms (slow movement), 666.7 ms (moderate movement), 363.6 ms (fast movement), and 250 ms (finalistic movement). Comparing the ISIs of the four speed modes, the fast and finalistic movements exceeded the repetitive content limits of ISI. Therefore, a visual refractory period will appear during the presentation of fast- and finalistic-movement AO tasks. Since the ERD suppressions are also affected by the visual refractory period during AO (Luck et al., 2000; Caravaglios et al., 2015), the hemisphere effects of ERD suppressions will be reduced by the visual refractory period. In other words, increased speed modes of AO within the limits of ISI will promote hMNS activations that benefit from the absence of a visual refractory period.

The patterns suggested that different speed modes of AO define the extent and distribution of hMNS activations. The purpose of our study was to explore the patterns of relationships between AO speed modes and hMNS activations, since the AO speed modes affect action imitation and action presentation during AO. Meanwhile, the speed modes of AO also play important roles in action information integration and the purpose of action in previous studies (Rizzolatti, 2005; Hobson and Bishop, 2016). Therefore, the speed factor must be considered in relevant studies and applications related to action, such as cognitive psychology, rehabilitation treatment, and BCI. The patterns in our study suggest that periodic actions performed at a moderate speed within decent ISI ranges as visual stimuli for observation will contribute to optimized rehabilitation treatment setups for healthy or unhealthy individuals. AO stimuli of moderate speed within decent ISI will also improve the performance of AO-BCI.

Recently, neurological diseases have been remedied by AO strategies, which are good at ameliorating motor recovery (Buccino et al., 2011). For healthy individuals, AO strategies are good at facilitating motor learning and increasing force (Stefan et al., 2005). Actions performed at a suitable speed mode for observation may enhance action motivation and optimize the recruitment of motor function (Porro et al., 2007). Current treatments for chronic stroke and motor impairment patients construct action execution (AE) strategies based on ecological values (Buccino, 2014), which are valid for improving motor function and ameliorating autonomy, such as grasping cups or cleaning the floor (Ertelt et al., 2007). However, since individual patients with neurological diseases usually have clinical impairments, these patients cannot accomplish such AE assignments in daily treatment without help. The limits for individual patients in AE will be remitted by an optimal AO strategy design. Because the AO assignments are visual stimuli, which need patients to focus their attention, AO assignments are easier to accomplish in daily treatment than AE assignments. In addition, regardless of whether AE or AO assignments are used, the treatment strategies of the medical rehabilitative field must be tailored to satisfy the individual needs of each subject. Treatment strategies should be designed with due consideration of environmental, social, and economic factors, as well as the pathology and characteristics of the subject. Our study suggests that an individually tailored speed within a decent ISI range must be considered when constructing rehabilitation AO treatments for individual subjects.

4.3. The Construction of BMBI Based on AO-BCI and MI-BCI

Our study suggests a new way to construct AO-BCI on a humanoid robot platform. Experimental paradigms are defined by parameters such that confounding factors are eliminated by precise settings. Compared with MI-BCI, AO-BCI offers a number of advantages. Typically, MI-BCI is constructed on the self-imaged spontaneous potential of ERD suppressions. Subjects may imagine self-motion, others-motion, wrist movement, arm movement, or hand movement without specific instruction. In addition, the familiarity factor also affects MI-BCI performance, since MI-BCI performance significantly increases for fully fledged subjects (Schroeder and Chestek, 2016; Subramanian et al., 2016). However, AO-BCI, as a visually evoked stimulus, eliminates the uncertainties of spontaneous MI-BCI. Moreover, familiarity in the BCI is controlled by the stimulus design. In experiments 1 and 2, none of the subjects had previous training experience. However, the BCI results of experiment 1 significantly outperformed those of experiment 2 for slow and moderate movements (See Figure 4). A comparison of the results demonstrated that fixed AO modes were less influenced by the subjects' experience, which is another advantage of constructing AO-BCIs.

The performance of AO-BCI and MI-BCI are mismatched in this study since the subjects for MI-BCI lacked MI training. With enough training in MI-BCI, the average accuracy and ITR among subjects can reach 75% and 1.8 bits/min for two classes of MI-BCI (Park et al., 2013). Due to confounding factors, achieving the same level of average accuracy and ITR in AO-BCI has thus far proven difficult. However, by adopting fine-tuned stimuli in AO, as in the present study, the average accuracy and ITR of AO-BCI reached 70% and 1.65 bits/min, which are similar to those of a well-trained MI-BCI (See Figure 4). These findings prompted us to construct a BMBI in healthy subjects using the stimulus paradigms of AO and well-trained MI. In recent studies, the majority of BMBIs were based on invasive signal processing technology and used animal carriers (Tessadori et al., 2012). However, BMBIs are rarely based on healthy human brains given the requirement for non-invasive signal processing techniques. As a system of bi-directional transmission, the BMBI must be built on two kinds of stimulators. The accuracy of performance and the ITR of efficiency must achieve the same level for both transmissions. Designing BMBIs that are influenced by variables remains a meaningful but difficult problem. The construction of BMBIs based on AO-BCI and MI-BCI in our study was based on EEG recordings, which will satisfy the requirement for a non-invasive method in healthy individual subjects.

5. Conclusion

Our experimental results suggest that the occipital region exhibits higher but less robust hMNS activations than the sensorimotor region. In addition, the patterns of hMNS activations first rise and then fall to a stable level during incremental-speed modes of AO. Moreover, the creation of an AO-BCI provides a novel means of building BMBI systems in healthy individuals with well-trained MI-BCI. However, as time passes, the subjects' refractory period, fatigue and pressure for waiting will lead limit the EEG recordings. Thus, our results provide only rough patterns of the relationships between AO speed modes and hMNS activations. Further studies with larger experimental trials are needed to explore the exact patterns. More modes of AO are also needed to explore confounding factors. Only then can accurate patterns of AO be completed to study the human brain.

Author Contributions

CZ, TL, and JL designed the experiments; TL and JL completed the experiments; TL analyzed the EEG data; CZ, FC, and TL wrote the paper.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (No.61673322 and 61673326), the Major State Basic Research Development Program of China (973 Program) (No. 2013CB329502), the Fundamental Research Funds for the Central Universities (No. 20720160126), and Natural Science Foundation of Fujian Province of China (No. 2017J01128 and 2017J01129).

References

Abbott, D. F. (2016). Probing the human brain functional connectome with simultaneous EEG and fMRI. Front. Neurosci. 10:302. doi: 10.3389/fnins.2016.00302

PubMed Abstract | CrossRef Full Text | Google Scholar

Abdi, H., and Williams, L. J. (2010). Principal component analysis. Wiley Interdisc. Rev. Comput. Stat. 2, 433–459. doi: 10.1002/wics.101

CrossRef Full Text | Google Scholar

Agnew, Z. K., Wise, R. J., and Leech, R. (2012). Dissociating object directed and non-object directed action in the human mirror system; implications for theories of motor simulation. PLoS ONE 7:e32517. doi: 10.1371/journal.pone.0032517

PubMed Abstract | CrossRef Full Text | Google Scholar

Ang, K. K., Chin, Z. Y., Wang, C., Guan, C., and Zhang, H. (2012). Filter bank common spatial pattern algorithm on BCI competition IV datasets 2a and 2b. Front. Neurosci. 6:39. doi: 10.3389/fnins.2012.00039

PubMed Abstract | CrossRef Full Text | Google Scholar

Barton, J. J., Simpson, T., Kiriakopoulos, E., Stewart, C., Crawley, A., Guthrie, B., et al. (1996). Functional MRI of lateral occipitotemporal cortex during pursuit and motion perception. Ann. Neurol. 40, 387–398.

PubMed Abstract | Google Scholar

Bauer, R., and Gharabaghi, A. (2015). Reinforcement learning for adaptive threshold control of restorative brain-computer interfaces: a Bayesian simulation. Front. Neurosci. 9:36. doi: 10.3389/fnins.2015.00036

PubMed Abstract | CrossRef Full Text | Google Scholar

Bello, J. P., Modroño, C., Marcano, F., and González-Mora, J. (2014). The mirror neuron system and motor dexterity: what happens? Neuroscience 275, 285–295. doi: 10.1016/j.neuroscience.2014.06.010

CrossRef Full Text | Google Scholar

Biagi, L., Cioni, G., Fogassi, L., Guzzetta, A., and Tosetti, M. (2010). Anterior intraparietal cortex codes complexity of observed hand movements. Brain Res. Bull. 81, 434–440. doi: 10.1016/j.brainresbull.2009.12.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Blakemore, S.-J., and Decety, J. (2001). From the perception of action to the understanding of intention. Nat. Rev. Neurosci. 2, 561–567. doi: 10.1038/35086023

PubMed Abstract | CrossRef Full Text | Google Scholar

Brisson, B., and Jolicœur, P. (2007). A psychological refractory period in access to visual short-term memory and the deployment of visual–spatial attention: multitasking processing deficits revealed by event-related potentials. Psychophysiology 44, 323–333. doi: 10.1111/j.1469-8986.2007.00503.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Buccino, G. (2014). Action observation treatment: a novel tool in neurorehabilitation. Philos. Trans. R. Soc. Lond. B Biol. Sci. 369:20130185. doi: 10.1098/rstb.2013.0185

PubMed Abstract | CrossRef Full Text | Google Scholar

Buccino, G., Gatti, R., Giusti, M. C., Negrotti, A., Rossi, A., Calzetti, S., et al. (2011). Action observation treatment improves autonomy in daily activities in Parkinson's disease patients: results from a pilot study. Mov. Disord. 26, 1963–1964. doi: 10.1002/mds.23745

PubMed Abstract | CrossRef Full Text | Google Scholar

Calvo-Merino, B., Glaser, D. E., Grèzes, J., Passingham, R. E., and Haggard, P. (2004). Action observation and acquired motor skills: an fMRI study with expert dancers. Cereb. Cortex 15, 1243–1249. doi: 10.1093/cercor/bhi007

PubMed Abstract | CrossRef Full Text | Google Scholar

Calvo-Merino, B., Grèzes, J., Glaser, D. E., Passingham, R. E., and Haggard, P. (2006). Seeing or doing? Influence of visual and motor familiarity in action observation. Curr. Biol. 16, 1905–1910. doi: 10.1016/j.cub.2006.07.065

PubMed Abstract | CrossRef Full Text | Google Scholar

Caravaglios, G., Muscoso, E. G., Di Maria, G., and Costanzo, E. (2015). Patients with mild cognitive impairment have an abnormal upper-alpha event-related desynchronization/synchronization (ERD/ERS) during a task of temporal attention. J. Neural Transm. 122, 441–453. doi: 10.1007/s00702-014-1262-7

PubMed Abstract | CrossRef Full Text | Google Scholar

Caspers, S., Zilles, K., Laird, A. R., and Eickhoff, S. B. (2010). ALE meta-analysis of action observation and imitation in the human brain. Neuroimage 50, 1148–1167. doi: 10.1016/j.neuroimage.2009.12.112

PubMed Abstract | CrossRef Full Text | Google Scholar

Coch, D., Skendzel, W., and Neville, H. J. (2005). Auditory and visual refractory period effects in children and adults: an ERP study. Clin. Neurophysiol. 116, 2184–2203. doi: 10.1016/j.clinph.2005.06.005

PubMed Abstract | CrossRef Full Text | Google Scholar

Davis, H., Osterhammel, P., Wier, C., and Gjerdingen, D. (1972). Slow vertex potentials: interactions among auditory, tactile, electric and visual stimuli. Electroencephalogr. Clin. Neurophysiol. 33, 537–545.

PubMed Abstract | Google Scholar

Delorme, A., and Makeig, S. (2004). EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J. Neurosci. Methods 134, 9–21. doi: 10.1016/j.jneumeth.2003.10.009

PubMed Abstract | CrossRef Full Text | Google Scholar

Demandt, E., Mehring, C., Vogt, K., Schulze-Bonhage, A., Aertsen, A., and Ball, T. (2012). Reaching movement onset-and end-related characteristics of EEG spectral power modulations. Front. Neurosci. 6:65. doi: 10.3389/fnins.2012.00065

PubMed Abstract | CrossRef Full Text | Google Scholar

Duffy, F. H., Burchfiel, J. L., and Lombroso, C. T. (1979). Brain electrical activity mapping (beam): a method for extending the clinical utility of EEG and evoked potential data. Ann. Neurol. 5, 309–321.

PubMed Abstract | Google Scholar

Ertelt, D., Small, S., Solodkin, A., Dettmers, C., McNamara, A., Binkofski, F., et al. (2007). Action observation has a positive impact on rehabilitation of motor deficits after stroke. Neuroimage 36, T164–T173. doi: 10.1016/j.neuroimage.2007.03.043

PubMed Abstract | CrossRef Full Text | Google Scholar

Filimon, F., Nelson, J. D., Hagler, D. J., and Sereno, M. I. (2007). Human cortical representations for reaching: mirror neurons for execution, observation, and imagery. Neuroimage 37, 1315–1328. doi: 10.1016/j.neuroimage.2007.06.008

PubMed Abstract | CrossRef Full Text | Google Scholar

Franceschini, M., Ceravolo, M. G., Agosti, M., Cavallini, P., Bonassi, S., Dall'Armi, V., et al. (2012). Clinical relevance of action observation in upper-limb stroke rehabilitation: a possible role in recovery of functional dexterity. a randomized clinical trial. Neurorehabil. Neural Repair 26, 456–462. doi: 10.1177/1545968311427406

PubMed Abstract | CrossRef Full Text | Google Scholar

Frenkel-Toledo, S., Bentin, S., Perry, A., Liebermann, D. G., and Soroker, N. (2013). Dynamics of the EEG power in the frequency and spatial domains during observation and execution of manual movements. Brain Res. 1509, 43–57. doi: 10.1016/j.brainres.2013.03.004

PubMed Abstract | CrossRef Full Text | Google Scholar

Frenkel-Toledo, S., Bentin, S., Perry, A., Liebermann, D. G., and Soroker, N. (2014). Mirror-neuron system recruitment by action observation: effects of focal brain damage on mu suppression. Neuroimage 87, 127–137. doi: 10.1016/j.neuroimage.2013.10.019

PubMed Abstract | CrossRef Full Text | Google Scholar

Gatti, R., Rocca, M. A., Fumagalli, S., Cattrysse, E., Kerckhofs, E., Falini, A., et al. (2017). The effect of action observation/execution on mirror neuron system recruitment: an fMRI study in healthy individuals. Brain Imaging Behav. 11, 565–576. doi: 10.1007/s11682-016-9536-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Geers, A. (1994). Techniques for assessing auditory speech perception and lipreading enhancement in young deaf children. Volta Rev. 96, 85–96.

Google Scholar

Gray, M. J., Frey, H.-P., Wilson, T. J., and Foxe, J. J. (2015). Oscillatory recruitment of bilateral visual cortex during spatial attention to competing rhythmic inputs. J. Neurosci. 35, 5489–5503. doi: 10.1523/JNEUROSCI.2891-14.2015

PubMed Abstract | CrossRef Full Text | Google Scholar

Händel, B. F., Haarmeier, T., and Jensen, O. (2011). Alpha oscillations correlate with the successful inhibition of unattended stimuli. J. Cogn. Neurosci. 23, 2494–2502. doi: 10.1162/jocn.2010.21557

PubMed Abstract | CrossRef Full Text | Google Scholar

Hobson, H. M., and Bishop, D. V. (2016). Mu suppression–a good measure of the human mirror neuron system? Cortex 82, 290–310. doi: 10.1016/j.cortex.2016.03.019

PubMed Abstract | CrossRef Full Text | Google Scholar

Huettel, S. A., and McCarthy, G. (2000). Evidence for a refractory period in the hemodynamic response to visual stimuli as measured by MRI. Neuroimage 11, 547–553. doi: 10.1006/nimg.2000.0553

PubMed Abstract | CrossRef Full Text | Google Scholar

Hwang, K., Shine, J. M., and D'Esposito, M. (2018). Frontoparietal activity interacts with task-evoked changes in functional connectivity. Cereb. Cortex. doi: 10.1093/cercor/bhy011. [Epub ahead of print].

PubMed Abstract | CrossRef Full Text | Google Scholar

Iacoboni, M., Molnar-Szakacs, I., Gallese, V., Buccino, G., Mazziotta, J. C., and Rizzolatti, G. (2005). Grasping the intentions of others with one's own mirror neuron system. PLoS Biol. 3:e79. doi: 10.1371/journal.pbio.0030079

PubMed Abstract | CrossRef Full Text | Google Scholar

Johannsen, J., and Röder, B. (2014). Uni-and crossmodal refractory period effects of event-related potentials provide insights into the development of multisensory processing. Front. Hum. Neurosci. 8:552. doi: 10.3389/fnhum.2014.00552

PubMed Abstract | CrossRef Full Text | Google Scholar

Kable, J. W., and Chatterjee, A. (2006). Specificity of action representations in the lateral occipitotemporal cortex. J. Cogn. Neurosci. 18, 1498–1517. doi: 10.1162/jocn.2006.18.9.1498

PubMed Abstract | CrossRef Full Text | Google Scholar

Klimesch, W. (2012). α-band oscillations, attention, and controlled access to stored information. Trends Cogn. Sci. 16, 606–617. doi: 10.1016/j.tics.2012.10.007

PubMed Abstract | CrossRef Full Text | Google Scholar

Lambooij, M., Fortuin, M., Heynderickx, I., and IJsselsteijn, W. (2009). Visual discomfort and visual fatigue of stereoscopic displays: a review. J. Imaging Sci. Technol. 53, 1–14. doi: 10.2352/J.ImagingSci.Technol.2009.53.3.030201

CrossRef Full Text | Google Scholar

Lei, X., Wu, T., and Valdes-Sosa, P. A. (2015). Incorporating priors for EEG source imaging and connectivity analysis. Front. Neurosci. 9:284. doi: 10.3389/fnins.2015.00284

PubMed Abstract | CrossRef Full Text | Google Scholar

Lingnau, A., and Downing, P. E. (2015). The lateral occipitotemporal cortex in action. Trends Cogn. Sci. 19, 268–277. doi: 10.1016/j.tics.2015.03.006

PubMed Abstract | CrossRef Full Text | Google Scholar

Luck, S. J., Woodman, G. F., and Vogel, E. K. (2000). Event-related potential studies of attention. Trends Cogn. Sci. 4, 432–440. doi: 10.1016/S1364-6613(00)01545-X

PubMed Abstract | CrossRef Full Text | Google Scholar

Maranesi, M., Ugolotti Serventi, F., Bruni, S., Bimbi, M., Fogassi, L., and Bonini, L. (2013). Monkey gaze behaviour during action observation and its relationship to mirror neuron activity. Eur. J. Neurosci. 38, 3721–3730. doi: 10.1111/ejn.12376

PubMed Abstract | CrossRef Full Text | Google Scholar

Molenberghs, P., Cunnington, R., and Mattingley, J. B. (2012). Brain regions with mirror properties: a meta-analysis of 125 human fmri studies. Neurosci. Biobehav. Rev. 36, 341–349. doi: 10.1016/j.neubiorev.2011.07.004

PubMed Abstract | CrossRef Full Text | Google Scholar

Mukamel, R., Ekstrom, A. D., Kaplan, J., Iacoboni, M., and Fried, I. (2010). Single-neuron responses in humans during execution and observation of actions. Curr. Biol. 20, 750–756. doi: 10.1016/j.cub.2010.02.045

PubMed Abstract | CrossRef Full Text | Google Scholar

Nam, C. S., Jeon, Y., Kim, Y.-J., Lee, I., and Park, K. (2011). Movement imagery-related lateralization of event-related (de) synchronization (ERD/ERS): Motor-imagery duration effects. Clin. Neurophysiol. 122, 567–577. doi: 10.1016/j.clinph.2010.08.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Neuper, C., Scherer, R., Wriessnegger, S., and Pfurtscheller, G. (2009). Motor imagery and action observation: modulation of sensorimotor brain rhythms during mental control of a brain–computer interface. Clin. Neurophysiol. 120, 239–247. doi: 10.1016/j.clinph.2008.11.015

PubMed Abstract | CrossRef Full Text | Google Scholar

Newman-Norlund, R. D., van Schie, H. T., van Zuijlen, A. M., and Bekkering, H. (2007). The mirror neuron system is more active during complementary compared with imitative action. Nat. Neurosci. 10:nn1911. doi: 10.1038/nn1911

PubMed Abstract | CrossRef Full Text | Google Scholar

Oberman, L. M., Hubbard, E. M., McCleery, J. P., Altschuler, E. L., Ramachandran, V. S., and Pineda, J. A. (2005). EEG evidence for mirror neuron dysfunction in autism spectrum disorders. Brain Res. Cogn. Brain Res. 24, 190–198. doi: 10.1016/j.cogbrainres.2005.01.014

PubMed Abstract | CrossRef Full Text | Google Scholar

Oberman, L. M., Pineda, J. A., and Ramachandran, V. S. (2007). The human mirror neuron system: a link between action observation and social skills. Soc. Cogn. Affect. Neurosci. 2, 62–66. doi: 10.1093/scan/nsl022

PubMed Abstract | CrossRef Full Text | Google Scholar

Oberman, L. M., Ramachandran, V. S., and Pineda, J. A. (2008). Modulation of mu suppression in children with autism spectrum disorders in response to familiar or unfamiliar stimuli: the mirror neuron hypothesis. Neuropsychologia 46, 1558–1565. doi: 10.1016/j.neuropsychologia.2008.01.010

PubMed Abstract | CrossRef Full Text | Google Scholar

Onishi, A., Takano, K., Kawase, T., Ora, H., and Kansaku, K. (2017). Affective stimuli for an auditory p300 brain-computer interface. Front. Neurosci. 11:522. doi: 10.3389/fnins.2017.00522

PubMed Abstract | CrossRef Full Text | Google Scholar

Orgs, G., Dombrowski, J.-H., Heil, M., and Jansen-Osmann, P. (2008). Expertise in dance modulates alpha/beta event-related desynchronization during action observation. Eur. J. Neurosci. 27, 3380–3384. doi: 10.1111/j.1460-9568.2008.06271

PubMed Abstract | CrossRef Full Text | Google Scholar

Park, C., Looney, D., ur Rehman, N., Ahrabian, A., and Mandic, D. P. (2013). Classification of motor imagery BCI using multivariate empirical mode decomposition. IEEE Trans. Neural Syst. Rehabil. Eng. 21, 10–22. doi: 10.1109/TNSRE.2012.2229296

PubMed Abstract | CrossRef Full Text | Google Scholar

Park, J. (2017). A neural basis for the visual sense of number and its development: a steady-state visual evoked potential study in children and adults. Dev. Cogn. Neurosci. doi: 10.1016/j.dcn.2017.02.011. [Epub ahead of print].

PubMed Abstract | CrossRef Full Text | Google Scholar

Pegado, F., Comerlato, E., Ventura, F., Jobert, A., Nakamura, K., Buiatti, M., et al. (2014). Timing the impact of literacy on visual processing. Proc. Natl. Acad. Sci. U.S.A. 111, E5233–E5242. doi: 10.1073/pnas.1417347111

PubMed Abstract | CrossRef Full Text | Google Scholar

Perry, A., Bentin, S., Shalev, I., Israel, S., Uzefovsky, F., Bar-On, D., et al. (2010). Intranasal oxytocin modulates EEG mu/alpha and beta rhythms during perception of biological motion. Psychoneuroendocrinology 35, 1446–1453. doi: 10.1016/j.psyneuen.2010.04.011

PubMed Abstract | CrossRef Full Text | Google Scholar

Pineda, J. A., Grichanik, M., Williams, V., Trieu, M., Chang, H., and Keysers, C. (2013). EEG sensorimotor correlates of translating sounds into actions. Front. Neurosci. 7:203. doi: 10.3389/fnins.2013.00203

PubMed Abstract | CrossRef Full Text | Google Scholar

Porro, C. A., Facchin, P., Fusi, S., Dri, G., and Fadiga, L. (2007). Enhancement of force after action observation: behavioural and neurophysiological studies. Neuropsychologia 45, 3114–3121. doi: 10.1016/j.neuropsychologia.2007.06.016

PubMed Abstract | CrossRef Full Text | Google Scholar

Pot, E., Monceaux, J., Gelin, R., and Maisonnier, B. (2009). “Choregraphe: a graphical tool for humanoid robot programming,” in The 18th IEEE International Symposium on Robot and Human Interactive Communication (Toyama), 46–51.

Google Scholar

Press, C. (2011). Action observation and robotic agents: learning and anthropomorphism. Neurosci. Biobehav. Rev. 35, 1410–1418. doi: 10.1016/j.neubiorev.2011.03.004

PubMed Abstract | CrossRef Full Text | Google Scholar

Rizzolatti, G. (2005). The mirror neuron system and its function in humans. Anat. Embryol. 210, 419–421. doi: 10.1007/s00429-005-0039-z

PubMed Abstract | CrossRef Full Text | Google Scholar

Rizzolatti, G., and Sinigaglia, C. (2010). The functional role of the parieto-frontal mirror circuit: interpretations and misinterpretations. Nat. Rev. Neurosci. 11, 264–274. doi: 10.1038/nrn2805

PubMed Abstract | CrossRef Full Text | Google Scholar

Rozzi, S., and Fogassi, L. (2017). Neural coding for action execution and action observation in the prefrontal cortex and its role in the organization of socially driven behavior. Front. Neurosci. 11:492. doi: 10.3389/fnins.2017.00492

PubMed Abstract | CrossRef Full Text | Google Scholar

Sardouie, S. H., and Shamsollahi, M. B. (2012). Selection of efficient features for discrimination of hand movements from meg using a BCI competition IV data set. Front. Neurosci. 6:42. doi: 10.3389/fnins.2012.00042

CrossRef Full Text | Google Scholar

Schroeder, K. E., and Chestek, C. A. (2016). Intracortical brain-machine interfaces advance sensorimotor neuroscience. Front. Neurosci. 10:291. doi: 10.3389/fnins.2016.00291

PubMed Abstract | CrossRef Full Text | Google Scholar

Schulte-Rüther, M., Markowitsch, H. J., Fink, G. R., and Piefke, M. (2007). Mirror neuron and theory of mind mechanisms involved in face-to-face interactions: a functional magnetic resonance imaging approach to empathy. J. Cogn. Neurosci. 19, 1354–1372. doi: 10.1162/jocn.2007.19.8.1354

PubMed Abstract | CrossRef Full Text | Google Scholar

Silsbee, P. L., and Bovik, A. C. (1996). Computer lipreading for improved accuracy in automatic speech recognition. IEEE Trans. Speech Audio Process. 4, 337–351.

Google Scholar

Smith, E. G., and Bennetto, L. (2007). Audiovisual speech integration and lipreading in autism. J. Child Psychol. Psychiatry 48, 813–821. doi: 10.1111/j.1469-7610.2007.01766.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Sprague, T. C., Saproo, S., and Serences, J. T. (2015). Visual attention mitigates information loss in small-and large-scale neural codes. Trends Cogn. Sci. 19, 215–226. doi: 10.1016/j.tics.2015.02.005

PubMed Abstract | CrossRef Full Text | Google Scholar

Stefan, K., Cohen, L. G., Duque, J., Mazzocchio, R., Celnik, P., Sawaki, L., et al. (2005). Formation of a motor memory by action observation. J. Neurosci. 25, 9339–9346. doi: 10.1523/JNEUROSCI.2282-05.2005

PubMed Abstract | CrossRef Full Text | Google Scholar

Subramanian, L., Morris, M. B., Brosnan, M., Turner, D. L., Morris, H. R., and Linden, D. E. (2016). Functional magnetic resonance imaging neurofeedback-guided motor imagery training and motor training for Parkinson's disease: randomized trial. Front. Behav. Neurosci. 10:111. doi: 10.3389/fnbeh.2016.00111

PubMed Abstract | CrossRef Full Text | Google Scholar

Summerfield, Q. (1992). Lipreading and audio-visual speech perception. Philos. Trans. R. Soc. Lond. B Biol. Sci. 335, 71–78.

PubMed Abstract | Google Scholar

Tangermann, M., Müller, K.-R., Aertsen, A., Birbaumer, N., Braun, C., Brunner, C., et al. (2012). Review of the BCI competition IV. Front. Neurosci. 6:55. doi: 10.3389/fnins.2012.00055

PubMed Abstract | CrossRef Full Text | Google Scholar

Tanji, K., Sakurada, K., Funiu, H., Matsuda, K., Kayama, T., Ito, S., et al. (2015). Functional significance of the electrocorticographic auditory responses in the premotor cortex. Front. Neurosci. 9:78. doi: 10.3389/fnins.2015.00078

PubMed Abstract | CrossRef Full Text | Google Scholar

Tessadori, J., Bisio, M., Martinoia, S., and Chiappalone, M. (2012). Modular neuronal assemblies embodied in a closed-loop environment: toward future integration of brains and machines. Front. Neural Circ. 6:99. doi: 10.3389/fncir.2012.00099

PubMed Abstract | CrossRef Full Text | Google Scholar

Vogt, S., Di Rienzo, F., Collet, C., Collins, A., and Guillot, A. (2013). Multiple roles of motor imagery during action observation. Front. Hum. Neurosci. 7:807. doi: 10.3389/fnhum.2013.00807

PubMed Abstract | CrossRef Full Text | Google Scholar

Wang, D., Miao, D., and Blohm, G. (2012). Multi-class motor imagery EEG decoding for brain-computer interfaces. Front. Neurosci. 6:151. doi: 10.3389/fnins.2012.00151

PubMed Abstract | CrossRef Full Text | Google Scholar

Weiner, K. S., and Grill-Spector, K. (2011). Not one extrastriate body area: using anatomical landmarks, hMT+, and visual field maps to parcellate limb-selective activations in human lateral occipitotemporal cortex. Neuroimage 56, 2183–2199. doi: 10.1016/j.neuroimage.2011.03.041

PubMed Abstract | CrossRef Full Text | Google Scholar

Wieser, M. J., Reicherts, P., Juravle, G., and von Leupoldt, A. (2016). Attention mechanisms during predictable and unpredictable threat-a steady-state visual evoked potential approach. Neuroimage 139, 167–175. doi: 10.1016/j.neuroimage.2016.06.026

PubMed Abstract | CrossRef Full Text | Google Scholar

Zumer, J. M., Scheeringa, R., Schoffelen, J.-M., Norris, D. G., and Jensen, O. (2014). Occipital alpha activity during stimulus processing gates the information flow to object-selective cortex. PLoS Biol. 12:e1001965. doi: 10.1371/journal.pbio.1001965

PubMed Abstract | CrossRef Full Text | Google Scholar