Motor-related brain activity during action observation: a neural substrate for electrocorticographic brain-computer interfaces after spinal cord injury

Introduction

Brain computer-interfaces (BCIs) transform brain activity into control signals for external devices including computers, communication aids, and prostheses (Wolpaw et al., 2002). Recent studies have demonstrated that BCIs may be able to restore functions that cannot be completed with the paralyzed upper limbs (Hochberg et al., 2012; Collinger et al., 2013b). The key component of a motor-BCI system is its neural decoder, or neuro-motor map, which is a set of decoding weights that transform motor cortical activity to intended movement. Typically, decoding weights can be calculated using motor cortical activity and the associated overt movement (Taylor et al., 2002; Wang et al., 2010a). However, this is not practical for clinical BCI systems as the targeted BCI users are often individuals who are unable to generate overt movement due to conditions such as spinal cord injury (SCI), amyotrophic lateral sclerosis (ALS), or upper limb amputation. An alternative strategy for identifying the neuro-motor map is to use motor cortical activity associated with action observation (AO) (Tkach et al., 2008; Velliste et al., 2008; Dushanova and Donoghue, 2010). Previous studies using magnetoencephalography (MEG) and electroencephalography (EEG) have shown a decrease in sensorimotor rhythm (10–30 Hz) power during both action execution and observation (Hari et al., 1998; Muthukumaraswamy et al., 2004; Caetano et al., 2007; Perry and Bentin, 2009; Press et al., 2011). Similarly, many motor cortical neurons recorded with microelectrodes show comparable firing rates during both conditions (Tkach et al., 2007; Dushanova and Donoghue, 2010). This suggests that AO may be used to identify the neuro-motor map without the need for overt movement.

Electrocorticography (ECoG) is a promising neural recording modality for BCI applications (Leuthardt et al., 2004; Schalk et al., 2007; Acharya et al., 2010; Miller et al., 2010; Flint et al., 2013; Wang et al., 2013). ECoG-based BCIs typically generate movement commands based on changes in ECoG signal power in specific frequency bands, including the sensorimotor rhythms and high-gamma band (>60 Hz). The sensorimotor rhythms typically show decreased spectral power during overt movement, and increased spectral power is observed in the high-gamma band (Crone et al., 1998a,b; Miller et al., 2007). It has been suggested that the high-gamma band carries rich movement information for BCI control, such as hand movement direction (Leuthardt et al., 2004; Heldman et al., 2006) and individual finger movement (Kubánek et al., 2009; Miller et al., 2009; Wang et al., 2009). However, there has not been any report to date that characterized high-gamma band modulation by observed hand movement using cortical surface recording in humans. Therefore, this study aims to determine whether action observation-related activity can be measured using ECoG during observed hand movement in both able-bodied subjects and an individual with tetraplegia due to SCI. We expect that significant and congruent motor-related cortical activity, defined as decreased spectral power of the sensorimotor rhythm and increased spectral power of the high-gamma band, will be recorded during execution and observation of grasping movements.

Materials and Methods

Participants

Four able-bodied subjects (F, G, I, and J) were recruited to participate in this study. All subjects were undergoing subdural ECoG monitoring for intractable epilepsy and were able to perform simple behavioral tasks and follow instructions as determined by the investigators. None of these participants had any motor impairment. We also collected data from an individual with cervical SCI (Subject S) who had an ECoG array implanted over his sensorimotor cortex for a BCI study (Wang et al., 2013). This trial is registered at clinicaltrials.gov (NCT01393444). Informed consent was provided by all subjects prior to participation in this research, which was approved by the Institutional Review Board at the University of Pittsburgh.

The ages of the able-bodied participants (F, G, I, and J) were 17, 23, 12, and 45 respectively. They all had a standard ECoG grid (Ad-Tech Medical Instrument Corporation, Racine, WI), with 3 mm diameter platinum disc electrodes and 10 mm center-to-center spacing, implanted to localize seizure foci prior to resection surgery. In addition, 2 subjects (F and J) were implanted with a small high-density ECoG grid (Ad-Tech Corp., Racine, WI) placed adjacent to the clinical grid as part of the research protocol with the goal of targeting motor cortical areas. The high-density ECoG grid contained 16 platinum disc electrodes with 1.5 mm diameter and 4 mm center-to-center spacing. Two corner electrodes on the high-density ECoG grid served as reference channels. Grid locations for the 4 able-bodied subjects are shown on x-ray and plotted on a standard brain model in Figure 1. The Location on Cortex (LOC) toolbox was used to compute the electrode position in Talaraich coordinates based on the lateral x-ray and to render these positions on a standard brain model (Miller et al., 2007). The current study focused on data collected from the high-density ECoG electrodes targeted to motor cortical areas for Subject F and J and data collected from standard ECoG electrodes over the motor cortical areas for Subject G and I. The high-density ECoG grids were targeted to motor cortical areas based on anatomical landmarks visible on the exposed cortical surface during the implantation surgery. Cortical stimulation mapping through the standard ECoG grids was performed as part of the clinical procedure to identify motor cortical areas highlighted in Figure 1 for Subjects G and I. No cortical stimulation was performed for the high-density research grid as this was not part of the standard clinical procedure.

FIGURE 1

Figure 1. ECoG grid locations. Left, ECoG grid locations shown on x-ray for Subjects F, G, I, and J. The motor cortical areas are highlighted in green. The schematic inset on each x-ray shows electrodes that were significantly modulated as indicated by the color. Red = Significant modulation during action execution (AE) and observation (AO), Blue = Significant modulation during AE only, Green = Significant modulation during AO only, White = no significant modulation. Electrode positions are shown on a standard brain model to the right of each x-ray using the same color scheme. The two large white arrows (Subjects F and J) mark the two electrodes whose data are shown in Figures 3, 4.

Subject S was a 30-year-old right-handed male with a complete C4 level SCI that occurred 7 years prior to participating in this study. The participant had no volitional arm or hand movement. A custom high-density ECoG grid (PMT Corp, Chanhassen, MN, USA) was implanted over the left sensorimotor cortex using stereotactic image guidance as shown in Wang et al. (2013). The grid was placed over cortical areas activated by video-guided attempted movement as measured with functional magnetic resonance imaging (fMRI) and MEG. The high-density ECoG grid contained 32 platinum disc electrodes with 28 electrodes facing the brain for recording and 4 ground and reference electrodes facing the dura. Two recording electrodes and 2 ground electrodes, located at the corners of the grid, were 3 mm in diameter. All other electrodes were 2 mm in diameter and spaced 4 mm apart. The platinum lead wires were tunneled to the subclavicular space and covered by a sterile dressing that allowed for connection to the neural recording system during recording sessions.

Data Collection

ECoG signals were sampled at 1200 Hz and band-pass filtered between 0.1 and 200 Hz using a g.USBamp recording system (g.tec Guger Technologies OEG, Austria). BCI2000, a general-purpose software for BCI research, was used to record neural and kinematic data (Schalk et al., 2004). Able-bodied subjects were asked to execute and observe grasping movements while hand and finger kinematics were recorded, either from the participant or the experimenter, with a 14-sensor 5DT data glove (5DT Inc., Irvine, CA, USA). Throughout the manuscript, the two conditions are referred to as action execution (AE) and AO. Subjects performed the grasping task with the hand contralateral to the ECoG grids, which in all cases was the dominant hand. Three objects (ball, toy hammer, and pen) were fixed with Velcro to a tray in front of the subject in an arrangement that minimized arm movement. Each object required a different type of grasp: power grasp, cylinder grasp, and pinch, respectively. The objects and grasp postures are shown in Figure 2. Participants began each repetition with their hand resting on the tray approximately 8 inches away from the objects and were prompted by a computer screen which object to grasp. The computer screen was placed approximately 12 inches in front of the tray on a bedside table. The height was adjusted for each subject such that they could comfortably view the objects and screen simultaneously. An image of the object to be grasped was shown for 2.5 s with an interstimulus interval ranging from 2 to 2.5 s between repetitions. Subjects were instructed to begin the movement as soon as the image appeared. They were to grasp and hold the object without lifting it until the picture disappeared signaling them to return to the rest position. The return to rest was completed during the interstimulus interval.

FIGURE 2

Figure 2. Experiment setup. Kinematics were recorded with a 5DT Data Glove while the subject (AE trials) or experimenter (AO trials) grasped 3 objects placed on a tray. The rest position is shown in (A). Pinch, power, and cylindrical grasps are shown in (B–D) respectively.

The placement order of the objects was shuffled by the experimenter for each block of repetitions which consisted of 5 grasps per object. Three of the participants performed 15–20 trials for each object, while Subject I had 5 AE trials and 10 AO trials per object. The number of sessions, and thus trials, was determined by the duration of the subject's stay in the epilepsy monitoring unit. During the observation trials, the experimenter stood next to the subject and grasped the objects when prompted by the computer using the same hand as the subject used to perform the task. Subjects were instructed to simply watch the experimenter performing the task and the experimenter watched the subjects to ensure that their hand remained at rest. Subject S completed 10 AO trials per object on day 24 post-implant as described above. The experimenter grasped the objects with her right hand since Subject S had the ECoG array implanted over the left hemisphere.

Data Analysis and Statistics

All analyses were performed using MATLAB (The Mathworks, Inc., Natick, MA, USA). Data were segmented into individual repetitions aligned to stimulus onset defined as the time at which the picture of an object appeared on the screen. Grasp onset was calculated by manual inspection of proximal interphalengeal (PIP) joint angles. Generally all of the fingers began to move together, but the earliest deviation from the rest angle for any finger was marked as grasp onset. Grasp completion time was defined when all PIP joints reached a constant angle signifying the steady-state grasp hold period. The spectral power of the neural data was computed from 0 to 200 Hz using the maximum entropy method with 3 Hz frequency bins (McFarland et al., 1997). The binned power spectra during each repetition were normalized by the average power recorded during all trials for a given electrode. This paper focuses on two frequency ranges: low frequency (10–40 Hz) and high frequency (65–115 Hz). The low frequency range corresponds to the mu and beta sensorimotor rhythms and the high frequency band represents the high-gamma band.

Data from all three grasps types were combined to form a single data set for each subject. For ease of comparison across participants, the trials were divided into three stages relative to stimulus onset: planning (0–0.5 s), early movement (0.5–1 s), and late movement (1–1.5 s). For each repetition, the normalized spectral power was calculated as the percent change relative to the average power recorded by a given electrode. For further analysis, the average normalized spectral power was calculated for the low and high frequency bands during the early movement phase. Depth of modulation is defined here as the average normalized spectral power in a given frequency band during the early movement phase. One-sided t-tests were used to test whether the mean depth of modulation measured during the early movement period was significantly different from 0. Essentially, this identified electrodes with significant activation during the early movement period as compared to the average activity recorded by a given electrode. Separate tests were completed for each electrode for both the low and high frequency band. Electrodes exhibiting a significant decrease in power in the low frequency band and a significant increase in power in the high frequency band were considered to have significant motor-related modulation.

The time series of normalized spectral power in the low and high frequency bands was calculated by averaging the normalized spectral power for all frequencies within the 10–40 and 65–115 Hz bands respectively (for an example, see Figure 4). For each subject, the normalized spectral power response for the low and high frequency band was averaged across trials for each object. The pairwise linear correlation coefficient was computed between the average time series of the normalized spectral power modulation during AE and AO for electrodes that showed a significant motor response during both AE and AO. The correlation coefficients for the modulated electrodes were combined across subjects into a single dataset so that the mean values could be computed for each object and frequency band. A fixed time window of 2 s following stimulus onset was used for this analysis. Two paired t-tests were used to test whether the depth of modulation of motor-related electrodes during the early movement period was significantly different between AE and AO for the low and high frequency bands.

Results

Kinematic Data

Grasp onset for the 4 able-bodied subjects ranged from 423 to 665 ms after stimulus onset and grasp completion time ranged from 1312 to 1808 ms after stimulus onset. The mean onset time was 531 ± 65 ms for AE and 524 ± 59 ms for AO. Grasps were performed by the experimenter during the AO condition. The mean grasp completion times were 1594 ± 180 and 1502 ± 103 ms for AE and AO respectively. Grasp kinematics were generally similar for the 3 objects. For all grasps, participants extended their fingers slightly to open the hand prior to flexing the metacarpophalengeal (MCP) and PIP joints to achieve the final grasp position. For the pinch grasp, the middle, ring, and pinky fingers remain extended. The objects and grasp postures are shown in Figure 2.

Identification of Motor-Related Responses in Able-Bodied Subjects

Electrodes that recorded a significant decrease in low frequency band (10–40 Hz) power and an increase in high frequency band power (65–115 Hz) during the early movement phase were considered to be motor-related electrodes. Of the 14 high-density ECoG electrodes for Subject F, 8 exhibited significant modulation during AE and 7 electrodes showed significant modulation during AO. Based on the projection of the x-ray onto a standard brain model (Figure 1), these electrodes were located over the middle frontal gyrus which contains the premotor cortex. For Subject G, 7 motor-related channels were identified during AE and 2 of these showed a significant modulation during AO. On the standard brain model, these electrodes spanned the pre- and post-central gyri which include the primary motor and somatosensory cortex respectively. Only 2 electrodes for Subject I (standard ECoG grid) showed a significant motor-related response during AE and none exhibited a significant response to AO. These electrodes appeared to be located over the pre-central gyrus. All 14 high-density ECoG electrodes were significantly modulated during AE and AO for Subject J. These electrodes were over premotor cortex spanning the superior and middle frontal gyrus. Figure 1 shows the location of significantly modulated electrodes for all able-bodied subjects. These electrodes covered various sensorimotor cortical areas across subjects, but all showed significant modulation during the motor task which involved observed or overt grasping.

Comparison between Action Execution and Observation

Table 1 summarizes the average depth of modulation for significantly modulated electrodes for each subject. For the able-bodied subjects who showed significant motor-related cortical activity during both AO and AE, the average depth of modulation of the low frequency band was 31.6 ± 10.6% in the AO condition and 42.7 ± 21.6% in the AE condition. For the high-gamma band, the average depth of modulation was 19.9 ± 7.7% in the AO condition and 25.0 ± 11.3% in the AE condition. The depth of modulation was significantly greater during AE for both the low (p = 0.002) and high frequency band (p = 0.02).

TABLE 1

Table 1. Average depth of modulation for electrodes significantly modulated by AE and AO.

Figure 3 shows time-frequency plots of the high-density ECoG activity recorded by an electrode for Subject F during AE and AO. During both conditions, for all grasps, the low frequency band power decreases and the high frequency band power increases (relative to each band's average power throughout an experiment session) immediately prior to and during movement. To quantify this similarity, the average low and high frequency power was computed over the trial duration and then averaged across trials for each object. Figure 4 shows the time course of the normalized power in the low and high frequency bands for a sample electrode that was significantly modulated during AE and AO. This figure illustrates the similarity of the modulation of spectral power during both conditions. For this particular subject, an increase in low frequency band power was recorded near the time of stimulus onset. Analysis of the kinematic data revealed that this increase corresponded to the low frequency band power rebound (Jurkiewicz et al., 2006) following movement back to the center resting position which occurred during the interstimulus interval. Figure 4 also illustrates the average correlation in normalized spectral power measured from all significantly modulated electrodes during AE and AO. Pairwise correlations of the average low and high frequency band power time series recorded during AE and AO for each object were computed for all channels that showed a significant motor-related response for a fixed time window of 2 s after stimulus onset. In general, the correlation values indicate a strong similarity in spectral power modulation between conditions. Across all objects, correlations were stronger for the high-gamma band (r = 0.779 ± 0.17) than for the low frequency band (r = 0.598 ± 0.22). For comparison, unmodulated electrodes on the high density ECoG grid or standard ECoG grids (n = 220) showed average correlations of 0.295 ± 0.20 for the high frequency band and 0.242 ± 0.18 for the low frequency band.

FIGURE 3

Figure 3. Example neural activity during action execution (AE) and observation (AO). Example time-frequency plot of neural activity (Subject F, high-density ECoG grid electrode #7) during AE and AO of three grasping tasks (Ball, Hammer, Pen). The color indicates normalized spectral power expressed as the percent change from the average activity of a given electrode. For both conditions, a decrease in low frequency spectral power and an increase in high frequency spectral power were observed with a similar temporal profile. Data was averaged based on stimulus onset (shown as a vertical dotted line at t = 0).

FIGURE 4

Figure 4. Correlation in spectral power during AE and AO. On the left, the mean correlation between normalized spectral power of the low and high frequency bands measured during AE and AO for three grasp conditions is shown. Correlations were computed for each of the 22 electrodes, from 3 subjects, that exhibited a significant motor response during both AE and AO and then the correlation coefficient values were averaged. Error bars represent the standard deviation. On the right, a sample plot of the average modulation of the low and high frequency bands is shown (Subject J, high-density ECoG electrode #8). Time 0 indicates stimulus onset. Normalized spectral power is reported as the percent change in power from the average response recorded by this electrode over all repetitions.

To further illustrate the similarity between AE and AO, Figure 5 and the Supplemental Movie 1 show the temporal evolution of the spatial distribution of the low and high frequency band responses across the high-density ECoG grid. Data was averaged across the three grasp types. Many channels on the high-density ECoG grid show a low frequency band power decrease followed by the typical rebound, which is an increase in low frequency band power after the movement ends (Jurkiewicz et al., 2006). The greatest power decrease of the low frequency band appeared to be centered at electrode 10 for both AE and AO. During the early and late movement periods, the high frequency band exhibits an increase in power for both AE and AO. Electrode 7 recorded the largest increase in high frequency band power for both conditions.

FIGURE 5

Figure 5. Spatiotemporal distribution of the low and high frequency band across the high-density ECoG grid for Subject F. The time intervals are relative to stimulus onset. The color indicates the percent change in the spectral power normalized to the average activity recorded by the electrode over all repetitions. The two corner electrodes marked as “0” served as reference channels.

Motor-Related Response to Action Observation after Spinal Cord Injury

The experimenter performed all grasps during AO trials with a mean grasp onset time of 561 ± 76 ms and a mean grasp completion time of 1674 ± 170 ms after stimulus onset. Fourteen of the 28 recording electrodes showed a significant motor-related response during AO. Electrodes showing significant motor-related modulation are shown on x-ray in Figure 6. When co-registered to the subject's structural MRI (shown in Wang et al., 2013), it appears that the majority of electrodes are located over primary somatosensory cortex. This reflects somatosensory activation resulting from observed grasping. The modulated electrodes were localized to contiguous electrodes on the medial and central portion of the ECoG grid. The average depth of modulation during AO was −13.6 ± 2.7% for the low frequency band and 22.6 ± 11.4% for the high-gamma band. Figure 7 shows time-frequency plots of the cortical activity recorded by an electrode for Subject S during AO of grasp. Cortical activation recorded with ECoG during AO was similar to that observed for able-bodied subjects.

FIGURE 6

Figure 6. ECoG grid location for Subject S, an individual with C4 complete spinal cord injury. The schematic inset shows electrodes that recorded a significant motor-related response during AO highlighted in green. The majority of these electrodes were located over primary somatosensory cortex. The large arrow marks the electrode whose data are shown in Figure 7.

FIGURE 7

Figure 7. Example time-frequency plot of neural activity (Subject S, electrode #12 as marked in Figure 6) during action observation (AO) of three grasping tasks (Ball, Hammer, Pen). The color indicates normalized spectral power expressed as the percent change from the average activity of a given electrode. In a person with spinal cord injury, we observed a decrease in low frequency spectral power and an increase in high frequency spectral power that was similar to the responses observed in able-bodied subjects (Figure 3).

Discussion

This paper provides evidence that observation-related activity can be recorded from the cortical surface with ECoG. To our knowledge this is the first study to document congruent activity in the high-gamma band during overt and observed movement. The high-gamma band carries significant information about movement, both overt and covert, and therefore may be important for developing ECoG-based BCI systems (Leuthardt et al., 2004; Miller et al., 2010; Wang et al., 2010b). This work extends the findings of others who observed congruence in the low frequency band during AE and AO in MEG and EEG studies (Hari et al., 1998; Muthukumaraswamy et al., 2004; Caetano et al., 2007; Perry and Bentin, 2009; Press et al., 2011). We observed that the temporal evolution of activity was similar during both conditions in the low and high frequency bands as evidenced by the strong correlations in normalized spectral power over the planning and movement phases (Figure 4).

Many investigators believe that observation-induced cortical activity is facilitated by the mirror neuron system which is a group of neurons that are activated similarly when a person acts and when the person observes the same action performed by another person (Fabbri-Destro and Rizzolatti, 2008). The mirror neuron system formally includes a subset of neurons in the inferior parietal lobe, ventral premotor cortex, and inferior frontal gyrus. Early studies of the mirror neuron system show that these neurons are selective for different grasp types (Gallese et al., 1996) and are maximally engaged for movements involving interaction between the hand and an object. Previous studies have also used kinesthetic motor imagery, during which a person imagines themselves performing an action, to activate the motor cortex in the absence of overt movement (McFarland et al., 2000; Ehrsson et al., 2003; Miller et al., 2010). We were able to estimate the location of modulated electrodes using the LOC toolbox (Miller et al., 2007). Although there are limitations with estimating electrode position from a single planar x-ray after a neurosurgical intervention, these results show that the modulated electrodes were over pre-motor, primary motor, or primary somatosensory cortex. Premotor cortex is formally part of the mirror neuron system and so one would expect that AO would result in modulation of these electrodes. Other studies have also shown that primary motor and somatosensory cortex are modulated during AO, although there is no clear consensus as to the mechanism of this activation (Hari et al., 1998; Avikainen, 2002; Caetano et al., 2007). Action observation and motor imagery likely leads to an internal representation of movement, which may activate the mirror neuron system (Jeannerod, 2001). It is possible that the mirror neuron system facilitates the imagination- or observation-related sensorimotor activity observed in this and other studies.

Previous studies have found that the depth of modulation for the low frequency band is lower during AO compared to AE (Hari et al., 1998; Caetano et al., 2007) and our results were generally consistent with this finding as summarized in Table 1. Some channels did show a larger modulation during the early movement period of AO compared to AE, however the magnitude of the difference was small. Overall, the depth of modulation for both the low frequency band and high-gamma band was significantly greater during AE compared to AO. For the high-gamma band, the average depth of modulation in able-bodied subjects was 19.9 ± 7.7% in the AO condition and 25.0 ± 11.3% in the AE condition, approximately 5% greater. Two of the participants (G and J) showed greater modulation of the low frequency band during AE. The average increase in depth of modulation was 10.2 ± 7.8% for Subject G and 20.4 ± 5.4% for Subject J. Interestingly, Subject F showed greater modulation of the low frequency band during AO compared to the AE condition, while high frequency band modulation was 9.5 ± 11.8% lower during AO. For this subject, we observed a similar spatiotemporal evolution of the low and high frequency neural activity during these two conditions (Figure 5). Visual inspection of the data on a single channel basis (Figures 3, 4) highlights the similarity in the cortical activation pattern between AE and AO as well as between grasp types.

For the purposes of developing a motor-BCI training strategy, we are interested studying methods to access motor-related activity in the absence of overt movement. To date, most ECoG studies have been conducted in patients undergoing intracranial monitoring for intractable epilepsy. We were able to collect ECoG data during AO from an individual with SCI when he participated in a BCI study with our research group (Wang et al., 2013). Approximately half of the electrodes on the ECoG grid implanted over his sensorimotor cortex were modulated by observed hand grasps. This is the first time that significant action observation-related activity has been measured with ECoG in someone with chronic tetraplegia. This response was observed on contiguous electrodes on the ECoG grid suggesting that the somatotopic representation of movement and the related somatosensory response may be preserved after SCI. Additionally, the spatiotemporal pattern and depth of modulation was similar for the motor-related responses recorded from the spinal cord injured subject and the able-bodied participants. These results are in line with previous studies that have shown that motor-related activation during covert movement is a robust and repeatable response among able-bodied individuals (Hari et al., 1998; Ehrsson et al., 2003; Neuper et al., 2005; Tkach et al., 2007; Miller et al., 2010). Previous studies with fMRI have shown that observation-related motor activity is still present after SCI (Sabbah et al., 2002). Further, recent evidence suggests that observation or imagery-related paradigms for BCI decoder training may be effective even after long-term paralysis (Hochberg et al., 2006; Kim et al., 2008; Collinger et al., 2013b). In these studies, subjects are often asked to imagine performing a movement while observing the visual display because motor imagery is known to activate motor cortex in a similar manner as overt movement. The ability to measure grasp-related activity through observed movement would facilitate BCI decoder training for controlling assistive devices for restoring or augmenting hand grasp such as a functional electrical stimulator or prosthetic hand. Restoration of upper limb function has repeatedly been identified as a top priority for individuals with tetraplegia (Anderson, 2004; Collinger et al., 2013a). Previous studies decoded detailed hand kinematics using ECoG signals recorded during overt hand movement (Kubánek et al., 2009; Wang et al., 2009) and an important next step will be to decode observed hand movement using ECoG data collected during AO. Further, it may be interesting to quantify differences in neural activation patterns between AO and AE whereas in this study the goal was to identify similarities between these two conditions.

The current study has several limitations. First, it studied a small number of subjects, including four able-bodied subjects who were undergoing ECoG monitoring for intractable epilepsy and one subject with tetraplegia. The number of subjects is comparable to previous ECoG studies in individuals undergoing ECoG monitoring (Schalk et al., 2007; Ball et al., 2009; Kubánek et al., 2009), and AO-related high-gamma band activity was captured in both able-bodied individuals and a person with chronic paralysis. Second, ECoG data reported here were collected either from a standard clinical grid or a high-density research grid. ECoG grid placement in subjects undergoing epilepsy monitoring was dictated by clinical needs, and the high-density research grid was placed over cortical areas not covered by clinical grids. The high-density research grids offer better spatial resolution than standard clinical grids, and are potentially more sensitive to subtle changes in local cortical activity (Wang et al., 2009; Slutzky et al., 2010; Wodlinger et al., 2011). In the current study, both high-density grids captured AO-related high-gamma band modulation. Last, one of the participants (Subject I) did not exhibit significant observation-related activity, even though two electrodes recorded a significant motor-related response during AE. A decrease in low frequency band power was observed in these channels during AO, although this was not statistically significant during the early movement phase. As shown in Figure 1, the position of the electrodes for Subject I is more medial than for the other subjects It is possible that the small number of repetitions (n = 10) may have limited our ability to measure a significant motor response during AO for this subject. This subject was much younger than the other subjects may have been less engaged during the AO task.

It is possible that eye movement can result in ECoG signal modulation, but this was mitigated in several ways. Eye movement often produces a transient time-domain response in scalp EEG recordings which results in an increase in power in the very low frequencies while high-gamma band activity resulting from eye movement is spatially restricted to the occipital lobe (Nagasawa et al., 2011; Plochl et al., 2012). We limited our analysis to the early movement phase when we expect that the subject's eyes were fixed on the object being grasped. The experimental setup was designed so that subjects could view the computer screen and objects with minimal eye movement and the object positions were varied between experimental blocks. Further, we would not expect that eye movement artifacts would be limited to localized cortical areas. We found that AE and AO resulted in modulation of focal areas of cortex in the pre-motor, motor, and somatosensory cortex. Finally, for the high-density ECoG grids, we use a corner electrode as a common reference which minimizes artifacts from sources away from the grid location.

ECoG recording in able-bodied subjects revealed congruent high-gamma band activity and sensorimotor rhythm modulation between observed and overt hand movements. Additionally, robust modulation of the high-gamma band and sensorimotor rhythm during AO was recorded from a participant with tetraplegia. These results suggest that AO can serve as a useful paradigm for neural decoder training for ECoG-based BCI applications to restore function to individuals with chronic paralysis.

Author Contributions

Jennifer L. Collinger, Douglas J. Weber, Michael L. Boninger, Elizabeth C. Tyler-Kabara, and Wei Wang designed the study. Jennifer L. Collinger, Ramana Vinjamuri, Alan D. Degenhart, Gustavo P. Sudre, and Wei Wang collected the data. Jennifer L. Collinger, Ramana Vinjamuri, Alan D. Degenhart, and Gustavo P. Sudre contributed to data analysis. Ramana Vinjamuri, Alan D. Degenhart, Gustavo P. Sudre, and Wei Wang developed the software used for data collection. Jennifer L. Collinger and Wei Wang wrote the manuscript and all authors provided critical review and approval of the manuscript.

Conflict of Interest Statement

The Associate Editor Martin Oudega declares that, despite being affiliated to the same institution as author(s) Jennifer L. Collinger, Ramana Vinjamuri, Douglas J. Weber, Michael L. Boninger, and Wei Wang, the review process was handled objectively and no conflict of interest exists. The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

This work was supported by Telemedicine and Advanced Technology Research Center (TATRC) Agreement W81XWH-07-1-0716, the Craig Neilsen Foundation, and the National Institutes of Health grants 5UL1RR024153, 3R01NS050256-05S1, 1R01EB009103-01, 1R21NS056136, and KL2TR000146. This material is supported in part by the Office of Research and Development, Rehabilitation Research & Development Service, VA Center of Excellence in Wheelchairs and Associated Rehab Engineering, Grant# B3142C and B6789C. The contents of this publication do not represent the views of the Department of Veterans Affairs or the United States Government. We would like to thank our subjects for volunteering to participate in this study. We would also like to thank the clinical and technical staff of the epilepsy monitoring units at the Children's Hospital of Pittsburgh and the University of Pittsburgh Medical Center, specifically: Dr. Deborah Holder, Dr. Anto Bagic, Dr. Gena Ghearing, Ms. Patricia Lordeon, Ms. Cheryl Plummer, Mr. Daniel Mosqueda, and Ms. Emily Oberst.

Supplementary Material

The Supplementary Material for this article can be found online at: http://www.frontiersin.org/journal/10.3389/fnint.2014.00017/abstract

Abbreviations

AO, action observation; AE, action execution; ALS, amyotrophic lateral sclerosis; BCI, brain-computer interface; EEG, electroencephalography; ECoG, electrocorticography; fMRI, functional magnetic resonance imaging; MCP, metacarpophalengeal; MEG, magnetoencephalography; PIP, proximal interphalengeal.

References

Acharya, S., Fifer, M. S., Benz, H. L., Crone, N. E., and Thakor, N. V. (2010). Electrocorticographic amplitude predicts finger positions during slow grasping motions of the hand. J. Neural Eng. 7:046002. doi: 10.1088/1741-2560/7/4/046002

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Anderson, K. D. (2004). Targeting recovery: priorities of the spinal cord-injured population. J. Neurotrauma 21, 1371–1383. doi: 10.1089/neu.2004.21.1371

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Avikainen, S. (2002). Modulated activation of the human SI and SII cortices during observation of hand actions. Neuroimage 15, 640–646. doi: 10.1006/nimg.2001.1029

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Ball, T., Schulze-Bonhage, A., Aertsen, A., and Mehring, C. (2009). Differential representation of arm movement direction in relation to cortical anatomy and function. J. Neural Eng. 6:016006. doi: 10.1088/1741-2560/6/1/016006

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Caetano, G., Jousmäki, V., and Hari, R. (2007). Actor's and observer's primary motor cortices stabilize similarly after seen or heard motor actions. Proc. Natl. Acad. Sci. U.S.A. 104, 9058–9062. doi: 10.1073/pnas.0702453104

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Collinger, J. L., Boninger, M. L., Bruns, T. M., Curley, K., Wang, W., and Weber, D. J. (2013a). Functional priorities, assistive technology, and brain-computer interfaces after spinal cord injury. J. Rehabil. Res. Dev. 50, 145–160. doi: 10.1682/JRRD.2011.11.0213

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Collinger, J. L., Wodlinger, B., Downey, J. E., Wang, W., Tyler-Kabara, E. C., Weber, D. J., et al. (2013b). High-performance neuroprosthetic control by an individual with tetraplegia. Lancet 381, 557–564. doi: 10.1016/S0140-6736(12)61816-9

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Crone, N. E., Miglioretti, D. L., Gordon, B., and Lesser, R. P. (1998a). Functional mapping of human sensorimotor cortex with electrocorticographic spectral analysis. II. Event-related synchronization in the gamma band. Brain 121(Pt 12), 2301–2315. doi: 10.1093/brain/121.12.2301

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Crone, N. E., Miglioretti, D. L., Gordon, B., Sieracki, J. M., Wilson, M. T., Uematsu, S., et al. (1998b). Functional mapping of human sensorimotor cortex with electrocorticographic spectral analysis. I. Alpha and beta event-related desynchronization. Brain 121(Pt 12), 2271–2299. doi: 10.1093/brain/121.12.2271

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Dushanova, J., and Donoghue, J. (2010). Neurons in primary motor cortex engaged during action observation. Eur. J. Neurosci. 31, 386–398. doi: 10.1111/j.1460-9568.2009.07067.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Ehrsson, H. H., Geyer, S., and Naito, E. (2003). Imagery of voluntary movement of fingers, toes, and tongue activates corresponding body-part-specific motor representations. J. Neurophysiol. 90, 3304–3316. doi: 10.1152/jn.01113.2002

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Fabbri-Destro, M., and Rizzolatti, G. (2008). Mirror neurons and mirror systems in monkeys and humans. Physiology (Bethesda) 23, 171–179. doi: 10.1152/physiol.00004.2008

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Flint, R. D., Wright, Z. A., Scheid, M. R., and Slutzky, M. W. (2013). Long term, stable brain machine interface performance using local field potentials and multiunit spikes. J. Neural Eng. 10:056005. doi: 10.1088/1741-2560/10/5/056005

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Gallese, V., Fadiga, L., Fogassi, L., and Rizzolatti, G. (1996). Action recognition in the premotor cortex. Brain 119(Pt 2), 593–609. doi: 10.1093/brain/119.2.593

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Hari, R., Forss, N., Avikainen, S., Kirveskari, E., Salenius, S., and Rizzolatti, G. (1998). Activation of human primary motor cortex during action observation: a neuromagnetic study. Proc. Natl. Acad. Sci. U.S.A. 95, 15061–15065. doi: 10.1073/pnas.95.25.15061

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Heldman, D. A., Wang, W., Chan, S. S., and Moran, D. W. (2006). Local field potential spectral tuning in motor cortex during reaching. IEEE Trans. Neural Syst. Rehabil. Eng. 14, 180–183. doi: 10.1109/TNSRE.2006.875549

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Hochberg, L. R., Bacher, D., Jarosiewicz, B., Masse, N. Y., Simeral, J. D., Vogel, J., et al. (2012). Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485, 372–375. doi: 10.1038/nature11076

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Hochberg, L. R., Serruya, M. D., Friehs, G. M., Mukand, J. A., Saleh, M., Caplan, A. H., et al. (2006). Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442, 164–171. doi: 10.1038/nature04970

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Jeannerod, M. (2001). Neural simulation of action: a unifying mechanism for motor cognition. Neuroimage 14, S103–S109. doi: 10.1006/nimg.2001.0832

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Jurkiewicz, M. T., Gaetz, W. C., Bostan, A. C., and Cheyne, D. (2006). Post-movement beta rebound is generated in motor cortex: evidence from neuromagnetic recordings. Neuroimage 32, 1281–1289. doi: 10.1016/j.neuroimage.2006.06.005

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kim, S.-P., Simeral, J. D., Hochberg, L. R., Donoghue, J. P., and Black, M. J. (2008). Neural control of computer cursor velocity by decoding motor cortical spiking activity in humans with tetraplegia. J. Neural Eng. 5, 455–476. doi: 10.1088/1741-2560/5/4/010

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kubánek, J., Miller, K., Ojemann, J., Wolpaw, J., and Schalk, G. (2009). Decoding flexion of individual fingers using electrocorticographic signals in humans. J. Neural Eng. 6:66001. doi: 10.1088/1741-2560/6/6/066001

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Leuthardt, E. C., Schalk, G., Wolpaw, J. R., Ojemann, J. G., and Moran, D. W. (2004). A brain-computer interface using electrocorticographic signals in humans. J. Neural Eng. 1, 63–71. doi: 10.1088/1741-2560/1/2/001

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

McFarland, D., Lefkowicz, A., and Wolpaw, J. (1997). Design and operation of an EEG-based brain-computer interface with digital signal processing technology. Behav. Res. Methods Inst. Comput. 29, 337–345. doi: 10.3758/BF03200585

CrossRef Full Text

McFarland, D. J., Miner, L. A., Vaughan, T. M., and Wolpaw, J. R. (2000). Mu and beta rhythm topographies during motor imagery and actual movements. Brain Topogr. 12, 177–186. doi: 10.1023/A:1023437823106

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Miller, K. J., Makeig, S., Hebb, A. O., Rao, R. P. N., Dennijs, M., and Ojemann, J. G. (2007). Cortical electrode localization from X-rays and simple mapping for electrocorticographic research: the “Location on Cortex” (LOC) package for MATLAB. J. Neurosci. Methods 162, 303–308. doi: 10.1016/j.jneumeth.2007.01.019

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Miller, K. J., Schalk, G., Fetz, E. E., Den Nijs, M., Ojemann, J. G., and Rao, R. P. N. (2010). Cortical activity during motor execution, motor imagery, and imagery-based online feedback. Proc. Natl. Acad. Sci. U.S.A. 107, 4430–4435. doi: 10.1073/pnas.0913697107

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Miller, K. J., Zanos, S., Fetz, E. E., Den Nijs, M., and Ojemann, J. G. (2009). Decoupling the cortical power spectrum reveals real-time representation of individual finger movements in humans. J. Neurosci. 29, 3132–3137. doi: 10.1523/JNEUROSCI.5506-08.2009

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Mukamel, R., Ekstrom, A. D., Kaplan, J., Iacoboni, M., and Fried, I. (2010). Single-neuron responses in humans during execution and observation of actions. Curr. Biol. 20, 750–756. doi: 10.1016/j.cub.2010.02.045

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Muthukumaraswamy, S. D., Johnson, B. W., and McNair, N. A. (2004). Mu rhythm modulation during observation of an object-directed grasp. Brain Res. Cogn. Brain Res. 19, 195–201. doi: 10.1016/j.cogbrainres.2003.12.001

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Nagasawa, T., Matsuzaki, N., Juhasz, C., Hanazawa, A., Shah, A., Mittal, S., et al. (2011). Occipital gamma-oscillations modulated during eye movement tasks: simultaneous eye tracking and electrocorticography recording in epileptic patients. Neuroimage 58, 1101–1109. doi: 10.1016/j.neuroimage.2011.07.043

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Neuper, C., Scherer, R., Reiner, M., and Pfurtscheller, G. (2005). Imagery of motor actions: differential effects of kinesthetic and visual-motor mode of imagery in single-trial EEG. Brain Res. Cogn. Brain Res. 25, 668–677. doi: 10.1016/j.cogbrainres.2005.08.014

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Perry, A., and Bentin, S. (2009). Mirror activity in the human brain while observing hand movements: a comparison between EEG desynchronization in the mu-range and previous fMRI results. Brain Res. 1282, 126–132. doi: 10.1016/j.brainres.2009.05.059

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Plochl, M., Ossandon, J. P., and Konig, P. (2012). Combining EEG and eye tracking: identification, characterization, and correction of eye movement artifacts in electroencephalographic data. Front. Hum. Neurosci. 6:278. doi: 10.3389/fnhum.2012.00278

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Press, C., Cook, J., Blakemore, S.-J., and Kilner, J. (2011). Dynamic modulation of human motor activity when observing actions. J. Neurosci. 31, 2792–2800. doi: 10.1523/JNEUROSCI.1595-10.2011

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Sabbah, P., De, S. S., Leveque, C., Gay, S., Pfefer, F., Nioche, C., et al. (2002). Sensorimotor cortical activity in patients with complete spinal cord injury: a functional magnetic resonance imaging study. J. Neurotrauma 19, 53–60. doi: 10.1089/089771502753460231

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Schalk, G., Kubánek, J., Miller, K. J., Anderson, N. R., Leuthardt, E. C., Ojemann, J. G., et al. (2007). Decoding two-dimensional movement trajectories using electrocorticographic signals in humans. J. Neural Eng. 4, 264–275. doi: 10.1088/1741-2560/4/3/012

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Schalk, G., McFarland, D. J., Hinterberger, T., Birbaumer, N., and Wolpaw, J. R. (2004). BCI2000: a general-purpose brain-computer interface (BCI) system. IEEE Trans. Biomed. Eng. 51, 1034–1043. doi: 10.1109/TBME.2004.827072

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Slutzky, M. W., Jordan, L. R., Krieg, T., Chen, M., Mogul, D. J., and Miller, L. E. (2010). Optimal spacing of surface electrode arrays for brain-machine interface applications. J. Neural Eng. 7, 26004. doi: 10.1088/1741-2560/7/2/026004

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Taylor, D. M., Tillery, S. I. H., and Schwartz, A. B. (2002). Direct cortical control of 3D neuroprosthetic devices. Science 296, 1829–1832. doi: 10.1126/science.1070291

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Tkach, D., Reimer, J., and Hatsopoulos, N. G. (2007). Congruent activity during action and action observation in motor cortex. J. Neurosci. 27, 13241–13250. doi: 10.1523/JNEUROSCI.2895-07.2007

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Tkach, D., Reimer, J., and Hatsopoulos, N. G. (2008). Observation-based learning for brain-machine interfaces. Curr. Opin. Neurobiol. 18, 589–594. doi: 10.1016/j.conb.2008.09.016

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Velliste, M., Perel, S., Spalding, M. C., Whitford, A. S., and Schwartz, A. B. (2008). Cortical control of a prosthetic arm for self-feeding. Nature 453, 1098–1101. doi: 10.1038/nature06996

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Wang, W., Chan, S. S., Heldman, D. A., and Moran, D. W. (2010a). Motor cortical representation of hand translation and rotation during reaching. J. Neurosci. 30, 958–962. doi: 10.1523/JNEUROSCI.3742-09.2010

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Wang, W., Collinger, J. L., Degenhart, A. D., Tyler-Kabara, E. C., Schwartz, A. B., Moran, D. W., et al. (2013). An electrocorticographic brain interface in an individual with tetraplegia. PLoS ONE 8:e55344. doi: 10.1371/journal.pone.0055344

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Wang, W., Collinger, J. L., Perez, M. A., Tyler-Kabara, E. C., Cohen, L. G., Birbaumer, N., et al. (2010b). Neural interface technology for rehabilitation: exploiting and promoting neuroplasticity. Phys. Med. Rehabil. Clin. N Am. 21, 157–178. doi: 10.1016/j.pmr.2009.07.003

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Wang, W., Degenhart, A. D., Collinger, J. L., Vinjamuri, R., Sudre, G. P., Adelson, P. D., et al. (2009). Human motor cortical activity recorded with Micro-ECoG electrodes, during individual finger movements. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2009, 586–589. doi: 10.1109/IEMBS.2009.5333704

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Wodlinger, B., Degenhart, A. D., Collinger, J. L., Tyler-Kabara, E. C., and Wang, W. (2011). The impact of electrode characteristics on electrocorticography (ECoG). Conf. Proc. IEEE Eng. Med. Biol. Soc. 2011, 3083–3086. doi: 10.1109/IEMBS.2011.6090842

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., and Vaughan, T. M. (2002). Brain-computer interfaces for communication and control. Clin. Neurophysiol. 113, 767–791. doi: 10.1016/S1388-2457(02)00057-3

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text