ORIGINAL RESEARCH article
Temporal coding of brain patterns for direct limb control in humans
- 1 Laboratory of Brain–Computer Interfaces, Institute for Knowledge Discovery, Graz University of Technology, Graz, Austria
- 2 Computer Science and Engineering, University of Washington, Seattle, WA, USA
For individuals with a high spinal cord injury (SCI) not only the lower limbs, but also the upper extremities are paralyzed. A neuroprosthesis can be used to restore the lost hand and arm function in those tetraplegics. The main problem for this group of individuals, however, is the reduced ability to voluntarily operate device controllers. A brain–computer interface provides a non-manual alternative to conventional input devices by translating brain activity patterns into control commands. We show that the temporal coding of individual mental imagery pattern can be used to control two independent degrees of freedom – grasp and elbow function – of an artificial robotic arm by utilizing a minimum number of EEG scalp electrodes. We describe the procedure from the initial screening to the final application. From eight naïve subjects participating online feedback experiments, four were able to voluntarily control an artificial arm by inducing one motor imagery pattern derived from one EEG derivation only.
Brain–computer interfaces (BCIs) are systems that establish a direct connection between the human brain and a computer (Wolpaw et al., 2002 ), thus providing an additional communication channel. For individuals suffering from severe palsy, amyotrophic lateral sclerosis (ALS) or brain stem stroke, such a BCI constitutes a possible way to communicate with the environment (Birbaumer et al., 1999 ; Nijboer et al., 2008 ; Kübler et al., 2009 ). BCIs can also be used to control neuroprostheses in patients suffering from a high spinal cord injury (SCI), for example by using functional electrical stimulation (FES) for grasp restoration with surface electrodes (Pfurtscheller et al., 2003 ) or implanted devices (Müller-Putz et al., 2006 ).
A major problem in high SCI patients (lesion above cervical level C4) is that they lose control over their grasp and elbow functions. In addition to these functional deficits, the ability to control external levers or joysticks also deteriorates. Eye-tracking systems in combination with a computer screen are a viable option for spelling purposes, whereas such systems are cumbersome in prosthesis control applications because the users have to watch their moving arm. However, Danóczy et al. (2008) described the use of eye gaze for target selection and control of grasp in a robotic arm. Another possibility for neuroprosthetic control is the use of facial and neck electromyogram (EMG) recordings as proposed by Kirsch (2005) and Williams and Kirsch (2008) . Whether patients can use such a configuration outside of their home is an open question. However, it seems that for those cases, a non-invasive BCI based on electroencephalographic (EEG) signals provides a good option to control such devices.
In the last couple of years, several attempts were made to control a neuroprosthetic device with the BCI. Two major approaches emerged: On the one hand, there is the invasive approach by decoding spike trains of motor neurons and transforming them into control signals for robotic arms or FES devices. Most of this research is done in non-human primates (Moritz et al., 2008 ; Pohlmeyer et al., 2009 ), only very few cases of paralysed humans were presented (Hochberg et al., 2006 ). Less invasive are electrocorticographic (ECoG) signals recorded from electrodes placed on the surface of the brain without penetrating the cortex. ECoG provides a high signal to noise ratio and spatial resolution. First attempts to use this signal for neuroprosthetic control were reported recently (Pisthol et al., 2008 ; Márquez-Chin et al., 2009 ; Waldert et al., 2009 ). Currently the medical risks using invasive technologies are still too high. The effort to reduce the risks and improve technology may make this technology feasible for future clinical applications.
On the other hand, there are efforts to restore movement in high-level SCI patients, by the use of an EEG-based, and therefore non-invasive, BCI. Heasman et al. (2002) reported on a neuroprosthetic control based on the modulation of the occipital alpha by opening and closing the eyes. Our group reported on the restoration of the grasp function in high SCI patients by implementing a so-called brain switch based on either foot motor imagery (Pfurtscheller et al., 2003 ) or left hand motor imagery (Müller-Putz et al., 2005 ). An EEG-based prosthetic control utilizing the modulation of steady-state visual evoked potentials (SSVEP) (Regan, 1989 ) has recently been reported (Müller-Putz and Pfurtscheller, 2008 ). In this work, a four-class SSVEP–BCI was implemented to control hand open and close movements as well as pronation and supination of the wrist.
One issue in non-invasive BCIs based on the detection and classification of imagery patterns is the limited number of control signals available. However, the number of control commands provided by a BCI is rather low (at least in motor imagery-based BCIs this number typically ranges from one to five classes, see for example Obermaier et al., 2001 ). To increase the number of commands, we implemented a temporal coding of the brain signal.
The Graz-BCI utilizes the midcentral event-related desynchronization and synchronization (ERD/ERS) (Pfurtscheller and Lopes da Silva, 1999 ) during motor imagery to control external devices (Pfurtscheller and Neuper, 2001 ). The neurophysiological basis for the use of such a mental strategy for an EEG-based BCI is that execution and imagination of limb movements activate overlapping sensorimotor cortical areas (Gerardin et al., 2000 ) and exhibit similar ERD/ERS patterns (Pfurtscheller and Neuper, 1997 ). In order to use the so-called ERD–BCI in patients, it is important to establish an experimental procedure that guides the participant to develop an appropriate mental strategy (Neuper et al., 2003 ) and keeps the motivation of the patients as high as possible.
The aim of this work can be summarized with the following three points:
• To establish a procedure starting from initial screening to the application of a control system,
• to evaluate the feasibility of only one motor imagery pattern for the control of an artificial arm (grasp and elbow functions),
• and finally, to further confirm that it is possible to establish such a control system with minimum number of EEG recordings (only one EEG derivation).
The procedure used to train naive BCI users to produce a brain pattern which can be used in a temporal coding manner is shown in Figure 1 . Subjects started with a screening session investigating three different types of motor imagery (MI) and step by step the most reactive brain pattern was identified and finally the subjects learned to establish this pattern over two different time durations. Finally, this pulse-width coded (PWC) brain switch was applied in an artificial arm control.
Figure 1. Procedure for naïve BCI users to learn to control an artificial arm with only one out of three motor imagery (MI) patterns (left, L; right, R, feet, F). Screening is performed with three types of MI, after feature extraction and classification, individual two types were selected and used in the Basket paradigm. From those results, the most reactive pattern was chosen to be used for the platform game and following in the arm control.
Materials and Methods
In order to set up the BCI in naive (inexperienced) BCI users to operate an artificial arm with only one motor imagery pattern, a set of experiments and feedback paradigms was devised. An overview of the whole procedure is given in Figure 1 .
Ten healthy subjects (mean age 28.1 years, median age 24.5 years, standard deviation 10.3), 4 females and 6 males, participated in this study. The subjects were without any medical or psychological diseases, had normal or corrected to normal vision and were paid for attending to the experiments. At the beginning of the study, all subjects were informed about the aim of this research project and gave their written consent to participate. The study was approved by the local ethic committee.
The participants were sitting comfortably in an armchair, 1.2 m in front of a computer screen. Before the experiment started, they were instructed via a presentation running on the screen in front of them. After that, one out of three different visual cues was presented in a random order on the computer screen, indicating the type of motor imagery to be performed (e.g., a left arrow for left hand motor imagery, right arrow for right hand motor imagery, and an arrow pointing downwards for foot motor imagery). Specifically, a single trial consisted of the following blocks: At second 0 a cross appeared in the middle of the black screen. After 2 s a beep sounded to catch the subject’s attention. The cue indicating the requested motor imagery appeared at second 3 and stayed on the screen until second 4.25. During a time period of 5 s, the subjects were requested to perform kinesthetic motor imagery (Neuper et al., 2005 ) according to the cue. At second 8 the cross disappeared and the screen was blank again. A variable pause lasting between 0.5 and 2.5 s was added before the next trial began. One run consisted of 30 cues, 10 for each class. The whole screening session consisted of eight runs resulting in 80 trials for each class. Short breaks were made between the runs. To get used to the experimental paradigm before the experiment was started, all participants had to execute the movements according to the cues presented on the screen.
EEG and EMG recordings
Thirty-two sintered Ag/AgCl electrodes were mounted over sensorimotor areas covering C3, Cz, and C4 with the reference at the left and the ground electrode at the right mastoid (see Figure 2 A). All electrode impedances were kept below 5 kOhms. A monopolar amplifier (Synamps, Compumedics Germany GmbH, Singen, Germany) was programmed to record EEG signals with a sampling frequency of 1000 Hz, 0.05–200 Hz bandpass including a notch filter at 50 Hz. Sensitivity was 100 μV. The EMG was recorded from three bipolar channels over the left/right finger extensor muscles at the forearm and the right musculus tibialis, respectively, using a bipolar amplifier (g.tec, Guger Technologies, Graz, Austria). Filter settings were set to 0.5 Hz for high and 1000 Hz for low pass, the sensitivity was set to 2 mV. The EMG data was digitized with 3000 Hz and stored for further analysis.
Figure 2. (A) Schematic EEG electrode arrangement. For electrode position Cz (#17) the required electrodes for the orthogonal source derivations are highlighted. (B) Median ERD/ERS maps for C3, Cz, and C4 and for three types of motor imagery. (C) Mean ERD/ERS values of defined frequency ranges (mu and beta) and three types of motor imagery. Significant differences (p < 0.05) are marked with a square bracket, an additional asterisk indicates a significance level of p < 0.01.
Orthogonal source derivations (Laplacian derivations) (Hjorth, 1975 ) were calculated to obtain reference-free data. Segments (trials) of 8 s duration were extracted from the data with respect to the visual cue onset (3 s before and 5 s after the cue).
To obtain time–frequency maps of the eight Laplacian channels, ERD/ERS analysis (Pfurtscheller and Lopes da Silva, 1999 ) was performed for frequency bands between 1 and 40 Hz with respect to a specific reference interval (0.5–1.5 s). To that end, sinusoidal wavelets were used to assess changes in the frequency domain by calculating the spectrum within a sliding window, squaring and subsequent averaging over the trials (Delorme and Makeig, 2004 ). The statistical significance of the ERD/ERS values was determined by applying a t-percentile bootstrap algorithm (Davision and Hinkley, 1997 ) with a significance level of α = 0.05. These so-called ERD/ERS maps were calculated for each subject.
For a more general overview, so-called “median ERD/ERS maps” were computed. The median of the ERD/ERS values of each time and frequency point of all 10 subjects was determined resulting in a new ERD/ERS map. To include the statistical significance of the individual ERD/ERS maps, all not significant values were set to zero. This procedure was performed for each of the three motor imagery classes.
For statistical analysis, the significant frequency ranges in the alpha and beta band were extracted from the median maps. Here the ERD/ERS was counted only if it lasted at least 0.75 s and its amplitude was more than 20%. This was done for each type of motor imagery and its somatotopically corresponding cortical area (e.g., left hand motor imagery and C4, right hand and C3, feet motor imagery and Cz). The obtained frequency ranges were then used to extract the ERD/ERS values of each subject for the three types of motor imagery and the three electrode positions C3, Cz, and C4 in the time range from second 3.5 to 7. For each frequency band (alpha and beta) an analysis of variance (ANOVA) was performed. Each ANOVA consisted of the within-subject variables “class” (left hand, right hand, or foot motor imagery) and “electrode” (C3, Cz, C4).
Identification of two best separable motor imagery classes
Each Laplacian EEG channel derivation was analyzed independently by means of the distinction sensitive learning vector quantization (DSLVQ) algorithm (Pregenzer et al., 1996 ). At each time two out of the three motor imagery tasks were compared. DSLVQ was applied to logarithmic band power features (calculated by filtering, squaring and averaging over a 1-s time window) from 6 to 34 Hz (step size 1 Hz, bandwidth 2 Hz) extracted from corresponding trials with the same latency to the beginning of the trial. These features were analyzed independently in steps of 0.5 s from 0.0 s to 8.0 s.
In order to obtain reliable values of the classification performance and the feature relevance the DSLVQ method was repeated 100 times (three codebook vectors per class, type C training, 10000 iterations, learning rate decreased from α = 0.05 to α = 0.0 and α′(t) = 0.1 α(t)).
The two best separable motor imagery tasks were chosen for further subject training. This selection implies one Laplacian channel and up to three frequency bands. For classification in the next experiment, a classifier based on Fisher’s linear discriminant analysis (LDA, 10-times 10-fold crossvalidation) was calculated.
Cue-Based Basket Paradigm
After the screening experiments, two of the subjects decided not to participate in further experiments, which left eight remaining subjects for the subsequent experiments. With the classifiers obtained from the screening session, online feedback training was performed using the “basket paradigm” (Krausz et al., 2003 ). The aim of this experiment was to hit a target (basket) with the ball-shaped control cursor at a predefined time. A red and a green “basket” (target) were presented at the bottom of the screen. After a pause with a fixed length of 1 s, a small green ball appeared at the top of the screen and began to fall downwards with a constant speed. The time the ball took to reach the target was set to 4 s. The subjects’ task was to hit the green basket (the position of which changed randomly from trial to trial). The horizontal position of the ball was directly controlled by the classification output (distance to the hyperplans), which was weighted by previously calculated gain factors to shift the mean deflection for each direction to the middle of the basket. Each session consisted of eight runs and each run consisted of 40 trials. After each run, the performance score (correct hits) was presented to the subjects on the screen. During this training they learned to establish two different brain patterns by imagining hand and/or foot movements. These experiments were repeated until subjects reached at least 70% accuracy over one session (320 trials).
Five Ag/AgCl electrodes comprising one Laplacian derivation were placed either around C3, Cz, or C4, depending on the results of the screening procedure. The electrodes were placed in a way that a single orthogonal derivation was possible, which means one electrode was directly over, for example, C3 whereas the remaining four were placed 2.5 cm anterior, posterior, lateral, and medial to this position. The reference electrode was placed at the left mastoid and the ground electrode was mounted at the right mastoid. The EEG was recorded using a g.BSamp amplifier (g.tec, Guger Technologies, Graz Austria), 0.5–100 Hz band pass filter, notch filter on (50 Hz), and a sensitivity of 100 μV. The sampling rate was 250 Hz.
The real-time Graz BCI system is based on Matlab and Simulink using the Real-Time Windows Target toolbox (The Mathworks Inc., Nattick, USA). The Laplace derivation CLAP was computed by applying equation (1) on a sample-by-sample basis. Here Ccenter was, for example, C3 and Csurr were the orthogonally surrounding electrodes (Hjorth, 1975 ).
Furthermore, logarithmic band power features were computed (again sample-by-sample). By applying Fisher’s LDA (the weights were obtained during the cue-based training period) a classification was realized. The LDA distance was used to evaluate the brain pattern. After each session the DSLVQ analysis was repeated and a new classifier was computed. Depending on the online/offline classification accuracy (highest value wins), the old classifier, the updated classifier, or the new classifier was used for the next feedback experiment.
For the next set of experiments only one brain pattern was selected and used as follows: Analyzing the classifier output time series of the two classes lead to the conclusion that one out of the two classes (class 1) did not significantly change from a period without motor imagery to a period with motor imagery. However, the other class (class 2) did show significant changes. It was therefore assumed that the first case (class 1) is a very general case, which means that the classifier would also select this class if no motor imagery was performed. Thus, this classifier output describes the so-called no-control (NC) state. Introducing a threshold TH into the class of the other motor imagery pattern (class 2), a switch function can be designed (intentional control, IC). Only when the motor imagery is recognized well enough to exceed the threshold a control signal is triggered.
Asynchronous Training: A Platform Game
For the purpose of subject training a computer game-like paradigm was created resembling a platform game. The participants were controlling a jumping ball and had to leapfrog obstacles presented in random intervals between 10 and 15 s along the way. The obstacles were “hills” with the length of either 1 or 3 s. Each time the LDA output was exceeding a selected threshold (TH was set to the class mean plus its standard deviation) the difference between the actual LDA output and the threshold was mapped to the height of the ball. The subjects were instructed to perform motor imagery only to jump over the obstacles and not in the periods in between. Six runs (each run lasted 300 s) with 10 short and 10 long obstacles (randomly placed) were performed. At the upper left corner of the screen, a score corresponding to the game performance was displayed. It increased when the ball successfully moved over the obstacles. For further analyses, the EEG data (recorded in the same way as for the basket training), the landscape and the trajectory of the ball were stored.
To conduct a more detailed analysis as compared to the performance measure during the game, the ball movement was analyzed. Therefore, four parameters were defined. The true positives (TPs) indicate whether the ball was correctly moving over the obstacles. The maximum number was 40 s (corresponding to 100%). The false positives (FPs) give the time when the ball was jumping without moving over the hills (maximum 260 s corresponding to 100%). Taking into account that a user typically starts jumping a bit before the hill begins and also jumps a little farther than necessary, the number of TPs and FPs was calculated in a second way. In addition to the duration of the hill (1 or 3 s), 1 s was padded before and after the actual hill, thus generating the hill size with 3 and 5 s.
Pulse-Width Modulated Brain Switch for Artificial Arm Control
The last stage consisted of controlling a robotic arm. For the arm a simple robot with six degrees of freedom was mounted on the chest of a manikin. For this study, only the gripper, serving as the hand, and the elbow were used.
The two different durations of the mental activity were used to operate a pulse-width coded switch. The output of the PWC switch depended on the threshold TH and the durations tshort and tlong. Each time the TH was exceeded for a duration t > tlong, the output was 2; for t > tshort and t < tlong, the output was 1; otherwise the output was 0. The two states were alternatively mapped to the commands hand open/close (state 1) and elbow flexion/extension (state 2) (see Figure 3 ). After each triggered movement, a so-called refractory period of 5 s was added. During this period commands given by the BCI were ignored, so that the corresponding hand/arm could move without any disturbances.
To get control and evaluate the performance, all subjects had to perform a predefined movement sequence: hand open (O), hand close (C), elbow flexion (F), elbow extension (E), O, and C. The training was performed in the “error ignoring” mode. This means that the robotic device was only accepting commands in the correct order. Wrong commands were ignored.
Evaluation of the PWC switch
To evaluate the performance, the experiment was repeated in two different ways (four runs each): First, after a long period (1 min) of rest (non-control state) subjects had to perform the sequence as fast as possible, followed by a period of 30 s of non-control. Then the sequence had to be performed again, and a non-control period of 1 min finalized one run. The periods without actions (2.5 min per run, total 10 min) were introduced to measure the number of false positives. While the experiment is performed, the number of TPs, false negatives (FNs), and FPs can be counted. In case of a FP during non-control periods – where the arm moved as well – participants had to finish the sequence first and then repeat the whole sequence.
In a second block of runs, the subjects had to perform the movement sequence according to the timing indicated by the experimenter. Again, non-control periods were included (1 min per run, total 4 min). This procedure allowed the identification TP and FP detections.
To get more insight into the results different performance measures were computed: the first one was the FN/TP rate calculated from both evaluations blocks. The number of FPs itself was related to the maximum number of FPs possible. Therefore, the time needed to trigger the hand and elbow was added separately for each subject. The maximum time of non-control periods (14 min) was divided by the individual trigger times of the subjects. The number of FP was then related to this number and presented as a percentage number. Finally, the reaction time was calculated (from run 5 to 8) to give an overview of how many seconds a subject took to carry out a specific movement. For both movements the times are presented. An illustration of the set-up can be seen in Figure 3 .
Classification accuracies, type of imagination as well as identified frequency bands after screening of 10 subjects are presented in Table 1 . The average accuracy was 83.8%. Furthermore, the positions of the Laplacian derivation as well as the time point of the classification accuracy complete this description.
Table 2. Frequency ranges obtained from median ERerD/S maps (at least 0.75 s ERerD of more than 20%).
The results of the two ANOVAs show a significant main effect “class” (F(2,18) = 35.86, p < 0.001 for the mu band; F(2,18) = 3.941, p = 0.038 for the beta band) and a significant interaction “class × electrode” (mu: F(4,36) = 7.941, p < 0.001; beta: F(4,36) = 3.776, p = 0.012). Paired t-tests were applied to investigate the differences of ERD/ERS for each electrode. In Figure 2 C the means for mu and beta are shown. Significant (p < 0.05) differences are market with a squared bracket, additional marked brackets (with ‘*’) show a significance level p < 0.01. This results show clearly, the lateralization that occurs corresponding to each type of motor imagery.
Electromyogram signals obtained from the screening recordings were classified according to the cues presented on the screen. It was found that there was no correlation between the EMG and classification results.
Cue-Based Basket Paradigm
Eight subjects participated in the basked BCI feedback paradigm. The goal was to reach at least 70% accuracy. Therefore, each subject performed an individual number of sessions. In Table 3 the results of the last session can be seen. Additionally, the used Laplacian derivations as well as the two motor imagery classes are displayed. Seven of the participating subjects reached an accuracy ranging from 70.4% to 91.6% (mean 81.6%). Only subject al2 was unable to hit the targets after four sessions (51.9%).
Table 4 presents the number of TPs and FPs for both the strict and weak conditions during six runs (total 30 min) of five subjects. After averaging the LDA output for short and long jumps, the mean and standard error give a qualitative overview of the system performance (not presented in the paper).
Table 4. Results of pulse-width modulated brain switch during six runs (total 30 min) platform game.
Artificial Arm Control
The results of the evaluation procedure with the artificial arm consisting of two parts (four runs each) are presented in Table 5 . The number of TPs and FNs during control and the number of FPs during non-control are presented in the first three columns. The times needed to trigger a movement are displayed in the next two columns separately for the hand and elbow movement. For a better overview, the FN/TP rates as well as a measure for FPs are given in the last two columns. One subject was not able to participate in runs 5–8. In Figure 4 , the time–frequency maps of the best subjects are presented for short motor imagery and long motor imagery, respectively. The learned motor imagery pattern represented by power decrease (ERD) in two frequency bands and the resulting LDA output can be clearly seen. The corresponding mean LDA output (± standard error) is shown in the right part of the figure. Here, also the results of the other participants are presented.
Figure 4. LDA output of the evaluation procedure with the artificial arm. Averaged LDA output (± standard error) of four subjects for short (upper) and long (lower) switches are given in the panels at the right. The horizontal line represents the threshold to be exceeded for the duration detection. Time duration between the vertical lines show give the time period for short and long motor imagery duration. At the left side an ERD/S map of subject al4 shows the reactive frequency bands during both, the short and long motor imagery.
Additionally, a movie is available as supplement. It shows one sequence performed by subject al4. Starting with opening, closing, elbow flexion, elbow extension, opening (all TP), elbow flexion (not performed, but counted as FN), and closing.
In this paper we described a BCI set-up procedure which was defined to start with naive BCI users with an initial screening and continues with different experiments to finally let them control an artificial arm in a self-paced way by means of a pulse-width coded brain switch. During the first screening, participants performed imagined movements with three limbs. Pattern analyses lead to the use of two patterns for cue-based feedback training. Further analysis led to one distinct pattern which was then trained to be elicited over two durations. The applied procedure was straightforward and led to satisfying results.
In the following paragraphs, the single procedural steps are discussed in more detail.
The mu and beta frequency ranges obtained from median ERD/S maps and the frequency bands obtained from DSLVQ show the same results. Median ERD/S maps show bands from 9.3 to 12.7 Hz (mu), 18.9 to 23.7 Hz (beta left, right motor imagery), and 26.9 to 27.8 Hz (beta, foot motor imagery). Mean frequency bands from DSLVQ are 9.7 to 12.9 Hz (mu) and 26.9 to 27.8 Hz. It is interesting to note that the beta ERD in foot motor imagery is higher than for left or right hand motor imagery. However, in a former study, short time foot motor imagery showed also a beta ERD in the range of 29.0 ± 4.4 Hz (Müller-Putz et al., 2007 ).
When inspecting the initial classification accuracies obtained from the DSLVQ analysis, it is worth mentioning that the mean accuracy was 83.8% (average over 10 subjects, cross-validated). Such a high accuracy may be a result of the careful preparation and involvement of the participants in the study. Moreover, they had to execute the movements prior the EEG recording. Another point was that the experimenter instructed the subjects to perform kinesthetic motor imagery and not only visual motor imagery (for more details see Neuper et al., 2005 ).
Further insights are obtained from the DSLVQ analysis (Table 2 ). All best accuracies show that there is always foot motor imagery involved when comparing two classes. So, it is either left hand motor imagery or right hand motor imagery versus foot motor imagery. The statistical analysis shows why. During left or right hand motor imagery there is always an ERD at the contralateral side (as often reported by e.g., Pfurtscheller and Lopes da Silva, 1999 ) and no ERD/S at the other electrode positions. On the other side, during foot motor imagery there is a weak ERD over Cz accompanied by an ERS at the lateral electrode positions. This phenomenon of local (Cz) ERD and surround ERS (C3, C4) was reported by Neuper and Pfurtscheller (2001) .
An important step in BCI set-up is to transfer good classification accuracies from initial screening into comparable classifications when presenting online feedback. Eight subjects participated in the Basket feedback paradigm. They performed several sessions until their classification accuracy reached a level of about 70%. From the results presented in Table 3 , only one person was not able to learn to control the ball of the paradigm and was therefore excluded from further studies. The average of their performances (without al2) was 81.6 ± 8.7% and was insignificantly lower (t-test, p < 0.05) than the results from initial screening (83.7%). Here, four subjects reached more than 80% and three more than 70%.
Five subjects remained in the study (two quitted their participation, one was excluded) and played the platform game. The idea behind this was to train one selected type of motor imagery to be used for different durations. Therefore, the subjects had to leapfrog hills using motor imagery lasting for one and three seconds, respectively. The results (see Table 4 ) show that only one (al7) out of the five subjects was not able to perform the task with certain accuracy. TPs and FPs were nearly equal which shows that there was no control. Also, calculating FP and TP with more weak boarders, there was no real improvement. The remaining four subjects were invited to participate in the final experiment.
One very important result found in this work and reported for the first time is that individuals can learn to modulate patterns with variable durations use those for different commands. On the basis of PWC method, the control of an artificial arm was realized. Participants were trained to establish one motor imagery pattern over two durations, depending on the task to be performed. It is important to mention that the durations of the motor imagery patterns varied over subjects and the adjustments for the timing threshold has to be done very carefully. It is necessary to make a clear difference between the codes. Here it is important to note that a pattern cannot be actively stopped by the BCI user. He/she just can stop in imagining the movement. Then the pattern needs some time to collapse. This should be in a timescale where the time threshold for the long motor imagery has not shown up. The critical point is to set both thresholds in a way that FNs are avoided (selecting the long period instead of the short or never selecting the long motor imagery). Another crucial point is the selection of the threshold for the LDA classifier to basically select the motor imagery pattern. Is it too low, the pattern will be selected easily – FNs occur during rest (non-control). Is the threshold to high, users will have problems in selecting a command, particularly the long motor imagery.
One out of four subjects reached very good accuracies in the evaluation procedure (comparing FN/TP), two were medium, and one was weak. The numbers of TPs give a kind of overview how many movements they had triggered. The number of FPs in the non-control task is comparable for all subjects except subject al10. He had weak results during the control phases but very low number of FP in the non-control parts of the evaluation procedure. This is an indicator that the threshold for the LDA was very high, and after more detailed analysis we found that he had only problems in reaching the long time threshold for elbow movement.
From these results it can be concluded that participants need individual training to receive a good performance, but also adjustments have to be made for the thresholds, especially for subject al10. In the future, an automatic adjustment method has to be implemented.
The whole procedure which was initially planned was applied with success and will be taken into account for our future work. Reducing an initially high number or EEG channels to one Laplacian channel led to good performances in a self-paced two-class paradigm. The new idea of using only one type of motor imagery for two commands was finally demonstrated by controlling an artificial arm.
Gernot R. Müller-Putz and Reinhold Scherer designed, implemented and performed experiments, analyzed data and wrote the paper. Reinhold Scherer implemented data analysis tools. Christa Neuper wrote the paper. Gert Pfurtscheller supervised the project and wrote the paper.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
The authors thank Dr. Clemens Brunner for his helpful comments on the manuscript. This study was supported by Wings for Life-Spinal Cord Research Foundation (project 06/002).
The Supplementary Material for this article can be found online at http://www.frontiersin.org/neuroscience/neuroprosthetics/paper/10.3389/fnins.2010.00034/
Danóczy, M., Fazli, S., Grozea, C., Müller, K. R., Popescu, F. (2008) “Brain2Robot: a grasping roboit arm controlled by gaze, and asynchronous EEG BCI,” in Proceedings of the 4th International Brain-Computer Interface Workshop, and Training Course, Graz. ISBN: 978-3-85125-005-6, 355–360.
Gerardin, E., Sirigu, A., Lehéricy, S., Poline, J. B., Gaymard, B., Marsault, C., Agid, Y., and Bihan, D. L. (2000). Partially overlapping neural networks for real and imagined hand movements. Cereb. Cortex, 10, 1093–1104.
Heasman, J. M., Scott, T. R., Kirkup, L., Flynn, R. Y., Vare, V. A., and Gschwind, C. R. (2002). Control of a hand grasp neuroprosthesis using an electroencephalogram-triggered switch: demonstration of improvements in performance using wavepacket analysis. Med. Biol. Eng. Comput. 40, 588–593.
Hochberg, L. R., Serruya, M. D., Friehs, G. M., Mukand, J. A., Saleh, M., Caplan, A. H., Branner, A., Chen, D., Penn, R. D., and Donoghue, J. P. (2006). Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442, 164–171.
Kirsch, R. (2005). “Development of a neuroprosthesis for restoring arm and hand function via functional electrical stimulation following high cervical spinal cord injury,” in Proceeding of the 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference, Shanghai. 4142–4144.
Krausz, G., Scherer, R., Korisek, G., and Pfurtscheller, G. (2003). Critical decision-speed and information transfer in the “Graz Brain-Computer Interface”. Appl. Psychophysiol. Biofeedback 28, 233–240.
Kübler, A., Furdea, A., Halder, S., Hammer, E. M., Nijboer, F., and Kotchoubey, B. (2009). A brain-computer interface controlled auditory event-related potential (p300) spelling system for locked-in patients. Ann. NY Acad. Sci. 1157, 90–100.
Márquez-Chin, C., Popovic, M. R., Cameron, T., Lozano, A. M., and Chen, R. (2009). Control of a neuroprosthesis for grasping using off-line classification of electrocorticographic signals: case study. Spinal Cord 47, 802–808.
Müller-Putz, G., Zimmermann, D., Graimann, B., Nestinger, K., Korisek, G., and Pfurtscheller, G. (2007). Event-related beta EEG-changes during passive and attempted foot movements in paraplegic patients. Brain Res. 1137, 84–91.
Müller-Putz, G. R., Scherer, R., Pfurtscheller, G., and Rupp, R. (2006). Brain-computer interfaces for control of neuroprostheses: from synchronous to asynchronous mode of operation. Biomed. Tech. 51, 57–63.
Neuper, C., Müller, G. R., Kübler, A., Birbaumer, N., and Pfurtscheller, G. (2003). Clinical application of an EEG-based brain-computer interface: a case study in a patient with severe motor impairment. Clin. Neurophys. 114, 399–409.
Neuper, C., Scherer, R., Reiner, M., and Pfurtscheller, G. (2005). Imagery of motor actions: differential effects of kinaesthetic versus visual-motor mode of imagery on single-trial EEG. Brain Res. Cogn. Brain Res. 25, 668–677.
Nijboer, F., Sellers, E. W., Mellinger, J., Jordan, M. A., Matuz, T., Furdea, A., Halder, S., Mochty, U., Krusienski, D. J., Vaughan, T. M., Wolpaw, J. R., Birbaumer, N., and Kübler, A. (2008). A P300-based brain-computer interface for people with amyotrophic lateral sclerosis. Clin. Neurophysiol. 119, 1909–1916.
Pfurtscheller, G., Müller, G. R., Pfurtscheller, J., Gerner, H. J., and Rupp, R. (2003). ‘Thought’ – control of functional electrical stimulation to restore hand grasp in a patient with tetraplegia. Neurosci. Lett. 351, 33–36.
Pfurtscheller, G., Neuper, C., Müller, G. R., Obermaier, B., Krausz, G., Schlögl, A., Scherer, R., Graimann, B., Keinrath, C., Skliris, D., Wortz, M., Supp, G., and Schrank, C. (2003). Graz-BCI: state of the art and clinical applications. IEEE Trans. Neural Syst. Rehabil. Eng. 11, 177–180.
Pohlmeyer, E. A., Oby, E. R., Perreault, E. J., Solla, S. A., Kilgore, K. L., Kirsch, R. F., and Miller, L. E. (2009). Toward the restoration of hand use to a paralyzed monkey: brain-controlled functional electrical stimulation of forearm muscles. PloS One 4, e5924, 1–8. doi:10.1371/journal.pone.0005924.
Waldert, S., Pistohl, T., Braun, C., Ball, T., Aertsen, A., and Mehring, C. (2009). A review on directional information in neural signals for brain-machine interfaces. J. Physiol. (Paris). 103, 244–254.
Williams, M. R., and Kirsch, R. F. (2008). Evaluation of head orientation and neck muscle EMG signals as command inputs to a human-computer interface for individuals with high tetraplegia. IEEE Trans. Neural Syst. Rehabil. Eng. 16, 485–496.
Keywords: brain–computer interface, electroencephalogram, motor imagery, neuroprosthesis, spinal cord injury
Citation: Müller-Putz GR, Scherer R, Pfurtscheller G and Neuper, C (2010) Temporal coding of brain patterns for direct limp control in humans. Front. Neurosci. 4:34. doi: 10.3389/fnins.2010.00034
This article was submitted to Frontiers in Neuroprosthetics, a specialty of Frontiers in Neuroscience.
Received: 10 November 2009;
Paper pending published: 25 January 2010;
Accepted: 20 May 2010; Published online: 18 June 2010
Edited by:José del R. Millán, Ecole Polytechnique Fédérale de Lausanne, Switzerland
Reviewed by:Silvestro Micera, Scuola Superiore Sant'Anna, Italy
Kim Dremstrup, Aalborg University, Denmark
Copyright: © 2010 Müller-Putz, Scherer, Pfurtscheller and Neuper. This is an open-access article subject to an exclusive license agreement between the authors and the Frontiers Research Foundation, which permits unrestricted use, distribution, and reproduction in any medium, provided the original authors and source are credited.
*Correspondence: Gernot R. Müller-Putz, Laboratory of Brain–Computer Interfaces, Institute for Knowledge Discovery, Graz University of Technology, Krenngasse 37, Graz 8010, Austria. e-mail: email@example.com