Towards Intelligent Environments: An Augmented Reality–Brain–Machine Interface Operated with a See-Through Head-Mount Display

The brain–machine interface (BMI) or brain–computer interface is a new interface technology that uses neurophysiological signals from the brain to control external machines or computers. This technology is expected to support daily activities, especially for persons with disabilities. To expand the range of activities enabled by this type of interface, here, we added augmented reality (AR) to a P300-based BMI. In this new system, we used a see-through head-mount display (HMD) to create control panels with flicker visual stimuli to support the user in areas close to controllable devices. When the attached camera detects an AR marker, the position and orientation of the marker are calculated, and the control panel for the pre-assigned appliance is created by the AR system and superimposed on the HMD. The participants were required to control system-compatible devices, and they successfully operated them without significant training. Online performance with the HMD was not different from that using an LCD monitor. Posterior and lateral (right or left) channel selections contributed to operation of the AR–BMI with both the HMD and LCD monitor. Our results indicate that AR–BMI systems operated with a see-through HMD may be useful in building advanced intelligent environments.

introduction The brain-machine interface (BMI) or brain-computer interface (BCI) is a new interface technology that uses neurophysiological signals from the brain to control external computers or machines (Birbaumer et al., 1999;Wolpaw and Mcfarland, 2004;Birbaumer and Cohen, 2007). Electroencephalography (EEG), in which neurophysiological signals are recorded using electrodes placed on the scalp, represents the primary non-invasive methodology for studying BMI. Our group applied EEG and developed a BMI-based system for environmental control and communication. In this system, we modified a P300 speller (Farwell and Donchin, 1988). The P300 speller uses the P300 paradigm and involves the presentation of a selection of icons arranged in a matrix. According to this protocol, the participant focuses on one icon in the matrix as the target, and each row/column or a single icon of the matrix is then intensified in a random sequence. The target stimuli are presented as rare stimuli (i.e., the oddball paradigm). We elicited P300 responses to the target stimuli and then extracted and classified these responses with respect to the target. In our former study, we prepared a green/ blue flicker matrix because this color combination is considered safest (Parra et al., 2007). We showed that the green/blue flicker matrix was associated with a better subjective feeling of comfort than was the white/gray flicker matrix, and we also found that the green/blue flicker matrix was associated with better performance (Takano et al., 2009a,b). The BMI system was satisfactorily used by individuals with cervical spinal cord injury (Komatsu et al., 2008;Ikegami et al., 2011). Towards intelligent environments: an augmented reality-brain-machine interface operated with a see-through head-mount display The brain-machine interface (BMI) or brain-computer interface is a new interface technology that uses neurophysiological signals from the brain to control external machines or computers. This technology is expected to support daily activities, especially for persons with disabilities. To expand the range of activities enabled by this type of interface, here, we added augmented reality (AR) to a P300-based BMI. In this new system, we used a see-through head-mount display (HMD) to create control panels with flicker visual stimuli to support the user in areas close to controllable devices. When the attached camera detects an AR marker, the position and orientation of the marker are calculated, and the control panel for the pre-assigned appliance is created by the AR system and superimposed on the HMD. The participants were required to control system-compatible devices, and they successfully operated them without significant Review Board at the National Rehabilitation Center for Persons with Disabilities. All subjects provided written informed consent in accordance with institutional guidelines.

experiMental design
Augmented reality techniques were combined with a BMI (Figure 1). The AR-BMI system consisted of an HMD (LE750A, Liteye Systems, Inc., Centennial, CO, USA) or LCD monitor (E207WFPc, Dell Inc., Round Rock, TX, USA), a PC, USB camera (QCAM-200V, Logicool, Tokyo, Japan), EEG amplifier (g.USBamp, Guger Technologies OEG, Graz, Austria), and EEG cap (g.EEGcap, Guger Technologies OEG). We used the ARToolKit C-language library for the system (Kato and Billinghurst, 1999). When the camera detects an AR marker, the pre-assigned infrared appliance becomes controllable. The AR marker's position and posture were calculated from the images detected by the camera, and a control panel for the appliances was created by the AR system and superimposed within sight of the subject. We prepared a TV and a desk light as controllable devices. AR markers for the control panels for the TV and desk light were prepared (Figure 2).
We prepared green/blue flicker matrices (Takano et al., 2009b) as control panels. The duration of intensification/rest was 100/50 ms. All icons flickered in random order, creating a sequence. One classification was carried out per 15 sequences (Figure 2). Subjects were required to send five commands to control both the TV and desk light. We asked the subjects to focus on one of the icons.

eeg recording and analysis
Eight-channel (Fz, Cz, Pz, P3, P4, Oz, Po7, and Po8) EEG data were recorded using a cap. All channels were referenced to the Fpz and grounded to the AFz. Electrode impedance was under 20 kΩ. The EEG data were amplified/digitized at a rate of 128 Hz using a gUSBamp. The gUSBamp internal digitization rate was higher than 128 Hz, so the data were down-sampled. The digitized data were filtered with an eighth-order high-pass filter at 0.1 Hz and a fourth-order 48-52 Hz notch filter.
In the analyses, recorded EEG data were filtered with a firstorder band-pass filter (1.27-2.86 Hz); 120 digitization points of ERP data were recorded according to the timing of the intensification. Data from the first 20 points (before intensification) FIgure 1 | Diagram of the Ar-BMI system. When the USB camera detects an AR marker, the control panels for a pre-assigned appliance (e.g., desk light) are added to the user's sight and the device becomes controllable. The subjects are able to operate the appliance by focusing on an icon on the augmented control panel.
FIgure 2 | experimental procedure. The icons change color from blue to green at the time of intensification, and the elicited ERPs are recorded. The red solid line in the EEG data indicates the segmented portion used for classification. The target was classified by Fisher's linear discriminant analysis after 15 sequences. The light control panel had four icons (turn on, turn off, light up, and dim), whereas the TV control panel had 11 icons (turn on, change channel, change video mode, volume up, and volume down).
The mean accuracy rate for light control was 84% (SD = 3.40) with the LCD monitor, compared to 76% (SD = 2.06) with the HMD; however, the results were not significantly different. The results were also not significantly different in an offline evaluation. Thus, our AR-BMI system could be operated not only by using a PC display, but also by using an HMD.

channel selection
We further investigated the effects of channel selection on the operation of the AR-BMI using an HMD and LCD monitor. We divided the EEG channels into different sets and evaluated their accuracy.
When we analyzed the data in two horizontal channel sets [A (P3, Pz, and P4) and B (Po7, Oz, and Po8; Figure 4A)], set B (posterior set) showed significantly higher accuracy than set A (anterior set) in all sessions and under all conditions (p < 0.05, two-way repeated ANOVA, no interaction).
When we analyzed the data in three vertical channel sets [C (P3 and Po7), D (Pz and Oz), and E (P4 and Po8; Figure 4B)], set D (middle set) showed significantly lower accuracy than the others (left and right sets) in all sessions and under all conditions (p < 0.05, two-way repeated ANOVA, no interaction, and Tukey-Kramer as a post hoc test).
These results show that the posterior and lateral (right or left) channel sets provided better performance in the operation of the AR-BMI with both the HMD and LCD monitor.
were used for baseline correction. The remaining 100 points (after intensification) were down-sampled to 25.6 Hz and used for classification.
In training sets, we recorded EEG data to create a feature vector beforehand. Subjects were required to focus on one of the target icons, and four target icons were used. Sixty (4 trials × 15 intensifications) sets of digitization points were recorded as the target data set, and 600 (4 trials × 15 intensifications × 10 non-target icons) sets of digitization points were recorded as the non-target data set. Each data set included 100 digitization points per each EEG channel, and these data sets were down-sampled to 20 digitization points per each EEG channel. In total, 160 dimension-feature vectors (20 dimensions per EEG channel) were calculated using the segmented data for each subject. Feature vectors were derived for each experimental condition (LCD and HMD).
In testing sets, using the feature vectors, target and non-target icons were discriminated using Fisher's linear discriminant analysis. The result of the classification, as the maximum of the summed scores, was used to determine the icon to which the subjects were attending.

results online perforMance and offline evaluation
In the current study, we prepared an AR-BMI to control systemcompatible devices. We used both a see-through HMD and an LCD monitor to further evaluate the effect of different types of visual stimuli on the AR-BMI.
Online performance was evaluated and the mean accuracy rate for the TV control panel was 88% (SD = 3.20) with the LCD monitor, compared to 82.7% (SD = 2.63) with the HMD; however, these AR-BMI with a head-mount display contributed favorably to the operation of the AR-BMI with both the HMD and LCD monitor. Important roles for postero-lateral channels in driving a P300-based BMI have been reported Rakotomamonjy and Guigue, 2008). Rakotomamonjy and Guigue (2008) scored the effectiveness of channels in a P300-based BMI using a support vector machine and found an advantage with Po7 and Po8. Krusienski et al. (2008) showed that the occipito-parietal (Po7, Oz, and Po8) and midline (Fz, Cz, and Pz) electrodes provided better accuracy.
The neuronal mechanisms of the P300 have been investigated, and it has been noted that that the P300 reflects stimulus-driven and top-down attentional processes with other cognitive process, including categorization (Bledowski et al., 2004;Polich, 2007). Our tasks used green/blue color stimuli so that the processing of chromatic information, which occurs primarily in the V4 area, was also required (Lueck et al., 1989;Plendl et al., 1993;Murphey et al., 2008). Additional studies are necessary to fully understand the neuronal processes underlying the P300 paradigm with green/blue color flickering stimuli; however, this study suggests the importance of posterior and lateral (right or left) channel sets in the operation of an AR-BMI with both an HMD and LCD display.

toward advanced intelligent environMents
Several combinations between BMI and other technologies have been attempted, such as BMI with eye tracking (Popescu et al., 2006), and BMI with robotics (Valbuena et al., 2007). AR was combined with SSVEP BMI to provide a rich virtual environment (Faller et al., 2010), and we used AR with an LCD monitor and an agent robot in P300 BMI so that the users could operate home electronics in the robot's environment (Kansaku et al., 2010). In this study, we developed an AR-BMI system operated with a see-through HMD, which may be useful in building advanced intelligent environments (Kansaku, in press).

discussion
In this study, we found that by applying an AR-BMI system operated with a see-through HMD, which can provide suitable control panels to users when they come into an area close to a controllable device, participants successfully operated system-compatible devices without significant training.

hMd vs. lcd Monitor
When visual-evoked potentials are applied to a BMI system, the effects of visual stimuli can be better evaluated. Townsend et al. (2010) reported that a checkerboard paradigm for visual stimuli increased accuracy. Our group found that green/blue flicker stimuli improved performance during operation of a P300-based BMI (Takano et al., 2009b). A BMI study that used an immersive HMD and LCD monitor to provide visual stimuli showed no significant difference between the two technologies (Bayliss, 2003).
In this study, we applied both a see-through HMD and an LCD monitor to an AR-BMI system to further evaluate the effect of different types of visual stimuli, and in the online evaluation, the performance with the HMD was not different from that with the LCD monitor. The percent accuracy in this study ranged from 76 to 88%; because the incidence of correct responses exceeded 70%, the system is considered to have reached the level of actual usage Nijboer et al., 2008).
In offline analyses, the see-through HMD provided significantly lower accuracy for TV control than the LCD monitor. Because icon size and the distance between icons can affect the accuracy of classification (Sellers et al., 2006), this may have been caused by the different types of visual stimuli between the HMD and LCD monitor. Thus, the effects of visual stimuli on BMI operation should be investigated further.

channel set
We also investigated the effects of channel selection on operation of the AR-BMI using an HMD and LCD monitor, and found that posterior and lateral (right or left) channel selections along these lines, using either non-invasive neurophysiological signals or neuronal firing data, may enable new daily activities not only for persons with physical disabilities, but also for ablebodied persons.

acknowledgMents
This study was supported in part by a grant from the Ministry of Health, Labor, and Welfare to K. Kansaku. K. Takano and N. Hata were supported by NRCD fellowships. We thank T. Komatsu. S. Ikegami, and M. Wada for their help, and Y. Nakajima and M. Suwa for their continuous encouragement.
The systems developed by our group use a modified P300 speller (Farwell and Donchin, 1988). Although the P300 speller has primarily been used for communication using spelling alphabets, the system has recently been used to control more complex systemcompatible devices, including robots (Bell et al., 2008;Komatsu et al., 2008). Thus, each icon expresses the user's thoughts by assigning more complex meanings.
Our AR-BMI with a see-through HMD can be used to control more types of devices; thus, the system may be helpful in expanding the range of activities for persons with disabilities. The future extension of the environment for human activities