In a typical letter by letter typing BCI application, the user has to select among a discrete set of task symbols from a Dictionary D={A,B,…Z}∪{<,-} where “−” represents space symbol and “<” represents backspace symbol. Here, we examine how a BCI can infer a task symbol from different EEG evidence and prior context information. In particular, we build a decision framework that takes into account two types of EEG evidence: FRP and ERP evidence. We propose several methods for combining FRP, ERP evidence and prior context information, using real-time posterior probability updates. This BCI application utilizes a visual presentation module to detect the user intent and the EEG collected during the visual stimulation is then employed in decision making procedure.

Different visual presentation methods can be considered in order to evoke visual potentials. Rapid serial visual presentation (RSVP) paradigm is a minimally gaze dependent alternative for matrix presentation paradigms, that is aimed to induce ERPs for intent detection. In the RSVP paradigm, the symbols are rapidly presented as a time series on a prefixed location on the screen in a pseudo-random order, to evoke the response when the target symbol appears (Acqualagna et al., 2010; Orhan et al., 2012; Moghadamfalahi et al., 2015). In this presentation scheme, each flashing letter is a trial and in each “sequence,” a subset of dictionary is presented. From now on, we will be referring to only inducing ERP (target) evidence when we mention RSVP trial.

Figures 1A,B illustrate a flash of a prospect symbol and RSVP trial respectively. Due to low signal-to-noise-ratio (SNR) of EEG, the system usually requires to query the user with more than one “sequence” and “prospect symbol” to achieve a desired confidence level before making a decision. The set of “sequence” and “prospect symbol” which leads to a decision is called an “epoch.” In every epoch, it is assumed that the target symbol remains unchanged. Figure 1C represents a schematic of an EEG epoch in the RSVP Keyboard™ including a series of letters in an ERP sequence and a feedback stimulus as a “prospect symbol.” The feedback stimulus is always presented at the end of the RSVP sequence (shown in green). In Figures 1A,B, “Press Space Bar or Enter to pause” indicates the Pause/Play button. “Esc to quit” indicates the exit button should the participant choose to end the experimental session. Both options are added to the experimental design for the convenience of the user.

The proposed probabilistic graphical model (PGM) that represents kth “epoch” for an EEG-based typing application is presented in Figure 2.

Here, ak* is a random variable which represents the user intent in epoch k, Ac(t)={ajt|j=1,…|Ac(t)|} is a subset from the dictionary D, treated as the “sequence” at instant t of the epoch k, c denotes for candidate, |A_c(t)| is the number of symbols presented in the t-th sequence, Ck represents the context information that has been provided with the language model for which we will provide a brief description, in section 2.5. Moreover, here we introduce Ap(t)∈D. This set is a singleton Ap(t)={āt} which includes the prospective symbol for the query set A_p(t) at instant t (p denotes for prospect). In addition, ec(ajt) and ep(āt) are the ERP and FRP evidence obtained in response to an RSVP trial ajt and feedback trial ā^t respectively. We assume that the user intent is not changing within an epoch. Hence, given that every ajt∈Ac(t) and āt∈Ap(t) are either target or non-target, the intent inference can be formulated as a binary decision problem. Therefore y(·), z(·) correspond to binary class labels for ERP, FRP responses. Hence, y(ajt):=δ(ajt;a*) has a one-to-one relationship with the true state ak* such that y(ajt)=1 if ak*=ajt and 0 otherwise. Similarly, z(ā^t): = δ(ā^t; a^*). N_c and N_p are the maximum number of “sequences” and “prospect symbols” that can be used in an epoch if a desired confidence level is not reached in reasonable duration. In the case that we do not use FRP evidence, the right box from the graphical model from Figure 2 will be eliminated and the rest will remain the same. We utilize the graphical model presented to compute the posterior distribution of the intended character ak* after collecting the EEG evidence and by utilizing the language model evidence. The details of the posterior distribution computation is given in section 2.4. In order to make inference on the user intent, we compare three different evidence acquisition paradigms (one for each strategy). These paradigms are discussed in section 2.3.

Here, we present three different evidence acquisition paradigms: (i) [P1], (ii) [P2], and (iii) [P3] as shown in Figure 3.

The decision making process utilizes a maximum a posteriori (MAP) inference mechanism for intent detection. The graphical model presented in Figure 2 is used to compute the posterior distribution of the intended symbol, after evaluating the ERP and FRP likelihoods in recorded EEGs during ERP and FRP sequences and using context priors. A general decision framework for the three evidence acquisition paradigms is presented in Figure 4. According to this framework, before making a final decision the ERP and FRP evidences corresponding to multiple sequences are aggregated and fused with the context prior. Different query selection methods [P_i] i = {1, 2, 3} are presented in Figure 4. (Please see section 2.3 for more details.)

We estimate the prospective symbol āt∈Ap(t) at instant t, as the mode of posterior distribution:

where âk* is the estimated user intent; Ec1:t={Ec(Ac(j))}j=1t is the ERP evaluations for all the sequences in epoch k up to t; Ec(Ac(t))={ec(ajt)}j=1|Ac(t)| is the set of observation for the query set A_c(t); Ep1:t-1={Ep(Ap(j))}j=1t-1 is the FRP EEG evidences for all the observed prospective sequences in epoch k at instant t; Ep(Ap(t))=ep(āt) is the set of observation vectors for the prospective set A_p(t). For [P2] and [P3] the FRP EEG evidence ep(āt) is obtained in response to ā^t.

To compute the posterior distribution in (1), we utilize the assumptions of the graphical model presented in Figure 2. According to this PGM, the ERP and FRP evidence and context information are independent when the intended symbol a_k is given. Then for epoch k and at time instant t, after observing the query sets A_c(t) and A_p(t), the maximum a posteriori can be computed using the objective function in (2).

We can further assume that conditioned on the unknown symbol a_k all EEG evidence from different trials are independent, and simplify the first two terms of Equation (2) as:

According to the inference equation defined in (2), we need to estimate (i) the context prior that we estimated using a language model P(ak*=a|C), (ii) class conditional distributions over the ERP evidence p(e_c|1) for target and p(e_c|0) for non-target classes, and (iii) class conditional distributions over the FRP EEG evidence p(e_p|0) and p(e_p|1). We have implemented the proposed ERP and FRP data acquisition paradigms using the RSVP Keyboard™ framework (Moghadamfalahi et al., 2015).

To compute P(ak*=a|C), we utilize an n-gram language model which provides a prior probability over every symbol in the dictionary. We have shown that context information when fused with EEG evidence improves the system performance effectively (Orhan et al., 2013; Moghadamfalahi et al., 2015). An n-gram LM is a Markov model of order n − 1. Let C={am*}m=n-1, …, 1, where am* is the m^th previously typed character. Then:

In our system, we use a 6-gram language model, which is trained on the NY Times portion of the English Gigaword corpus (Roark et al., 2010).

In RSVP Keyboard™, the EEG signal is acquired using a g.USBamp biosignal amplifier with active g.Butterfly electrodes at a sampling rate of 256 Hz, from 16 EEG sites (according to the International 10/20 configuration): Fp1, Fp2, F3, F4, Fz, Fc1, Fc2, Cz, P1, P2, C1, C2, Cp3, Cp4, P5, and P6. To improve the signal-to-noise ratio (SNR), and to eliminate drifts, signal is filtered by an FIR linear-phase bandpass filter with cutoff frequencies [1.5,42] Hz and a notch filter at 60 Hz. Typically, a wideband filter, such as a [0.05–30] Hz filter is recommended to avoid the potential distortion of ERP waveforms (Luck, 2014). In our work, temporal-windowed EEG signals are filtered by [1.5,42] Hz bandpass filter (FIR, linear phase, length 153, 0 DC-gain) to eliminate the low frequency deviations and high frequency noise. Lower high-cutoff frequencies may be used (Orhan et al., 2016).

In order to capture the ERP and FRP, while omitting the possible motor reposes (Moghadamfalahi et al., 2015), EEG from a time window of [0, 500) ms after each flash's onset is processed as the corresponding raw data for each trial. To further pre-process after filtering, the EEG data for each channel are first down-sampled by 2 and projected to a lower dimensional space using principal component analysis (PCA), and finally data from every channel is concatenated to form the feature vector yji for trial ith, of type j in response to a trial, as we defined in Equation (7). More specifically, ypi represents FRP evidence for the prospective symbol trial ith; and yci represents ERP evidence for the query trial ith. After pre-processing,

where vji[n] is the multivariate measurement collected from channel n. Note that here N_ch = 16 is the number of channels and N_t is the number of time samples for each channel after applying PCA.

We then perform a quadratic projection of these feature vectors on to a one dimensional space so that it maximizes the separation between two possible classes of non-target and target. This projection is obtained as the log-likelihood ratio of two multivariate normal density functions estimated using regularized discriminant analysis (RDA) over target and non-target classes. eji(aci) and eji(api), are the one dimensional ERP and FRP evidences, respectively. We estimate the class conditional distributions of p(e_c(a)|1), p(e_c(a)|0) over the ERP evidences; and p(e_p(a)|1), p(e_p(a)|0) over the FRP evidences, using kernel density estimation (KDE). We employ Gaussian kernel with a bandwidth computed using the Silverman's rule from the recorded labeled data (Silverman, 1986). Note that these distributions are computed after collecting data in a calibration session. Then, the estimated densities are used in test sessions.

Recall that the EEG (ERP and FRP) evidence and language model prior are fused using the assumptions of the graphical model presented in Figure 2 to obtain the posterior probability mass function (PMF). The posterior probabilities is then used in MAP inference framework to make a joint decision as described in section 2.

All users participate in three copy phrase tasks, each task being performed on a separate day. In each day, the user performs the task pursuing one of [P1], [P2], and [P3] paradigms. The order of the paradigms are randomly assigned to the users to avoid the learning impact on the typing performance.

A copy phrase task includes typing the following ten different phrases.

Each phrase includes a missing word and the users are asked to complete these words. Here, the target words are written in bold. The entire sentence is shown to the user before each phrase is being typed. We use different phrases with different difficulty levels in terms of prior probability provided by the language model. For instance, the words such as “THE” or “WILL” are very easy to type because their initial letters are very likely based on the LM prior. However, the words such as “PAID” or “BUYS” are very difficult to type. Figure 5 demonstrates an example of a user performing the copy phrase task.

Prior to each copy phrase task all participants perform two calibration tasks: calibration_ERP and calibration_FRP. Calibration_ERP is used to learn the statistics of the ERP classifier (target vs. non target), using the calibration mode of the system to record labeled EEG data. Typically, each calibration_ERP session consists of 100 sequences of symbols. Before each sequence, the user is asked to attend to a particular symbol. Then a sequence consisting of the target symbol and 9 other non-target symbols is presented to the user in a random order. Calibration_FRP is used to learn the statistics of the FRP classifier (correct vs. non correct) using the copy mode of the system to record labeled EEG data. To obtain compatible evidence, we simulated [P2] and [P3] paradigms to collect supervised FRP EEG data.

During calibration_FRP, we modify the LM probabilities, in order to record enough labeled data for correct and non correct classes. Users are asked to rest between calibrations and copy phrase tasks and continue once they felt ready.

The length of each trial is 500 ms for all paradigms, there are 10 trials in one ERP sequence and 1 trial (i.e., the prospect symbol followed by a question mark) in one FRP sequence. A_c(t) is selected based on the posterior probability (fusion of evidence + LM). In [P1], the trial symbol is shown for 150 ms followed by a 50 ms blank screen (i.e., the inter-trial interval). The interval between successive sequences is 500 ms. In [P2] and [P3], after ERP evidence is collected, the trial symbol is shown for 0.9 s followed by a 0.1s blank screen for the FRP evidence. Decision symbol is shown for 2 s. The maximum number of sequences allowed in an epoch is 100 for calibration tasks (Simply because during calibration, we have a single combined epoch and we do not make decisions). In copy phrase tasks, the maximum number of sequences allowed in an epoch is 8 (that is, a decision is made after max 8 sequences in an epoch). In paradigm [P3], the posterior probability for showing the prospect symbol is set as α_p = 0.66. Note that, we do not employ an α_p during paradigm [P2] because we already show the prospect symbol after every ERP sequence. For all three paradigms ([P1], [P2], and [P3]), the posterior probability threshold for decision is set as α_d = 0.9.

Using the supervised data collected during the calibration_FRP, we first analyze the average EEG recorded in response to correct and incorrect feedback for the two evidence acquisition, [P2] and [P3]. Figure 6 shows the average FRPs for the correct and incorrect feedback trials for 12 users for the two scenarios that use FRP; [P2] and [P3]. The results show the statistical presence of the ErrP response in both [P2] and [P3]. As we can see in Figures 6A,B, the waveform in response to the incorrect feedback is characterized by a positive component observed ([350 ms]) after the delivery of the incorrect feedback, representing a visually evoked potential (VEP). We do not observe this positive response after the correct feedback, as shown in Figures 6C,D. Upon the presentation of the incorrect feedback, a negative ([50–100] ms) component is also observed. In addition, Figures 6A–D show the scalp topography at different time windows {100, 320, 400} ms. Based on these results, we observe higher separability between the EEG time series recorded in response to non-correct and correct feedback around the time window of 320–350 ms. Finally, Figures 7A,B show the average FRPs for the correct and incorrect feedback for 12 users and 16 electrodes; for [P2] and [P3], respectively. From Figures 7A,B, we can observe that when [P3] paradigm is performed, the amplitude of this positive component ([350 ms]) for the incorrect stimuli is slightly lower compared to [P2] (it could be inferred from the amplitude difference between the green line (non-ErrP) vs. red line (ErrP) for [P2] and [P3]).)

We then compare classification accuracies across different acquisition paradigms by employing AUC values as the measure of EEG evidence classification accuracy. In particular, using the calibration data obtained in the Human-in-the-loop calibration experiment (calibration_ERP and calibration_FRP) as described in section 3.2, we compare the offline target vs. non-target stimuli and correct vs. incorrect feedback classification results for the three data acquisition paradigms, [P1], [P2], and [P3]. Figure 8A compares the areas under the receiver operating characteristics curves (AUCs) for the FRP evidences of each user in different acquisition paradigms, [P2] and [P3]. Similarly, Figure 8B shows the ERP classification AUCs for each user for different acquisition paradigms, [P1], [P2], and [P3]. AUC values are calculated based on the cross validation of the classifier's performance on the training (calibration) data sets. In 10 out of 12 users tested, the classification AUC for paradigm [P2] is larger than [P3], as observed in Figure 8A. This can be a result of the experiment [P2] being more controlled. In other words, since each RSVP sequence is appended with a prospect symbol in paradigm [P2], the user always knows when the feedback is going to be presented in [P2] as opposed to [P3]. Comparing the calibration results from Figures 8A,B we can see that for most users, the ERP calibration results have higher AUCs compared to the FRP classification. This difference in the classification AUCs can be due to the fact that the number of observations that we collect during ERP calibration is higher than the number of observations that we can collect during FRP calibration.

Using the data collected during three copy phrase tasks, we analyze the typing performance for the three evidence acquisition paradigms. As explained in section 3.2, each copy phrase task includes typing ten different phrases with different difficulty levels. Table 1 shows the typing accuracy performance of the three evidence acquisition paradigms for all users in terms of two measures: accuracy in typing a letter correctly (ATL), which is the total number of correctly typed letters divided by the total number of typed letters; and probability of the phrase completion (PPC) which is the total number of correctly typed phrases divided by the total number of phrases. We observe that both [P2] and [P3] paradigms improve the typing accuracy performance compared to [P1]. As shown in Table 1, none of the users are able to complete the 10 copy phrase tasks correctly using [P1]. A paired t-test is also performed on ATLs to compare the typing accuracies among different paradigms across 12 users. In most EEG-based BCI systems, signal recorded from multiple channels along the scalp is assumed to be a Gaussian process with an unknown covariance and mean (Gonzalez-Navarro et al., 2016b). Assuming the Gaussianity of the recorded signal, we believe that applying t-testing is plausible. The result is presented in Table 3. From Table 3, we observe very low p-values for [P2] vs. [P1], and [P3] vs. [P1]. However, no significant differences between [P2] and [P3] are observed.

Here, we use information transfer rate (ITR) (Obermaier et al., 2001) as another performance measure. ITR summarizes the accuracy and speed into a single metric and it is commonly used to measure BCI performance. Figure 9 illustrates the ITR (bits/sequence) values for all subjects; and Table 2 reports the mean of the ITR values among 12 subjects for the three strategies. From Figure 9 and Table 2 it can be observed that [P2] (in red) along with [P3] (in blue) yield considerable improvements in both speed and accuracy. Among all the results, [P1] displays the lowest performance. Using paired t-test, a hypothesis testing is also performed to compare the ITR values obtained from different paradigms across 12 participants. The results are presented in Table 3. Table 3 also represents [P2] as the slightly better paradigm, although the difference between [P2] and [P3] is not statistically significant.

Human-in-the-loop copy phrase experiment results in Figure 9 and Table 1 show that the proposed strategies [P2] and [P3] outperform the strategy [P1] in terms of accuracy (with [P2] leading the race); and result in significant improvements in both speed and accuracy when compared to [P1]. We believe that improving not only accuracy but also speed is highly desired for BCI systems that are designed for real-life applications.

Finally, using online copy phrase and calibration results, we report ITR as a function of AUC obtained from the FRP and the ERP classifiers for each paradigm in Figures 10A,B. There are two different AUC values for paradigms [P2] and [P3] since they both use ERP and FRP evidences, whereas there is only one AUC value for [P1] corresponding to the ERP evidence. A linear regression model is fit to the observed data. We also report the coefficient of determination R². From these figures it can be observed that in all three cases, there is a positive correlation between AUC and ITR. In the case of AUC_ERP, the correlation is higher, which indicates that AUC_ERP is an effective factor to improve ITR for the three paradigms. For FRP, the correlation is much lower, so we can conclude that AUC_FRP has a small effect on ITR for both paradigms [P2] and [P3].

Compared to the benchmark BCI spellers which rely on visually evoked potentials (VEPs) such as SSVEPs, our ERP/ErrP based BCI speller has a slight advantage in accuracy (Liu et al., 2020), (Wong et al., 2020). Compared to the vision-independent BCI paradigms which rely on ERP elicitation via auditory and tactile stimulation, our visually evoked ERP/LM/FRP fusion BCI-speller has a significant advantage in ITR, and the accuracies we obtain with paradigms [P2] and [P3] compete with state-of-the-art P300 BCIs in the literature (Eidel and Kübler, 2020), (Kawala-Sterniuk et al., 2021).