For the reconstruction of EEG signals, let z∈RN×T1×S denote the LSS-EEG signals from distribution P_L, and x∈RN×T2×S denote the HSS-EEG signals from the real distribution P_H. In the definition, N denotes the number of channels, and T₁ and T₂ denote the samples of one trial for LSS-EEG signals and HSS-EEG signals during recordings, respectively. S denotes the number of trials for the motor-based tasks. The reconstruction goal is to formulate a function f(z) that projects LSS-EEG signals z to HSS-EEG signals x:

In fact, the reconstruction function maps the LSS-EEG samples from P_L into a certain distribution P_C, and our goal is to adjust a certain distribution P_C to make it close to the real distribution P_H by varying the function f(z). The reconstruction has two procedures with GAN. In the generation procedure, the object is to adjust EEG samples from distribution P_L to distribution P_C. In the discriminator procedure, the object is to adjust EEG samples from distribution P_C to distribution P_H. The reconstruction procedure can ultimately be treated as a procedure to adjust EEG samples from one distribution to another.

Typically, since EEG signals are nonlinear and non-stationary, the noise model in such signals is complicated, and the reconstruction mapping relationship is non-uniformly distributed across the signals. Thus, there is no clear indication of how the distributions of LSS-EEG and HSS-EEG signals are related to each other. It is difficult to reconstruct LSS-EEG signals using conventional methods. However, the uncertainties in the noise model and the reconstruction mapping relationship can be ignored by using deep neural networks (DNNs), as the DNNs can efficiently learn high-level features from nonlinear and non-stationary signals and reconstruct a representation of the data distribution from modest-sized signal patches. Therefore, the GAN framework based on DNN is suitable for EEG signal reconstruction. In summary, a modified GAN framework with the Wasserstein distance and temporal-spatial-frequency (TSF) loss is introduced to reconstruct HSS-EEG signals from LSS-EEG signals.

The GAN framework consists of two opposing neural networks, a generator G, and a discriminator D, that are optimized to minimize a two-player min-max problem (Goodfellow et al., 2014). The discriminator is trained to distinguish the generated samples from the real samples, while the generator is trained to generate fake samples that are not determined as fake by the discriminator. For the reconstruction of EEG signals, we further defined the discriminator D_θD and the generator G_θG to solve the min-max problem:

where E(·) denotes the expectation operator. When the discriminator meets the real data, it will satisfy D_θD(x) = 1 to discriminate the real data. Here, D_θD(x) = 1 reaches the expectation for logD_θD(x). When the discriminator meets the generated data, it will satisfy D_θD(G_θG(z)) = 0 to discriminate the generated data. Here, D_θD(G_θG(z)) = 0 reaches the expectation for log(1−D_θD(G_θG(z))). Therefore, the minimax optimal function is designed by the expectation operator. The general reconstruction idea is to train a generator for the purpose of fooling a differentiable discriminator that is trained to distinguish reconstructed HSS-EEG signals from real HSS-EEG signals. In constructing EEG signals, GANs suffer from remarkable training difficulty due to the nonlinear and non-stationary characteristics of EEG signals. To overcome the training problem of the original GAN framework, instead of using Jensen–Shannon divergence, the WGAN framework uses the Wasserstein distance to compare sample distributions (Gulrajani et al., 2017). From the definition of WGAN, the min-max problem optimized by D_θD and G_θG can be written:

In the min–max problem, the Wasserstein distance is estimated by the first two terms. The last term is the gradient penalty for network regularization. In the penalty term, P_R denotes the distribution of uniform samples x~ along straight lines connecting pairs of generated and real samples. ∇x~(·) is the gradient calculator, and the parameter λ is a constant weighting parameter for the penalty term. In fact, the WGAN framework removes the log function and drops the last sigmoid layer to keep the gradient while training the min-max problem. The discriminator D_θD and the generator G_θG are trained alternatively by optimizing one and updating the other.

To allow the generator to transform the data distribution from a low sampling rate and sensitivity to a high sampling rate and sensitivity, another part of the loss function needs to be added to the GAN/WGAN architecture to retain the detail and information content of the EEG signals. A widely used loss function for signal details and information contents is the mean square error (MSE) loss function (Yang et al., 2018). Typically, as the common MSE is computed by minimizing the point-wise error in image processing, the temporal MSE is computed by minimizing the time sampling point-wise error between a LSS-EEG patch and a HSS-EEG patch by the time step:

where ||·||_F denotes the Frobenius norm, L_T−MSE denotes the temporal MSE for the generator G_θG, t is the time step of real EEG signals and generated EEG signals, and T is the number of time steps for each batch. In contrast to images, EEG signals are multi-channel time-series data, and the spatial and frequency features must be considered for reconstruction. Therefore, in addition to the temporal MSE L_T−MSE between time steps, the spatial MSE L_S−MSE between channels and the frequency MSE L_F−MSE between signal batches also need to be considered for encouraging the GAN/WGAN architecture to construct more accurate HSS-EEG signals. Recently, common spatial patterns (CSP) have been widely used to extract spatial features from EEG signals (Luo et al., 2018a), and power spectral density (PSD) features are widely used to extract frequency features from EEG signals (Petroff et al., 2016). The CSP algorithm is used to compute the optimal projection vectors to project the original EEG signal to a new space to obtain good spatial resolution and discrimination between different classes of EEG signals. The PSD algorithm is used to compute the power values on specific frequencies to compose a spectra. Using these two algorithms, the spatial MSE L_S−MSE and the frequency MSE L_F−MSE are defined for the generator:

where CSP(·) and PSD(·) are the CSP feature and PSD feature extractor, respectively. c is the channel of real EEG signals and the same of the generated EEG signals, C is the number of channels, n is the batch of real EEG signals and the same as that of the generated EEG signals, and N is the number of batches. For convenience, the TSF loss is computed by weighting three such MSE losses:

where λ_T, λ_S, λ_F are the weights of three such different MSE losses, respectively. Datasets with different sampling rates and sensitivities will obtain different weights, and, thus, the values of the weights will be determined by experiments.

In addition, to confirm that the EEG signals are temporally and spatially coherent, a regularization loss L_TV(G_θG) based on total variation is used in the generator:

where ∇_z(·) is the gradient calculator; the gradient regularization loss will encourage temporal and spatial coherence of the reconstruction. Combining Equations (3), (7), and (8), the overall joint reconstruction loss function is expressed as

where λ₁ and λ₂ are the weights for controlling the trade-off among the WGAN adversarial loss, the TSF-MSE loss and the TV loss.

The proposed WGAN-EEG reconstruction framework is illustrated in Figure 1. The WGAN-EEG framework consists of three parts to reconstruct HSS-EEG signals from LSS-EEG signals. For the first part of the deep generator G_θG, “B residual blocks” with an identical layout that was proposed by “Kaiming He” (He et al., 2016) are employed in the generator network. To facilitate the high sensitivity of EEG signals, 16 “B residual blocks” are applied to LSS-EEG signals to extract deep features for the generator. In each “B residual block,” following the common usage of the deep learning community, two convolutional layers with small 3*3 kernels, 1 stride, and 64 feature maps (k3n64s1) are followed by a batch-normalization layer (BN) and the ReLU activation function (Ioffe and Szegedy, 2015). To increase the sampling rate of the input EEG signals, the trained deconvolutional layer (stride = 0.5) is followed by “B residual blocks” to increase the sampling rate. In real-world application, the WGAN-EEG architecture is trained well to fit HSS-EEG signals before usage. In the usage scenario, the recorded LSS-EEG signals are incorporated into the well-trained architecture to reconstruct HSS-EEG signals to improve the sensitivity.

The second part of the WGAN-EEG framework is the TSF-MSE loss calculator, which is realized in Figure 1. The reconstructed output HSS-EEG signals G_θG(z) from the generator G_θG and the ground truth HSS-EEG signals x are fed into the calculator to extract the CSP features and the PSD features. Then, using the extracted features, the TSF-MSE loss is computed by Equations (4), (5), (6). The reconstruction error computed by the loss function is then back-propagated to update the generator network's weights.

The third part of the WGAN-EEG used to discriminate real HSS-EEG signals from generated HSS-EEG samples, the discriminator network D_θD, is shown in Figure 2. Here, we followed the architectural guidelines for the discriminator to use the LeakyReLU activation function and avoid max-pooling along the network (Zhang et al., 2017). The discriminator network contains eight convolutional layers with an increasing number of filter kernels by a factor of 2. In fact, the convolutional kernels are increased from 64 to 512 kernels, and the stride is alternatively increased from 1 to 2 to reduce the EEG signal sampling rate when the number of features is doubled. In the discriminator, each convolutional layer is followed by a LeakyReLU activation function and a batch-normalization layer. After eight convolutional layers, there are two FCN layers, of which the first layer has 1,024 outputs with the LeakyReLU activation function, and the second layer has a single output. Following the instructions of the WGAN (Gulrajani et al., 2017), the discriminator of the WGAN-EEG has no sigmoid cross entropy layer.

The WGAN-EEG framework is trained by using EEG signal batches and applied on the entity of each signal trial. The details of training the WGAN have been described in the experiments.

To explore the feasibility and performance of the proposed algorithm, three EEG signal datasets with different sampling rates and sensitivities are applied to train and evaluate the proposed networks. Table 1 illustrates the details of these three different EEG datasets.

In the training procedure, we trained six models using the GAN/WGAN framework within three different datasets. All down-sampled training EEG samples were fed into the generator, and the real training EEG samples were fed into the discriminator. The generated EEG samples and the real EEG samples were discriminated by the TSF-MSE loss function to update the generator and the discriminator for solving the min-max problem. Because the AO dataset and the GAL dataset have high sampling rates and a high number of channels, models for these two datasets were trained over 30 epochs. However, the MI dataset has a lower sampling rate and fewer channels, and, therefore, this dataset was trained over 20 epochs. Each epoch traverses all the data in the corresponding dataset. According to the different devices used to record EEG signals, the generators of the GAN/WGAN frameworks were specified by different scopes of generation for different datasets. We specified the generation scopes of [−40, 40 μV], [−50, 50 μV], and [−100, 100 μV] for the AO dataset, the GAL dataset, and the MI dataset, respectively.

In our experiments, we randomly extracted pairs of signal patches from down-sampled EEG signals and real EEG signals as our training inputs and labels. The patch size is N*τ, where N is the channel number for different datasets, and τ is the EEG samples from the temporal domain. Since the limited trials of EEG signals (<500 trials for one subject) and smaller values of τ will construct more accurate sequential relationships for the EEG signals, following our previous research (Luo et al., 2018a), we cropped a minimal length for the training of the deep neural network. According to the pre-experiment, we set τ = 12 to satisfy the minimal length for the convolution in the GAN/WGAN architecture. In the optimization of the generator and the discriminator, according to current research (Basu et al., 2018), the GAN models were optimized by the Adam algorithm (Basu et al., 2018), and the WGAN models were optimized by the RMSprop algorithm (Mukkamala and Hein, 2017). The optimization procedure for the GAN/WGAN architectures is shown in Figure 3. The mini-batch size was set to 32. Following the instructions of the GAN/WGAN frameworks (Goodfellow et al., 2014; Gulrajani et al., 2017), the Adam optimizer's hyperparameters were set as α=10-5,β1=0.5,β2=0.9, and the RMSprop optimizer's hyperparameters were set as α = 10⁻⁵, β = 0.9. The hyperparameter for the gradient penalty of WGAN framework was set as λ = 10 according to the suggestion in the reference (Gulrajani et al., 2017). The hyperparameters for the SRGAN/SRGAN frameworks in Equation (9) were set as λ1=10-3 and λ2=2*10-8 by the suggestions of reference (Ledig et al., 2017). The hyperparameters in the TSF-MSE loss function of Equation (7) and the joint reconstruction were set of different values according to the experimental experience of each reconstruction round, and the average values with standard deviations of all parameters in three datasets are given in Table 2. The optimization processes for the GAN framework and the WGAN framework are similar; however, some places are changed to the corresponding optimizer and the loss functions (see Figure 3).

The GAN/WGAN frameworks were implemented in Python 2.7 with the Tensorflow 1.8 library. Two NVIDIA 1080Ti GPUs were used in this study.

To visualize the convergence of the GAN/WGAN frameworks, the conventional temporal MSE, frequency MSE, spatial MSE, the proposed TSF-MSE losses, and the Wasserstein distance for validation of three different datasets were computed according to Equations (2), (3), (4), and (5). Figure 4 shows the averaged temporal MSE, frequency MSE, spatial MSE, and TSF-MSE losses vs. the number of epochs for different datasets within the GAN/WGAN frameworks.

From Figures 4A–D, for a given framework and dataset, we have compared the variations and differences between the conventional temporal MSE, frequency MSE, spatial MSE, and our proposed TSF-MSE. In the four figures, all of the iterative curves are shown to have decreased rapidly within the first 10 epochs (each epoch contains 10 error recordings), and the initial decreases indicated that these two metrics are positively correlated for the EEG signal reconstruction. However, for each dataset or when using GAN/WGAN frameworks, the loss results of TSF-MSE were lower than the loss results of conventional temporal MSE, frequency MSE, and spatial MSE. In addition, of these four losses, the WGAN frameworks oscillated in the convergence process, while the GAN frameworks smoothed in the convergence process. Comparing the oscillation of losses, the TSF loss exhibited varied smoothing for the WGAN framework compared to the GAN framework for each dataset. These observations of network convergence suggested that the conventional MSE losses and our proposed TSF-MSE loss have different focuses within the GAN/WGAN frameworks. By applying the generators, the difference between conventional MSE losses and our proposed TSF-MSE loss will be further revealed in the reconstructed EEG signals.

Figure 5 illustrates the Wasserstein distance estimation vs. the number of epochs for three different datasets. The plotted Wasserstein values were estimated by the definition of −E_x~PH[D_θD(x)]+E_z~PL[D_θD(G_θG(z))] in Equation (3). From the figure, we have found a reduction in the Wasserstein distances as the number of epochs increased, but different datasets have different decay rates of the reducing Wasserstein distance. For the curves of the three datasets, we noted that the Wasserstein distance we computed is a surrogate that has not been normalized by the total number of EEG signal samplings, and, therefore, the curves would have decreased to close to zero after 100 epochs by using the normalization for the EEG signals.

To show the reconstruction effects of the GAN/WGAN frameworks with our proposed TSF-MSE loss function, we considered two different aspects of the reconstruction results. The first one was the sampling rate reconstruction by the same sensitivity signals' GAN/WGAN frameworks, which is shown in Figure 6. The second one was the sensitivity rate reconstruction by the different sensitivity signals' GAN/WGAN frameworks, which is shown in Figure 7. Since the proposed reconstruction method used a novel TSF-MSE loss function for the training of GAN/WGAN architectures, the statistical temporal, frequency, and spatial results were also compared between the original signals and the reconstructed signals. Figures 8–10 illustrated the mean temporal error, mean spectra difference, and brain electrical activity mapping on 12 Hz of a single trial compared with the original EEG signals and all reconstructed EEG signals.

To plot the reconstruction results of different models and situations, we chose the same trial from each dataset for the comparison experiments. Because the number of channels differs for each dataset, we choose the “FPz” channel for the experiments to plot the figures. In addition, as one trial over a long period of time will hide some details of the reconstruction signals, we chose the 50 ms range of (500 and 550 ms) for the AO and MI datasets and the 50 ms range of (100 and 150 ms) for the GAL dataset to plot the details of the reconstruction results. From the reconstruction results by the same details shown in Figure 6, we have found that the signals' proximity between the reconstructed data and the original data decreased in the following order for the three datasets: AO > GAL > MI. The difference between the GAN framework and the WGAN framework cannot be realized at the signal level. In the figures shown in Figure 7, the high sensitivity EEG signals' GAN/WGAN frameworks reconstructed the low sensitivity EEG signals well, such as the AO and GAL data reconstructed by the MI GAN/WGAN frameworks. However, the low sensitivity EEG signals' GAN/WGAN models cannot reconstruct accurate high sensitivity EEG signals, such as MI data reconstructed by the AO and GAL GAN/WGAN frameworks.

For the statistical results in Figures 8–10, we have found that excepting for the temporal errors, reconstructed EEG signals show the same regulations on frequency and spatial features. For the reconstructions of the same sensitivity, the mean spectra results have shown that WGAN architectures outperform than GAN architectures, so do the brain electrical activity mapping (BEAM) results for reconstructions of the same sensitivity. As for the reconstructions of different sensitivity, we have found that higher sensitivity models bring lower spectra difference and more distinct ERS/ERD phenomenon on BEAMs, while lower sensitivity models bring higher spectra difference and less distinct ERS/ERD phenomenon on BEAMs.

In fact, the qualitative analysis could not yield promising insight regarding HSS-EEG signals reconstructed by LSS-EEG signals. Hence, a quantitative analysis was applied to explore the performance of reconstructed EEG signals. In this paper, because the AO dataset corresponded to action observation, the GAL dataset corresponded to action execution, and the MI dataset corresponded to motor imagery, these three datasets caused the same event-related desynchronization/event-related synchronization (ERD/ERS) phenomenon, which can be classified by filter bank common spatial patterns (FBCSP) and a support vector machine (SVM) (Luo et al., 2018a,b). The ERS/ERD phenomenon from EEG signals is common on three motor-related datasets, and such phenomena are usually used for the motor-based BCI. Therefore, the ERS/ERD phenomenon will be the key index with which to measure the performance of BCI system by EEG signals. This study thus selected the ERS/ERD phenomenon from EEG signals as a quantitative measure, and FBCSP features with an SVM classifier were applied to explore the performances of the original signals and the reconstructed signals. For comparison with different models and different sensitivities, there were several hyperparameters for the FBCSP features, SVM classifier, and deep learning classifier:

Classification results for the sampling rate reconstruction by the same sensitivity signals' GAN/WGAN frameworks are shown in Tables 4–6 for AO dataset, GAL dataset, and MI dataset, respectively. In addition, classification results for the sensitivity rate reconstruction by the different sensitivity signals' GAN/WGAN frameworks are shown in Tables 7–9 for AO datset, GAL dataset, and MI dataset, respectively. In all tables, the results are presented by classification accuracy forms, and a paired t-test statistical technique was used to detect whether the reconstructed EEG signals significantly outperform than the original EEG signals. P-value of the t-test statistics are provided in the tables, and ^*p <0.05 and ^**p <0.01 represent the results compared among two columns are significantly different and extremely significantly different.

Tables 4–6 illustrate the up-sampling classification results compared with the original data, the spline reconstructed data, the GAN reconstructed data, and the WGAN reconstructed data. Among the three datasets, we have found that the WGAN reconstructed data achieved the best classification performance. In the AO dataset, the WGAN reconstructed signals achieved the best classification accuracy (67.67%), which was higher than those of the original data (63.57%), the spline reconstructed data (60.91%), and the GAN reconstructed data (65.41%). In the GAL dataset, the WGAN reconstructed signals achieved the best classification accuracy (73.89%), which was higher than those of the original data (69.78%), the spline reconstructed data (68.25%), and the GAN reconstructed data (73.63%). In the MI dataset, the WGAN reconstructed signals achieved the best classification accuracy (64.01%), which was higher than those of the original data (61.98%), the spline reconstructed data (60.41%), and the GAN reconstructed data (63.61%).

From the t-test statistical results that computed compared signals, the reconstructed GAN/WGAN model signals exhibited significant improvement of classification, producing a better performance than the original signals, while spline reconstructed signals exhibited significant reduction of classification performance, lower that of the original signals. The significant improvement and reduction are presented for all AO/GAL/MI datasets (^*p <0.05). Specifically for the WGAN model in AO dataset and GAN/WGAN model in MI dataset, the classification performances presented were extremely significant (^**p <0.01). Therefore, we have concluded that the GAN/WGAN models with proposed TSF-MSE loss function showed a significant improvement for reconstructing EEG signals with the same sensitivity.

Tables 7–9 give the classification results compared with the GAN/WGAN models trained with different sensitivities. Table 7 gives the classification results of the AO data reconstructed by the GAL/MI trained GAN/WGAN models. Table 8 gives the classification results of the GAL data reconstructed by the AO/MI trained GAN/WGAN models. Table 9 gives the classification results of the MI data reconstructed by the AO/GAL trained GAN/WGAN models. For the AO dataset, signals reconstructed by the GAL-GAN model achieve the best average classification accuracy (64.55%), which was higher than those of the original data (63.51%) and the data reconstructed by the GAL-WGAN (64.40%), the MI-WGAN (62.07%), and the MI-GAN (62/08%). For the GAL dataset, signals reconstructed by the AO-GAN model achieve the best average classification accuracy (70.60%), which is higher than those of the original data (69.78%) and the data reconstructed by the AO-WGAN (70.34%), the MI-WGAN (69.57%), and the MI-GAN (70.21%). For the MI dataset, signals reconstructed by the AO-GAN model achieved the best average classification accuracy (64.93%), which was higher than those of the original data (61.98%) and the data reconstructed by the AO-WGAN (63.29%), the MI-WGAN (63.66%), and the MI-GAN (63.39%). The GAN model performed better than the WGAN model for reconstructing EEG signals by different sensitivities, and LSS-EEG signals reconstructed by HSS-EEG models will increase the sampling rate and sensitivity of signals, which will increase the classification performance.

From the t-test statistical results that computed between compared signals, the AO dataset reconstructed signals by GAL-WGAN and GAL-GAN, showing significant improvement of classification performance than the original signals (*p <0.05), while other datasets reconstructed signals showed no significant performance compared to the original signals(*p > 0.05). In addition, AO dataset reconstructed signals by MI-GAN a classification performance that was significantly worse than the original signals (*p <0.05). Therefore, we have concluded that the GAN/WGAN models with proposed TSF-MSE loss function showed significant performance improvement with enough data and no significant performance improvement without enough data for reconstructing EEG signals with the same sensitivity. Besides, if there is a large gap of sensitivity between two EEG signals datasets, the lower sensitivity based GAN model will cause significant worse performance of reconstructing high sensitivity signals to low sensitivity signals (such as MI-GAN applied to AO dataset).

Since this study has proposed a novel loss function to build the GAN/WGAN architectures for reconstructions, we have also compared the mean classification accuracy between temporal-MSE based GAN/WGAN architectures and TSF-MSE based GAN/WGAN architectures. Due to the single spatial-MSE and frequency-MSE cannot reconstruct signals, these two losses were not included in the comparison. Table 10 illustrates the comparison results for all reconstructions and datasets. We have also used a paired t-test statistical technique to detect whether the TSF-MSE based GAN/WGAN architectures significantly outperform than the temporal-MSE based GAN/WGAN architectures. In Table 10, AO−>AO means AO dataset reconstructed by the same sensitivity AO dataset, GAL−>AO/MI−>AO represents AO dataset reconstructed by the different sensitivity GAL/MI datasets, and so forth. Experimental results have shown that no matter GAN architecture or WGAN architecture, TSF-MSE loss function outperformed the conventional temporal-MSE loss function (*p <0.05). Therefore, the novel loss function proposed by us will significantly improve the performance of the reconstructed EEG signals.

Classification of reconstructed signals between the “FBCSP+SVM” classifier and “FBCSPNet” classifier Schirrmeister et al. (2017) are illustrated in Table 11. The results have shown average classification results of “FBCSP+SVM” and “FBCSPNet” for both GAN and WGAN models on three datasets. The improved ratios have shown that the GAN model and WGAN model bring 3.75 and 5.25% improvement on the average, respectively, to all three datasets for the “FBCSP+SVM” classifier. In addition, the GAN model and WGAN model bring 1.68 and 2.21% improvement on average, respectively, for all three datasets for “FBCSPNet” classifier. Therefore, we have concluded that EEG signals reconstructions by GAN/WGAN model are advantageous to the classification performance for different classifiers. If the classifier exhibits the a better performance, it has the ability to obtain more discriminant ERD patterns, so the improvement of the deep learning classifier is less than the conventional classifier.

In order to intuitively represent the differences between EEG signals reconstruction by the same sensitivities or different sensitivities of EEG signals, Figure 11 illustrates the average results of these comparisons. In Figure 11, the Figure 11A shows the average results of Tables 4–6 and Figure 11B shows the average results of Tables 7–9. From the average figures, the disciplines of EEG signals reconstruction by the GAN/WGAN models analyzed above can be found.

The purpose of this paper is to reconstruct HSS-EEG signals from LSS-EEG signals by using GAN/WGAN frameworks with a carefully designed loss function. In this paper, among the experiments of three different EEG datasets, we have compared the performance of GAN/WGAN frameworks for up-sampling with the same sensitivity and reconstruction with different sensitivities. The classification performances show significant improvement in terms of reconstructions of the same sensitivity. For AO dataset, the classification performances also show significant improvement by reconstructions of GAL-WGAN and GAL-GAN. However, other datasets reconstruction signals with different sensitivity have no significant improvement than original signals. There are two possible reasons for the statistical results. One possible reason is that the AO dataset has enough subsets (a total of 24) to compute the t-test index. However, datasets GAL and MI with 12 subsets and nine subsets, respectively, are not sufficient to compute the t-test index. Another possible reason may be due to the signal amplitude range for the GAN/WGAN reconstruction. In our experiments, the reconstructed signals amplitude range was set as the same as the original signals, and the amplitude range may have prevented the variations of reconstructed signals brought by the signals with different sensitivity. Therefore, in future works, more experiments for different ranges are also needed for a same dataset to confirm the relationship between signal amplitude range and patterns classification performance. For the average classification accuracy for all experiments, the up-sampled EEG signals performed better than the original data, and we think this might be due to the fact that the reconstruction procedure obtains more discriminant signals. In addition, the original temporal-MSE and the proposed TSF-MSE as loss functions were also compared.

The up-sampling reconstruction with the same sensitivity results have demonstrated that using the WGAN helps to improve signal qualities and statistical properties. Comparing the reconstruction HSS-EEG signals and the original real HSS-EEG signals in Figures 6, 8A, 9A, 10A, we can see that the WGAN framework helps to solve the problem of the over-smoothing effect suffered by the conventional temporal-MSE signal generators (Aydin et al., 2015). Although the reconstructed HSS-EEG signals shared a similar result, as in Figures 6, 8A, 9A, and 10A,B, the quantitative analysis of classifying signals by a machine learning model, as given in Tables 4–6, Figure 11A, have shown that the WGAN framework yields a higher classification accuracy and obtains more reliable statistical properties due to more discriminant patterns. However, if we use GAN/WGAN frameworks alone, the critical ERD/ERS of brain activity characteristics in the EEG signals will be reduced along with the single temporal loss. Theoretically, the GAN/WGAN frameworks are based on generative models, and such models generate naturally appearing HSS-EEG signals but cause severe distortion of the ERD/ERS characteristics in the EEG signals (Choi et al., 2017). Therefore, an additive loss function should be included to guarantee that the ERD/ERS characteristics remain the same for the reconstruction.

Beyond the above analysis, the TSF-MSE loss function was introduced to guarantee the ERD/ERS characteristics during the training of the GAN/WGAN frameworks, and the classification performance of ERD/ERS characteristics can be found in the compared results in Table 10. As is well known, the temporal-MSE loss was the basis of the time-series data, and such loss will guarantee the reconstructed shape of the temporal domain. However, EEG signals are multi-channel time-series data, and the spatial domain is thus also important in the reconstruction. In addition, most ERD/ERS characteristics are reflected in the frequency domain, making the frequency domain also important in the reconstruction. Therefore, the TSF-MSE constructed by the original signals from the temporal domain, the FB-CSP features from the spatial domain, and the PSD features from the frequency domain have been introduced in this paper to guarantee the EEG signals temporal characteristics, spatial characteristics, and ERD/ERS characteristics (Strohmeier et al., 2016). Additionally, the TSF-MSE-based GAN/WGAN models cause lower losses than the temporal MSE, frequency MSE, and spatial MSE-based GAN/WGAN models (see Figure 4). Our proposed TSF-MSE-based WGAN framework outperformed the other models in reconstructing up-sampled EEG signals with the same sensitivity. These results demonstrate that we can use this method to increase the sampling rate of EEG signals to achieve higher performance in brain-computer interfaces (BCIs) or EEG-based rehabilitation treatments.

In this paper, in addition to reconstructing HSS-EEG signals from the same sensitivity, we also reconstructed HSS-EEG signals from different sensitivities. In fact, if EEG signals with low sensitivity can be reconstructed into high-sensitivity signals, the reconstructed HSS-EEG signals will contain more details of the ERD/ERS characteristics, which will improve the classification performance for many applications. Among the experimental results shown in Tables 4–9, we can conclude that the average classification accuracies of WGAN framework are higher than GAN framework for reconstruction with the same sensitivity on all datasets, while the GAN framework obtained better average classification accuracies for reconstruction with different sensitivities on all datasets. In addition, a larger gap in the sensitivity will significantly increase the average classification accuracies of all datasets, while a smaller gap in the sensitivity will result in a smaller difference in the average classification accuracies of all datasets (see the comparison results in Tables 7–9, Figure 11B). We can also find indicators for different sensitivity gaps in Figure 7. For example, considering the AO data reconstructed by the MI-GAN and MI-WGAN models (see Figure 7B), a high-sensitivity signal reconstructed by the low-sensitivity GAN/WGAN models caused the signals to be overfitted and exceed the original data range. Hence, the reconstructed results contained fewer ERD/ERS characteristics to classify the EEG signals, and the classification accuracy was lower than the results using the original data. Conversely, for the MI data reconstructed by the AO-GAN and AO-WGAN models (see Figure 7F), we can see that the low-sensitivity MI data reconstructed by the high sensitivity models presented more variations in the temporal domain. Because the variations in the time-series represented detailed characteristics of ERD/ERS, the reconstructed high sensitivity EEG signals performed better in the classification of ERD/ERS characteristics. Therefore, in practical applications, we can train a high-sensitivity GAN model for EEG signal reconstruction. By applying the GAN/WGAN models, the ERD/ERS characteristics extracted from low sensitivity devices can be enhanced for use in real-time and real-application BCI or rehabilitation treatment.

In contrast to the results of reconstructing HSS-EEG signals with the same sensitivity, the GAN framework performed better than the WGAN framework for reconstructing HSS-EEG signals with different sensitivities. An approaching value range caused a smaller difference between the GAN framework and the WGAN framework (the AO dataset and the GAL dataset), but a separated value range caused a large difference between the GAN framework and the WGAN framework. Therefore, the difference in the classification performance was caused by the different value ranges of different sensitivities. We suggest two reasons for this difference: first, the WGAN framework contained a gradient penalty, and such a penalty would be out of the value ranges for different value ranges. The penalty then influenced the convergence of the WGAN framework (Mescheder et al., 2018), and, thus, the results of the WGAN framework were lower than the results of the GAN framework. Second, the WGAN framework used an RMSprop optimizer to train deep neural networks, but the GAN framework used an Adam optimizer (Basu et al., 2018). In fact, the Adam optimizer has a momentum gradient procedure, which will be fitted for regressing different value ranges. Hence, the different value ranges can be reconstructed by the Adam optimizer (Zou et al., 2018). In all of these, if we have recorded the highest sensitivity EEG signals, we must also record low-sensitivity EEG signals. We can use the highest sensitivity EEG signals to train a GAN/WGAN model to reconstruct the low sensitivity EEG signals, and the reconstructed EEG signals can be used to improve classification performance for the construction of real-time and real-application BCIs or rehabilitation treatment.

Over the past decade, most EEG-based studies have been focused on constructing BCIs or developing rehabilitation treatments (Ang et al., 2015). However, there are two main limitations to the application of EEG signals when constructing such systems, namely, the cost and portability of EEG recording devices. In fact, HSS-EEG signals will yield the best performance in BCIs and rehabilitation treatments, although HSS-EEG signals are usually recorded by expensive devices, posing an inconvenience. For example, in the “NeuroScan SymAmp2” device (Chu et al., 2016), the recording system consists of two computers and one device to link them together. One computer is used to present a stimulus for the BCI or rehabilitation treatment, and the other computer is used to record and store the EEG signals for computing the BCI or rehabilitation results. Subjects must sit in a room to wear a “NeuroScan Quik Cap” to collect data. The collection procedure is complex, and the resistance must be maintained under 5 kΩ by using conductive paste on each electrode (Agnew et al., 2012). Because the resistance is kept low and the device has a high sensitivity, the recorded EEG signals will have the ERD/ERS characteristics required for classification in BCI and rehabilitation treatment.

In general, the “NeuroScan SymAmp2” device is expensive, and the EEG signals must be recorded indoors in a limited environment (e.g., a dimly lit, sound-attenuated room). Hence, it is difficult to implement the results of the “NeuroScan SymAmp2” device (the same sensitivity as signals in AO dataset) in applications such as BCI and rehabilitation treatment. Nevertheless, low-cost and portable devices, such as “Emotiv” (the same sensitivity as signals in MI dataset), have high electrode resistance and a low sampling rate and sensitivity for recording EEG signals. The device only provides poor ERD/ERS characteristics for classification in BCI and rehabilitation treatment applications. The “Emotiv” device can be worn at any time via a simple process without requiring the resistance to be kept level (Neale et al., 2017). The energy supply for the device is a battery, and the device uses WiFi or Bluetooth communication. These advantages allow the device to be inexpensive, portable, and convenient for constructing BCIs and developing rehabilitation treatment. These mutual contradictions for signal precision and signal cost and portability inspire us to train a model to reconstruct HSS-EEG signals from LSS-EEG signals. The trained model meets the requirements of high precision and portability with low cost and can be used to improve EEG-based applications.

In fact, signal reconstruction is a difficult problem in digital signal processing, but an effective and feasible reconstruction method could significantly promote the application of signals. In this study, by using a GAN framework with Wasserstein distance and the carefully designed TSF-MSE loss function, well-trained reconstruction models have been shown to be able to reconstruct HSS-EEG signals from LSS-EEG signals. Experimental results reveal that LSS-EEG signals (just like those recorded by “Emotiv”) reconstructed by the HSS-EEG signals (just like those recorded by “NeuroScan SymAmp2”) trained models and enhanced the average classification accuracies of ERD/ERS characteristics for action observation, action execution, and motor imagery. These results inspire new ways to construct BCIs or develop novel rehabilitation treatments, but more researches need to be done to explore significant enhancement reconstruction methods across EEG signals with different sensitivities.

Based on the method of this paper, the improvement of sampling rate and sensitivity will improve the specific ERD/ERS phenomenon of MI, AO, and AE, so as to improve the performance of the BCI system. Although the CNN- based GAN/WGAN architectures will take a significant amount of time to build an available GAN/WGAN architecture, once the reconstruction model is built, the use of such a model will not take long, and the reconstructed EEG time series can be obtained within a specific time (<1 s for a trial). In future works, we can either reduce the complex of GAN architecture or improve the computational efficiency to reduce the usage time for reconstructing GAN/WGAN architecture. Then, the GAN/WGAN architectures will be used for real-time inference. In general, we used a low-cost, portable device to collect LSS-EEG signals for use in BCI or rehabilitation treatment. Before analyzing the collected data, the GAN/WGAN reconstruction models were applied to reconstruct HSS-EEG signals. The reconstructed HSS-EEG signals can significantly improve the classification performance and information transfer rate for use in BCIs or rehabilitation treatments.