Some of the common metrics used to evaluate the performance of a seizure prediction model include sensitivity (Sen), specificity (Spe), accuracy (Acc), false prediction rates per hour (FPR/h), warning time, warning time ratio (41, 42). The performance evaluation metrics of “segment-based” and “event-based” prediction models are different in their meaning as the models themselves are based on different assumptions. Table 1 describes the relationship and conceptual issues regarding evaluation metrics between the “segment-based” and “event-based” prediction models (68–70). Table 2 highlights the evaluation metrics of the performance of the “event-based” prediction models. In most of the “event-based” prediction models, the performance is evaluated primarily by the sensitivity and FPR/h metrics (22, 42, 68). Table 3 highlights the evaluation metrics of the performance of the “segment-based” prediction models.

Different ML and DL algorithms can influence the performance of the constructed epilepsy prediction models, with some generally applicable algorithms resulting in more stable performances. In a study (43) of a seizure prediction model combining a Long Short-Term Memory (LSTM) network with convolutional neural networks (CNN), authors used a public dataset of scalp EEG (sEEG) recordings from 23 patients with 185 seizures. The sensitivity of the model in “event-based” prediction was up to 100%, with FPR = 0.02/h and a predetermined preictal window (period) of 2 h. However, the authors also therein state that their approach does not allow to evaluate the prediction time, that is, the duration from the first preictal segment identification to the actual seizure onset. Another study (33) used Long-term Recurrent Convolutional Network (LRCN) to construct a seizure prediction model from EEG data of 15 epilepsy patients and achieved 100% sensitivity and FPR of 0.04/h in the “event-based” model. Zhao et al. (71) reportedly achieved a sensitivity of 99.81% and FPR of 0.05/h in a CNN-based prediction model constructed using sEEG recordings from 10 epilepsy patients in a public dataset.

Although models constructed using these conventional algorithms have shown satisfactory results, it is still difficult to apply them to clinical practice, largely because DL-guided models can easily cause overfitting, resulting in weak generalization when their assumptions (e.g. the predetermined values of their parameters) are not realized. Additionally, there is a need to minimize the computational load (time and power consumption) while maintaining a high level of performance, especially in implantable and wearable devices where such a load may be critical (71). The introduction of new concepts through innovative algorithms and models to enable prospective real-time clinical applications while maintaining the their retrospective high performance levels is the current mainstream approach to developing high-end AI-guided seizure prediction models.

The performance of models constructed by new algorithms and technologies is generally lower than that of models constructed based on matured algorithms, which may be related to the fact that the new algorithms are not yet perfectly optimized for the desired tasks. Bomela et al. (81) exploited “Inferring The Connectivity Of Networks (ICON)”, a new technique for analyzing the connectivity features of functional brain networks, and the final prediction model obtained a sensitivity of 93.62% and FPR of 0.16/h in the public dataset of sEEG data from 17 patients. Recently, another author proposed the Consistency-based Semisupervised Seizure Prediction Model (CSSPM) for epileptic seizure prediction (68), which obtained a mean sensitivity of 78.5% and the FPR of 0.44/h in sEEG recordings from 11 patients in a public dataset. It has been argued that a FPR value above 0.15/h is not acceptable (82) because even though the sensitivity is high, a high FPR would lead to too frequent false alarms. In another study, authors reported a sensitivity of 89.81% and FPR of 0.08/h in sEEG recordings of 10 patients using the Hilbert Vibration Decomposition (HVD) method to extract features and construct a CNN-based seizure prediction model (53).

In the event-based seizure prediction models, the preictal period (PP), SOP and SPH are predetermined, PP = SOP + SPH, and SOP and SPH are optimized for the model to achieve high sensitivity and low FPR (i.e., high specificity). There should be less uncertainty about the occurrence of the seizure after the end of SPH (i.e., ideally SOP = 0), and SPH should be long enough (83) so that measures can be taken by the patient and/or healthcare provider to prepare for or intervene and avert an upcoming seizure. The SPH refers to the period from the issue of the warning of an upcoming seizure to the beginning of SOP, within which no seizures occur; if seizures do occur within SPH, the model is considered to be underperforming. The preictal period (PP) is sometimes called the preictal interval length (PIL), the SPH the prediction period, and the SOP the prediction horizon (Figure 3) (71, 84, 85).

It has been suggested that the optimal SOP and SPH ranges to accurately predict seizures with high sensitivity and specificity should be distinct for each individual, that is, there should be no uniform, standardized SOP and SPH values across patients (86). In a conducted study, the best performance of a prediction model could be achieved when the SPH was <2 min (87), which is a low value and might be related to the algorithm used to construct the respective model as well as the quality of the raw data. SOP is defined as 30 min in the majority of the relevant literature, arguing that a 30 min SOP is appropriate as it is short enough so that it does not cause much anxiety to the patient then waiting for a seizure to occur after the end of SPH (33, 41, 42, 68, 71, 73, 88, 89). Zhang et al. (42) defined SOP as the 30 min and SPH as 0 min windows in practice and evaluated EEG records of 23 patients from a public dataset. The sensitivity of the shallow CNN model created to predict sEEG seizures was 92.2%, with the FPR of 0.12/h. Another study (73) reported the utilization of intracranial EEG (iEEG) data from 20 patients in a generalized dataset for the construction of a threshold-based model for predicting seizures. Here, the authors defined SOP as 30 min and 50 min and the SPH as 2 min. The final model achieved a mean sensitivity of 90.42 and 91.67%, and the FPR of 0.12 and 0.10/h, respectively. Another study that defined SOP as 30 and 50 min, respectively, used the Bayesian Linear Discriminant Analysis (BLDA) to construct models with sensitivities of 85.11 and 93.62%, and an average FPR of 0.08/h (41). The goal in constructing a good prediction model is to minimize SOP and maximize SPH, a combination that achieves the lowest possible FPR and the highest possible sensitivity.

Although many of the predictive models reviewed in the current study have achieved excellent performance indices, there is still a great deal of uncertainty as to whether they can be used in the clinical setting. Firstly, most studies use the same public dataset, such as the CHB-MIT dataset or the Freiburg dataset, but there is a lack of validation of the constructed models in other datasets, which makes it difficult to guarantee that the same model will work well in different datasets. In addition the lack of available continuous datasets, which have to include multiple interictal, preictal, ictal, postictal EEG recordings, makes it difficult to obtain adequate validation of the constructed models. There are few eligible datasets, and only a few competitions have been conducted on the same datasets, the first one in 2002 at the First International Workshop on Seizure Prediction in Bonn, Germany (44). Not all datasets used in subsequent competitions were continuous over time. Again, the advanced algorithms that emerged from these competitions have rarely been validated on other datasets. In addition, most studies balance the interictal and preictal segments in order to obtain a high-performance model, but this is not true from a practical standpoint, since interictal periods are much longer than the preictal periods. For this reason, future studies should be validated on multiple continuous datasets without artificially balancing the number of segments over time or assuming constant preictal periods, SPH and SOP across seizures, even within the same patient. Adaptive individualized seizure prediction algorithms, such as the first one published one by Iasemidis et al. (18), should be capable of dealing with these shortcomings.

The post-processing techniques play important roles in the construction of the seizure prediction model as the interface between the “segment-based” model and the “event-based” prediction model. The post-processing methods help obtain the window segment outputs from the segment-based (classification) model (80). The most commonly used methods are filters, such as Kalman and Bayesian filters (35, 42, 54, 75, 77). Choosing the right filter length and alert threshold can improve the performance of the model to some extent. Determining the length of the moving filter is the first step in post-processing. The use of a filter can facilitate the elimination of most random errors from the classification model and reduce a large number of false predictions that may occur in the short term (Figure 4A). A study published in 2019 (55) used a causal moving average filter of 1 min in length to smooth and filter the classification results of 30 window segments, and the used 3D CNN (3D CNN) model obtained a sensitivity of 85.7%, and FPR of 0.096/h, and 10.5% of warning time ratio, as well. Another study (56) set the length of moving average filter to a range of 0–15 min, with a step size of 0.5 min, and found by comparison that the best performance of the prediction model was achieved with a moving average filter of 1 min in length, which is consistent with the findings of Ozcan et al. (55). Setting a very long filter length can enhance the smoothing of warnings resulting in delayed alerts or even missed seizures (low sensitivity), while too short a filter length can lead to opposite effects (high FPR and hence low specificity).

The alarm is the second step in the post-processing technique and it is ON when the detection rate of preictal segments within the pre-specified duration of the used filter reaches a set threshold (Figure 4B). Khan et al. (74) defined the probabilities of a segment in the classification model to be interictal, preictal or ictal as p0, p1 and p2, respectively, and assuming that p0 + p1 + p3 = 1. For monitoring and predicting seizures, the value of “1 – p0” was taken as the index of the model that would characterize a segment as preictal or ictal using a predetermined threshold of 0.6. When the probability exceeded 0.6, the model was programmed to issue an alert predicting an upcoming seizure. This setting was later verified by another group (56), using a threshold range of 0.1–0.9 in steps of 0.05. This study showed that the best performance of the prediction model could be achieved with a threshold value of 0.6, a sensitivity of 95.5% and FPR as low as 0.109/h. It was found that (57) the classification model constructed by combining CNN and features from a Directed Transfer Function (DTF) analysis was post-processed by a 20 point moving average filter reached sensitivity of 90.8% and FPR of 0.08/h. Theoretically, the predictive performance of the model can also be improved by estimating and not by having the threshold predetermined. For example, In the case of an artificial neural network (ANN)-based prediction model (80) constructed using a single nonlinear feature, the model generated alerts in the test dataset when the average eigenvalue in these periods was higher than or equal to those in the windows in the ictal period of the training dataset. The model achieved an average FPR of 0.014/h, with an average seizure prediction time of 26.73 min in advance of a seizure. These studies emphasize the importance of the employed thresholds since a very low threshold would end up sending too many false alarms, while an unnecessarily high threshold could lead to missing emergencies.

Although most false alarms can be eliminated in the post-processing stage, the ones that survive and repeatedly predict the same seizure will send out multiple warnings causing panic to patients (53). Hence, an “absolute refractory period (ARP)”, i.e., a period between two consecutive warning messages, should be set to prevent frequent warnings for the same seizure. The sensitivity and FPR of the model were 89.8% and 0.081/h, respectively, when the ARP was set to 20 min. In other studies, this ARP was equated to the SOP (55, 73), which improved the performance of the final model. Furthermore, the “absolute refractory period” setting can be useful in effectively removing incorrect warnings which escape the post-processing filtering stage. Ideally, only one correct warning before a seizure is sufficient for the necessary precautionary measures to be taken.

The “k-of-n” approach means that if “≥k” of the consecutive “n” time window segments output from the classification (segment-based) model are determined “preictal”, then all “n” segments are considered preictal (Figure 5) (33, 36, 68). When the same post-processing method was applied to construct a CNN-based classification model for interictal and preictal periods, including only one layer of output regularization, the “8-of-10” was used to give an early warning. The proposed method exhibited better performance in all three datasets (Freiburg dataset, CHB-MIT dataset, American Epilepsy Society Seizure Prediction Challenge dataset) with sensitivities of 81.4, 81.2, and 75%, respectively (58). Parvez et al. (90) used an iEEG dataset of 27 patients to construct a binary classification model by Least-Squares Support-Vector Machine (LS-SVM) and defined the output of the classification model as “1” for the preictal window segments and “0” for the interictal window segments. Here, the authors compared different window lengths and numbers to derive the best combination. When 3 out of 5 consecutive 10-sec windows were identified as preictal, then all 50 sec were considered as preictal. If 2 out of 6 consecutive 50-sec windows were identified as preictal, then all 50 sec were considered as preictal. In case 2 out of 6 consecutive 50-sec windows were thus characterized preictal, an early warning would be issued. This prediction model was 91.95% accurate with an average of 2.14% prediction errors per patient. A subsequent study (36) used the same two levels of “k-of-n” (“3-of-5” and “2-of-6”), and the prediction model achieved the same excellent performance. The multi-level “k-of-n” method is superior to the single-level method.

Considering the uniqueness of EEG data from each epilepsy patient, studies suggested the importance of individualized seizure prediction models for precision medicine. So, the classification model has also been constructed without cross-validation from different patient datasets, thus ensuring a specific prediction model for each patient (91). In the post-processing steps, 3 levels of layer (window lengths of 5 sec, 2 and 6 min) of the “k-of-n” method were used for filtering. Each 2 min window that exceeded a certain percentage of 5 sec preictal or interictal windows was classified accordingly. When the characterized preictal 2 min windows within a 6 min window exceeded a different threshold, a warning of an impending seizure was issued. The average sensitivity of this optimized model was 94%, and the FPR was 0.111/h. These findings revealed that the “k-of-n” method was a simple and easy to implement post-processing technique. But the ideal “k” and “n” and the number of layers that optimized prediction were not estimated. The major drawback was that larger values of “k“ and “n“ would render the model less sensitive (delayed warnings or even missed alarms). Smaller values of “k” and “n” would impact the specificity of the model.

In addition to the commonly used filter techniques and the “k-of-n” approach in post-processing, some studies have invoked novel post-processing techniques to further improve the performance of the epileptic seizure prediction model. A study (92) on real-time seizure prediction over a long period (169–364 days) used a new adaptive post-processing approach by updating the alarm threshold the alarm threshold every 7 days based on the prediction results of the previous week. The authors, considering the large number and overlapping outputs of the classification model, distinguished between preictal and interictal periods by comparing the mean and standard deviation of the Support Vector Machine (SVM) model outputs, which yielded a final model with a prediction sensitivity of 84%. Although SVM is one of the most commonly used traditional ML methods, a DL method can automatically extract features. But, if DL is to be used in seizure prediction models, the new methods need to be optimized and validated in the distinction of preictal and interictal periods. Direito et al. (22) emphasized the concept of Firing Power (93), first classifying windows from the test dataset into different periods through a classification model, then using this information to calculate the firing power parameters through a sliding window, and once the threshold threshold for Firing Power exceeded 0.5, the system generated an alarm of an upcoming seizure. So, new post-processing techniques are used today in the field of seizure prediction, but there is a lack of systematic validation of these algorithms on existing and new datasets.

In an “event-based” seizure prediction model, post-processing techniques and the length of SOPs and SPHs are important for the model performance. Also, we found that models built from invasive EEG (iEEG) recordings were generally superior to ones from scalp EEG (sEEG) recordings. Because iEEG has the advantages of less artifact and power interference, recording from brain sites closer to the epileptogenic focus, and without any attenuation from the skull, generates high quality of data with fewer confounding factors. A significant decrease in features important for the epileptic seizure detection and prediction algorithms was observed within 3 months after iEEG electrodes implantation. This may be due to trauma and tissue inflammation from the implantation. Also, the mean power of the high gamma band was found to decrease in many patients, which could inevitably affect the performance of the model (71). The sEEG is simple to perform, is non-invasive, standardized, and has fewer side effects, making it popular among epilepsy patients (72). Since sEEG is a non-invasive recording modality, it has significant disadvantages in terms of signal quality, signal artifacts, and lower spatial resolution of electrical brain activity. Peng et al. (94) used the same DL algorithm to construct seizure prediction models from sEEG and iEEG datasets. They showed the sensitivity of the prediction model constructed from the iEEG dataset exceeded that from the sEEG dataset by 5.8%, and the FPR was 0.08/h (lower than that of the sEEG model). The datasets used in most studies are mostly public, open-source datasets, such as the CHB-MIT dataset (physionet.org/content/chbmit) (95) (the most commonly used dataset for non-invasive EEG), and the Freiburg dataset for invasive EEG (which is now subject to a fee, and integrated into the European Epilepsy Dataset, www. fdm. uni-freiburg.de/EpilepsyData) (81); the American Epilepsy Society Seizure Prediction Challenge dataset, www.kaggle.com, and others (96, 97). These datasets have public and easily accessible characteristics, which facilitate the comparison of models' performance on the same EEG datasets, as well as support algorithms' optimization. The European Epilepsy Dataset (98) is the most enriched dataset available to date (www.epilepsiae.eu), including sEEG data from 225 patients and iEEG data from 50 patients to date. Moreover, it is essentially a combination of datasets from different recording sites, including Coimbra, Freiburg, Paris, and Treviso. However, since some of these integrated datasets are non-open access, it is difficult for most researchers to run their algorithms on these data. Importantly, most existing public datasets have a low number of cases and insufficient or discontinuous data. In the future, we should emphasize developing a large open-source platform to validate models and improve their performance while maintaining a good generalization capability.

The sampling frequency of the EEG may also affect the performance of seizure prediction models. For example, some studies have found high-frequency (80–500 Hz) oscillations (HFOs) to be a reliable biomarker for predicting seizures (82, 99). Since many commonly used public datasets contain EEG sampled at 256 Hz, HFOs cannot be detected. In addition, EEG recorded in long term [e.g., over days in the epilepsy monitoring unit (EMU)] is influenced by the patient's daily routine, the dose of anti-seizure medication etc. The above, in addition to the actual preictal period being very different and changing over time from one assumed by a model, constitute barriers for the use of predictive seizure models in practice.

ML has been widely used to build seizure predictive models (100), where common algorithms such as SVM (101), random forests (102), and cluster analysis (32) are used. The principle of constructing predictive seizure models involves the pre-processing of interictal and preictal EEG datasets, followed by computational extraction of commonly used features, and then constructing classification models by ML. In a “segment-based” epileptic seizure prediction model (70), the sensitivity reached was 95.8% using SVM. DL is derived from ML, and the biggest difference between DL and ML is the ability of DL to better mine data features (29). Automatic feature extraction means less human error, and the underlying mathematical logic of DL algorithms is more rigorous and suitable for processing and handling big datasets. The performance of the classification model constructed by the algorithm determines, to some extent, the performance of the epilepsy prediction model processed by post-processing techniques. In general, models constructed by DL outperform ML, and ML often outperforms DL in terms of its ability to perform on brand new data (generalization capability/robustness).

a. Importance of large open-source databases and exchangeable platforms: Currently, there are no big open-source datasets available to validate prediction algorithms, while both ML and DL rely on the support of big data. Although many current models have achieved high performances with pre-determined parameter values and assumptions on particular datasets, they would perform poorly in clinical practice when these assumptions and values are not valid. It is expected that good communication would allow validation of internationally advanced ideas as well as techniques and inspiration from the GitHub platform in computing.

b. Refinement of clinical characteristics of the data: It is well known that differences in age, gender, medication status, seizure type, and other patient characteristics and demographics, which leads to a high number of confounding factors in the raw data, thereby affecting the performance of models to varying degrees and hindering their real-life application. The current EEG datasets still lack the necessary inclusion and exclusion criteria to compensate for the above, probably due to less integration of action between clinicians and model developers.

c. Addressing the shortcomings of the low signal-to-noise ratio of sEEG data: Noninvasive BCI technology using sEEG has unique advantages. sEEG is safe, simple, convenient, and universal, which makes it easier to record from more patients under different conditions. Importantly, collaboration between engineers and computer scientists has led to development of pre-processing techniques for sEEG that eliminate interference artifacts with high fidelity.

a. Disclosure and availability of algorithms: The vast majority of research studies are currently focusing on the innovation, upgrade and validation of new algorithms, and comparison of old and new seizure prediction models on the same dataset. Establishing an open-source platform that allows validation and modification of models using different datasets would improve the generalization (robustness) of models and help address key issues of model-to-clinic transition.

b. The choice between ML and DL: ML requires manual feature extraction, which is prone to feature mismatch. While DL needs no manual feedback for feature extraction, there is a problem of over-extraction of features, which leads to weak generalization of the resulting model. It is believed that ML should be combined with DL, that is, to extract EEG features by DL and build classification models using traditional ML methods, thus obtaining generalizable and high-performance models.

Post-processing techniques play an important role in bridging the (segment-based) classification and the (event-based) prediction models with a significant impact on their performance in seizure prediction. Currently, researchers in the field of EEG-based seizure prediction has not paid enough attention to post-processing techniques. Although multiple methods and techniques have emerged, there is no metric to evaluate the advantages and disadvantages of these techniques. In the future, relevant evaluation metrics should be developed to evaluate post-processing techniques.

a. Sensitivity and false prediction rate: Sen and FPR are important metrics of performance of prediction models in clinical practice.

b. SOP and SPH: The SOP and SPH are pre-determined parameters that affect Sen and FPR of any prediction model. The longer the SPH and shorter the SOP, a better seizure prediction model is constructed. In today's precision medicine paradigm, the values of SOP and SPH could be optimized to get the best Sen and FPR for an individual or class of individuals.

Many high-performance algorithms have been developed and tested on collected data retrospectively. It is thus difficult to determine whether they generalize well. A small number of epilepsy patients have been recruited for prospective analysis by companies and teams, but overall progress remains limited. Future validation on multiple datasets, which could later be replicated in a number of competitions, is highly recommended.

The application of models to clinical practice should also consider certain practical issues, such as running in real time, low power consumption for implantation in the brain or embodiment in wearable devices, user-friendliness of inputs and outputs to and from the models.