To select appropriate assessment parameters for EEW, two aspects were considered, namely, operations and management, which are distinct but interrelated. Operation refers to the day-to-day activities and tasks of a system (Whipple and Frankel, 2000). In EEWs, operations focus on the reliability of alert production (Medina-Cetina and Nadim, 2008). In contrast, management involves EEW planning and control to achieve high performance (Ittner and Lacker, 1997), considering alert reliability and seismic information accuracy (Ruhl et al., 2019; Esposito et al., 2022). Managers are responsible for overseeing objective reviews and implementing improvements, ensuring that a balance is maintained between safety, effectiveness, and sustainability (Too and Weaver, 2014). Figure 2 shows the EEW algorithm analysis process with both aspects. The assessment steps are shown in the EEW flow, considering algorithm analysis based on simulations.

As regards the operational aspects, system behavior was categorized into three steps. Step 1 is the evaluation preparation stage, comprising input of the earthquake scenario and setting the minimum targeted nS. Step 2 is the detection phase of the analysis process [i.e., short-time-average over long-time-average trigger (Allen, 1982)] or the filter picker (Lomax et al., 2012). If the nS condition is satisfied, it is verified to determine whether it becomes an associated event (Cho et al., 2022). If the alarm fails to generate an event, verification is required whether this event was alarm worthy. Verification allows the determination of the alarm being operationally a false alarm or normal operation. Step 3 comprises predicting the source and assessing the alarm based on associated event information (Weyrich et al., 2021). Alarms can be produced by EEWs based on the predicted source information, which is termed decision making of warning (DMW).

As regards the management aspect, it is essential to verify the success of the alert and the source information that influenced the decision to issue an early warning. By considering all the parameters that impacted the final alert, it could be ensured that the algorithm and system were free of issues. Figure 2 presents representative error sources that could be easily identified from an administrator viewpoint. However, various errors could occur in operations, and this procedure is only applicable to natural earthquakes.

In the review of EEW assessment, we ignored the associate event time, which depends on the nS. Additionally, waveform data used in KMA seismic stations are applied only to locations that pass strict criteria for latency, background noise, and trigger rate (KMA, 2017; Ahn et al., 2021). Consequently, data lagging or failed equipment rarely occur for KMA EEW in real-time operations. However, from a management perspective, a station detection rate review within a certain time frame could be performed separately, because of trigger checking at the observatory.

We selected four parameters for this study based on aspects. 1) The first parameter is “prevent missed alerts (PM)” in the system, i.e., confirming whether a targeted earthquake alert was missed. Alerts in normal operations are issued when the national observation network detects a warning event using n stations. 2) The second parameter is “prevent false alerts (PF)” in the system, which determines whether a targeted earthquake alert is incorrect. This involves checking alarms that are normal operations, non-earthquakes, or source errors when EEW alerts occur. 3) The third parameter is the accuracy of magnitude (AM), and 4) is the accuracy of location (AL). These four parameters were reviewed from operational and administrative perspectives and represent all the information that the public sees when an alert is triggered.

To help respondents better understand the assessment parameters, we provided training before conducting the research. During the training, we introduced various existing analytical examples, explained the unique features of the criteria, and received approval for ethical review surveys involving human participants. Further, we discussed the EEW technical and theoretical limitations. These are 1) Inability to fully understand the underground geological structure, implying that seismic information is subject to uncertainty at source (Iervolino et al., 2009; Tidlund et al., 2022; Murray et al., 2023; Ni et al., 2023). 2) Detection accuracy varies depending on the observation network because of wave propagation attenuation (Jackson and Anderson, 1970; Vipin and Sitharam, 2011; Goda et al., 2023). 3) Accuracy limitations of source detection in coastal earthquakes (Angove et al., 2019; Lim et al., 2020; Takabatake and Kojima, 2023), and the difficulty of installing seismometers in the seabed (Podolskiy et al., 2021). The system operators understood and agreed with these EEW limitations. In addition, we emphasized understanding of the characteristics of earthquakes in the Korean Peninsula, as well as the psychological conditions of the public. Given the infrequent occurrence of seismic events in the Korean Peninsula, even weak shaking could elicit feelings of surprise and unease among the general population (Kwon et al., 2020; Yeon et al., 2020). Therefore, unlike other countries, the KMA issues warning criteria from even slight magnitudes (M_L 3.5 and over).

Considering KMA policy and technology characteristics, we designed a questionnaire for the EEW assessment. In the criteria establishment stage of the survey, we created five response items to establish the assessment criteria. We classified the criteria between inland areas and outside (i.e., ocean, outside the country) of the seismic observation network. Both groups 1) within the seismic observation network (= inside seismic network, ISN) and 2) outside the seismic observation network (= outside seismic network, OSN) were considered. Categorization by observation network was included in all five questions.

The survey questions were:

Q.1: “What is the minimum magnitude of an earthquake to be analyzed for performance assessment?” We needed to establish a baseline progression for our evaluation, considering the margin of error in our prediction scale. However, detecting initial waveforms from slight magnitude earthquakes is complicated owing to their small amplitudes and poor observation environments. The EEW performance results could, therefore, vary depending on the criteria and magnitude, with greater sensitivity for weaker events than larger ones.

Q.2: “Compared with M_L ≥ 5.0, how important is an earthquake of M_L < 5.0?” This item was included in the questionnaire, as the level of ensuing damage could differ according to the magnitude of the event. While seismic intensity is important from a damage perspective, the basic information transmitted is magnitude and location. Therefore, we investigated the relativity of magnitude. The initial standard for early warnings of earthquakes in Republic of Korea was M_L > 5.

Q.3: “How important is the comparison of earthquake locations, both inside and outside the observation network?” This question was included because it is associated with the observation network conditions, and the current criteria classify inland and outside events (i.e., teleseismic or occurring outside the country). These criteria represent the KMA strategy for considering the impact of earthquake detection relative to the observation network. Inland earthquakes can be detected rapidly and analyzed as they occur between observation networks, but oceanic analysis could likely be inaccurate in relation to the source owing to biased and sparse observation networks.

Q.4: “What is the maximum allowable error range when evaluating the accuracy of the magnitude?”

Q.5: “What is the maximum allowable error range when evaluating the accuracy of the epicenter?” The fourth and fifth questions are related to accuracy, as accuracy could affect missed and false alarms. Additionally, if the difference between the predicted information and precise information analyzed afterward is significant, an operating system check could be required.

In the criteria establishment stage of our survey, we analyzed the results based on the five questionnaires and collected the majority opinion regarding operational recommendations. Table 1 shows a summary of the established criteria from the questionnaire. The criteria in Q.1 allowed us to define the analysis object range. We established the range for earthquakes to be considered by ISN and OSN, i.e., 0.5 smaller than the magnitude of the alert criteria.

We set the weight for event sources based on Q.2 and Q.3. From an operational side, all earthquakes are valued equally because the evaluation pertains only to whether an alert should be issued. For instance, in a point-source assessment by Cochran et al. (2018), all earthquakes were also valued equally. However, the damage caused by an earthquake depends on the location and magnitude of the event. Therefore, we proposed weighing of the source effects. The score for each earthquake event was: Event Value  E V = δ M × δ L (1) where EV is event value, which is the judgment of earthquakes from a management perspective. δM is the weight for magnitude, and δL is the weight for the source location (i.e., ISN, OSN) of the earthquake. For example, if an earthquake of M_L 5.5 occurred inland, the event value would be 1. Conversely, if an earthquake with magnitude M_L 3.2 occurred inland near the coast, the event value would be 0.18.

The criteria for an EEW false alarm was selected based on answers to Q.4 and Q.5. The criteria set for a false alarm were assessed based on the estimated source information produced by EEWs. In practice, the results of both source location and magnitude of EEWs depend on the nS (Lim et al., 2020). The more stations are involved in collecting information, the better would be the accuracy of epicenter analysis. From a management perspective, the criteria for a false alarm could be a crucial factor for planning system improvements.

Assessments of EEWs must consider both the appropriateness of the alerts and the accuracy of earthquake source estimates. The source estimates in EEWs are updated rapidly as they increase because of a higher number of nS. However, we do not recommend reviewing all the steps that are updated over time, because the public responds to the first warning message (Lassa, 2008; Vihalemm et al., 2012); therefore, the information at the moment the alarm occurs is the only concern. Alert time depends on nS, which is the most important factor in EEW alarm decision making. Our proposed assessment is designed to evaluate performance according to the target nS.

The first step was to score alert appropriateness (AA). We designed the scoring to include only events where the alert was a normal operation under the target nS conditions. If an earthquake required an alert but resulted in a false or missed alarm, we scored it as zero. This allowed us to determine if the system has effective controls for false or non-alarms. Consequently, our assessment could result in a low score if the system failed to control alarms for multiple cases. The formula for this is: Prevent false alerts  P F = E V × A A (2) Prevent Miss alerts  P M = E V × A A (3) where EV is the event value by Eq. 1, AA is scoring of alert appropriateness (only 1 or 0). Therefore, a normal alert is 1, and false and missed alarms are 0.

The second step is to score the accuracy of an estimated source. By default, we designed the accuracy calculation to be based only on positive cases (normal alerts). The basic framework is similar to that of the Cochran et al. (2018) model. The expression is: Accurate of magnitude  A M = E V × 1 − M N o − M E E W Δ M × A A (4) Accurate of location  A L = EV × 1 − L N o − L E E W Δ L × A A (5) where M _NO and L _NO are the notified magnitude and epicenter of the earthquake according to the KMA. The notified earthquake information is calculated through manual analysis by seismic analysts at the KMA. The M _EEW and L _EEW are the calculated source information (i.e., magnitude and epicenter) according to the target nS in the EEW. The maximum allowable error values, ΔM and ΔL, are based on the answers to Q.4 and Q.5. The accuracy value is close to 1 for a small error and close to 0 for a large error.

We assumed that different experts would place different values on the four parameters; therefore, we performed a relative comparison of the four parameters to obtain their values. For the weights of the four assessment parameters, we used the Analytic Hierarchy Process (AHP) that involves pairwise comparison, weight calculation, and consistency verification, as described by Saaty et al. (1990). The AHP process could be used as a rational decision-making tool for risk assessment to stratify, simplify, and systematize multiple criteria (Wen, 2015; Thaker et al., 2018; Providakis et al., 2022). This survey aimed to identify the most important parameters for operators by employing relative comparability.

During the pairwise comparison process, the study assessed the relative importance of the four assessment parameters using a nine-point scale, as shown in Figure 3. Following this step, the results were subjected to consistency verification to ensure their reliability. The consistency index (CI) used to assess the reliability of the responses was calculated through the maximum Eigenvalue operation of the comparison matrix. A lower CI value indicated higher reliability. For this study, an average weight of 11 questionnaires with a consistency index of 0.2 or less was used for the analysis. A summary of the results of these weights is presented in Table 2.

In pairwise analysis, the most important factor from the perspective of managers and operators is controlling false alarms. This factor is crucial because the issuing of a warning and the level of the warning depend on the accuracy of magnitude and location. Although the algorithm itself does not reveal this, operators consider the activation of a warning without an actual earthquake a significant problem. Therefore, false alarm control was allocated the highest weight, followed by non-alarm control, location accuracy, and magnitude accuracy.

To determine the assessment of the overall score (OS), we used the weights and assessment points of the four parameters. The final score was obtained by multiplying the score of each element by its weight, as follows: Overall Score  O S = α ∙ P M + β ∙ P F + γ ∙ A M + κ ∙ A L (6) where PM, PF, AM, and AL refer to parameters such as prevent missed alerts, prevent false alerts, accuracy of magnitude, and accuracy of location, with α, β, γ, and κ as their respective weights (Table 2).