Detection and evaluation of signals for immune-related adverse events: a nationwide, population-based study

Background Immune checkpoint inhibitors (ICIs) are one of the main pillars of cancer therapy. Since other studies such as clinical trial and retrospective study have limitations for detecting the immune-related adverse events (irAEs) characterized by unpredictable onset, nonspecific symptoms and wide clinical spectrum, we aimed to identify the incidence of irAEs and to detect and evaluate the signals using real-world data. Methods Cancer patients treated with anticancer medications were analyzed using the nationwide health insurance claims database of South Korea from 2017 to 2019, and Clinical Data Warehouse (CDW) database of Asan Medical Center (AMC), a tertiary referral hospital, from 2012 to 2019. AEs of ICI users were compared with those of non-ICI anticancer medication users. PD-1 inhibitors (nivolumab and pembrolizumab) and PD-L1 inhibitors (atezolizumab) were evaluated. We defined an AE as a newly added diagnosis after the ICI prescription using an ICD-10 diagnostic code. A signal was defined as an AE that was detected by any one of the four indices of data mining: hazard ratio (HR), proportional claims ratio (PCR), claims odds ratio (COR), or information component (IC). All detected signals were reviewed and classified into well-known or potential irAEs. Signal verification was performed for targeted AEs using CDW of AMC using diagnostic codes and text mining. Results We identified 118 significant signals related to ICI use. We detected 31 well-known irAEs, most of which were endocrine diseases and skin diseases. We also detected 33 potential irAEs related to disorders in the nervous system, eye, circulatory system, digestive system, skin and subcutaneous tissues, and bones. Especially, portal vein thrombosis and bone disorders such as osteoporosis with pathological fracture and fracture of shoulder, upper arm, femur, and lower leg showed high HR in ICI users than in non-ICI users. The signals from hospital database were verified using diagnostic codes and text mining. Conclusion This real-world data analysis demonstrated an efficient approach for signal detection and evaluation of ICI use. An effective real-world pharmacovigilance system of the nationwide claims database and the EMR could complement each other in detecting significant AE signals.


Introduction
Immune checkpoint inhibitors (ICIs), which activates the immune system to fight against cancer, have evolutionally changed the recent trend of cancer therapy.Monoclonal antibodies targeting T-cell immune checkpoint molecules, including programmed cell death protein 1 (PD-1), programmed cell death ligand-1 (PD-L1), and cytotoxic T lymphocyte associated protein-4 (CTLA-4), are the most effective examples of ICIs.Randomized clinical trials involving ICIs have demonstrated the durable clinical benefits of these medications across a diverse range of cancers (1)(2)(3).
However, patients treated with ICI may experience immunerelated adverse events (irAEs), which are autoimmune disease-like toxicities caused by immune reaction associated with this treatment.These events exhibit distinct characteristics compared to those observed with conventional cytotoxic chemotherapy or targeted agents.They can affect any part of the body with diverse severity and onset times (4,5).The unique profile of irAEs, characterized by unpredictable onset, even after the completion of treatment, nonspecific symptoms, a broad clinical spectrum, and in some cases, irreversibility and fatality poses a significant challenge in the pharmacovigilance of ICIs.Clinical trials have reported various adverse events (AEs), but they have limitations since they collect data from a limited number of selected populations for a restricted duration of time.In particular, ICIs have often received accelerated approval from regulatory agencies based on results from earlier phase studies with small samples sizes and short study durations (6,7).This may increase the risk of the AEs not being fully characterized in pre-marketing clinical trials.Therefore, postmarketing pharmacovigilance of ICIs is particularly crucial to capture real-world experiences.Several efforts have been undertaken to identify AE signals using post-market pharmacovigilance databases (8)(9)(10)(11)(12)(13).However, these databases primarily consist of voluntary AE reports, making them susceptible to various reporting biases and rendering them inadequate for calculating AE incidence.Large-scale observational studies using real-world data, such as electronic health records, medical claims and billing data, and registries, have the potential to overcome the inherent limitations of clinical trials and spontaneous reporting databases.Given that the National Health Insurance (NHI) system in South Korea provides healthcare coverage to the entire population residing the country and that all reimbursing claims with diagnostic codes are evaluated by the Health Insurance Review and Assessment Service (HIRA), the claim data of HIRA can be considered an appropriate data source for pharmacovigilance research.Several attempts have been made to utilize health insurance claims data as a pharmacovigilance tool for conducting post-marketing surveillance, employing multiple data mining methods (14,15).
In the present study, our objective was to monitor known irAEs in a real-world setting and identify potential new AE signals in patients undergoing treatment with ICIs.To achieve this, we utilized a nationwide, population-based claim database as a primary data source and further validated these signals using a clinical data warehouse.

Data source for signal detection
We defined the potential risk associated with ICI usage as a "signal", a term frequently used in pharmacovigilance.This term denotes "Information originating from one or several sources, indicating a possible new causal relationship or a novel aspect of a known association between an intervention and an event or a group of related events, whether adverse or beneficial" (16, 17).Signal detection in this study was conducted using the HIRA database.The HIRA database includes all information on approximately 50 million people, the whole Korean population covered by the NHI program since 2000.Every resident in South Korea is provided with a unique civil registration number.The NHI program provides coverage for all residents in a form of compulsory social insurance, which ensured the complete follow-up of study participants.In South Korea, the NHI system started to reimburse ICI treatment for non-small cell lung cancer (NSCLC) and bladder cancer from August 2017, and for melanoma from February 2018 (18).We obtained the claims data of patients with NSCLC, melanoma, and bladder cancer that had been submitted by healthcare providers between January 1, 2014, and January 31, 2019, with anonymized identifiers provided by the HIRA to protect their privacy, in accordance with the Act on the Protection of Personal Information Maintained by Public Agencies.The database contains information on demographic records, diagnosis, procedures performed, and prescriptions.Demographic information included age and sex.All diagnoses were coded using the Korean Standard Classification of Disease, seventh edition (KCD-7), a modified version of the International Classification of Disease and Related Health Problems, 10th revision (ICD-10).The prescription data included generic name of the drug according to the HIRA drug formulary code, prescription date, and duration.Access to the HIRA database was limited to researchers who completed an application process and were granted approval for research access.

Ethics
This study was approved by the Institutional Review Board of Asan Medical Center (IRB No. 2019-1300, 2019-1247), and informed consent was waived.

Study subjects for signal detection
In this study, we retrospectively identified patients with NSCLC, bladder cancer, and melanoma by using the KCD-7 codes for primary and secondary diagnoses (C34, C67, and C43, respectively), as well as the corresponding cancer-specific deductible insurance codes (V027 for NSCLC, V193 for bladder cancer, and V194 for melanoma).These patients were then categorized into two groups: the ICI group and the non-ICI group.Utilizing the 5-year HIRA claims database (from January 1, 2014, to January 31, 2019), we selected patients who received anticancer medication between August 2017 and January 2019, based on the initiation date (August 2017) of the NHI coverage for ICIs.The ICI group consisted of patients who received at least one cycle of treatment with PD-1 inhibitors (nivolumab and pembrolizumab) or PD-L1 inhibitor (atezolizumab) inhibitor between August 2017 and January 2019.The non-ICI group, serving as the comparator, included patients who received non-ICI systemic anticancer agents such as cytotoxic chemotherapy and molecular targeted therapy (Supplementary Table S1) for NSCLC, bladder cancer, and melanoma during the same period as the ICI group.The index date for each patient was defined as the first prescription date of the corresponding drug, and all study patients were followed up for a maximum of 18 months from the index date.AEs in the ICI group were compared with those in the non-ICI anticancer medication group.

Definition of adverse events
In this study, AEs were identified from the claims study database by considering newly added diagnosis codes during treatment.AEs were defined as diagnoses that did not exist prior to exposure to the drug of interest but were added after exposure.The KCD-7 coding system, which consists of 21 chapters with various code blocks, was used for classification.Physicians in South Korea are required to report the most specific diagnosis code available according to the KCD-7 coding system, ranging from three to seven digits, in the medical care cost statements submitted to the NHI system.
Initially, a low-level classification was performed using threedigit KCD codes, resulting in a group of 1,348 codes.Subsequently, a mid-level classification was conducted using groups of three-digit KCD codes to categorize single conditions within each block, resulting in a total of 222 codes.However, certain disease categories were excluded from the study.The neoplasm category (KCD-7 codes, C00-D48) was excluded due to the difficulty in distinguishing it from existing cancers, which are already indications for ICIs.Additionally, pregnancy-related diagnosis including ''Pregnancy, childbirth, and the puerperium'' (O00-O99), ''Certain conditions originating in the perinatal period'' (P00-P96), and ''Congenital malformations, deformations, and chromosomal abnormalities'' (Q00-Q99) were excluded to focus on non-pregnant cancer patients.

Signal detection criteria
The hazard ratio (HR) was computed by dividing the incidence of ICI-specific AE pairs (A) by the total observed duration of claims (T 1 ), and then dividing the results by the incidence of non-ICI-AE pairs (C) divided by the total observed duration of claims (T 0 ).If the lower limit of the 95% confidence interval (CI) of an AE's HR was greater than 1, it was considered a potential signal possibly associated with ICIs.
Disproportionality measures were also employed to detect AE signals, using proportional claims ratios (PCRs), claims odds ratios (CORs), and Bayesian confidence propagation neural networks (BCPNNs) of information components (ICs).These measures were specifically adapted for use with the claims database, as opposed to spontaneous reporting databases.Two-by-two tables were constructed, combining the frequency of drug-AE pairs and total claims, along with the estimated indices for each AE (Table 1).
PCR is defined as the ratio between the incidence of a specific AE associated with a particular ICI and the incidence of AEs related to non-ICI drugs listed in the database (19).COR represents the odds ratio of developing a specific AE compared to all other events for the ICI group compared with the odds of developing AEs when using non-ICI drugs listed in the database (18,19).A signal was deemed significant if the lower limit of the 95% CIs for PCR or COR was greater than 1, or if the number of ICI-specific AE pairs (A) was equal to or greater than 3.For the BCPNN analysis, a signal was considered statistically significant if the lower limit of 95% credible interval was above 0 (20) (Table 2).In this study, any AE that was detected by at least one of the four indices (HR, PCR, COR, and IC) was considered a signal.The statistical analyses were conducted using SAS Enterprise Guide (version 7.1; SAS institute, NC, USA) and R version 3.6.2for Windows (R Foundation for Statistical Computing, Vienna, Austria).Significant signals only in the ICI group compared to the non-ICI group were presented.

Signal categorization
The detected AEs from the HIRA database were reviewed independently by two medical oncologists.They assessed the expectedness and frequency of the AEs compared to previous literature (21,22).The detected AEs were categorized based on the following conditions: 1) Well-known irAEs that were established as related to immunotherapy based on previous literature, and/or are associated with the pathogenesis of the disease entity involving autoimmune activation; 2) potential irAEs that were not previously recognized as irAEs, but were considered clinically relevant and important.

Signal verification
Signal verification was conducted using the Clinical Data Warehouse database of Asan Medical Center (AMC) to validate targeted AEs.This validation involved utilizing diagnostic codes and text mining techniques.Recognizing the limitation of the HIRA database, which relies on health insurance claims data, the electronic medical record (EMR) from the tertiary care hospital, AMC, was employed to verify targeted AEs based on symptoms or laboratory abnormalities.The research team, consisting of clinical experts, reviewed and discussed the detected signals to choose the targeted AEs.Access to the CDW database is restricted to inhospital researchers who have received approval from both the Institutional Review Board and the Data Subcommittee.Among 21,205 cancer patients diagnosed with NSCLC, melanoma, or bladder cancer between July 1, 2012 and January 31, 2019 from the hospital EMR database, 9236 patients who received systemic anticancer therapy were identified using diagnostic and prescription codes (Supplementary Figure S1).These patients were classified into two groups: ICI users (n=836) and non-ICI users (n=8,400).The observation period for all participants commenced from their respective first prescription date, and they were followed up until the end of the study period.
In addition, we identified the other validation group of 1,706 ICI users from the hospital EMR database, comprising patients diagnosed with any type of cancer, for clinical text analysis (text mining).This validation group included patients who were not reimbursed for ICI treatment.Then, these 1,706 ICI users were matched in a 1:2 ratio to 3,412 non-ICI users based on age, sex, and cancer subtype to compare the detailed information documented in the narrative chart.
Among the detected AE signals, targeted AEs that could be easily detected using diagnostic codes or diagnostic codes required on prescriptions were searched using the diagnostic code information ('diagnostic code from the hospital EMR data', n=9236).On the other hand, text mining techniques were applied to search for targeted AE signals that could be easily detected in narrative EMR records such as nursing records and progression notes ('text mining applied to the hospital EMR data', n=5118; 1,706 ICI users and 3,412 non-ICI users).Thus, there were not only signals that were verified by both methods, but also other signals excluded from each analysis.
To explore the unstructured data obtained from the EMR, including the clinical notes written by medical staff, we utilized Standard Query Language with the LIKE function in the R package (SQLDF) to identify meaningful words.The operational definition of the verified signals, including the disease category and detailed search terms, can be found in Supplementary Table S2.

Characteristics of the study subjects
A total of 314,423 patients were diagnosed with NSCLC, melanoma, or bladder cancer between July 2014 and January 2019.Among these patients, 25,817 were prescribed with anticancer medications from August 2017 to January 2019.Among them, 4,516 patients received ICIs, while 21,301 patients received non-ICI anticancer medications (Supplementary Figure S2).
Baseline characteristics of the study patients are shown in Supplementary Table S3.While there were statistically significant differences between the ICI group and the non-ICI group in all variables, the differences were not clinically significant.Overall, 74.1% of the patients were male, with a mean (± standard deviation; SD) age of 66.4 (± 10.0) years and a mean (± SD) Charlson comorbidity index (CCI) of 7.5 (± 3.6).The majority of patients had NSCLC (88.5%), while bladder cancer and melanoma accounted for 9.0% and 2.6% of the cases, respectively.In the ICI group, 75.6% of the patients were male, with a mean (± SD) age of 64.9 (± 10.0) years and a mean (± SD) CCI of 8.6 (± 3.6).The majority of patients in this group had NSCLC (83.1%), while bladder cancer and melanoma accounted for 11.4% and 5.6% of the cases, respectively.

Number of detected signals from the National Health Insurance claims database
Figure 1 illustrated the number of detected signals and the logical relations in which the detection criterion was satisfied by each index.Out of the 1,570 screened AEs using the KCD code, a total of 118 distinct AEs were identified as signals associated with ICIs using the HR index (n=115), PCR index (n=67), COR index (n=49), and IC index (n=43).The HR index demonstrated the highest number of detections, followed by PCR, COR, and IC, in descending order.

Detected AE signals 3.3.1 Incidence of detected AE signals
When detected AEs were categorized by the organ system, the ICI group exhibited the highest incidence of AEs in the endocrine, respiratory, dermatologic, and musculoskeletal systems, which included disorders in the thyroid gland (E00-E07, 20.8%), other disorders of the thyroid gland (E07, 16.0%), influenza and pneumonia (J09-J18, 21.1%), pneumonia, (J18, 18.6%), pruritus, (L29, 11.4%), and arthropathies (M00-25, 17.5%) (Table 3; Supplementary Table S4).Mental and behavioral disorders (F00-99) as well as infectious and parasitic diseases (A00-B39) were also common AEs observed in the ICI group, ranging from 5% to 10% in incidence.Hepatic organ-related AEs (K71) were present in approximately 4% of the ICI group.Other AEs, such as neurologic (G00-99), circulatory (I00-99), and digestive diseases (K00-93), were relatively infrequent, with an incidence of less than <5%.The Table 3 displays 43 AE signals that were found to be significant in all four criteria (HR, PCR, COR, IC) among the 118 detected signals of AEs associated with the use of ICIs in the HIRA database.These signals met the threshold for significance across all four indices, indicating a strong association with ICI treatment.

Potential ICI-related AE
Detected AEs that were not previously well-known but judged as clinically relevant and important were defined as potential ICI-   vii) Digestive system-related AEs-acute appendicitis; viii) Skin and subcutaneous tissue-related AEs-eccrine sweat disorders and decubitus ulcer; and viii) Bone disorders-related AEs-osteoporosis with pathological fracture, disorders in the continuity of the bone, fracture of the shoulder and upper arm, fracture of the femur, and fracture of the lower leg, including the ankle (Figure 2B).

Signal verification with hospital EMR database
Among the detected AE signals, signal verification for 17 targeted AEs was performed using additional hospital EMR database to address the limitation of the HIRA claims data (Figure 3; Supplementary Table S2).The incidence of detected AE signals was compared between population-level and hospital-level database.The signal verification was performed in two ways, using the diagnostic code and text mining.
After defining targeted AEs by search term through text mining aside from using diagnostic codes (Supplementary Table S2), AE signals were verified from the hospital EMR database (Figure 3).In patients available for text mining (n=5118; 1,706 ICI users and 3,412 non-ICI users), the detected AEs were i) related to infections: other types of sepsis, herpes zoster, and pneumocystis; ii) related to metabolic disease: disorders in mineral metabolism; iii) related to mental disorders: delirium; iv) related to nervous disease: epilepsy; v) related to circulatory system: cardiac arrest: vi) related to skin and subcutaneous tissue: eccrine sweat disorders; and vii) related to symptom: somnolence and convulsion.These results indicate that AE signals detected from the NHI database were verified from the hospital EMR database, using both diagnostic codes and text mining.

Discussion
In the present study, we utilized both nationwide claims databases and hospital EMR to comprehensively monitor and detect signals of irAEs possibly associated with ICIs.The hospital EMR data, which includes structured and unstructured data such as clinical notes, was utilized to verify the detected irAE signals in the claim data.By integrating these two different real-world databases, we were able to identify irAE signals that were not well-established in the literature and assess their association with the use of ICIs.This integrated approach highlights the importance of utilizing both nationwide claims databases and hospital EMR data in real-world pharmacovigilance system.This comprehensive approach provides valuable insights into the safety profile of ICIs and contributes to the ongoing monitoring and management of irAEs in clinical practice.
Our analysis of claim data revealed the spectrum and incidences of well-known irAEs at the population level.The most common well-known irAEs observed were thyroid disorders, such as hypothyroidism, subclinical hypothyroidism, hyperthyroidism, and thyroiditis (20.8% in the ICI group vs. 7.6% in the non-ICI group; HR = 4.0, 95% CI 3.6-4.4).The incidence of thyroid irAEs in our study appeared to be higher than reported in phase III clinical trials, where the mean incidence was around 10.8% ( 23), but consistent with previous real-world studies, which have reported incidence rates ranging from 13% to 30% (24-26).In addition, our claim data analysis revealed other well-known irAEs across various organ systems, such as dermatologic AEs such as pruritus, dermatitis, and atopic dermatitis, musculoskeletal AEs such as systemic lupus erythematosus and other systemic involvement of connective tissue, gastrointestinal AEs such as hepatitis, respiratory AEs such as pneumonitis, ocular AEs such as iridocyclitis, and neurologic AEs such as meningitis (Figure 2).The incidence of these well-known typical irAEs was similar to previous studies involving PD-1/PD-L1 inhibitors (5,(27)(28)(29)(30).Of note, the incidence of irAEs in respiratory system we detected was higher than that of previously reported (5,31).These findings demonstrate the utility of nationwide claims databases in real-world AE monitoring and risk measurement.Furthermore, our analysis unveiled new AE signals occurring more frequently in patients receiving ICIs compared to those receiving non-ICI agents.In the endocrine system, we identified type 2 diabetes mellitus (DM) (9.0% in the ICI group vs. 8.7% in the non-ICI group; HR = 1.2, 95% CI 1.1-1.4)and Cushing's syndrome (0.1% in the ICI group vs. 0.0% in the non-ICI group; HR = 5.7, 95% CI 1.7-19.8)as a new signal associated with ICI use.While ICIs are known to rarely cause type 1 DM (32, 33), their association with new-onset type 2 DM has not been well established.However, recent studies have indicated that ICIs may exacerbate pre-existing type 2 DM (33,34) and increase the risk of developing incident diabetes (35), which aligns with our findings.Regarding Cushing's syndrome, it has been described in case reports of patients receiving ICIs (36, 37), but the specific details and relationship between Cushing's syndrome and ICIs have not been elucidated due to its rarity.These findings underscore the value of utilizing large-scale population-based claim data to uncover rare AE profiles.Nevertheless, caution is warranted in interpretation, as there is potential for increased steroid administration in ICI-treated patients due to irAEs.This may partially contribute to bias, leading to a higher prevalence of diabetes or Cushing 's disease in the ICI group compared to the non-ICI group.
Another notable finding was the association between ICIs and bone disorders, including osteoporosis with pathological fracture and fractures of various sites.After first case series describing skeletal irAEs associated with ICI in 2018 (46), another largescale pharmacovigilance analysis using the Food and Drug Association (FDA) Adverse Event Reporting System recently suggested a possible cause-effect relationship between serious vertebral fractures and ICI use in patients without apparent preexisting risk factors (47).Emerging evidence suggests that systemic activation of T cells leads to an osteoprotegerin ligandmediated increase in osteoclastogenesis and bone loss.ICIs can enhance bone resorption by activating T cells (48), causing bone fragility and increasing the risk of fractures (49)(50)(51).
The verification of AE signals using the hospital EMR database demonstrated the complementarity of diagnostic codes and text mining.Text mining analysis has the potential of capturing AEs that clinicians have not yet assigned a diagnosis code to, or for which diagnostic codes could not assigned.The integration of these two approaches can enhance the detection of significant AE signals.
To the best of our knowledge, this population-based study is the first to investigate the nationwide incidence of AEs and compare the incidence of AEs between users of ICIs and conventional chemotherapy.Previous large-scale studies utilizing the FDA Adverse Event Reporting System or U.S. commercial insurance database have only reported the incidence of irAEs among ICI users in observational settings (48,52).Moreover, our study included a large number of cancer patients treated with ICIs, surpassing the sample size of most randomized controlled trials (53).In addition, we applied multiple data mining indices and conducted a comprehensive review of the identified signals.If an AE signal was detected by multiple data mining indices, it was considered more robust compared to signals detected by a single index.The majority of signals reported in our study were detected using the HR, which reflects the time until the occurrence of an event.Consistent with our findings, another study that utilized a national health insurance claims database reported that the HR exhibited the highest sensitivity among the data mining indices (15).Therefore, it is recommended to apply multiple indices and compare the results, as it is challenging to determine the superior method of data mining (15, 54).Finally, we validated the signals identified from the hospital EMR database using two approaches: diagnostic codes and text mining.
Our study has several limitations.Firstly, due to the observation nature of this study, there is a possibility of misclassifying AEs.Since AEs were defined based on diagnostic codes, symptoms or nondiagnosed conditions may have gone undetected.Similarly, even if keywords were found in the hospital records, we cannot be certain if AEs actually occurred.Secondly, we were unable to assess the severity of AEs and detailed patient information such as cancer stage, performance status, and treatment history.Thirdly, it is conceivable that AEs associated with other anticancer medications may have been identified after transition to different anticancer medications within 18 months of initiating either ICI or non-ICI.Despite the potential for this bias, our intention was to capture signals for a sufficient duration after beginning study drug administration.The decision was motivated by the fact that various and significant irAEs such as endocrine AEs, colitis, nephritis or hepatitis can manifest at any time, even after the discontinuation of ICI therapy, and may wax and wane over time (55,56).Fourthly, we only included PD1/PD-L1 inhibitors such as nivolumab, pembrolizumab, and atezolizumab.Hence, the safety profiles of other types of ICIs, such as CTLA-4 inhibitors, should be evaluated in the future.Finally, beyond signal detection and validation, which involved comparing the frequency of AEs between the ICI and non-ICI groups, we recognize the absence of in-depth analysis in this study.We are planning further research with more comprehensive investigations, including retrospective analyses using large-scale, long-term hospital databases, and analyses in prospective trials.These will aim to explore potential irAEs identified in our study, such as portal vein thrombosis and bone disorders.

Conclusion
This real-world data analysis utilizing both nationwide claims databases and hospital EMR, has demonstrated an effective approach for signal detection and evaluation of the irAEs.The development of an effective real-world pharmacovigilance system utilizing nationwide claims database can complement existing EMR systems and contribute to the early diagnosis and prompt treatment of rare irAEs.Further research and development of near real-time or earlier access to pharmacovigilance systems are warranted to enhance patient safety and improve outcomes.

FIGURE 1
FIGURE 1 Number of detected signals related to immune checkpoint inhibitor use by data-mining indices in the nationwide claims database.HR, hazard ratio; PCR, proportional claims ratio; COR, claims odds ratio; IC, information component; KCD, Korean Standard Classification of Diseases.* means number of detected signals in mid-level.

2
FIGURE 2 Categorization of detected signals related to immune checkpoint inhibitor use from the nationwide claims database.(A) Well-known immunerelated adverse events.(B) Potential immune-related adverse events.AE, adverse events; HR, Hazard ratio; CI, Confidence interval; PCR, Proportional Claims Ratio; COR, Claims Odds Ratio; IC, Information Component.* detected statistically significant results.

FIGURE 3
FIGURE 3Signal verification from the nationwide claims database and hospital electronic medical record database.AE, adverse events; HR, Hazard ratio; CI, Confidence interval,.

TABLE 1
Study framework of the disproportionality analysis of the health insurance claims database.

TABLE 2
Definition and signal detection criterion for data mining indices.
CI, confidence interval; P, probability.See Table 1 for what A, B, C, and D mean.

TABLE 3
Adverse event signals significant in all four criteria (HR, PCR, COR, IC) in the HIRA database.*The IC value is statistically significant when it is greater than zero.KCD, Korean Standard Classification of Diseases; AE, adverse events; HR, hazard ratio; CI, confidence interval; PCR, proportional claims ratio; COR, claims odds ratio; IC, information component lower credible interval, NEC; not elsewhere classified.
*W is for Well-known immune-related AEs (dark shade), and P is for Potential ICI-related AEs (light shade).*