Background incidence rates from electronic healthcare databases for vaccine safety monitoring: review of challenges from the COVID-19 vaccination campaign and proposal for best practices

Abstract

Background incidence rates (BIRs) are essential for contextualizing adverse event rates in vaccine safety monitoring, particularly through observed-to-expected (O/E) analyses. The unprecedented rapid development of vaccines during the coronavirus disease 2019 (COVID-19) pandemic necessitated rigorous and continuous safety monitoring. While BIRs are traditionally obtained from literature reviews, the pandemic accelerated the large-scale generation of BIRs from electronic healthcare databases through various initiatives such as the vACCine covid-19 monitoring readinESS (ACCESS) and the Biologics Effectiveness and Safety (BEST) to support COVID-19 vaccine safety surveillance strategies. The Beyond COVID-19 Monitoring Excellence (BeCOME) initiative, launched in 2022, established seven working groups, including one focused on best practices for BIR generation and utilization in pharmacovigilance activities. The BeCOME BIR working group conducted a targeted literature review to enable focused analysis of challenges that emerged during large-scale vaccination campaigns through December 2023. The group also used structured group discussion following the nominal group technique over 10 months to develop consensus-based judgments on critical factors for BIR best practices. The findings were organized into four domains: key initiatives summary, pandemic-related challenges, implications for O/E analyses, and best practice recommendations. To identify key BIR initiatives supporting COVID-19 vaccine safety assessment, the review employed multiple strategies including scientific literature examination, public health authority website reviews, and reference list searches; data on study characteristics, limitations, challenges, and recommendations were extracted. The targeted review focused on five major initiatives, such as ACCESS and BEST, that generated BIRs for adverse events of special interest (AESIs) during the pandemic. The group identified persistent challenges during the vaccination campaign including timeliness constraints during rapid vaccine deployment, substantial heterogeneity across data sources, inconsistent case definitions, limited information for key subpopulations, and difficulties addressing emerging AESIs. These challenges directly impacted O/E analyses, potentially leading to biased safety signal assessments. We propose a coordinated action plan among key stakeholders to establish sustainable mechanisms for regular BIR delivery with methodological improvements, develop consensus on best practices for BIR selection, and secure resources to ensure pandemic preparedness. Implementing these recommendations will strengthen vaccine safety monitoring systems for both routine vaccination programs and future public health emergencies.

Introduction

Background incidence rates (BIRs) of potential adverse events of special interest (AESIs) - rates at which these outcomes naturally occur in the general or specific populations of interest - are important for contextualizing the rates observed in populations exposed to vaccines or drugs of interest (Black et al., 2021). BIR data play a key role in safety monitoring across various settings, including clinical trials, observational real-world evidence (RWE) studies, and spontaneous reporting systems. They are also central to conducting advanced quantitative analyses, such as observed-to-expected (O/E) analyses. In the context of vaccines, O/E analyses compare the observed number of adverse events in a vaccinated population (typically obtained from a spontaneous reporting system) to the expected number calculated using BIRs from general or specific populations (comparator population) to assess safety signals (Mahaux et al., 2016).

After the World Health Organization (WHO) declared the coronavirus disease 19 (COVID-19) outbreak caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) a global pandemic on 11 March 2020 (World Health Organization, 2025), COVID-19 vaccines were developed at an unprecedented speed and were made available for mass vaccination campaigns within a year (Ledford et al., 2020; Fortner and Schumacher, 2021). While the rapid development of vaccines was a crucial tool for controlling the pandemic, it also underscored the need for rigorous and ongoing safety monitoring. As with all medicinal products, monitoring the safety of COVID-19 vaccines in real-world settings, in addition to data generated through clinical trials, was (and still is) a key component of pharmacovigilance (PV). Real-world monitoring is particularly important since rare adverse events not detected/detectable during the clinical development phases could emerge during large-scale vaccination campaigns.

Public health or national health institutions, regulatory agencies, academic institutions, healthcare organizations, and vaccine manufacturers proactively developed vaccine safety surveillance strategies in anticipation of the COVID-19 vaccine rollout during the pandemic. Lists of AESIs for COVID-19 vaccines were issued by multiple organizations, including but not limited to the Safety Platform for Emergency Vaccines (SPEAC), the US Centers for Disease Control and Prevention (CDC), the European Medicines Agency (EMA), the US Food and Drug Administration (FDA), and the Medicines and Healthcare products Regulatory Agency (MHRA) in the UK (Black et al., 2021). These lists were extensive, covering multiple systems—neurologic, immunologic, cardiologic, hematologic, among others—due to the emerging nature of the COVID-19 disease, and the use of different and/or novel vaccine platforms. They were also dynamic and not necessarily consistent, with some AESIs being included in certain lists but not in others (Black et al., 2021). In addition, there was no single definition of AESIs. For example, the WHO defines an AESI in the context of COVID-19 as “a pre-specified medically-significant event that has the potential to be causally associated with a vaccine product that needs to be carefully monitored and confirmed by further special studies” with an emphasis of specificity (World Health Organization, 2020). In contrast, the Council for International Organizations of Medical Sciences (CIOMS) adopts a more sensitivity-focused approach, defining an AESI as a noteworthy event related to a specific product or class of products that a sponsor wants to monitor carefully. These events could be either serious or non-serious and might include potential precursors or prodromes for more serious medical condition (Council for International Organizations of Medical Sciences CIOMS, 2023). The heterogeneity of AESI lists might have reflected different interpretations of AESI definitions across these organizations during the pandemic, thus affecting operational implementation across regulatory and public health settings.

Prior to the COVID-19 pandemic, BIRs for given AESIs were typically obtained from literature reviews of published population-based epidemiological studies or generated from retrospective analyses of electronic healthcare databases (Black et al., 2021; Mahaux et al., 2016; Esposito et al., 2018; Gubernot et al., 2021; Klein et al., 2010; Willame et al., 2021a). However, this approach is constrained by inconsistent design or methodological choices across studies and reliance on historical data might not have reflected current population health patterns or emerging conditions. The urgency of the COVID-19 pandemic drove a move toward large-scale generation of BIRs from electronic healthcare databases, enabling rapid evaluation of potential safety signals from the extensive and evolving lists of AESIs. To prepare for the aforementioned safety surveillance strategies for COVID-19 vaccines, various initiatives were launched over the course of the pandemic to generate BIRs for many overlapping AESIs from electronic healthcare databases to support analyses by health authorities and marketing authorization holders (MAHs) in line with regulatory requirements for systematic O/E analyses in their risk management plans. In Europe, the EMA funded the vACCine covid-19 monitoring readinESS (ACCESS) project, which generated rates for 41 AESIs in 10 European healthcare databases (Willame et al., 2023; Willame et al., 2021b). In the US, the FDA Biologics Effectiveness and Safety (BEST) initiative generated BIRs for 17 AESIs in six US administrative claims databases (Moll et al., 2023; Tworkoski et al., 2021). In addition, the Observational Health Data Sciences and Informatics (OHDSI) community generated and published rates of 15 AESIs in eight countries in 2021 (Li et al., 2021), and rates of 16 AESIs in eleven countries in 2023 (Voss et al., 2023). The Global COVID Vaccine Safety (GCoVS) project was established in 2021 under the multinational Global Vaccine Data Network (GVDN) consortium with the aim to gather rates on 13 AESIs in 9 countries (Phillips et al., 2023). Other groups from Canada, Korea, Scotland, and Australia also published BIRs of prespecified AESIs (Nasreen et al., 2021; Nasreen et al., 2022; Huh et al., 2021; Cullen et al., 2023; Pillsbury et al., 2023). The abundance of newly available BIR data combined with BIRs from existing literature facilitated the ability to conduct O/E analyses while also presenting challenges in utilizing and interpreting the results.

Beyond COVID-19 Monitoring Excellence (BeCOME) was launched in 2022 to identify lessons learned from non-competitive cooperation among vaccine manufacturers during the COVID-19 pandemic regarding the real-world monitoring of benefits and risks of vaccines (Bauchau et al., 2023; Bauchau et al., 2025). Seven working groups were established, with one dedicated to identifying common challenges, possible solutions, and priorities across stakeholders for sustainable BIR generation from electronic healthcare databases and its utilization. Specific objectives of this BIR working group were to (i) identify and summarize information on published initiatives that were conducted to generate BIRs for COVID-19 vaccine safety monitoring, (ii) review challenges encountered in generating and using BIRs in the framework of PV activities, and (iii) propose best practices for BIR generation and use. This paper summarizes the working group’s findings in relation to these objectives and offers a set of broad recommendations intended to cover the PV scope of any vaccine, both within and outside the context of pandemic preparedness, drawing from the experience gained during the COVID-19 pandemic.

Methods

Working group structure and experience

The manuscript was prepared by nine volunteer members of the BeCOME BIR working group, who are current or former PV and pharmacoepidemiology experts of various research affiliations (pharmaceutical companies, contract research organizations and independent consultancy) based in Canada, Europe, UK and US with specific experience in COVID-19 vaccine safety surveillance and/or safety studies. While this manuscript describes MAH experiences and lessons learned, objectivity is maintained through the use of published evidence, systematic consensus development, and integration of diverse stakeholder perspectives (see targeted literature review and consensus approach below for methodological details).

Identification of published initiatives on BIRs through targeted literature review

Rather than conducting a systematic review, a targeted literature review approach was selected and conducted by working group members to identify and synthesize key initiatives that generated BIRs to support the assessment of COVID-19 vaccine safety by the MAHs during the WHO’s Public Health Emergency of International Concern (ending on 5 May 2023) (World Health Organization, 2025). The search covered publications and reports available through 31 December 2023. This approach enabled a focused analysis and review of the challenges encountered by the MAHs during post-marketing monitoring, with particular emphasis on issues that emerged during large-scale vaccination campaigns. To this end, the working group used multiple strategies including (i) a targeted search and review of the scientific literature that were known, available and actively used by the MAHs during this specific period (ii) a targeted search and review of prespecified Public Health and Health Authority websites such as EMA, FDA, and CDC and (iii) a snowball search of the reference lists from the reviewed documents in the first two strategies. Additionally, BIR resources and knowledge were exchanged with other subject matter experts or task forces within trade associations including International Federation of Pharmaceutical Manufacturers and Associations (IFPMA) and Vaccines Europe – a specialized group of European Federation of Pharmaceutical Industries and Associations (EFPIA) (Bauchau et al., 2023). As the targeted literature component focused on identifying key BIR publications and datasets that were actively utilized by the MAHs, they were determined through the approaches outlined above. A formal systematic literature review was beyond the scope of this work; this is acknowledged in the limitations section.

Data were extracted on characteristics of the key initiatives from their reports, publications, dashboard or web apps, where available. These initiatives were selected based on their collaborative efforts to produce general population-based incidence rates across diverse geographic regions or countries, while establishing the infrastructure to integrate multiple data sources including health insurance, general practice, hospital, and registry data. For each initiative and where available, study characteristics (study countries, observation period, type of data sources, study population, number of AESI, etc.), information describing challenges, limitations, and recommendations for future practice were summarized.

Consensus approach to challenges and recommendations for best practices

Results

Summary of key initiatives that generated BIRs during the COVID-19 pandemic

The BIR working group identified five key initiatives: ACCESS (Willame et al., 2023; Willame et al., 2021b), GVDN (Phillips et al., 2023), OHDSI (Li et al., 2021; Voss et al., 2023), BEST (Moll et al., 2023; Tworkoski et al., 2021), and Ontario studies (Nasreen et al., 2021; Nasreen et al., 2022). These initiatives generated BIRs for varying numbers of AESIs ranging from 8 (first Ontario study) (Nasreen et al., 2021) to 41 (ACCESS) (Williame et al., 2023; Williame et al., 2021b), with significant overlap among them. These BIRs were widely used by vaccine manufacturers to support COVID-19 vaccine safety monitoring during the pandemic period and continue to be used up to the present for other vaccines as well. While additional BIR publications from countries such as Korea, Scotland, and Australia were identified during this period (Huh et al., 2021; Cullen et al., 2023; Pillsbury et al., 2023), they were not considered as key initiatives due to their limited applicability for the MAHs. Table 1 summarizes these initiatives. The detailed description of these initiatives, including their respective reports, dashboards or publications on BIRs for selected AESIs, is provided in Supplementary Table S1.

ACCESS, GVDN, and OHDSI generated BIRs using different types of databases from multiple countries (primarily from Europe or North America), while BEST utilized exclusively US claims databases and Ontario studies generated BIRs from a single Canadian provincial linked health administrative database. These initiatives primarily targeted general pediatric and adult populations, with ACCESS and BEST additionally generating BIRs for specific subpopulations. For example, ACCESS covered subpopulations with underlying conditions who were at risk for developing severe COVID-19 and pregnant women (Willame et al., 2023; Willame et al., 2021b), and BEST included subpopulations such as nursing home residents and influenza vaccinated individuals (Moll et al., 2023; Tworkoski et al., 2021). Although all these initiatives generated BIRs stratified by age and sex separately, only the GVDN dashboard (Global Vaccine Data Network, 2023), OHDSI studies (Li et al., 2021; Voss et al., 2023), and Ontario studies (Nasreen et al., 2021; Nasreen et al., 2022) provided BIRs with dual stratification by both age and sex. Depending on data source and study design, these initiatives also generated BIRs stratified by other factors such as time periods (pre-COVID and peri-COVID), data source types (electronic health records [EHR], claims, registries), and/or healthcare settings (inpatients, emergency department visits, outpatients). Due to the urgent need to monitor COVID-19 vaccine safety, ACCESS and BEST released their reports in 2021 (Willame et al., 2021b; Tworkoski et al., 2021) and subsequently published their findings as manuscripts in 2022 (Willame et al., 2023; Moll et al., 2023). OHDSI and Ontario studies published their initial manuscripts in 2021 (Li et al., 2021; Nasreen et al., 2021), while GVDN published their BIRs in 2023 (Phillips et al., 2023) and simultaneously released them on their public dashboard (personal communication, May 2025).

The initiatives acknowledged several common challenges or limitations. These included the inability to validate case definitions since diagnostic codes do not always represent true disease occurrence, variable data quality and completeness across sources, and considerable variability (heterogeneity) in BIRs across populations, regions, and data sources. Some initiatives faced specific challenges related to their data sources. For example, governance issues (such as administrative constraints or lengthy governance approval process) affected French data access in the ACCESS initiative (Willame et al., 2021b), resulting in delay which was especially problematic during a pandemic response. In addition, the COVID-19 pandemic introduced specific challenges affecting BIRs, such as altered healthcare utilization patterns due to resourcing constraints, limits on elective procedures/encounters during the early pandemic months and increasing use of virtual healthcare. These factors could potentially lead to underestimation of less severe events. Overall, these challenges or limitations underscore the complexity of establishing reliable BIRs for vaccine safety monitoring during pandemic situations and emphasize the need to consider the context for generation of BIRs and careful interpretation when using these data for safety signal detection.

Additionally, the ACCESS, GVDN, OHDSI, and BEST initiatives offered high-level recommendations for generation and/or utilization of an appropriate BIR: (i) use the same database for comparing background and post-vaccination rates; (ii) select data sources with the most complete event identification; (iii) consider population characteristics when selecting appropriate BIRs; and (iv) account for temporal trends and seasonal variations when selecting BIRs.

Challenges encountered during the COVID-19 vaccination campaign

The unprecedented scale and urgency of the COVID-19 vaccination campaign presented new levels of challenges for vaccine safety surveillance worldwide, although some similar challenges had been experienced during the 2009 H1N1 pandemic. Furthermore, the pandemic environment created multiple obstacles to effectively generate and utilize these rates for the extensive and evolving lists of AESIs. Built on the review of key initiatives that generated the BIRs during the COVID-19 pandemic, five major challenges were further identified through a combination of the BIR working group members’ professional experience and structured discussions and collaborative discussions held during the inaugural BeCOME meeting. These challenges are described below.

Timeliness

Timeliness refers to whether the data are collected and available at the right time for its intended use (Data Analytics and Methods Task Force EMA, 2023). Vaccines manufacturers developed COVID-19 vaccines at an unprecedented speed, making them available for mass vaccination campaigns within a year after the pandemic began (Ledford et al., 2020; Fortner and Schumacher, 2021). However, some vaccines received authorization for use earlier than others and their MAHs had regulatory obligations to rapidly initiate passive and active safety surveillance based on RWE generation. At the time of administration of the first COVID-19 vaccines in December 2020, BIR data from large scale electronic healthcare database extraction were not yet publicly available. For example, OHDSI’s publication by Li et al. (2021) became available in June 2021, and the ACCESS final report (v2.0) was released in August 2021 (Willame et al., 2021b) (Figure 1). The timing and availability of BIRs likely reflected differences in stakeholder priorities: regulatory agencies required early data for product safety monitoring, while other organizations focused on longer-term immunization program monitoring objectives.

FIGURE 1

To enable an early start and fulfilment of regulatory systematic O/E analyses requirements, MAHs, through non-competitive collaboration in the framework of the COVID Research and Development Alliance - PV and Vaccines Europe PV subgroups (COVID Research and Development Alliance, 2022; Vaccines Europe, 2020), shared experience on BIRs to achieve alignment with those obtained from published population-based epidemiological studies. As the BIR data generated from large-scale healthcare database initiatives became available over time, they were progressively incorporated as an additional source for systematic O/E analyses and other safety signal contextualization addressing evolving regulatory requirements. Specifically, the EMA requested the use of the BIRs generated from the ACCESS initiative to perform O/E analyses to be included in COVID-19 vaccine monthly summary safety reports (SSR), as outlined in core RMP19 requirements and guidance v3.1 (European Medicine Agency, 2022).

Heterogeneity

Heterogeneity in BIRs refers to the variability or differences in the rates of a single AESI across different populations, regions, or time periods. Among the key initiatives presented in Table 1, large variability of BIRs was observed across all multi-database studies, regardless of whether they used multiple data sources from different countries or within the same country.

Despite the implementation of a common protocol, common data models, and standardized analysis methods in multi-database initiatives, substantial disparities in BIRs for the same AESIs emerged. For instance, there was a six-fold difference in the 2019 BIRs of transverse myelitis between the lowest and highest estimates across eight European databases spanning four countries in the ACCESS initiative (Willame et al., 2021b) (Figure 2). Similar variability in BIRs was observed across different initiatives, even when using similar data sources and outcome definitions within comparable observation periods. For example, when examining transverse myelitis across BEST (Moll et al., 2023) and OHDSI (Li et al., 2021) initiatives utilizing US Optum’s pre-adjudicated commercial claims and Optum’s De-identified Clinformatics Data Mart, respectively, the BIR varied considerably across age groups. In both studies, individuals aged 55–64 years exhibited the highest rates, yet the rates differed by 4 to 6-fold between the studies (Figure 3).

FIGURE 2

Willame et al., 2021b). ACCESS, vACCine covid-19 monitoringreadinESS; BIR, backgroundincidencerate; CI, confidence interval.

FIGURE 3

Moll et al., 2023; Li et al., 2021). (A) FDA BEST Optum pre-adjudicated commercial claims in 2019. (B) OHDSI de-identified Clinformatics Data Mart Socio-Economic Status from 2017-2019. BEST, Biologics Effectiveness and Safety; FDA, Food and Drug Administration; ICD‐10‐CM, International Classification of Diseases, 10th Revision, Clinical modification; OHDSI, Observational Health Data Sciences and Informatics; US, United States.

While the variability in results might reflect genuine differences between studied populations and offer valuable insights, heterogeneity caused by other factors (e.g., methodological differences) complicates the selection of an appropriate BIR (Table 2). Disparities can arise from different coding systems, healthcare practices, provenance of diagnosis (primary care, hospitalized care), and type of electronic healthcare database (claims, EHR), which have all been shown to impact BIR estimates (Willame et al., 2023; Ostropolets et al., 2022; Russek et al., 2023).

One study published in 2022 systematically examined the influence of various parameters on BIR calculation. This study used 12 data sources for 15 AESIs related to COVID-19 vaccines as an example, and found that in addition to age and sex, study design choices such as anchoring (e.g., healthcare visit, vaccination, or arbitrary date), clean window choice (minimum interval to classify an event as incident), and time-at-risk duration or risk window (risk period for the event to occur) significantly influence BIR estimates (Ostropolets et al., 2022).

Another recent case study of BIRs for venous thromboembolism specifically assessed between-database heterogeneity using 11 healthcare databases across six European countries. This study found that databases collecting data from different segments of the healthcare system contributed most significantly to heterogeneity in BIR estimates. Moreover, the authors advise that when pooling data, careful consideration between stratified and pooled estimates is necessary for effective pharmacoepidemiological and regulatory evaluations (Russek et al., 2023).

Absence of consistent AESI algorithms and validation

Code-based algorithms are typically used to identify outcomes in large electronic healthcare databases. While some AESIs can be easily identified using algorithms consisting of existing simple diagnosis codes (e.g., Guillain-Barré Syndrome, myocardial infarction), others may require using or developing more complex algorithms. This is particularly true for syndromic conditions like thrombosis with thrombocytopenia syndrome (TTS) or very specific/rare adverse events such as corneal transplant rejection, where even specific diagnosis codes may not exist.

It is important to develop accurate and sometimes very complex algorithms based on available diagnostic codes, laboratory tests, drug treatments, procedures, etc. to align AESI case definitions with clinical, Brighton Collaboration, or Medical Dictionary for Regulatory Activities (MedDRA) terms. However, data on specific diagnostic tests or procedures required for accurate AESI identification may not be collected/coded in healthcare databases sometimes, or even not be generated at the healthcare provider level. For example, US claims databases often lack laboratory test results in general, while US EHR databases may have incomplete patient histories. Therefore, reconciling case definitions used for BIR calculations in healthcare databases with those employed in spontaneous reporting systems or clinical trials may pose significant methodological challenges, e.g., for O/E analyses.

Many published BIRs are “crude” rates using unvalidated code-based algorithms, potentially introducing misclassification bias. This limitation is explicitly acknowledged by all initiatives (Supplementary Table S1). In addition, when risk of outcome misclassification is expected, it is important to validate these algorithms to inform on their accuracy in identifying outcomes of interest. Medical chart extraction and review is often considered the gold standard outcome validation method, but usually represents a lengthy and resource intensive process, especially when conducting large scale BIR estimation. It may also be infeasible, especially for rare outcomes, or when none or few of the relevant medical charts are accessible (Praet et al., 2024).

Moreover, code-based algorithm performances (sensitivity and specificity) can vary significantly across database type (e.g., EHR vs. claims databases) due to differences in data elements and quality. Alternative validation methods, such as validation through database linkage, longitudinal patient profile review, or generation of a probabilistic reference standard (e.g., PheValuator) are available to circumvent some of these technical and operational limitations, especially when BIR generation is time-sensitive (Praet et al., 2024).

Information unavailable for specific subpopulations or geographic locations

Most publications provided BIRs only for general population and lacked estimates within specific subpopulations such as racial/ethnic subgroups, pregnant women, or populations with pre-specified co-morbidities, etc. Although all five initiatives in Table 1 included age- and/or sex-stratified BIRs, some lacked the granular data necessary for robust safety signal assessment. For instance, while the Ontario study provided more granular estimates for paediatric population (including 0-4, 5-11, 12-15 and 16–19 years) (Nasreen et al., 2022), ACCESS used a broad category of 0–19 years (Willame et al., 2023; Willame et al., 2021b) and BEST used <18 years (Moll et al., 2023; Tworkoski et al., 2021).

Furthermore, BIRs are usually not available for many low- and middle-income countries (LMICs), particularly Africa, as compared to Europe and North America. These gaps stem from two main factors. First, the compressed timelines during public health emergencies made generating comprehensive background rates across various subpopulations impractical, as it would have delayed rapid response. Second, some regions, particularly LMICs, have limited or absent infrastructure for electronic healthcare data and resources. Together, these challenges hinder effective safety signal evaluation in specific populations and LMICs. This disparity in BIR availability raises important ethical concerns about equitable safety monitoring, as populations in LMICs may receive less comprehensive post-marketing surveillance despite potentially facing different risk profiles or healthcare contexts. Addressing these disparities require targeted solutions including capacity-building initiatives, enhanced international support, and collaborative partnerships to ensure equitable vaccine safety monitoring globally.

Emerging AESIs or potential risks

During the COVID-19 pandemic, the rapid accumulation of new knowledge and the need for swift decision-making created a unique environment where the evolving list of AESIs due to emerging or potential risks, such as TTS and vaccine-associated enhanced disease (VAED), among others, posed another significant challenge for risk assessment.

Thrombosis with thrombocytopenia syndrome is an extremely rare, potentially life-threatening novel adverse event following adenovirus vector COVID-19 vaccination. After initial cases reported in March 2021 (Greinacher et al., 2021; Schultz et al., 2021), TTS case definitions and management guidelines were developed quickly. The Brighton Collaboration published an interim TTS case definition in May 2021 (Brighton Collaboration, 2021), with subsequent updates in November 2021 and again later in December 2023 (Schonborn et al., 2024). In parallel, the WHO published an interim guidance for clinical case management of TTS in July 2021 (World Health Organization, 2021) and updated it in July 2023 (World Health Organization, 2023). In addition to the rapidly evolving TTS case definition, availability of information in healthcare databases might be limited (lack of information on D-dimer levels, anti-PF4 antibodies, platelet counts, and timing of thrombocytopenia diagnosis) to establish an algorithm that has sufficient specificity for robust assessment of the incidence of vaccine-induced TTS. Consequently, estimating the BIR for TTS by utilization of a standardized approach across different data sources and time periods was and remains a challenge.

Vaccine-associated enhanced disease occurs when an individual who has received a vaccine develops a more severe form of the disease upon subsequent exposed to the virus, compared to unvaccinated individuals (Munoz et al., 2021). While this phenomenon was first observed with an investigational respiratory syncytial virus vaccine (Kim et al., 1969) and an inactivated measles vaccine in the 1960s (Fulginiti et al., 1967), VAED risk for COVID-19 vaccines remains theoretical (Gartlan et al., 2022). To date, no evidence of VAED from COVID-19 vaccines has been found from clinical trials or post-marketing surveillance. Recognizing the potential risk, the Brighton Collaboration formed a VAED working group of multidisciplinary experts in March 2020 (Munoz et al., 2021). The working group acknowledged significant diagnostic challenges, noting the absence of confirmatory tests for VAED, and the potential for confusion with vaccine failure as both may present with similar clinical symptoms after vaccination. Despite establishing three levels of diagnostic certainty, generating appropriate BIRs for VAED applied for any vaccines remains problematic or impossible as the case definition composition varies.

Implications of BIRs on O/E analyses

During the COVID-19 pandemic, regulatory requirements mandated COVID-19 vaccine manufacturers to conduct O/E analyses on numerous prespecified AESIs as part of their signal detection activities (European Medicine Agency, 2022). This deviated from the traditional application of O/E analyses, which typically focuses on signal refinement for safety concerns arising from various sources (Mahaux et al., 2016). The results and interpretations of these O/E analyses were required to be included in the monthly SSR to facilitate a rapid review of the safety profile and enable swift signal identification for COVID-19 vaccines.

As a critical parameter of O/E analysis, the BIRs should closely reflect the true incidence in the target population with minimal bias. Any inaccuracies in the BIRs can substantially distort the O/E ratio and consequently affect the interpretation of safety signals. The absence of standardized methods for selecting a suitable BIR led different organizations to apply varying criteria when choosing a BIR estimate, making it difficult to compare O/E results across different COVID-19 vaccines. However, stakeholders may use BIRs differently in O/E analysis, depending on their objectives. For example, vaccine manufacturers may conduct safety signal assessments during clinical development or post-marketing surveillance, while regulatory agencies use BIRs to monitor the safety profile of vaccines under development or newly approved vaccine products for regulatory decisions.

The selection of an appropriate BIR was further complicated by the heterogeneity in the estimates, as described above. EMA strongly recommended using ACCESS-derived BIRs in Europe (European Medicine Agency, 2022), but significant inter-source variability was observed, as illustrated in Figure 2. This variability introduced uncertainties in calculating the number of expected AESI cases, which consequently affected the O/E ratio. For instance, assuming 5 million COVID-19 vaccine doses administered and a fixed 42-day risk window for transverse myelitis, the minimum number of observed cases needed for a statistically significant O/E ratio varies substantially based on the selected ACCESS BIR: 4-6 reported cases when using lower BIRs (0.14-0.33 per 100,000 person-years) versus 8-11 cases with higher BIRs (0.57-0.92 per 100,000 person-years). This variability could lead to biased O/E estimates and incorrect safety signal assessments. Therefore, multiple sensitivity analyses across a range of BIR estimates were needed to address these uncertainties.

The availability of granular-level BIRs (e.g., age- and sex-specific BIRs) was also essential. For example, with myocarditis, one of the prespecified AESIs under investigation in the O/E analysis requested by regulators, age- and sex-stratified analyses proved particularly important. While overall O/E analyses indicated no elevated ratios, age-and sex-specific analyses revealed statistically significant O/E ratios within the younger age group, predominantly in men following the second dose of mRNA COVID-19 vaccines (European Medicines Agency, 2021). This finding underscores the importance of stratified BIR and granular vaccine exposure data for effectively generating evidence while fine tuning potential safety signals.

Beyond BIR selection challenges, it should be noted that O/E analyses also face other challenges and methodological considerations when estimating the “observed” component from passive reporting systems. These challenges and proposed improvements to current analytical methods are detailed in a separate publication by the BeCOME dedicated working group on O/E analyses (Serradell et al., 2025).

Recommendations for best practices: practical guide

Background incidence rates serve as a cornerstone of vaccine safety surveillance, providing essential context for interpreting adverse events following vaccination. By establishing a critical baseline for O/E analyses, BIRs help distinguish true safety signals from coincidental events that would have occurred regardless of vaccination. This baseline provides the context necessary for effective safety signal assessment and supports evidence-based regulatory decision-making. Therefore, generating unbiased and precise BIRs in target populations is essential.

For a BIR to be unbiased and precise, the observed BIR value would have minimal uncertainty from the two fundamental sources of error: (i) random error (reduces with increasing sample size and is reflected in the width of the confidence interval) and (ii) systematic error (not affected by sample size and results in underestimation or overestimation of the true incidence) (Figure 4). Random error occurs by chance and in an unpredictable direction. On the other hand, systematic error causes BIRs to deviate from the true incidence in directions that may be predicted based on the type of systematic error.

FIGURE 4

To inform the best practices in estimating BIRs, two factors should be considered: (i) is a candidate bias certain to be present? and (ii) if present, is the candidate bias likely to introduce a small or large deviation from the true incidence? Generators and users of BIRs should therefore consider which major sources of error are likely to profoundly impact the results and which may have more inconsequential effects. For example, random error may well be the highest priority if estimating a BIR from a small database (e.g., for rare diseases), whereas if estimating a BIR using large databases, minimizing sources of systematic error may be most important. Factors such as the condition being studied, the context of care, the level of missingness, and the specific research question all play a role in determining which data source to prioritize when BIRs are heterogeneous (Clothier et al., 2024).

While random error can be reduced by increasing the sample size, when possible, systematic error requires careful consideration of many epidemiologic study design principles. Table 3 presents the BeCOME BIR working group expert opinion on a decision tool to reduce systematic error through study design that applies both in producing new BIRs from electronic healthcare databases and in assessing the appropriateness of published BIRs for a given use case. The suggested decision tool includes five methodological categories: geographic area, time period, sampling scheme, phenotype validation, and coverage of healthcare settings.

Researchers are encouraged to assess the relative importance of each category when generating or using BIRs. For example, if an AESI has recently undergone a major change in diagnostic practices, time period should be considered a key priority, since it would largely bias the estimated BIR to use data from prior to the diagnostic change. To aid in assessing BIR appropriateness, we suggest assigning a quality score of 1 (low weight) through 3 (high weight) to each methodological category based on the specific scenario and recommendation. Scores can thus range from 5 to 15 (5: all scores = 1; 15: all scores = 3). Higher total scores can be generally interpreted as having less bias compared to the optimal BIR in the given application. However, critical consideration of all sources of bias, the specific objective, and intended use beyond this simplified tool is necessary in downstream uses of BIRs. Table 3 includes an illustrative example assessing a published BIR for autoimmune hepatitis (AIH) (Gronbaek et al., 2020) to be used in a hypothetical O/E analysis in the US. In this example, the BIR of AIH received a total score of 11 which can be interpreted as appropriately robust to proceed with the O/E analysis, noting potential bias introduced through extrapolation of the BIR from the UK to the US and potential measurement error due to lack of validation of the coding algorithm used to define cases.

It is imperative that all factors in Table 3 be considered together and in the context of the use case. Specifically, it is possible that a published BIR may have a high total score suggestive of low risk of biases, yet there is a key consideration that deems the particular BIR not fit-for-purpose. For example, a high-quality study using recent population-based data may have produced a BIR that is substantially higher than other published rates. For applications to PV where the goal is to be conservative in order to capture safety signals, use of a high outlier BIR may not be recommended despite receiving a high-quality score.

Background incidence rates generated from systematic reviews with meta-analyses were not considered in scope for the application of Table 3. It may be preferable to use such systematic reviews as a source to identify a single primary study of high quality and use that in downstream analyses for the relevant population rather than directly applying the meta-analytic BIR, which may be heterogenous. Considering the multifactorial study design elements critically will aid in the goal of selecting and/or generating BIRs that are unbiased and precise and therefore serve as good proxies for the source population BIRs.

It should be noted that this tool may encounter feasibility challenges, particularly in LMICs where BIRs are frequently unavailable. This highlights broader ethical concerns about inequality in vaccine safety monitoring capabilities between regions. The BeCOME LMIC working group has identified challenges in resource-constrained settings, including inadequate infrastructure, insufficient trained personnel, and poor stakeholder coordination (Bauchau et al., 2025). To address these challenges, strategic priorities have been identified: securing sustainable human and financial resources for PV systems; developing context-appropriate training programs for local staff; and fostering coordination among multiple stakeholders (e.g., vaccine manufacturers, donors, non-governmental organizations, regulators, and health ministries) (Bauchau et al., 2025). While international collaborations and WHO support are crucial, they must focus on building local capacity rather than creating dependency. Implementing these strategies will enable local BIR generation and strengthen indigenous expertise, advancing equitable vaccine safety monitoring across all populations regardless of economic circumstances.

In addition to the recommendations in Table 3, BIR generation from electronic healthcare databases should adhere to established RWE study guidelines including, but not limited to, recommendations issued by the EMA, the FDA, and the CIOMS Working Group XIII (EMA, 2024; FDA, 2023; Hennessy et al., 2025). Digitalization of healthcare data has expanded over the past few decades, allowing proactive development of more granular subpopulation data during non-emergency periods. These efforts would enhance preparedness for future public health emergencies by enabling more nuanced and equitable safety signal assessment across diverse populations. Although valuable, even BIRs derived from robust, recent, population-based studies with validated case definitions have inherent limitations that must be considered in BIR utilization and interpretation, particularly when utilized for O/E analyses.

Methodological and contextual limitations

Several limitations warrant explicit acknowledgment. First, the methodological transparency of our targeted review process may be limited, as we did not employ a systematic review with predefined search strategies and inclusion criteria. However, this approach allowed us to conduct a focused analysis of challenges that emerged during large-scale vaccination campaigns. Second, our analysis predominantly relies on key initiatives where datasets originate primarily from high-income countries, which may not adequately represent the situations in resource-constrained settings such as LMICs. Third, some initiatives discussed depend on case definitions that have not been fully validated across diverse populations and healthcare contexts, potentially affecting the accuracy and comparability of BIRs. Finally, this paper presents experiences and perspectives from industry professionals. As stated in methods, we have endeavored to provide a balanced and objective assessment throughout this work. We encourage readers to interpret our findings and recommendations within this context, recognizing the unique insights that industry experience brings to understanding the practical challenges and solutions in vaccine safety monitoring. These limitations underscore the need for continued collaboration between industry, academia, regulatory agencies, and public health organizations to ensure comprehensive and unbiased approaches to vaccine safety surveillance.

Conclusion

The COVID-19 pandemic highlighted both the critical importance and significant challenges of generating and utilizing BIRs for vaccine safety monitoring. Through collaborative initiatives such as ACCESS, BEST, OHDSI, GVDN, and others, substantial progress was made in developing BIR resources across multiple countries and healthcare databases. However, these efforts revealed persistent challenges including timeliness constraints, heterogeneity across data sources, limited or no BIRs available for key subpopulations and populations in LMICs, inconsistent or unvalidated case definitions, and difficulties addressing emerging AESIs. These challenges directly impacted O/E analyses, potentially leading to biased signal detection or assessment.

Moving forward, we propose a coordinated action plan among key stakeholders (regulatory agencies, industry, and BIR producers) to: (i) establish sustainable mechanisms for regular, periodic BIR delivery with continuous methodological improvements (such as harmonized case definitions, validation of AESI algorithms, etc.), enhanced stratification capabilities, and coverage of new AESIs; (ii) develop consensus on best practices for BIR selection and develop robust instruments to help guide decisions on relevance of factors and how they should be handled; and (iii) secure upfront resources to ensure sustainability and pandemic preparedness. Proactive development of more granular subpopulation data and data from LMICs during non-emergency periods would significantly enhance preparedness for future public health emergencies, enabling more nuanced and equitable safety signal assessment across diverse populations. By implementing these recommendations, we can build more robust vaccine safety monitoring systems capable of responding effectively to future public health emergencies while maintaining the highest standards of PV during routine vaccination programs.

References

• 1

  BauchauV.DavisK.FriseS.Jouquelet-RoyerC.WilkinsJ. (2023). Real-world monitoring of COVID-19 vaccines: an industry expert view on the successes, challenges, and future opportunities. Drug Saf.46 (4), 327–333. 10.1007/s40264-023-01290-8

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 2

  BauchauV.BollaertsK.BryanP.ButteryJ.DavisK.ChenR. T.et al (2025). Multi-Stakeholder call to action for the future of vaccine post-marketing monitoring: proceedings from the first beyond COVID-19 monitoring excellence (BeCOME) conference. Drug Saf.48 (5), 577–585. 10.1007/s40264-024-01510-9

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 3

  BlackS. B.LawB.ChenR. T.DekkerC. L.SturkenboomM.HuangW. T.et al (2021). The critical role of background rates of possible adverse events in the assessment of COVID-19 vaccine safety. Vaccine39 (19), 2712–2718. 10.1016/j.vaccine.2021.03.016

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 4

  Brighton Collaboration (2021). Updated Proposed Brighton Collaboration process for developing a standard case definition for study of new clinical syndrome X, as applied to Thrombosis with Thrombocytopenia Syndrome (TTS). Available online at: https://brightoncollaboration.org/wp-content/uploads/2023/08/TTS-Interim-Case-Definition-v10.16.3-May-23-2021.pdf (Accessed July 1, 2024).

  • Google Scholar
  • View reference in article

• 5

  ClothierH. J.ShettyA. N.MesfinY.MackieM.PearceC.ButteryJ. P. (2024). What would have happened anyway? Population data source considerations when estimating background incident rates of adverse events following immunisation to inform vaccine safety. Vaccine42 (5), 1108–1115. 10.1016/j.vaccine.2024.01.025

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 6

  Council for International Organizations of Medical Sciences (CIOMS) (2023). CIOMS Cumulative glossary with a focus on pharmacovigilance. Available online at: https://cioms.ch/wp-content/uploads/2023/04/CIOMSGlossary_v2.1_1stMay2023.pdf (Accessed September 5, 2025).

  • Google Scholar
  • View reference in article

• 7

  COVID Research and Development Alliance (2022). Available online at: https://covidrdalliance.com/index.html (Accessed February 7, 2023).

  • Google Scholar
  • View reference in article

• 8

  CullenL. A.GrangeZ.AntalK.WaughL.AlsinaM. S.GibbonsC. L.et al (2023). COVID-19 vaccine safety in Scotland - background rates of adverse events of special interest. Public Health224, 1–7. 10.1016/j.puhe.2023.08.006

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 9

  Data Analytics and Methods Task Force EMA (2023). Data Quality Framework for EU medicines regulation. Available online at: https://www.ema.europa.eu/en/about-us/how-we-work/data-regulation-big-data-other-sources/data-quality-framework-medicines-regulation (Accessed May 21, 2025).

  • Google Scholar
  • View reference in article

• 10

  EMA (2024). Reflection paper on use of real-world data in non-interventional studies to generate real-world evidence - scientific guideline. Available online at: https://www.ema.europa.eu/en/reflection-paper-use-real-world-data-non-interventional-studies-generate-real-world-evidence-scientific-guideline (Accessed September 15, 2024).

  • Google Scholar
  • View reference in article

• 11

  EspositoD.TitievskyL.BeachlerD. C.HawesJ. C. L.IsturizR.ScottD. A.et al (2018). Incidence of outcomes relevant to vaccine safety monitoring in a US commercially-insured population. Vaccine36 (52), 8084–8093. 10.1016/j.vaccine.2018.10.052

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 12

  European Medicine Agency (2022). Consideration on core requirements for RMPs of COVID-19 vaccines. coreRMP19 Guidance V3.1. Available online at: https://www.ema.europa.eu/en/documents/other/consideration-core-requirements-rmps-covid-19-vaccines_en.pdf (Accessed October 1, 2022).

  • Google Scholar
  • View reference in article

• 13

  European Medicines Agency (2021). Updated Signal assessment report on Myocarditis, pericarditis with Tozinameran (COVID-19 mRNA vaccine (nucleoside-modified) – COMIRNATY). Available online at: https://www.ema.europa.eu/en/documents/prac-recommendation/updated-signal-assessment-report-myocarditis-pericarditis-tozinameran-covid-19-mrna-vaccine-nucleoside-modified-comirnaty_en.pdf (Accessed June 30, 2024).

  • Google Scholar
  • View reference in article

• 14

  FDA (2023). Considerations for the use of real-world data and real-world evidence to support regulatory decision-making for drug and biological products. Available online at: https://www.hhs.gov/guidance/sites/default/files/hhs-guidance-documents/FDA/54767258fnl.pdf (Accessed September 15, 2024).

  • Google Scholar
  • View reference in article

• 15

  FortnerA.SchumacherD. (2021). First COVID-19 vaccines receiving the US FDA and EMA emergency use authorization. Discov. (Craiova).9 (1), e122. 10.15190/d.2021.1

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 16

  FulginitiV. A.EllerJ. J.DownieA. W.KempeC. H. (1967). Altered reactivity to measles virus. Atypical measles in children previously immunized with inactivated measles virus vaccines. JAMA202 (12), 1075–1080. 10.1001/jama.202.12.1075

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 17

  GartlanC.TiptonT.SalgueroF. J.SattentauQ.GorringeA.CarrollM. W. (2022). Vaccine-Associated enhanced disease and pathogenic human coronaviruses. Front. Immunol.13, 882972. 10.3389/fimmu.2022.882972

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 18

  Global Vaccine Data Network (2023). Background rates dashboards. Available online at: https://www.globalvaccinedatanetwork.org/Data-Dashboards/Background-Rates-Dashboards (Accessed December 1, 2024).

  • Google Scholar
  • View reference in article

• 19

  GreinacherA.ThieleT.WarkentinT. E.WeisserK.KyrleP. A.EichingerS. (2021). Thrombotic thrombocytopenia after ChAdOx1 nCov-19 vaccination. N. Engl. J. Med.384 (22), 2092–2101. 10.1056/NEJMoa2104840

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 20

  GronbaekL.OteteH.BanL.CrooksC.CardT.JepsenP.et al (2020). Incidence, prevalence and mortality of autoimmune hepatitis in England 1997-2015. A population-based cohort study. Liver Int.40 (7), 1634–1644. 10.1111/liv.14480

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 21

  GubernotD.JazwaA.NiuM.BaumblattJ.GeeJ.MoroP.et al (2021). U.S. Population-Based background incidence rates of medical conditions for use in safety assessment of COVID-19 vaccines. Vaccine39 (28), 3666–3677. 10.1016/j.vaccine.2021.05.016

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 22

  HarveyN.HolmesC. A. (2012). Nominal group technique: an effective method for obtaining group consensus. Int. J. Nurs. Pract.18 (2), 188–194. 10.1111/j.1440-172X.2012.02017.x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 23

  HennessyS.AtsutaY.HillS.RagoL.JuhaeriJ.Council for International Organizations of Medical Sciences CIOMS Working Group XIII (2025). Real-World data and real-world evidence in regulatory Decision Making: report Summary from the council for international organizations of medical sciences (CIOMS) working Group XIII. Pharmacoepidemiol Drug Saf.34 (3), e70117. 10.1002/pds.70117

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 24

  HuhK.KimY. E.RadnaabaatarM.LeeD. H.KimD. W.ShinS. A.et al (2021). Estimating baseline incidence of conditions potentially associated with vaccine adverse events: a call for surveillance System using the Korean national health insurance claims data. J. Korean Med. Sci.36 (9), e67. 10.3346/jkms.2021.36.e67

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 25

  KimH. W.CancholaJ. G.BrandtC. D.PylesG.ChanockR. M.JensenK.et al (1969). Respiratory syncytial virus disease in infants despite prior administration of antigenic inactivated vaccine. Am. J. Epidemiol.89 (4), 422–434. 10.1093/oxfordjournals.aje.a120955

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 26

  KleinN. P.RayP.CarpenterD.HansenJ.LewisE.FiremanB.et al (2010). Rates of autoimmune diseases in Kaiser Permanente for use in vaccine adverse event safety studies. Vaccine28 (4), 1062–1068. 10.1016/j.vaccine.2009.10.115

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 27

  LedfordH.CyranoskiD.Van NoordenR. (2020). The UK has approved a COVID vaccine - here's what scientists now want to know. Nature588 (7837), 205–206. 10.1038/d41586-020-03441-8

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 28

  LiX.OstropoletsA.MakadiaR.ShoaibiA.RaoG.SenaA. G.et al (2021). Characterising the background incidence rates of adverse events of special interest for covid-19 vaccines in eight countries: multinational network cohort study. BMJ373, n1435. 10.1136/bmj.n1435

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 29

  MahauxO.BauchauV.Van HolleL. (2016). Pharmacoepidemiological considerations in observed-to-expected analyses for vaccines. Pharmacoepidemiol Drug Saf.25 (2), 215–222. 10.1002/pds.3918

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 30

  MollK.LufkinB.FingarK. R.Ke ZhouC.TworkoskiE.ShiC.et al (2023). Background rates of adverse events of special interest for COVID-19 vaccine safety monitoring in the United States, 2019-2020. Vaccine41 (2), 333–353. 10.1016/j.vaccine.2022.11.003

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 31

  MunozF. M.CramerJ. P.DekkerC. L.DudleyM. Z.GrahamB. S.GurwithM.et al (2021). Vaccine-associated enhanced disease: case definition and guidelines for data collection, analysis, and presentation of immunization safety data. Vaccine39 (22), 3053–3066. 10.1016/j.vaccine.2021.01.055

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 32

  NasreenS.CalzavaraA. J.SundaramM. E.MacDonaldS. E.RigholtC. H.PaiM.et al (2021). Background incidence rates of hospitalisations and emergency department visits for thromboembolic and coagulation disorders in Ontario, Canada for COVID-19 vaccine safety assessment: a population-based retrospective observational study. BMJ Open11 (12), e052019. 10.1136/bmjopen-2021-052019

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 33

  NasreenS.CalzavaraA.BuchanS. A.ThampiN.JohnsonC.WilsonS. E.et al (2022). Background incidence rates of adverse events of special interest related to COVID-19 vaccines in Ontario, Canada, 2015 to 2020, to inform COVID-19 vaccine safety surveillance. Vaccine40 (24), 3305–3312. 10.1016/j.vaccine.2022.04.065

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 34

  OstropoletsA.LiX.MakadiaR.RaoG.RijnbeekP. R.Duarte-SallesT.et al (2022). Factors influencing background incidence rate calculation: systematic empirical evaluation across an international network of observational databases. Front. Pharmacol.13, 814198. 10.3389/fphar.2022.814198

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 35

  PhillipsA.JiangY.WalshD.AndrewsN.ArtamaM.ClothierH.et al (2023). Background rates of adverse events of special interest for COVID-19 vaccines: a multinational Global Vaccine Data Network (GVDN) analysis. Vaccine41 (42), 6227–6238. 10.1016/j.vaccine.2023.08.079

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 36

  PillsburyA.PhillipsA.DengL.QuinnH.MacartneyK.GiddingH. (2023). Background incidence rates of selected adverse events of special interest (AESI) to monitor the safety of COVID-19 vaccines. Vaccine41 (22), 3422–3428. 10.1016/j.vaccine.2023.04.041

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 37

  PraetN.WillameC.HaynesK.SwerdelJ. N.WeaverJ.DiddenE. M.et al (2024). Methodological framework to conduct outcome validation in administrative or insurance claims and/or electronic healthcare record databases: a practical guide for pharmacoepidemiologists [poster]. Berlin, Germany: ISPE Annual Meeting.

  • Google Scholar
  • View reference in article

• 38

  RussekM.QuintenC.de JongV. M. T.CohetC.KurzX. (2023). Assessing heterogeneity of electronic health-care databases: a case study of background incidence rates of venous thromboembolism. Pharmacoepidemiol Drug Saf.32 (9), 1032–1048. 10.1002/pds.5631

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 39

  SchonbornL.PavordS.ChenV. M. Y.PaiM.GwarzoD. H.ButteryJ.et al (2024). Thrombosis with thrombocytopenia syndrome (TTS) and vaccine-induced immune thrombocytopenia and thrombosis (VITT): Brighton Collaboration case definitions and guidelines for data collection, analysis, and presentation of immunisation safety data. Vaccine42 (7), 1799–1811. 10.1016/j.vaccine.2024.01.045

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 40

  SchultzN. H.SorvollI. H.MichelsenA. E.MuntheL. A.Lund-JohansenF.AhlenM. T.et al (2021). Thrombosis and thrombocytopenia after ChAdOx1 nCoV-19 vaccination. N. Engl. J. Med.384 (22), 2124–2130. 10.1056/NEJMoa2104882

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 41

  SerradellL.FrettaA.NachbarJ.DreyfusJ.GarofaloD.LuciniA.et al (2025). The use of observed-to-expected analyses as a signal detection tool in COVID-19 vaccine safety surveillance: lessons learned from an industry perspective. Front. Drug Saf. Regul.5, 1650992. 10.3389/fdsfr.2025.1650992

  • CrossRef
  • Google Scholar
  • View reference in article

• 42

  TworkoskiE.ZhouC. K.WongH. L.LufkinB.MollK.FingarK. R.et al (2021). Background rates of adverse events of special interest for COVID-19 vaccine safety monitoring. Available online at: https://bestinitiative.org/wp-content/uploads/2022/01/C19-Vaccine-Safety-AESI-Background-Rate-Report-2021.pdf (Accessed July 23, 2022).

  • Google Scholar
  • View reference in article

• 43

  Vaccines Europe (2020). Available online at: https://www.vaccineseurope.eu/(Accessed February 7, 2023).

  • Google Scholar
  • View reference in article

• 44

  VossE. A.ShoaibiA.Yin Hui LaiL.BlacketerC.AlshammariT.MakadiaR.et al (2023). Contextualising adverse events of special interest to characterise the baseline incidence rates in 24 million patients with COVID-19 across 26 databases: a multinational retrospective cohort study. EClinicalMedicine58, 101932. 10.1016/j.eclinm.2023.101932

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 45

  WillameC.DoddC.van der AaL.PicelliG.EmborgH. D.KahlertJ.et al (2021a). Incidence rates of autoimmune diseases in European healthcare databases: a contribution of the ADVANCE Project. Drug Saf.44 (3), 383–395. 10.1007/s40264-020-01031-1

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 46

  WillameC.DoddC.GiniR.DuranC. E.ThomsenR. M.WangL.et al (2021b). Background rates of Adverse Events of Special Interest for monitoring COVID-19 vaccines. Genève, Switzerland and Zenodo. Available online at: https://zenodo.org/records/5255870#.YSdGIktxfIV (Accessed October 22, 2021).

  • Google Scholar
  • View reference in article

• 47

  WillameC.DoddC.DuranC. E.ElbersR.GiniR.BartoliniC.et al (2023). Background rates of 41 adverse events of special interest for COVID-19 vaccines in 10 European healthcare databases - an ACCESS cohort study. Vaccine41 (1), 251–262. 10.1016/j.vaccine.2022.11.031

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 48

  World Health Organization (2023). Guidance for clinical case management of thrombosis with thrombocytopenia syndrome (TTS) following vaccination to prevent coronavirus disease (COVID-19). Available online at: https://www.who.int/publications/i/item/9789240061989 (Accessed July 1, 2024).

  • Google Scholar
  • View reference in article

• 49

  World Health Organization (2020). Covid-19 vaccines: safety surveillance manual. Available online at: https://www.who.int/docs/default-source/covid-19-vaccines-safety-surveillance-manual/covid19vaccines_manual_aesi.pdf?sfvrsn=239020a2_1 (Accessed September 5, 2025).

  • Google Scholar
  • View reference in article

• 50

  World Health Organization (2021). Guidance for clinical case management of thrombosis with thrombocytopenia syndrome (TTS) following vaccination to prevent coronavirus disease (COVID-19); interim guidance. Available online at: https://iris.who.int/bitstream/handle/10665/342999/WHO-2019-nCoV-TTS-2021.1-eng.pdf (Accessed July 1, 2024).

  • Google Scholar
  • View reference in article

• 51

  World Health Organization (2025). Coronavirus disease (COVID-19) pandemic. Available online at: https://www.who.int/europe/emergencies/situations/covid-19 (Accessed March 1, 2025).

  • Google Scholar
  • View reference in article