The use of Observed-to-Expected analyses as a signal detection tool in COVID-19 vaccine safety surveillance: lessons learned from an industry perspective

LAURENCE SERRADELL^1*ANTONELLA FRETTA²JENNY NACHBAR³JILL DREYFUS⁴DIANA GAROFALO⁴ARIANNA LUCINI²SUSAN MATHER⁵DAINA ESPOSITO⁶ANNE-LAURE CHABANON⁷VINCENT BAUCHAU⁸SARAH SELLERS³

• ¹Epidemiology & Benefit Risk, Patient Safety & Pharmacovigilance, Sanofi, Lyon, France
• ²Worldwide Medical and Safety, Pfizer Inc, Milan, Italy
• ³Global Vaccine Safety, Novavax Inc, Gaithersburg, MD, United States
• ⁴Global Medical Epidemiology, Pfizer Inc, New York NY, United States
• ⁵Worldwide Medical and Safety, Independant, previously Pfizer Inc, Collegeville, PA, United States
• ⁶Global Safety Epidemiology, Moderna Inc, Cambridge, MA, United States
• ⁷Patient Safety & Pharmacovigilance, Sanofi, Lyon, France
• ⁸Vaccine Safety, Independent, Previously GSK, Wavre, Belgium

During the pandemic, the accelerated review and authorization of coronavirus disease 19 (COVID-19) vaccines by regulatory authorities elicited the need for rapid and thorough worldwide signal detection and evaluation. To meet this need, the European Medicines Agency and other health authorities expected that, in addition to routine signal detection, COVID-19 vaccine manufacturers should leverage observed-to-expected (O/E) analyses unconventionally as a quantitative method for signal detection of adverse events of special interest (AESIs). The objective of O/E analyses in vaccine signal detection was to determine if AESIs were occurring at a higher-than-expected rate in the vaccinated population in comparison with an unexposed population. The use of O/E was intended to mitigate the challenge of analyzing large volumes of individual case safety reports (ICSRs) received over a very short period following mass vaccination campaigns. The "Beyond COVID-19 Monitoring Excellence" (BeCOME) initiative, a non-competitive voluntary initiative launched in 2022 by COVID-19 vaccine Marketing Authorization Holders (MAHs) and key stakeholders, was established to align systems, enhance processes, and foster innovation in post-marketing vaccine monitoring, building on lessons from the pandemic. A dedicated working group was created to review and share MAHs' experience on O/E analyses used as an additional tool for signal detection during the COVID-19 pandemic. This review presents the industry perspective on using O/E analyses for COVID-19 vaccine signal detection, including challenges and limitations encountered, and proposes best practices for future improvement. Despite the priority and resources devoted to O/E analyses, no de novo signals resulting in the identification of safety concerns were detected using this methodology during the COVID-19 pandemic. O/E analyses are most useful when source data are accurate and there is a high level of confidence in the assumptions and parameters used. In the context of the COVID-19 pandemic, confidence in certain assumptions and parameters was low, limiting the value of O/E analyses in signal detection. Nevertheless, O/E analyses applied for signal refinement, as traditionally used, proved to be useful. Industry experiences support maintaining O/E analyses as a tool for signal refinement and standardizing methodological approaches as much as possible to enhance its future application and comparability across stakeholders.