Skip to main content

Computational Methods and Systems to Support Decision Making in Pharmacovigilance

Cover image for Research Topic "Computational Methods and Systems to Support Decision Making in Pharmacovigilance"
40kviews
Editorial21 April 2023
Editorial: Computational methods and systems to support decision making in pharmacovigilance

The World Health Organization defines pharmacovigilance (PV) as “the science and activities related to the detection, assessment, understanding, and prevention of adverse events or any drug-related problem.”[1] Government agencies, clinical institutions, the pharmaceutical industry, and other entities manage PV programs as vital safeguards for supporting the early detection and analysis of safety signals. These rely primarily on expert judgment and concrete business practices specific to each organization [2]. Computational methods and decision-support systems may enable more timely, consistent, and comprehensive analysis and processing of real-world data (RWD) and may free up time for domain experts to focus on higher-value contributions. As a positive side-effect, their development can help standardize business practices by specifying the steps human reviewers follow. In this special topic issue, we primarily invited papers that assess contributions to PV which may improve efficiency in established business practices, minimize manual effort, and maximize the quality of human decision-making by enhancing existing processes transparently. We also welcomed case studies of method implementation and perspectives that might significantly contribute to the domain.

Primary use cases in PV, such as case processing and prioritization or signal detection and evaluation, incorporate several steps that the complete decision-support system should augment. As data, user-related, and other challenges of the entire workflow vary, most efforts, to this date, have delivered solutions addressing more narrowly defined tasks. Historically, computational methods were initially proposed and deployed to enable signal detection and analysis in large databases using only structured data. More recently, several studies have presented methods for processing unstructured free texts, prioritizing case reports for clinical and regulatory review, and improving data quality in spontaneous reporting systems. For example, Painter et al. conducted a meta-analysis of a recent scoping review on machine learning in PV [3] and found that the pharmaceutical industry mainly applied machine learning to process RWD and social media [4]. They also highlighted the need to develop consistent systems that can learn and incorporate human-in-the-loop mechanisms and called for best practices for adopting and validating these systems.

To build effective solutions, existing workflows and business practices must be understood. These are rarely documented in stepwise algorithmic forms that can easily be translated into programmatic tools. It is, therefore, necessary to take a step back and, in close collaboration with the end users, identify the overall desired framework for a selected workflow. The papers in this special topic highlight several important aspects of this. Fong et al. pursued a strategy to support the medication error categorization workflow in the US Food and Drug Administration Adverse Event Reporting System (FAERS) [5]. Interestingly, they first defined a framework that helped them clarify the use case and then developed a prototype user-centered Artificial Intelligence (AI) application based on this framework. This process might work both ways, i.e., the first version of the framework implemented in a prototype tool could be informed by the pilot use of this prototype and, subsequently, updated to support new rounds of development.

Kreimeyer et al. focused on detecting duplicate cases in FAERS using the complete structured and free-text information from FAERS Individual Case Safety Reports (ICSRs) [6]. They built the confidence required to operationalize it by engaging safety reviewers in developing, evaluating, and validating their deduplication as a supplement to human review. Dang et al. developed a natural language processing approach to extract four demographic variables (gender, weight, ethnicity, and race) from FAERS ICSR narratives and complete the missing values in the corresponding structured fields [7]. While their approach excellently retrieved ethnicity, gender, and race, it would meaningfully improve only gender information in the actual practice, as reporters infrequently provide weight, ethnicity, or race in the FAERS narratives. This demonstrates that operationalizing an AI solution must evaluate multiple factors, not solely performance.

Dimitriadis et al. presented an open-source platform that integrated ICSRs, synthetic clinical data, social media data from Twitter, and scientific literature from PubMed [8]. Their prototype web-based tool allowed for various analyses in an interactive user interface. However, a comparison with the EudraVigilance Data Warehouse and Analysis System (EVDAS) provided by the European Medicines Agency revealed that PV experts preferred the existing technology. This study illustrates how implementing new tools in an actual work environment requires significant effort in presenting complete and validated systems and educating end users on their application.

When generative AI hits the headlines and heated discussions ensue [9], it is essential to emphasize that PV is a tightly controlled, multi-step process. The rapid improvement of large language models across multiple applications should benefit PV, but to build trust with PV experts, we must systematically engage with users when collecting requirements and designing decision-support systems. Incorporating AI and other sophisticated algorithms into these systems requires knowledgeable interdisciplinary teams, including end-users and software developers, and several validation rounds aligned with the supported tasks and business workflows. Some of the studies in our research topic followed this strategy. This is the recipe for success in most cases as long as the right expectations are set from the beginning and the proposed solutions do not bring dramatic changes to existing workflows without proven benefit. Including a human-in-the-loop further ensures that a system’s limitations are carefully examined and that a quality assurance process to control them is in place. Ultimately, as no automated solution is likely to be perfect, decision-support systems must be constructed to take advantage of the respective strengths of machines and humans to improve efficiency, improve quality, and add value beyond what each might achieve on their own.

2,100 views

3 citations

PERSPECTIVE
1 February 2023
An industry perspective on the use of machine learning in drug and vaccine safety
Jeffery L. Painter, 1 more and Andrew Bate
An industry perspective on the use of machine learning in drug and vaccine safety
In recent years there has been growing interest in the use of machine learning across the pharmacovigilance lifecycle to enhance safety monitoring of drugs and vaccines. Here we describe the scope of industry-based research into the use of machine learning for safety purposes. We conducted an examination of the findings from a previously published systematic review; 393 papers sourced from a literature search from 2000–2021 were analyzed and attributed to either industry, academia, or regulatory authorities. Overall, 33 papers verified to be industry contributions were then assigned to one of six categories representing the most frequent PV functions (data ingestion, disease-specific studies, literature review, real world data, signal detection, and social media). RWD and social media comprised 63% (21/33) of the papers, signal detection and data ingestion comprised 18% (6/33) of the papers, while disease-specific studies and literature reviews represented 12% (4/33) and 6% (2/33) of the papers, respectively. Herein we describe the trends and opportunities observed in industry application of machine learning in pharmacovigilance, along with discussing the potential barriers. We conclude that although progress to date has been uneven, industry is very interested in applying machine learning to the pharmacovigilance lifecycle, which it is hoped may ultimately enhance patient safety.

6,820 views

15 citations

BRIEF RESEARCH REPORT
14 November 2022
Evaluation of a natural language processing tool for extracting gender, weight, ethnicity, and race in the US food and drug administration adverse event reporting system
Vivian Dang, 5 more and Monica A. Muñoz
Evaluation of a natural language processing tool for extracting gender, weight, ethnicity, and race in the US food and drug administration adverse event reporting system
The U.S. Food and Drug Administration Adverse Event Reporting System (FAERS) contains over 24 million individual case safety reports (ICSRs). In this research project, we evaluated a natural language processing (NLP) tool’s ability to extract four demographic variables (gender, weight, ethnicity, race) from ICSR narratives. Specificity of the NLP algorithm was over 94% for all demographics, while sensitivity varied between the demographics: 98.6% (gender), 45.5% (weight), 100% (ethnicity), and 85.3% (race). Among ICSRs missing weight, ethnicity, and race in the structured field, few cases had this information in the narrative (>95% missing); consequently, the positive predictive value (PPV) for these three demographics had wide 95% confidence intervals. After NLP implementation, the total number of ICSRs missing gender was reduced by 33% (i.e., NLP identified 472 thousand reports having a gender value in the narrative that was not in the structured field), while the total number of ICSRs missing weight, ethnicity, or race was reduced by less than 4%. This study demonstrated that the implementation of an NLP tool can provide meaningful improvements in the availability of gender information for pharmacovigilance activities conducted with FAERS data. In contrast, NLP tools targeting the extraction of weight, ethnicity, or race from free-text fields have minimal impact largely because the information was infrequently provided by the reporter. Further gains in completeness of these fields must originate from increases in provision of demographic information from the reporter rather than informatic solutions.

4,255 views

9 citations

Increased Confidence in Deduplication of Drug Safety Reports with Natural Language Processing of Narratives at the US Food and Drug Administration
BRIEF RESEARCH REPORT
15 June 2022
Increased Confidence in Deduplication of Drug Safety Reports with Natural Language Processing of Narratives at the US Food and Drug Administration
Kory Kreimeyer, 6 more and Taxiarchis Botsis
The U.S. Food and Drug Administration (FDA) receives millions of postmarket adverse event reports for drug and therapeutic biologic products every year. One of the most salient issues with these submissions is report duplication, where an adverse event experienced by one patient is reported multiple times to the FDA. Duplication has important negative implications for data analysis. We improved and optimized an existing deduplication algorithm that used both structured and free text data, developed a web-based application to support data processing, and conducted a six-month dedicated evaluation to assess the potential operationalization of the deduplication process in the FDA. Comparing algorithm predictions with reviewer determinations of duplicates for twenty-seven files for case series reviews (with a median size of 281 reports), the average pairwise recall and precision were equal to 0.71 (SD±0.32) and 0.67 (SD±0.34). Overall, reviewers felt confident about the algorithm and expressed their interest in using it. These findings support the operationalization of the deduplication process for case series review as a supplement to human review.

5,664 views

14 citations

BRIEF RESEARCH REPORT
10 November 2022
Exploring opportunities for AI supported medication error categorization: A brief report in human machine collaboration
Allan Fong, 4 more and Jo Wyeth

3,839 views

3 citations