Abstract
Purpose:
Electronic medical records (EMRs) are essential for pharmacoepidemiologic research; however, information on the quality of Canadian hospital EMR data is scarce. We aimed to test data validity in the Epic EMR for key pharmacoepidemiologic themes using QT-prolonging medications and major adverse cardiac events (MACEs) as the research question.
Methods:
An entity relationship diagram (ERD) was developed to navigate >20,000 tables. Computational data validation (comparison with Epic SlicerDicer and Canadian Institute for Health Information Discharge Abstract Database (CIHI-DAD) data) and manual data validation (chart review) were applied with iterative adjustments. Percent agreement with 95% confidence intervals was used to estimate data validity.
Results:
The ERD demonstrated that 43 tables are needed for key pharmacoepidemiologic research data. A cohort of 70,078 adult inpatients was used for computational validation and 2,281 patient charts for manual validation. Demographics (for example, age, sex, gender, admitting service, and critical care transfers); exposures (medication administrations); many potential confounders (lab results, interacting drugs, and electrocardiograms [EKGs]); and most timestamping (admission, medication administrations, death, lab results, and EKGs) were validated with >95% agreement. Inadequate agreement for MACE (other than death) brought the primary outcome MACE agreement to <90%, requiring patient-level linkage to CIHI-DAD to improve agreement. Comorbidities, including hypertension and heart failure, also required this linkage.
Conclusion:
A world-leading proprietary EMR after the painstaking development of an ERD and validation of key data fields showed that not all data are of sufficient quality for pharmacoepidemiologic research. Linked human-coded data are required for some diagnoses.
Key points
Electronic medical records are rich in patient data that must be validated for research use.
Epic does not provide an entity relationship diagram identifying data field locations or linkages.
A few available data tables are required for pharmacoepidemiologic research.
Most key data concepts had >95% agreement with manual chart review or computational validation.
Accurately identifying diagnoses and their timing requires linked data or natural language processing.
Introduction
Electronic medical records (EMRs) capture an abundance of data from patients while helping with clinical workflows and administrative processes and are also very useful for research to improve the quality of care (). However, clinical data may not necessarily be of adequate quality for research. Making actionable conclusions that will alter clinical flow or health policies relies on data of low bias, which requires ensuring the relevance, accuracy, and completeness of the data used (; Wang et al., 2023).
Epic is a leading EMR worldwide and is the most common EMR in the United States and Canada, despite only being first implemented in Canada in 2017 (; Witowski, 2025; ). The Epic EMR has been used for many types of research, including clinical, health service, population health, quality improvement, registry, and economic analyses (). For pharmacoepidemiologic research, EMR data have been linked to other data sets such as pharmacy records, claims data, and clinical research networks (Wu et al., 2025; ; ).
Previous studies have validated parts of an EMR but not all required themes for pharmacoepidemiologic research. EMR data from two hospitals in the Netherlands were validated, showing ≥98% agreement for patient demographics and blood products used (). An EMR data validation study in Spain was computationally validated against a research database that reported positive predictive values (PPVs) for 12 sections of the EMR, including demographics, comorbidities, medications, and microbiology, which ranged from 58% to 99% (). The Hospital for Sick Children developed a research database with 18 curated tables for Epic, which was validated iteratively using manual chart review until all errors were corrected (). However, the validation does not describe how diagnoses or medication administrations were validated ().
Conducting validation studies is not straightforward. Most proprietary EMR systems, including Epic, provide no clear entity relationship diagram (ERD) outlining where data reside, table organization, or relationships. Epic contains three databases: Chronicles, Clarity, and Caboodle (Figure 1) (; ; ). Chronicles is the real-time clinical database, Clarity is used for large and complex data reports and is most suitable for research, and Caboodle is used to produce reports and supply data for tools such as SlicerDicer, which provides high-level, deidentified data on selected patient populations (; ; ). Identifying relevant data fields, tables, and their linkages is challenging because the Epic Clarity database contains thousands of data tables (; ). Before using EMR data for research, we must be assured that the data are relevant, accurate, and complete. This requires an understanding of key concepts and requirements for data validation, as well as the ability to internally and externally validate data and iteratively correct errors ().
FIGURE 1
As part of a research program to clarify the association between “known” QT-prolonging (QTP) medications (QTPmeds) and relevant major adverse cardiac events (MACEs) using the Epic EMR at our institution, we required the validation of key pharmacoepidemiologic variables in our EMR. This research could lead to streamlining the plethora of medication-related safety alerts for QTPmeds, many of which are based on theoretical or very low-quality data (; ; ).
Methods
Study design
This study is reported on the basis of the recommendations of the STAtement on the Reporting of Evaluation studies in Health Informatics (Supplementary Material Table A.1) (). This is a retrospective validation study using Epic (institutionally known as Dovetale) EMR data of all adult inpatients aged ≥18 years admitted to our hospital from 2 December 2017 to 24 March 2023. The Hamilton Integrated Research Ethics Board provided a waiver of consent for this study due to its retrospective nature and large cohort. Our hospital is in Canada and is composed of three sites: an acute care hospital with over 600 beds, which includes a women’s and infant’s department, an inpatient mental health hospital, and an outpatient urgent care and services center ().
Methods for data acquisition and measurement
Demographic and clinical data for eligible patients were extracted from the Epic Clarity database by an Epic-certified senior research data analyst using structured query language (SQL) queries (see Figure 2 for full data workflow). Data tables and fields in the Epic Clarity database were identified using the Clarity Dictionary in the Epic Data Handbook (). The data were manually and computationally validated (if data were available for comparison). The SQL queries were iteratively adjusted to improve the completeness and correctness of the data extractions by changing the data tables, fields, and filters. Epic-certified clinical informatics specialists at our institution were consulted when fields were missing data. When the accurate data tables and fields were located, they were added to the ERD.
FIGURE 2
Five key data themes required for pharmacoepidemiologic research were selected for validation: demographics, exposures, outcomes, potential confounders, and timestamping. In this study, MACEs were our primary outcome and followed the Food and Drug Administration QTP standard outcomes but adapted for the additional data available in the hospital: death, ventricular tachycardia including Torsades de Pointes (TdP), ventricular fibrillation, nonfatal cardiac arrest, syncope, and seizure (). These outcomes were extracted using the International Classification of Diseases, 10th Revision (ICD-10) codes or diagnosis names (Supplementary Material Table A.2). Comorbidities were also extracted using ICD-10 codes or diagnosis names. The ability to identify MACE occurring after admission was essential because research requires exposures to predate outcomes. For validation, outcomes were considered to have happened if they were documented in a chart note as being witnessed by a healthcare professional, documented in a physician note, or confirmed with testing.
Manual data validation was performed as the gold standard for completeness and correctness of each item by extracting data from the Epic Clarity database and comparing it with data found through random patient chart review from the research view of Epic (Verma et al., 2021). Search strings and Boolean operators were used for chart review, including comorbidities and lab results (Supplementary Material Table A.3). Computational data validation was also performed when data were available by comparing Epic extractions against the Canadian Institute for Health Information Discharge Abstract Database (CIHI-DAD) to check for completeness and correctness, or Epic was computationally validated against itself using similar items or the Epic tool SlicerDicer for internal concordance.
CIHI-DAD is the reference database for demographic, clinical, and administrative data for hospital inpatient discharges from facilities across Canada, except Quebec (). CIHI-DAD data are recorded by professional coders who review patient charts and are used by leading health services research organizations in Canada (). The agreement in CIHI-DAD for demographics and hospital admission information is very high (>90%) (). The agreement for diagnoses was validated in CIHI-DAD against the chart with agreement of 77%–80%, and agreement for diagnoses in the chart present in CIHI-DAD was 83%–84% (). CIHI-DAD was used to validate age, hospital admission including the admission date, MACE including death, and comorbidities. SlicerDicer was used to internally check the completeness of our Epic Clarity extractions, specifically the number of eligible patients, age, and sex distribution. SlicerDicer provides total numbers, percentages, and minimums and maximums based on the selected criteria (; ).
We followed the recommended eight methods to assess the dimensions of data quality: element presence by ensuring expected and desired data concepts are available; data element agreement by comparing items within the EMR for similar information; distribution comparison using SlicerDicer for aggregate statistics; data source agreement by comparing EMR data with another source (SlicerDicer) for agreement; comparing EMR data with a gold standard such as CIHI-DAD data; log review using timestamping to assist with validation; validity checks by looking at extremes of age- and sex-based procedures; and structural agreement by looking at lab units and diagnostic codes for formatting agreement (; Weiskopf and Weng, 2013).
Methods for data analysis
Data analysis was conducted using Microsoft® Excel. Descriptive statistics (mean, median, standard deviation, and interquartile range) were used to characterize the cohort. Percent agreement reported with 95% confidence intervals (CIs) calculated with WINPEPI was used for all items validated by comparing the data extraction with the chart or comparator data set (). Diagnostic accuracy measures of sensitivity, specificity, PPV, and negative predictive value (NPV) were also used for medication exposures and outcomes (). Each data concept was manually and computationally validated if data for comparison were available. Percent agreement was considered excellent if it was >95% and inadequate if it was <80%.
Results
Entity relationship diagram
An ERD was developed and converted to the style of the Observational Medical Outcomes Partnership (OMOP) common data model showing example linkages between relevant data tables (Figure 3) (). The Epic Clarity database at our institution included 20,764 tables (L. Bishop, email, 17 April 2024). Iterative searches for relevant research data eventually identified 43 tables containing the data required for this project.
FIGURE 3
Study cohort and data validation
The characteristics of the study cohort are shown in Table 1. The details of data validation are shown in Table 2.
TABLE 1
| Characteristic | Number (%) |
|---|---|
| Patients | 70,078 |
| Hospital admissions | 108,598 |
| Hospital admissions per patient - Mean - Median | 1.55 (SDa 1.5) 1 (IQRb 1) |
| Age at admission (mean [SD]; median [IQR]) - Mean - Median | 55.5 (SD 20.8) 58 (IQR 37) |
| Sex (admissions) - Female - Male | 64,105 (59.0%) 44,493 (41.0%) |
| Gender identity (admissions) - Female - Male - Non-binary - Other - Missing data | 3,409 (3.1%) 3,517 (3.2%) 218 (0.2%) 118 (0.1%) 101,336 (93.3%) |
| Admitting unit - General internal medicine - Nephrology - General surgery - Orthopedics - Urology - Thoracic surgery - Head and neck - Obstetrics - Gynecology - Acute mental health | 25,021 (23.0%) 4,728 (4.4%) 14,113 (13.0%) 6,863 (6.3%) 6,649 (6.1%) 4,425 (4.1%) 2,715 (2.5%) 16,951 (15.6%) 2,169 (2.0%) 7,125 (6.6%) |
| Patients transferred to critical care | 12,138 (17.3%) |
| Patients exposed to at least 1 “known” QTPmedc | 50,487 (72.0%) |
| Patients not exposed to any “known” QTPmeds | 19,591 (28.0%) |
| Patients not exposed to any QTPmeds | 5,174 (7.4%) |
| Patients who experienced at least 1 MACEd outcome - Nonfatal MACE - Death - Total | Nonfatal MACE: 5,369 Death: 3,775 Total: 9,144 unique patients (13.0%) |
| EKGse - Patients - Number of EKGs | 32,767 (46.8%) 98,814 |
| Lab tests – potassium - Patients - Number of lab tests | Patients: 54,467 (77.7%) Number of lab tests: 557,577 (54.9%) |
| Lab tests – calcium - Patients - Number of lab tests | 30,515 (43.5%) 169,839 (16.7%) |
| Lab tests – magnesium - Patients - Number of lab tests | 30,404 (43.4%) 180,858 (17.8%) |
| Lab tests – troponin - Patients - Number of lab tests | 28,401 (40.5%) 107,141 (10.6%) |
| Telemetry - Patients - Number of telemetry orders | 10,467 (14.9%) 17,232 |
Characteristics of the study cohort.
aSD, standard deviation.
bIQR, interquartile range.
cQTPmed, QT-prolonging medication.
dMACE, major adverse cardiac events.
eEKG, electrocardiogram.
TABLE 2
| Item to be validated | Validation standard (SlicerDicer, CIHI) | Number of charts checked | Results (% agreement) | 95% confidence interval |
|---|---|---|---|---|
| 1. Demographics | ||||
| Total number of patients in cohorts | SDa | 70,078 | >99% | (99.8, 99.9) |
| Patients with >1 MRN | N/A | 70,078 | 100% | (100, 100) |
| Correct age >18 years on admission date | DADb | 10,065 | >99% | (99.3, 99.6) |
| Age <18 years | DAD | 10,065 | 100% | (100, 100) |
| Date of birth | MV | 200 | 100% | (98.2, 100) |
| Implausible age | MV, SD | 13 | 100% | (75.3, 100) |
| Hospital encounters (potential missed admissions) | MVc | 300 | 100% | (98.8, 100) |
| Sex | MV, SD | 131 | 100%, 99% | (97.2, 100), (99.8, 99.9) |
| Gender | MV | 131 | 100% | (97.2, 100) |
| Implausible sex-based procedures (e.g., males delivering babies and females having prostate surgery) | MV | 4 | >99% | (99.9, 100) |
| Postal code | MV | 200 | 100% | (98.2, 100) |
| Admission | MV, DAD | 50, 10,065 | 100%, 99% | (92.9, 100), (99.7, 99.9) |
| Admitting service | MV | 50 | 100% | (92.9, 100) |
| Presence in critical care | MV | 79 | 100% | (95.4, 100) |
| 2. Exposures | ||||
| Patients given “known” QTPmeds | MV | 235 | 97% | (94.0, 98.8) |
| Patients not given any “known” QTPmeds | MV | 100 | 99% | (94.6, 99.9) |
| Patients not given any QTPmeds | MV | 100 | 96% | (90.0, 98.9) |
| 3. Outcomes | ||||
| Patients with at least 1 MACE outcome (ventricular arrhythmia composite, cardiac arrest, syncope, seizure, and death) | MV | 250 | 70% | (63.9, 75.6) |
| Death | MV, DAD | 200,444 | 100%, 100% | (98.2, 100), (99.2, 100) |
| Ventricular arrhythmia composite (any of ventricular tachycardia including TdPd and ventricular fibrillation) | MV | 50 | 42% | (28.2, 56.8) |
| Cardiac arrest | MV | 50 | 84% | (70.9, 92.8) |
| Syncope | MV | 50 | 66% | (51.2, 78.8) |
| Seizure | MV | 50 | 58% | (43.2, 71.8) |
| Patients with no MACE outcome | MV | 100 | 98% | (93.0, 99.8) |
| 4. Potential confounders | ||||
| Lab values (potassium, magnesium, calcium, and troponin) | MV | 378 lab results for 50 patients | 100% | (99.0, 100) |
| EKGe data (for QTc-Bf and cardiac rhythm interpretation) | MV | 117 EKGs for 63 patients | 99% | (95.3, 99.9) |
| Telemetry order | MV | 87 orders for 50 patients | 95% | (88.6, 98.7) |
| Telemetry indication | MV | 84 orders for 50 patients | 98% | (91.7, 99.7) |
| Diabetes | MV, DAD | 75,560 | 100%, 56% | (95.2, 100), (51.9, 60.2) |
| Hypertension | MV, DAD | 75,502 | 100%, 84% | (95.2, 100) (80.4, 87.0) |
| Heart failure | MV, DAD | 75,167 | 96%, 69% | (88.8, 99.2), (61.9, 76.3) |
| 158, 210, and 172 administrations for 6, 25, and 25 patients, respectively | MV | 158, 210, and 172 administrations for 6, 25, and 25 patients | 100% | (99.3, 100) |
| Metabolic inducer (phenytoin) | MV | 199 administrations for 25 patients | 100% | (98.2, 100) |
| Diuretics (furosemide, hydrochlorothiazide, indapamide, and metolazone) | MV | 384 administrations for 25 patients | 93% | (90.5, 95.7) |
| Electronic decision support alerts shown to providers | MV | 65 alerts for 25 patients | 88% | (77.2, 94.5) |
| Alerts being shown to the listed provider | MV | 57 alerts | 100% | (93.7, 100) |
| Alerts being overridden | MV | 57 alerts | 100% | (93.7, 100) |
| 5. Timestamping | ||||
| Date of admission | DAD | 10,065 | 99% | (98.7, 99.1) |
| Minute of medication administration | MV | 56 patients | 100% | (93.6, 100) |
| Date of nonfatal MACE | MV | 25 patients | 36% | (18.0, 57.5) |
| Date of death | MV | 200 | 98% | (94.3, 99.2) |
| Minute of lab draw | MV | 378 lab results | 100% | (99.0, 100) |
| Minute of EKG | MV | 67 EKGs | 100% | (94.6, 100) |
| Minute of telemetry order | MV | 84 orders | 100% | (95.7, 100) |
| Minute of telemetry discontinuation | MV | 84 orders | 87% | (77.8, 93.3) |
| Minute of alerts | MV | 57 alerts | 100% | (93.7, 100) |
Data validation items and results for Epic.
aSD, SlicerDicer.
bCIHI-DAD, Canadian Institute for Health Information Discharge Abstract Database data.
cMV, manual validation.
dTdP, Torsades de Pointes.
eEKG, electrocardiogram.
fQTc-B, QT-interval corrected using Bazett’s formula.
Study findings and outcome data
Demographics
The total number of patients in the cohort was 70,078; this was computationally validated against SlicerDicer, which returned 70,170 patients with >99% agreement (95% CI: 99.8, 99.9). Patients with >1 medical record number (MRN) were validated by checking for health card numbers associated with multiple MRNs. A sample of patients manually validated for >1 MRN, age at the time of admission and implausible age, date of birth, hospital admission, sex and implausible sex, postal code, presence in critical care, and provider service showed 100% agreement. Gender identity was manually validated for 131 charts showing 100% agreement; however, it was only available for 7,262 (6.7%) admissions. Sex was computationally validated against SlicerDicer, in which both Epic and SlicerDicer resulted in 60.8% and 39.2% female and male patients, respectively. Admission and age were computationally validated against CIHI-DAD with >99% agreement. Overall, all demographic items had excellent validity (>95%).
Exposures
Exposures included administration of QTPmeds. The “known,” “possible,” and “conditional” lists of QTPmeds were extracted from CredibleMeds (). The drug name and administration of “known” QTPmeds were manually validated for 235 charts with 97% (95% CI: 94.0, 98.8) agreement. Patients not given any “known” QTPmeds and no QTPmeds of any type were manually validated for 100 charts with 99% (95% CI: 94.6, 99.9) and 96% (95% CI: 90.0, 98.9) agreement, respectively. Diagnostic accuracy for exposure to “known” QTPmeds based on manual validation of 335 charts showed a sensitivity of 99% (95% CI: 97.6, 99.9), a specificity of 93% (95% CI: 86.9, 97.3), a PPV of 97% (95% CI: 94.1, 98.5), and an NPV of 99% (95% CI: 93.3, 99.9) (Table 3). Some medication administrations were missed because of limitations when combining Anatomical Therapeutic Chemical (ATC) code and generic name data extractions, where certain dosage forms such as half tablets could not be extracted. Overall, exposures to QTPmeds had excellent validity (>95%).
TABLE 3
| True positive: 228 | False positive: 7 | PPVa: 97% (95% CI: 94.1, 98.5) |
| False negative: 1 | True negative: 99 | NPVb: 99% (95% CI: 93.3, 99.9) |
| Snc: 99% (95% CI: 97.6, 99.9) | Spd: 93% (95% CI: 86.9, 97.3) | n = 335 patients |
Diagnostic accuracy measures for Epic exposures to “known” QTPmeds.
aPPV, positive predictive value.
bNPV, negative predictive value.
cSn, sensitivity.
dSp, specificity.
Outcomes
Our MACE outcomes of interest included death, ventricular arrhythmia composite (any ventricular tachycardia including TdP and ventricular fibrillation), nonfatal cardiac arrest, syncope, and seizure. MACEs were manually and computationally validated, demonstrating 100% agreement for death. Epic does not differentiate between the phase of care when storing diagnoses in structured data fields; thus, manual validation was conducted on the basis of whether the outcome occurred while the patient was admitted or as a presenting diagnosis rather than in the past. Agreement for the ventricular arrhythmia composite was 42% (95% CI: 28.2, 56.8), nonfatal cardiac arrest was 84% (95% CI: 70.9, 92.8), syncope was 66% (95% CI: 51.2, 78.8), and seizure was 58% (95% CI: 43.2, 71.8). The diagnostic accuracy of MACE outcomes based on manual validation of 250 patients with MACEs and 100 patients without MACEs showed a sensitivity of 99% (95% CI: 96.0, 99.9), a specificity of 57% (95% CI: 48.9, 64.2), a PPV of 70% (95% CI: 66.3, 73.5), and an NPV of 98% (95% CI: 92.5, 99.5) (Table 4). This indicated an unacceptably high rate of false-positive diagnoses in Epic due to two main problems with the EMR data: (a) diagnostic uncertainty (e.g., unwitnessed events like syncope or seizure) and (b) Epic not differentiating the phase of care as some diagnoses were historical but remain on the chart. By adding patient-level linkage of Epic data to CIHI-DAD, which differentiates the phase of care, manual validation for 300 patients identified as having MACE outcomes after admission (other than death) improved to 79% (95% CI: 74.3, 83.8) agreement.
TABLE 4
| True positive: 175 | False positive: 75 | PPVa: 70% (95% CI: 66.3, 73.5) |
| False negative: 2 | True negative: 98 | NPVb: 98% (95% CI: 92.5, 99.5) |
| Snc: 99% (95% CI: 96.0, 99.9) | Spd: 57% (95% CI: 48.9, 64.2) | n = 350 patient admissions |
Diagnostic accuracy measures for Epic MACE outcomes.
aPPV, positive predictive value.
bNPV, negative predictive value.
cSn, sensitivity.
dSp, specificity.
Potential confounders
Manual validation for lab values, EKG data (QT-interval corrected by Bazett’s formula and cardiac rhythm), and telemetry order and indication showed ≥95% agreement. Manual validation of the presence of relevant comorbidities (diabetes, hypertension, and heart failure) showed 100%, 100%, and 96% agreement, respectively. However, computational validation of these comorbidities against CIHI-DAD showed 56%, 84%, and 69% agreement due to Epic missing the diagnoses in some patients with these comorbidities (Table 5). Interacting drugs were manually validated for select metabolic inhibitors (ketoconazole, clarithromycin, and bupropion), a metabolic inducer (phenytoin), and diuretics (furosemide, hydrochlorothiazide, indapamide, and metolazone) for up to 25 charts each for all administrations which showed 100%, 100%, and 93% (95% CI: 90.5, 95.7) agreement, respectively. Certain strengths of furosemide were missed in both generic name and ATC code extractions, leading to lower agreement than other interacting drugs. For electronic decision support alerts, 850,672 medication-related alerts were shown to providers in 2023. The override rate of these alerts was 98%. Pharmacists saw the largest proportion of these alerts, followed by medical residents and attending physicians. Manual validation of the override history of 65 overridden medication alerts showed 88% (95% CI: 77.2, 94.5) agreement due to the absence of some alerts from the chart. Overall, potential confounders had excellent validity (>95%) for most items but required CIHI-DAD linkage for chronic comorbidities due to false negatives in Epic.
TABLE 5
| Diabetes | Hypertension | Heart failure | ||||||
|---|---|---|---|---|---|---|---|---|
| True positive: 314 | False positive: 23 | PPVa: 93% (95% CI: 90.1, 95.4) | True positive: 421 | False positive: 643 | PPV: 40% (95% CI: 38.0, 41.2) | True positive: 116 | False positive: 207 | PPV: 36% (95% CI: 32.3, 39.7) |
| False negative: 246 | True negative: 1,253 | NPVb: 84% (95% CI: 82.3, 84.8) | False negative: 81 | True negative: 691 | NPV: 90% (95% CI: 87.4, 91.3) | False negative: 51 | True negative: 1,462 | NPV: 97% (95% CI: 95.8, 97.3) |
| Snc: 56% (95% CI: 51.9, 60.2) | Spd: 98% (95% CI: 97.3, 98.9) | n = 1,836 patients | Sn: 84% (95% CI: 80.4, 87.0) | Sp: 52% (95% CI: 49.1, 54.6) | n = 1,836 patients | Sn: 69% (95% CI: 61.9, 76.3) | Sp: 88% (95% CI: 85.9, 89.1) | n = 1,836 patients |
Diagnostic accuracy measures for comorbidities.
aPPV, positive predictive value.
bNPV, negative predictive value.
cSn, sensitivity.
dSp, specificity.
Timestamping
The date of admission was computationally validated against CIHI-DAD for 10,065 admissions, which showed 99% agreement. Manual validation for patients for date of admission, medication administrations, lab values, EKGs, telemetry order, and medication alerts showed 100% agreement. Manual validation of telemetry discontinuation time for 84 orders showed 87% agreement (95% CI: 77.8, 93.3). Manual validation for the date of MACE (other than death) based on 25 patients showed only 36% agreement (95% CI: 18.0, 57.5), as many diagnoses were not added to the problem list or medical history until days after the event or were already present from a previous event. Unfortunately, CIHI-DAD does not abstract the precise date of diagnosis. Manual validation for the date of death for 200 patients showed 98% agreement (95% CI: 94.3, 99.2) as the date of death was off by 1 day for 5 patients. Overall, timestamping had excellent validity (>95%) for most items except nonfatal MACE outcomes and telemetry discontinuation.
Discussion
Principal findings
We believe that our study is the first to present a thorough and comprehensive validation of key EMR data fields for pharmacoepidemiology research using the ubiquitous Epic EMR. ERD development to provide a clear roadmap to understand which data tables and fields were required was time consuming given the vast Epic Clarity database. The iterative process required for the ERD and data validation took approximately 11 months for both the senior research data analyst and lead pharmacist author. In the end, we showed that 34 of 46 data concepts required for pharmacoepidemiologic research had excellent validity with agreement over 95%. Specifically, the demographics, exposures, most potential confounders, and most timestamping items had very high levels of agreement (>95%), as these items reside primarily in structured data fields. Exposures and their timing with very high levels of agreement are important for causal analysis with outcomes so that the exposure occurred before the outcome. However, more work is required to improve diagnoses and timing for both outcomes and comorbidities.
These problematic diagnosis data extractions for our primary outcome of MACE are important data validity issues and are due to Epic not coding acute and chronic diagnoses in a structured data field from clinical notes unless they are manually added by providers in the problem list or medical history. Even if manually added by a provider, high rates of false-positive diagnoses may occur because of timing as these outcomes may remain on the problem list or medical history even years later, and clinicians may not update the timing of a diagnosis already in a problem list if the event recurred. This led to poor agreement for ventricular arrhythmia, which can be confirmed with EKGs and telemetry (Whitaker et al., 2023). However, our data had many historical diagnoses that were not updated for the hospital admission of interest. Linking EMR data to CIHI-DAD improved the validity of diagnoses, including the phase of care. However, we join others in advocating for improvements in CIHI’s handling of diagnoses. For some comorbidities, such as hypertension and diabetes, diagnoses are mandatory to be coded in CIHI-DAD, as well as any diagnosis that lengthens hospital stay, needs additional treatment compared with maintenance, or remarkably affects the hospital treatment (; ). This can lead to false negatives for some comorbidities, such as heart failure, which may not always be coded.
Natural language processing (NLP) using machine learning, by parsing clinician notes, could be helpful in finding diagnoses and making calculated assumptions about timing (). For example, studies are promising for identifying pneumonia as a diagnosis and classification of heart failure severity, with PPVs and sensitivities ranging from approximately 50% to 100% (; ). However, further work may be required for NLP to provide timestamping as the onset of psychosis symptoms in another study was correct only 71% of the time (Viani et al., 2021). Future work for this study will incorporate NLP of clinician notes for identification and timing of diagnostic outcomes, which could work well for those containing specific terms such as TdP.
Limitations of the study
Some limitations of this study include the nature of the comparator data set used, relatively small manual validation numbers for some items, and using percent agreement when there could be multiple areas of missing data. Given the >20,000 tables in Epic Clarity, it is possible, but unlikely, that there are other relevant tables that we missed or have not been activated that might validate better than the ones we used for this work. However, we iteratively validated the data to try to find the most appropriate tables for pharmacoepidemiologic research. Manual validation was conducted using a modest number of charts for each data concept because it is quite labor intensive. Validating more charts or different chart selection methods might change the level of agreement for each item. Using agreement as a validation statistic helped identify whether data items found in Epic were true, where CIHI-DAD was helpful in identifying missing data from Epic, but sometimes CIHI-DAD did not collect the required data.
Meaning and generalizability of the study
EMR research has been growing in popularity due to the efficiency of using large numbers of real-time observational data for many important clinical and health service questions where randomized controlled trials cannot be conducted or would not be appropriate (; ). There are many different potential sources of error in EMR data that necessitate data validation to identify and correct errors (Wang et al., 2023; Verma et al., 2021; ). Although this study was only conducted at a single site using Epic, the results and example ERD show the types of data that could be relied on based on our specific extractions from Epic Clarity versus data that require additional work to ensure validity, such as diagnoses. However, EMR data from any institution should be validated before conducting research to check the validity of each key data field.
Epic is a rapidly growing EMR that is already dominant in North America (; ). Sharing detailed ERDs for Epic among researchers and institutions could help reduce major inefficiencies when setting up an EMR data set for research. The ERD we created is more conceptual than the OMOP ERD, which is a standard for EMR data extraction (). However, it shows the types of data and linkages in Epic Clarity that we validated.
Epic also has an anonymized data set known as COSMOS from >300 million patients and 1,800 hospitals worldwide (). Institutions can opt-in to join the COSMOS data set, which contains demographics, hospital admission details, diagnoses, medications, and more (). However, free text in clinical notes is not submitted to COSMOS, which could miss diagnoses not documented in structured data fields (). Converting EMR data to a common data model, such as OMOP or the developing logical data model from CIHI, would lead to a standard structure of data allowing for collaborative research with institutions using Epic and other EMRs (; ).
Conclusion
Once an accurate ERD was formulated, most required data concepts for pharmacoepidemiologic research, including patient demographics and admission information, exposures, most potential confounders such as interacting drugs and lab results, and timestamping, were validated well. However, some diagnoses require a linked data source or the development of advanced techniques, such as NLP, to improve validity.
Statements
Data availability statement
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.
Ethics statement
The studies involving humans were approved by the Hamilton Integrated Research Ethics Board. The studies were conducted in accordance with local legislation and institutional requirements. Written informed consent for participation was not required from the participants or the participant’s legal guardians or their next of kin in accordance with the national legislation and institutional requirements.
Author contributions
TS: Visualization, Project administration, Validation, Formal Analysis, Methodology, Conceptualization, Data curation, Writing – review and editing, Investigation, Writing – original draft. TP: Data curation, Validation, Investigation, Resources, Writing – review and editing, Software, Formal Analysis. JS: Validation, Investigation, Writing – review and editing. LM: Writing – review and editing, Methodology. PG: Writing – review and editing, Methodology. AH: Resources, Conceptualization, Investigation, Funding acquisition, Writing – review and editing, Supervision, Project administration, Formal Analysis, Methodology, Writing – original draft.
Funding
The author(s) declared that financial support was received for this work and/or its publication. This work was supported by the Hamilton Academic Health Sciences Organization (HAHSO) #HAH-22-003, Canadian Institutes of Health Research (CIHR) #PJT-197768, and Mitacs Accelerate #IT35208.
Conflict of interest
The author(s) declared that this work was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Generative AI statement
The author(s) declared that generative AI was not used in the creation of this manuscript.
Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fphar.2026.1812502/full#supplementary-material
References
1
AbramsonJ. H. (2011). WINPEPI updated: computer programs for epidemiologists, and their teaching potential. Epidemiol. Perspect. Innov.8 (1), 1. 10.1186/1742-5573-8-1
2
AdejumoP.ThangarajP. M.DhingraL. S.AminorroayaA.ZhouX.BrandtC.et al (2024). Natural language processing of clinical documentation to assess functional status in patients with heart failure. JAMA Netw. Open7 (11), e2443925. 10.1001/jamanetworkopen.2024.43925
3
BayleyK. B.BelnapT.SavitzL.MasicaA. L.ShahN.FlemingN. S. (2013). Challenges in using electronic health record data for CER: experience of 4 learning organizations and solutions applied. Med. Care51 (8 Suppl. 3), S80–S86. 10.1097/MLR.0b013e31829b1d48
4
Becker’s Hospital Review (2024). Epic is no. 1 worldwid after adding 356 hospitals last year. Available online at: https://thisweekhealth.com/news_story/epic-is-no-1-worldwide-after-adding-356-hospitals-last-year (Accessed May 4, 2025).
5
BergerF. A.van der SijsH.van GelderT.van den BemtP. M. L. A. (2021). The use of a clinical decision support tool to assess the risk of QT drug-drug interactions in community pharmacies. Ther. Adv. Drug Saf.12, 2042098621996098. 10.1177/2042098621996098
6
BeukelmanT.ChenL.AnnapureddyN.OatesJ.ClowseM. E. B.LongM.et al (2023). Using pooled electronic health records data to conduct pharmacoepidemiology safety studies: challenges and lessons learned. Pharmacoepidemiol Drug Saf.32 (9), 969–977. 10.1002/pds.5627
7
BrenderJ.TalmonJ.de KeizerN.NykänenP.RigbyM.AmmenwerthE. (2013). STARE-HI - statement on reporting of evaluation studies in health informatics: explanation and elaboration. Appl. Clin. Inf.4 (3), 331–358. 10.4338/ACI-2013-04-RA-0024
8
Canadian Institute for Health Information (2015). Discharge abstract database (DAD) metadata. Available online at: https://www.cihi.ca/en/discharge-abstract-database-dad-metadata (Accessed May 4, 2025).
9
Canadian Institute for Health Information (2016). Data quality study of the 2015–2016 discharge abstract database: a focus on hospital harm. Available online at: https://www.cihi.ca/sites/default/files/document/data-quality-study-dad-2015-2016-report-en.pdf (Accessed March 26, 2026).
10
Canadian Institute for Health Information (2022). Canadian Coding Standards for Version 2022 ICD-10-CA and CCI. Available online at: https://secure.cihi.ca/free_products/canadian-coding-standards-2022-en.pdf (Accessed May 6, 2026).
11
Canadian Institute for Health Information (2024). Logical data model. Available online at: https://www.cihi.ca/en/connected-care/canadas-health-data-content-standard-and-architecture/logical-data-model (Accessed December 22, 2025).
12
ChapmanA. B.PetersonK. S.RutterE.NeversM.ZhangM.YingJ.et al (2022). Development and evaluation of an interoperable natural language processing system for identifying pneumonia across clinical settings of care and institutions. JAMIA Open5 (4), ooac114. 10.1093/jamiaopen/ooac114
13
ChishtieJ.SapiroN.WiebeN.RabatachL.LorenzettiD.LeungA. A.et al (2023). Use of epic electronic health record system for health care research: scoping review. J. Med. Internet Res.25, e51003. 10.2196/51003
14
Clinical & Translational Science Institute (2025). EHR Data for Research. Rochester, NY:University of Rochester Medicine. Available online at: https://www.urmc.rochester.edu/clinical-translational-science-institute/services-and-support/electronic-health-record-data-for-research (Accessed May 4, 2025).
15
CredibleMeds (2024). Risk categories for drugs that prolong QT & induce Torsades de Pointes (TdP). Available online at: https://crediblemeds.org/index.php/druglist (Accessed May 4, 2025).
16
DyrdaL. (2024). Epic is no. 1 worldwide after adding 356 hospitals last year. Available online at: https://www.beckershospitalreview.com/ehrs/epic-is-no-1-worldwide-after-adding-356-hospitals-last-year.html (Accessed May 4, 2025).
17
EhrensteinV.KharraziH.LehmannH.TaylorC. O. (2019). “Obtaining data from electronic health records,” in Tools and Technologies for Registry Interoperability, Registries for Evaluating Patient Outcomes: A User’s Guide Editors GliklichR. E.LeavyM. B.DreyerN. A.3rd Edition (Rockville (MD): Agency for Healthcare Research and Quality US) Available online at: https://www.ncbi.nlm.nih.gov/books/NBK551878 (Accessed May 4, 2025).
18
Epic Systems (2025a). Epic Data Handbook. Available online at: https://datahandbook.epic.com (Accessed May 4, 2025).
19
Epic Systems (2025b). Epic Systems. Cosmos. Available online at: https://cosmos.epic.com (Accessed December 21, 2025).
20
FernainyP.CohenA. A.MurrayE.LosinaE.LamontagneF.SourialN. (2024). Rethinking the pros and cons of randomized controlled trials and observational studies in the era of big data and advanced methods: a panel discussion. BMC Proc.18 (Suppl. 2), 1. 10.1186/s12919-023-00285-8
21
Food and Drug Administration (2005). E14 clinical evaluation of QT/QTc interval prolongation and proarrhythmic potential for non-antiarrhythmic drugs. Available online at: https://www.fda.gov/media/71372/download (Accessed February 2, 2026).
22
FordE.KimJ.KharraziH.GleasonK.GumasD.DeCampL. (2018). A guide to using data from EPIC, MyChart, and cogito for behavioral, social and systems science research. Available online at: https://ictr.johnshopkins.edu/wp-content/uploads/Phase1.Epic_.Social.Guide_2018.04.30_final.pdf (Accessed December 22, 2025).
23
GarciaM. C.TsangK.LohitS.DengJ.SchneiderT.Matos SilvaJ.et al (2024). Hydroxychloroquine-chloroquine, QT-Prolongation, and major adverse cardiac events: a meta-analysis and scoping review. Ann. Pharmacother.58 (7), 742–755. 10.1177/10600280231204969
24
GeddesJ.KatzJ.AsiimweA.BrookhartM. A.CarrollC.EldemirL.et al (2025). Quantification of information gained by linking claims data to an electronic health record cohort of patients with metastatic breast cancer. Pharmacoepidemiol Drug Saf.34 (10), e70213. 10.1002/pds.70213
25
GuoL. L.CalliganM.VetteseE.CookS.GagnidzeG.HanO.et al (2023). Development and validation of the SickKids enterprise-wide data in azure repository (SEDAR). Heliyon9 (11), e21586. 10.1016/j.heliyon.2023.e21586
26
GuranC. (2017). First in Canada epic end-to-end electronic medical record. Available online at: https://hospitalnews.com/first-canada-epic-end-end-electronic-medical-record/ (Accessed May 4, 2025).
27
HoevenL. R. V.BruijneM. C.KemperP. F.KoopmanM. M. W.RondeelJ. M. M.LeyteA.et al (2017). Validation of multisource electronic health record data: an application to blood transfusion data. BMC Med. Inf. Decis. Mak.17 (1), 107. 10.1186/s12911-017-0504-7
28
LewisA. E.WeiskopfN.AbramsZ. B.ForakerR.LaiA. M.PayneP. R. O.et al (2023). Electronic health record data quality assessment and tools: a systematic review. J. Am. Med. Inf. Assoc.30 (10), 1730–1740. 10.1093/jamia/ocad120
29
LougheedT. (2019). How American EMRs came to dominate Canadian health care. CMAJ191 (26), E749–E750. 10.1503/cmaj.109-5765
30
NewmanD. (2024). Epic systems modules & software – hyperspace, healthy planet. ClinDoc. Available online at: https://healthcareitskills.com/epic-systems-modules (Accessed May 4, 2025).
31
NewtonK. M.PeissigP. L.KhoA. N.BielinskiS. J.BergR. L.ChoudharyV.et al (2013). Validation of electronic medical record-based phenotyping algorithms: results and lessons learned from the eMERGE network. J. Am. Med. Inf. Assoc.20 (e1), e147–e154. 10.1136/amiajnl-2012-000896
32
NhamT.GarciaM. C.TsangK. J.SilvaJ. M.SchneiderT.DengJ.et al (2024). Proarrhythmic major adverse cardiac events with donepezil: a systematic review with meta-analysis. J. Am. Geriatr. Soc.72 (8), 2552–2565. 10.1111/jgs.18909
33
NordoA. H.LevauxH. P.BecnelL. B.GalvezJ.RaoP.StemK.et al (2019). Use of EHRs data for clinical research: historical progress and current applications. Learn Health Syst.3 (1), e10076. 10.1002/lrh2.10076
34
Observational Health Data Sciences and Informatics (2025). Standardized data: the OMOP common data model. Available online at: https://www.ohdsi.org/data-standardization (Accessed May 4, 2025).
35
Pedrera-JimenezM.Garcia-BarrioN.Rubio-MayoP.MaestroG.LaluezaA.Garcia-ReyneA.et al (2022). Making EHRs trustable: a quality analysis of EHR-derived datasets for COVID-19 research. Stud. Health Technol. Inf.294, 164–168. 10.3233/SHTI220430
36
PennDnA (2025). Epic Clarity. Available online at: https://www.med.upenn.edu/penndna/epic-clarity-data-warehousing.html (Accessed May 4, 2025).
37
SainiV.JaberT.ComoJ. D.LejeuneK.BhanotN. (2021). Exploring ‘slicer dicer’, an extraction tool in EPIC, for clinical and epidemiological analysis. Open Forum Infect. Dis.8 (1), S414–S415. 10.1093/ofid/ofab466.821
38
SchuemieM.LosR. (2025). OMOP CDM v5.4 ERD. Available online at: https://ohdsi.github.io/CommonDataModel/cdm54erd.html (Accessed May 4, 2025).
39
SheikhalishahiS.MiottoR.DudleyJ. T.LavelliA.RinaldiF.OsmaniV. (2019). Natural language processing of clinical notes on chronic diseases: systematic review. JMIR Med. Inf.7 (2), e12239. 10.2196/12239
40
ShrefflerJ.HueckerM. R. (2023). “Diagnostic testing accuracy: sensitivity, specificity, predictive values and likelihood ratios,” in Statpearls (Treasure Island (FL): StatPearls Publishing). Available online at: https://www.ncbi.nlm.nih.gov/books/NBK557491 (Accessed May 15, 2025).
41
St. Joseph’s Healthcare Hamilton (2025). Get to know St. Joe’s. Available online at: https://www.stjoes.ca/about/get-to-know-st.-joe-s (Accessed May 15, 2025).
42
TarabichiY.FreesA.HoneywellS.HuangC.NaidechA. M.MooreJ. H.et al (2021). The cosmos collaborative: a vendor-facilitated electronic health record data aggregation platform. ACI Open5 (1), e36–e46. 10.1055/s-0041-1731004
43
UC Davis Health (2025). Epic - SlicerDicer. Available online at: https://health.ucdavis.edu/data/epic-slicer-dicer.html (Accessed December 22, 2025).
44
VermaA. A.PasrichaS. V.JungH. Y.KushnirV.MakD. Y. F.KoppulaR.et al (2021). Assessing the quality of clinical and administrative data extracted from hospitals: the general medicine inpatient initiative (GEMINI) experience. J. Am. Med. Inf. Assoc.28 (3), 578–587. 10.1093/jamia/ocaa225
45
VianiN.BotelleR.KerwinJ.YinL.PatelR.StewartR.et al (2021). A natural language processing approach for identifying temporal disease onset information from mental healthcare text. Sci. Rep.11 (1), 757. 10.1038/s41598-020-80457-0
46
WangW.FerrariD.Haddon-HillG.CurcinV. (2023). “Electronic health records as source of research data. 2023 Jul 23,” in Machine Learning for Brain Disorders. Editor ColliotO. (New York NY: Humana). Available online at: https://www.ncbi.nlm.nih.gov/books/NBK597466 (Accessed May 4, 2025).
47
WeiskopfN. G.WengC. (2013). Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J. Am. Med. Inf. Assoc.20 (1), 144–151. 10.1136/amiajnl-2011-000681
48
WhitakerJ.WrightM. J.TedrowU. (2023). Diagnosis and management of ventricular tachycardia. Clin. Med. (Lond).23 (5), 442–448. 10.7861/clinmed.2023-23.5.Cardio3
49
WitowskiN. (2025). Top 10 EHR vendors in Canadian hospitals. Defin. Healthc.Available online at: https://www.definitivehc.com/blog/top-canadian-hospitals-ehr-vendors (Accessed December 22, 2025).
50
WuP.HurstJ. H.FrenchA.ChrestensenM.GoldsteinB. A. (2025). Linking electronic health record prescribing data and pharmacy dispensing records to identify patient-level factors associated with psychotropic medication receipt: retrospective study. JMIR Med. Inf.13, e63740. 10.2196/63740
Summary
Keywords
data accuracy, data validation, electronic medical records, epic, pharmacoepidemiology
Citation
Schneider T, Punjani T, Silva JM, Mbuagbaw L, Gibson P and Holbrook AM (2026) Characterization and validation of electronic medical record data for pharmacoepidemiologic research. Front. Pharmacol. 17:1812502. doi: 10.3389/fphar.2026.1812502
Received
16 February 2026
Revised
08 May 2026
Accepted
19 May 2026
Published
21 July 2026
Volume
17 - 2026
Edited by
Mohammed Salahudeen, University of Tasmania, Australia
Reviewed by
Andrea K. Wright, Zoetis, United States
Lucretia Udrescu, Victor Babes University of Medicine and Pharmacy, Romania
Smith Heavner, Critical Path Institute, United States
Updates
Copyright
© 2026 Schneider, Punjani, Silva, Mbuagbaw, Gibson and Holbrook.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Tyler Schneider, tschneid@stjoes.ca
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.