Estimating real-world treatment effects in the presence of measurement error and sparse outcome data using propensity score methods

Jane Burnell¹Amitava Banerjee²Gordon Prescott¹Chris Sutton³Svetlana Tishkovskaya^1*

• ¹Lancashire Clinical Trials Unit, University of Central Lancashire, Preston, England, United Kingdom
• ²Institute of Health Informatics, Faculty of Population Health Sciences, University College London, London, England, United Kingdom
• ³School of Health Sciences, Faculty of Biology, Medicine and Health, The University of Manchester, Manchester, England, United Kingdom

Introduction: The real-world treatment effect of a novel treatment can be estimated by analysing routinely collected patient data, in the form of Electronic Health Records (EHR). Any treatment allocation in EHR is not randomised and there may be systematic differences between the treatment groups. Propensity Score (PS) methods are commonly used to correct for these differences and reduce the bias in the treatment effect estimate. The aims of the study were to compare the performance of the most popular PS methods in the estimation of the treatment effect in the presence of two common issues in EHRs: covariate measurement error and sparse data. Methods: The motivational example for this study was the assessment of the treatment effect of the novel oral anti-coagulant Rivaroxaban compared with the previous standard treatment Warfarin for the prevention of future stroke in patients with atrial fibrillation. Using simulation experiments based on a dataset comparing Rivaroxaban with Warfarin, we evaluated the performance of four PS methods. Results: In the simulations with characteristics of the original dataset, using 3:1 PS matching generated a largest bias of +0.0428 (corresponding ratio of HRs (rHR) 1.0437), whereas for the other PS methods it was smaller and in negative direction: IPTW for ATE -0.0181 (rHR=0.9821); IPTW for ATT -0.0110 (rHR=0.9891); PS stratification -0.0099 (rHR=0.9901), with relative differences between rHRs being small to negligible. Fifty percent under-recording of a covariate (stroke) in the PS model, increased the MSE between 6%-11% compared to the MSE with no introduced measurement error. While 50% over-recording reduced the MSE by around 35%. The difference in the bias of the low prevalence outcome (0.5%) and the high prevalence outcome (10%) was: IPTW for ATE 0.1514 (rHRs=1.1635); IPTW for ATT 0.0160 (rHRs=1.0161); 3:1 PS matching 0.0758 (rHRs=1.0787); PS Stratification 0.0177 (rHRs=1.0179). A similar pattern for outcome prevalence was seen for all the simulation scenarios. Conclusions:This study showed that PS methods proposed in the literature may not all perform well for individual datasets. The findings produced recommendations for using PS methods in the estimation of real-world treatment effect when the covariate measurement error and sparse outcome data are present.