AUTHOR=Capers Patrice L. , Brown Andrew W. , Dawson John A. , Allison David B. 

TITLE=Double Sampling with Multiple Imputation to Answer Large Sample Meta-Research Questions: Introduction and Illustration by Evaluating Adherence to Two Simple CONSORT Guidelines

JOURNAL=Frontiers in Nutrition

VOLUME=2

YEAR=2015

URL=https://www.frontiersin.org/journals/nutrition/articles/10.3389/fnut.2015.00006

DOI=10.3389/fnut.2015.00006

ISSN=2296-861X

ABSTRACT=<p><bold>Background:</bold> Meta-research can involve manual retrieval and evaluation of research, which is resource intensive. Creation of high throughput methods (e.g., search heuristics, crowdsourcing) has improved feasibility of large meta-research questions, but possibly at the cost of accuracy.</p><p><bold>Objective:</bold> To evaluate the use of double sampling combined with multiple imputation (DS + MI) to address meta-research questions, using as an example adherence of PubMed entries to two simple consolidated standards of reporting trials guidelines for titles and abstracts.</p><p><bold>Methods:</bold> For the DS large sample, we retrieved all PubMed entries satisfying the filters: RCT, human, abstract available, and English language (<italic>n</italic> = 322, 107). For the DS subsample, we randomly sampled 500 entries from the large sample. The large sample was evaluated with a lower rigor, higher throughput (R<sub>LO</sub>T<sub>HI</sub>) method using search heuristics, while the subsample was evaluated using a higher rigor, lower throughput (R<sub>HI</sub>T<sub>LO</sub>) human rating method. Multiple imputation of the missing-completely at-random R<sub>HI</sub>T<sub>LO</sub> data for the large sample was informed by: R<sub>HI</sub>T<sub>LO</sub> data from the subsample; R<sub>LO</sub>T<sub>HI</sub> data from the large sample; whether a study was an RCT; and country and year of publication.</p><p><bold>Results:</bold> The R<sub>HI</sub>T<sub>LO</sub> and R<sub>LO</sub>T<sub>HI</sub> methods in the subsample largely agreed (phi coefficients: title = 1.00, abstract = 0.92). Compliance with abstract and title criteria has increased over time, with non-US countries improving more rapidly. DS + MI logistic regression estimates were more precise than subsample estimates (e.g., 95% CI for change in title and abstract compliance by year: subsample R<sub>HI</sub>T<sub>LO</sub> 1.050–1.174 vs. DS + MI 1.082–1.151). As evidence of improved accuracy, DS + MI coefficient estimates were closer to R<sub>HI</sub>T<sub>LO</sub> than the large sample R<sub>LO</sub>T<sub>HI</sub>.</p><p><bold>Conclusion:</bold> Our results support our hypothesis that DS + MI would result in improved precision and accuracy. This method is flexible and may provide a practical way to examine large corpora of literature.</p>