AUTHOR=Dharmarathne Gayan , Hanea Anca , Robinson Andrew P. 

TITLE=Improving the Computation of Brier Scores for Evaluating Expert-Elicited Judgements

JOURNAL=Frontiers in Applied Mathematics and Statistics

VOLUME=Volume 7 - 2021

YEAR=2021

URL=https://www.frontiersin.org/journals/applied-mathematics-and-statistics/articles/10.3389/fams.2021.669546

DOI=10.3389/fams.2021.669546

ISSN=2297-4687

ABSTRACT=Structured expert judgement (SEJ) is a suite of techniques that are used to elicit expert predictions, for example probability predictions of the occurrence of events, for situations in which data are too expensive or impossible to obtain. The quality of expert probability predictions can be assessed using Brier scores and calibration questions. In practice, these scores are computed from data that may have a correlation structure due to sharing the effects of the same levels of grouping factors of the experimental design. For example, asking common questions from experts may result in correlated probability predictions due to sharing common question effects. Mixed-effects models incorporate design-based correlation structures of observations by permitting suitable random effects. Here, we focus on (i) improving the computation of standard error estimates of expert Brier scores, and (ii) imputation of missing probability predictions in computing expert Brier scores to enhance the comparability of the prediction accuracy of experts using mixed-effects models with question effects as random effects. We show that the accuracy of estimating standard errors of expert Brier scores and imputing missing probability predictions can be improved by incorporating the within-question correlations due to asking common questions. We also discuss the implications of adopting a formal experimental design approach for SEJ exercises.