AUTHOR=de Graaf Tom A. , Sack Alexander T. 

TITLE=When and How to Interpret Null Results in NIBS: A Taxonomy Based on Prior Expectations and Experimental Design

JOURNAL=Frontiers in Neuroscience

VOLUME=Volume 12 - 2018

YEAR=2018

URL=https://www.frontiersin.org/journals/neuroscience/articles/10.3389/fnins.2018.00915

DOI=10.3389/fnins.2018.00915

ISSN=1662-453X

ABSTRACT=In the dominant statistical approach of hypothesis testing, frequentist inference, null results are not informative. They provide no evidence for or against the alternative hypothesis, and do not confirm or reject the null hypothesis either. Despite this formal limitation of the statistical framework, researchers informally often do find null results meaningful to a greater or lesser extent. Specifically in the context of brain stimulation research, we previously discussed conceptually why, and under which circumstances, one might interpret and draw inferences from null results (de Graaf and Sack, 2011, Null Results in TMS: From Absence of Evidence to Evidence of Absence). We here add to that discussion in several ways. Firstly, we discuss how the increasingly common alternative approach, Bayesian inference, does allow quantification of the support for the null hypothesis, relative to support for the alternative hypothesis. But until Bayesian analyses are commonplace, it may be helpful to offer concrete handholds to assess the level to which traditional null results are informative. Along one gradient, from Replication nulls through Exploration nulls to Hypothesized nulls, null results can be less or more surprising, in the context of prior expectations, research, and theory. Orthogonal to this, experimental design choices create a gradient along which null results of an experiment, considered in isolation, become more informative; determined by target localization procedure, neural efficacy checks, and power and effect size evaluations. Along the latter dimension, we concretely propose three ‘levels of null evidence’. With caveats, these proposed levels A, B, and C reflect the strength of information provided by an empirical null result along concrete criteria. However, this exercise remains conceptual and informal, and is best complemented by Bayesian analysis whenever possible.