Trafimow () rejects the notion that data can militate against a theory if the data is wholly irrelevant to the theory. This very reasonable-sounding argument may not hold in the important (and common) case where competing models are being compared. The underlying issues may be clarified by way of example.
Consider the active debate on the contributions of fMRI to cognitive theory (e.g., Mather et al., ). It has been argued that fMRI adds little to cognitive theorizing, because data from fMRI are fundamentally irrelevant to cognitive theories pitched at the traditional computational level of analysis (Marr, ). A so-called “consistency fallacy” is said to be committed when data is treated as informative simply because some theory is consistent with it (Loosemore and Harley, ; Coltheart, ). This fallacy putatively arises because no other result could possibly obtain; neuroscientific data is simply irrelevant to cognitive theories at this level of analysis.
Both this argument and Trafimow's () more general variant of it are, however, incorrect when any competing theory assigns a differential prior to the obtained result. Take the general case described by Trafimow (): some theory assigns equal likelihood to all possible outcomes of an experiment. (This is just another way of saying the data are irrelevant to it.) The observed data will nonetheless reduce our belief in the original theory as compared to some competing theory, all else being equal, if this competitor had assigned a higher prior to the obtained outcome. Conversely, the observed data will increase our belief in the original theory as compared to a competing theory, if this competitor had assigned a lower prior to the obtained outcome. This argument holds regardless of whether the observed outcome is a significant difference or a null effect; Bayes' rule of course applies in both these conditions.
That said, it is true that no evidence can be offered regarding existing theories if they all assign equal likelihood to all outcomes, or if they cannot be subject to model comparison. In practice, such a situation might less often reflect sloppy inference than a failure to adequately articulate, develop or embellish a theory.
Statements
Acknowledgments
Thanks to David Badre for helpful discussion, and the National Institute of Neurological Disease and Stroke (R01 NS065046) for funding.
References
1
ColtheartM. (2013). How can functional neuroimaging inform cognitive theories?Perspect. Psychol. Sci. 8, 98–103. 10.1177/1745691612469208
2
LoosemoreR.HarleyT. (2010). 17 brains and minds: on the usefulness of localization data to cognitive psychology, in Foundational Issues in Human Brain Mapping, eds HansonS. J.BunzlM. (Cambridge, MA: MIT Press), 217–240. 10.7551/mitpress/9780262014021.003.0018
3
MarrD. (1982). Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. New York, NY: Henry Holt and Co, Inc.
4
MatherM.CacioppoJ. T.KanwisherN. (2013). 20 years of fMRI—what has it done for understanding cognition? [Special Section]. Perspect. Psychol. Sci. 8, 41–113. 10.1177/1745691612469036
5
TrafimowD. (2013). To disconfirm or not to disconfirm: a null prediction vs. no prediction. Front. Psychol. 4:733. 10.3389/fpsyg.2013.00733
Summary
Keywords
model comparison, consistency fallacy, accounting for data, Bayes Theorem, null prediction
Citation
Chatham CH (2013) The consistency fallacy and failures of theory embellishment. Front. Psychol. 4:965. doi: 10.3389/fpsyg.2013.00965
Received
04 November 2013
Accepted
07 December 2013
Published
25 December 2013
Volume
4 - 2013
Edited by
Rick Dale, University of California, Merced, USA
Copyright
© 2013 Chatham.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: chathach@gmail.com
This article was submitted to Theoretical and Philosophical Psychology, a section of the journal Frontiers in Psychology.
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.