Commentary: Cognitive reflection vs. calculation in decision making

Sinayev and Peters (2015; hereafter SP namely, that the propensity or disposition to think analytically plays an important role in CRT performance (Pennycook et al., 2015b). We discuss recent empirical evidence that supports the claim that the CRT is more than just a measure of numeracy or, more generally, cognitive ability.


DISTINGUISHING COGNITIVE ABILITY AND ANALYTIC COGNITIVE STYLE
Dual process theorists often distinguish between disposition and ability as factors that determine good reasoning (e.g., Stanovich and West, 2000;Stanovich, 2009;Evans and Stanovich, 2013). The logic is as follows: If someone does not have the disposition or willingness to think analytically, they will not fully exercise their cognitive ability and will not do as well on the problem. Naturally, the converse is also true: If someone does not have sufficient cognitive ability, it will not matter how much time and effort they are willing to spend thinking about the problem.
This distinction has been applied to CRT performance. For example, according to Toplak et al. (2014): "the CRT is a measure of the tendency toward the class of reasoning error that derives from miserly processing. This may be why the predictive power of the CRT is in part separable from cognitive ability. The latter measures computational power that is available to the individual, but not necessarily the depth of processing that is typically used in most situations" (p. 165). That  Pennycook et al. (2014).

Dependant variable
Step Age Gender Education Income Cognitive Reflection Calculation Numeracy

Religious Belief
Step 1 0.35 0.15 −0.09 −0.03 Step 2  Traditional moral values Step 1 0.14 <0.01 −0.14 0.06 Step 2 0.14 −0.02 −0.12 0.07 −0.20 Step 4  each question in the CRT cues a compelling intuitive response means that responding correctly requires that individuals expend cognitive effort despite having what initially appears to be a suitable response . Importantly, this focus on thinking disposition does not imply that cognitive abilities (such as numeracy) are irrelevant for CRT performance. Rather, the claim is that the CRT indexes, to some degree, a disposition or propensity to think analytically (i.e., "analytic cognitive style") in addition to cognitive ability. Prima facie evidence for the importance of thinking disposition in solving the CRT comes from the finding that few people provide the correct responses (e.g., 30.3% for the bat and ball problem among undergraduate students; Pennycook et al., 2015b) despite the apparent simplicity of the math required to check the accuracy of the intuitive response (e.g., for the bat and ball problem: 0.10 + 1.10 = 1.20 = 1.10).

IS THE CRT JUST ANOTHER NUMERACY TEST?
If the CRT captures some aspect of analytic cognitive style, it should be predictive of a wide range of judgments and decisions. However, if the CRT is "primarily a measure of numeric ability" (S&P, p. 3), then it should only robustly predict judgments and decisions that require some sort of mathematical operation. There is emerging evidence that analytic cognitive styleand the CRT in particular-is predictive of diverse psychological outcomes that are not traditionally associated with research in decision making (Pennycook et al., 2015c). For instance, higher scores on the CRT are associated with religious disbelief (Gervais and Norenzayan, 2012;Pennycook et al., 2012;Shenhav et al., 2012), paranormal disbelief (Pennycook et al., 2012;Cheyne and Pennycook, 2013), less traditional moral values (Pennycook et al., 2014;Royzman et al., 2014), improved scientific understanding and reasoning (Shtulman and McCallum, 2014;Drummond and Fischhoff, 2015), belief in evolution (Gervais, 2015), creativity on complex tasks , less reliance on Smartphone technology as an external information source (Barr et al., 2015b), and lowered receptivity to pseudo-profound bullshit . Indexes of cognitive ability were included as controls in many of these studies (Pennycook et al., 2015c), including, in some cases, established numeracy tests (Pennycook et al., 2014Barr et al., 2015a,b;Trippas et al., 2015). With few exceptions, analytic cognitive style measures (including the CRT) were predictive after controlling for cognitive ability (including numeracy; Pennycook et al., 2015c).
What, then, of the two new studies presented by S&P? That CRT performance was not predictive over-and-above numeracy may simply provide evidence that the aspect of CRT performance that reflects thinking disposition does not play a role in the types of decisions that S&P investigated. This seems particularly likely with respect to the incentivized outcomes in Study 2 (as discussed by S&P, p. 12) because the very goal of incentivizing tasks in behavioral research is to minimize dispositional variance. We suggest that a stronger test of the role of thinking disposition over-and-above numeracy would be in judgments or decisions in "naturalistic" contexts where there is no clear prompt or direct incentive to think analytically . S&P did measure some real-world outcomes (e.g., saving money for retirement). However, examined outcomes all included direct incentives (e.g., monetary reward). The evidence highlighted above indicates that CRT performance is predictive for beliefs or judgments that not only lack incentives, but lack normatively correct or incorrect outcomes.
As a case study, consider the results of Pennycook et al. (2014) presented in Table 1. This study focused on predicting religious belief, traditional moral values (e.g., trust in authority, concerns over bodily purity), and disgust-based moral dilemmas. Importantly, none of these constructs has any theoretical association with numeracy, but they do involve compelling intuitions or societal defaults that could be influenced by the disposition to think analytically (for further detail, see Pennycook et al., 2014). As expected given our account, numeracy and "calculation" (using S&P's CRT scoring technique) are not significant predictors of any outcome variable, whereas "cognitive reflection" is a robust predictor for all three (see Supplementary Materials for further details about this analysis). Note, however, that it would be inappropriate to conclude on this basis that numeracy has nothing to do with CRT performance. Rather, the purpose of this analysis is to show that there are some instances where the CRT predicts an outcome even after controlling for numeracy. This indicates that the CRT is more than just a numeracy measure. Similar analyses have been done with a variety of outcome variables and with a variety of cognitive abilities as control variables (see Pennycook et al., 2015c).

CONCLUSION
What is the role of analytic cognitive style and cognitive ability in decision making? Although, the answer undoubtedly depends on the sort of decision being made, we have drawn attention to evidence that the CRT is predictive of a wide range of outcomes, even after controlling for cognitive ability (Pennycook et al., 2015c). This provides evidence that CRT performance reflects, at least to some degree, a propensity or willingness to think analytically and the CRT, therefore, is not "primarily a measure of numeric ability" (S&P, p. 3).
Nevertheless, we acknowledge that a propensity to think analytically does not play an important role in all (or perhaps even most) decisions that people make in their day-to-day lives. Moreover, it is clear that the role of numeracy in CRT performance has not been adequately acknowledged by dual-process theorists. Future research could profitably follow S&P's lead by further investigating which types of decisions depend on numeracy, but not cognitive style (and vice versa). This will require indices of both analytic cognitive style and cognitive ability (and, in particular, numeracy), as well as a more nuanced hypotheses about what factors explain performance on reasoning and decision making tasks.

AUTHOR CONTRIBUTIONS
GP wrote the initial draft of this manuscript. RR provided critical revisions.

SUPPLEMENTARY MATERIAL
The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fpsyg. 2016.00009