Signal detection theory (SDT) has occupied a prominent role in the study of perception (Green and Swets, 1966), categorization (Ashby and Gott, 1988), memory (Pazzaglia et al., 2013, for a review), and more recently, reasoning (e.g., Rotello and Heit, 2009; Heit and Rotello, 2010). A key feature of SDT is that it provides tools to disentangle response strategies from other cognitive processes by formally specifying the underlying assumptions. SDT assumes that evidence is normally distributed on a strength dimension (e.g., brightness, line length, familiarity, or argument strength), and that the overlap between target and nontarget distributions determines judgment sensitivity. As the distance between the means of the distributions increases (i.e., as the distributional overlap gets smaller), sensitivity increases. It is also assumed that a participant will only respond “yes” if the strength of the item under consideration exceeds an internal criterion. This criterion can be shifted independently from sensitivity as a function of task demands or individual preferences. The placement of the criterion determines response bias. Certain conditions cause people to adopt a more conservative criterion, meaning that a larger strength value is required to respond “yes,” resulting in a decrease in overall positive responses. Other conditions cause people to adopt a more liberal criterion leading to more “yes” responses overall. The distinction between sensitivity and response bias is a critical one that has not always been adequately addressed by theories of reasoning.

In deductive reasoning, participants are asked to evaluate the logical validity of arguments. A standard belief bias experiment has logic and belief crossed to yield four types of arguments: valid-believable (VB), valid-unbelievable (VU), invalid-believable (IB), and invalid-unbelievable (IU). The dependent measure is how often each item type is endorsed as valid. Traditionally, three indexes are derived from the data: logic index=VB+VU−IB−IUbelief index=VB+IB−VU−IUinteraction index=VU+IB−VB−IU

The logic index is interpreted as a measure of reasoning sensitivity. The belief index is typically thought to indicate belief-based response bias. The interaction index is used as a measure of the effect of beliefs on accuracy, that is, enhanced reasoning in the face of unbelievable conclusions.

These interpretations of the effects of logic, belief, and the logic × belief interaction have been called into question by critics who note that the analysis of raw validity judgments relies on unstated assumptions about the nature of evidence that may not be valid (Klauer et al., 2000; Dube et al., 2010). Klauer et al. (2000) pointed out that the interaction index is difficult to interpret for psychometric reasons: changes in proportions starting out from different initial values, as is the case for valid and invalid arguments, cannot readily be compared across conditions. Klauer and colleagues addressed this issue by assuming that argument strength was uniformly distributed, specifying a threshold model as the underlying decision mechanism. Using the multinomial processing tree (MPT) modeling framework, they conducted an impressive series of experiments which culminated in the specification of the selective processing theory of belief bias, according to which beliefs affect both the response stage (response bias) and the reasoning stage (sensitivity, i.e., accuracy). Following up on this work, Dube et al. (2010) pointed out that MPT models and ANOVA of the traditional reasoning indices both predict a linear relationship between the hit rate (responding “valid” to valid problems) and the false alarm rate (responding “valid” to invalid problems). Empirical tests of this assumption, however, demonstrated that it did not hold for reasoning (Dube et al., 2010; see also Trippas et al., 2013, 2014; Heit and Rotello, 2014). Dube et al. put forward SDT as a theoretical framework capable of dealing with this nonlinear relationship for examining how conclusion believability affects reasoning sensitivity and response bias. In applying various SDT models to their own data from syllogistic reasoning tasks, they observed that beliefs did not affect reasoning sensitivity. Instead, the traditional logic × belief interaction was interpreted as originating from a belief driven response bias. This finding was surprising given that most extant theories predict changes in the quality of reasoning (we focus on these theories in more detail in the general discussion). Klauer and Kellen (2011) replied to Dube et al. arguing that MPT threshold models do not necessarily predict linear ROCs. This led them to propose an alternative MPT model capable of fitting curvilinear ROCs (Bröder and Schütz, 2009). Crucially, according to this updated MPT model, there was an effect of believability on both reasoning sensitivity and response bias. Furthermore, the superior fit of this alternative MPT model suggested that conclusions drawn from it were preferable to the SDT model. In turn, Dube et al. (2011) responded to Klauer and Kellen using model recovery simulations to demonstrate that the updated MPT model only fit the data better because it was more flexible than the SDT model, concluding that the SDT model was preferable. In other words, beliefs were thought to affect response bias only.

Following up on this discussion, Trippas et al. (2013) demonstrated that the picture may be more complex. Taking an SDT approach, they observed that while beliefs seemed to only influence response bias under some conditions, changes in sensitivity were evident in other conditions. They pointed to individual differences as an important mediating factor: reasoners with higher levels of cognitive ability were more likely to be influenced by their prior beliefs in the reasoning stage, in contrast to those with lower levels of cognitive ability, who mainly exhibited a belief-based response bias. Although individual differences are also studied in perception (Klein, 1951), categorization (Whitfield, 1984), and memory (e.g., Aminoff et al., 2012; Kantner and Lindsay, 2012), they have played a particularly important role in the development of reasoning theories (see Stanovich and West, 2000, 2008, for reviews), suggesting that it is perhaps unsurprising that ignoring them has led to conflicting findings.

The fine-grained conclusions drawn by Trippas et al. (2013) could not have been confidently reached without the use of a formal modeling procedure such as SDT. In fact, recent work by Heit and Rotello (2014) combined statistical simulations with an experimental approach to underscore that the issue extends even beyond the interaction index. According to their data, all traditional belief bias indices are problematic, with simulations showing that they lead to an (unacceptably) inflated Type 1 error rate—and thus unreliable conclusions. In this paper, we follow up on Heit and Rotello's work by describing three SDT indices which are designed to disentangle sensitivity and response bias, by explicitly estimating the various parameters of the underlying distributions of argument strength for each participant. In the final part of this paper, we then apply these indices to a case study in which the examination of individual differences (cognitive ability) is illuminating: the role of perceptual fluency on belief bias in syllogistic reasoning.

Fluency, the ease of processing a stimulus, has been a focus of interest in a variety of cognitive domains. In memory, fluency has been shown to influence both response bias and sensitivity. Some fluency-related memory illusions seem to be pure bias effects (e.g., Verde et al., 2010). On the other hand, perceptual disfluency (i.e., visual degradation) at encoding can produce better long-term recognition (e.g., Mulligan, 1996), a counterintuitive effect that has been linked to the enhanced effort needed to encode difficult-to-perceive materials. A similar effect of disfluency has been reported in reasoning. Alter et al. (2007) found that presenting reasoning problems in difficult to read font led to more accurate judgments. However, their finding has been difficult to replicate (Thompson et al., 2013). As Dube et al. (2010) noted in the study of belief bias, apparent inconsistencies in results can sometimes be traced to the use of analytic techniques that fail to appropriately distinguish between response bias and sensitivity. The studies of Alter et al. and Thompson et al. measured accuracy by analysing proportion correct, equivalent to the use of the traditional reasoning indices. They also exclusively focused on the effect of accuracy, ignoring any potential effects of fluency on response bias. This motivated our examination of the effect of disfluency on belief bias in syllogistic reasoning using the alternative SDT indices.

Although SDT has taken a prominent role in the development of theoretical accounts in domains such as perception, categorization, and memory, its application to reasoning has been a fairly recent development (e.g., Heit and Rotello, 2005). As has been previously argued in the context of belief bias, failure to specify assumptions about the nature of evidence can lead to interpretations of data that are misleading or incorrect (Klauer et al., 2000; Dube et al., 2010). The benefit of a formal model like SDT is in its specification of assumptions. The provision of analytic tools that allow the separation of sensitivity and response bias is an added advantage when it comes to the study of phenomena like fluency effects, which are known (in the domain of memory) to potentially impact both. Our findings from the manipulation of fluency in a reasoning task differed from those of two previous studies: we found that disfluency led to a reduction in reasoning sensitivity, in contrast to the improvement reported by Alter et al. (2007) and the null effect reported by Thompson et al. (2013). While we can only speculate about the extent to which differences in measurement tools may have contributed to inconsistent findings, it is important that the SDT indices used here do not suffer from the theoretical shortcomings noted with traditional indices. For the current study, the main advantage of the SDT-indices was a decreased probability of Type 1 errors, leading to increased confidence in our findings compared to previous research where the conclusions are drawn on the basis of more traditional analysis techniques.

Perceptual fluency mediated the link between cognitive ability and the belief-accuracy effect. When arguments were presented in a perceptually disfluent fashion, the link between cognitive ability and motivated reasoning disappeared. It may be that increased difficulty discourages or distracts from the use of what is essentially a higher-level strategy. On a similar note, Trippas et al. (2013) found that time pressure reduced both overall reasoning sensitivity and the belief-accuracy effect among higher ability participants. Drawing this comparison, it may be that a more parsimonious interpretation of the role of disfluency in reasoning is simply the following: making a task more difficult hogs the necessary resources typically used to afford an accuracy advantage in the face of unbelievable arguments.

While we failed to replicate the disfluency advantage, we did replicate other findings with respect to belief bias. Trippas et al. (2013) observed a logic × belief interaction effect, mediated by cognitive ability, that is usually interpreted as a belief-accuracy effect. Certain theories of belief bias (e.g., mental models theory: Oakhill et al., 1989; selective processing theory: Klauer et al., 2000; Evans et al., 2001; modified selective processing theory: Stupple et al., 2011; dual process theory (DPT): Evans, 2007) predict that the accuracy effect stems from a type of “motivated reasoning,” whereby the unbelievable nature of a conclusion triggers more effortful reasoning, leading to increased normative performance. Other theories (e.g., misinterpreted necessity: Evans et al., 1983; metacognitive uncertainty: Quayle and Ball, 2000; verbal reasoning theory: Polk and Newell, 1995; modified verbal reasoning theory: Thompson et al., 2003) predict that the interaction stems from an asymmetrical belief-based response bias for valid and invalid arguments, driven by various metacognitive backup strategies. These strategies are thought to be engaged when reasoning becomes too complicated. Finally, some theories (the response bias-only account: Dube et al., 2010; the probability heuristics model: Chater and Oaksford, 1999) do not predict a logic × belief interaction, although there may be ways to reconcile our findings with these theories. Pure response bias may hold under some conditions but not others (Heit and Rotello, 2014). The probability heuristics model might be extended to allow for a reasoning advantage in the face of unbelievable problems by assuming that believability influences the probability of using the p-entailment heuristic. Of the various theories just described, only three make explicit predictions about the role of individual differences in belief bias.

According to DPT, motivated reasoning requires working memory capacity (WMC), in contrast to response bias, which is a by-product of effortless heuristic processing (Evans, 2007). The finding that higher cognitive ability (a correlate of WMC) leads to increased motivated reasoning and decreased response bias is consistent with this theory. Modified selective processing theory, an algorithmic level reasoning theory part of the DPT framework similarly predicts that reasoning ability is linked to an increase in motivated reasoning and a decrease in response bias. One notable addition is the prediction of a curvilinear relationship between motivated reasoning and cognitive capacity: excellent reasoners who are near ceiling necessarily resist the influence of their prior beliefs, leading to perfect normative performance, thus showing no difference in accuracy as a function of believability. Metacognitive uncertainty, finally, predicts that those of lower WMC are more inclined to show the logic × belief interaction. The current findings run counter to this prediction.

How can future research further distinguish between these reasoning theories? Many critics would argue that the study of belief bias has focused on traditional or classical syllogisms for too long, ignoring other forms of reasoning more prevalent in daily life. The study of belief bias in statistical syllogisms, reasoning problems which include the quantifiers “Most” and “Few” alongside the traditional ones, will provide a fruitful direction. All of the belief bias theories discussed here evaluate reasoning competence with relation to the norm of classical logic. However, according to this framework, statistical syllogisms are invalid. If future experiments were to demonstrate accuracy changes as a function of beliefs and/or fluency for traditional and statistical syllogisms in equal measures, then this would suggest that traditional belief bias theories only capable of dealing with classical logic must be incorrect in their current form.