A fundamental challenge for cognitive neuroscience is to explain how the primitives of psychological theory are neurally implemented (Davis and Poldrack, 2013). Theories and models aimed at meeting this challenge are the modern manifestation of what Fechner (1860/1966) called “inner” psychophysics: the theory of the precise mapping between mental quantities and the brain. In Fechner's own time, inner psychophysics remained a dream (Scheerer, 1992). Even today, concrete proposals remain elusive. Indeed, many have wondered whether cognitive neuroscience is even up to the challenge (Price and Friston, 2005; Coltheart, 2006; Feldman Barrett, 2009; Poldrack, 2010). We side with those who have argued, more optimistically, that the field requires a shift in thinking for progress to continue (de Wit et al., 2016).

The key to inner psychophysics, we believe, is using psychological models, applied to neural activity, to predict behavior (Werner and Mountcastle, 1963; Britten et al., 1992). Consider the traditional motivation for using behavioral measures in (“outer”) psychophysics. Since the mind cannot be measured directly, Fechner and others reasoned that behavior can serve as a proxy to estimate stimulus-driven variation in mental quantities and processes. Similar reasoning supports behavioral measures as the key to developing “linking” hypotheses from psychological theory to the brain (Brindley, 1960; Teller, 1984). If a neural component implements a primitive identified by some theory or model, then we should be able to predict behavioral variation from its functional organization (Forstmann et al., 2011). We focus on representational linking hypotheses: how are the representations posited by psychological theory implemented by the brain in a manner that predicts behavior?

We propose that multi-variate pattern analysis (MVPA), or neural “decoding,” methods provide one starting point for the development of an inner psychophysics, and representational linking hypotheses. These methods have allowed researchers to investigate the information latent in neural activity patterns, and uncover the structure and content of the brain's population code (Kriegeskorte and Kievit, 2013; Haxby et al., 2014; Haynes, 2015). A cornerstone of these methods are linear classifiers applied to high-dimensional neural activation spaces. Here we present a simple approach for developing representational linking hypotheses based on the shared architecture of linear classifiers and observers under decision boundary models such as signal detection theory (Green and Swets, 1966). We also review our work on visual object categorization that lends empirical support to the approach (Carlson et al., 2014; Ritchie et al., 2015), and connect the approach to research on the neural loci of decision-making (Gold and Shadlen, 2007).

The suitability of MVPA methods for investigating neural representation, and developing representational linking hypotheses, can be motivated in part by their biological and (potential) psychological plausibility.

A common assumption in cognitive neuroscience is that the brain utilizes “population codes”: internal representations are implemented in distributed patterns of neural activity—incidentally, an idea somewhat anticipated by Fechner's (1882/1987) discussion of memory. If the brain uses population codes it may face a multivariate classification problem when differentiating these neural patterns. If this differentiation is achieved by a linear combinations of inputs, then we should be able to decode the contents of the encoding patterns of activity using classifiers that mirror the linear operations the brain employs. In decoding analyses, activation spaces are reconstructed from patterns of neural activity, and a linear classifier is trained to discriminate between the patterns for different experimental conditions. If the classifier performs significantly above chance, then minimally it can be inferred that information about the conditions is latent, and accessible, from the patterns of neural activity (Kriegeskorte and Bandettini, 2007). The biological plausibility of the linear classifiers also suggests that the information may be explicitly represented by the patterns.

While MVPA offers one starting point for developing an inner psychophysics, the biological plausibility of linear classifiers does not alone establish a connection between activation spaces and observer psychology. As de Wit et al. (2016) emphasize, that a classifier can learn to discriminate patterns of neural activity shows that information is latent, and perhaps represented, but not necessarily how it is being used, or is usable, by the observer (Cox and Savoy, 2003; Williams et al., 2007). In other words, the biological plausibility of linear classifiers is not enough to show that they are psychologically plausible, which also requires linking a psychological theory to an activation space. Fortunately, there is a long tradition in psychology of modeling the structure of psychological spaces to predict behavior (Attneave, 1950; Shepard, 1964; for a more recent perspective, see: Gärdenfors, 2000). All quantitative models of categorization within this tradition hold that tokens of a representation occupy different points in a space, and how these points are positioned in the space, based on some similarity or distance function, drives mental processes and behavior (Ashby and Maddox, 1993). For example, in prototype models (e.g., Posner and Keele, 1968) discriminability of a stimulus is determined by the distance of a representation to the central tendency of a category distribution in the space, and in exemplar models (e.g., Nosofsky, 1986) discriminability is determined by the relative similarity of the representation to all other exemplar representations in the space. The high-dimensional activation spaces reconstructed using MVPA may conform to similar principles of organization identified in these quantitative models of psychological space (Op de Beeck et al., 2008; Davis and Poldrack, 2013; Kriegeskorte and Kievit, 2013; Haxby et al., 2014). In which case, representational linking hypotheses can be developed by applying principles from models of psychological space to activation spaces.

One straightforward approach is to directly compare the structure of a psychological space to an activation space. Several studies using fMRI (Edelman et al., 1998; Mur et al., 2013; Charest et al., 2014; Sha et al., 2015; Bracci and Op de Beeck, 2016), cellular recordings (Op de Beeck et al., 2001) and MEG (Wardle et al., 2016), have constructed psychological spaces for stimuli from judgments of visual similarity, and compared them to activation spaces constructed using methods such as representational similarity analysis (RSA), which estimates the pair-wise (dis)similarity between patterns of neural activity for different conditions (Kriegeskorte et al., 2008a). A robust correlation between the two spaces suggests the activation spaces might implement the representations that are driving the similarity judgments. This similarity-based approach reflects the psychological plausibility of methods like RSA. Although seldom noticed in cognitive neuroscience, linear classifiers are also psychologically plausible, as we will illustrate.

As used in MVPA, linear classifiers specify a decision boundary through an activation space in order to discriminate between neural patterns produced by different experimental conditions. In general form this decision process is equivalent to that of the human observer as posited by (linear) decision boundary models of categorization for multivariate stimuli (Ashby and Gott, 1988; Ashby and Maddox, 1990). To illustrate the close correspondence of decoding methods with these models, consider that Naïve Bayes classifiers, applied after linear discriminate analysis (LDA), and observers under signal detection theory (SDT) share a common organization. More specifically, they make the same assumptions concerning: (i) distributions of evidence/data, and (ii) the evaluation of evidence/data by the classifier/observer.

At an abstract level, SDT specifies a number of primitives that mediate the relationship between stimulus and behavior (Figure 1A). Consider a simple task in which an observer must discriminate and map two stimuli (stim1, stim2) to two responses (resp1, resp2). The input produced by a stimulus is characterized as a sample from one of two distributions (f_stim1, f_stim2) along an evidence dimension, e.g., brightness, with response choice resulting solely from a rule applied to a decision variable (assuming equal stimulus probabilities and outcome utilities). Traditionally, the decision variable was the value of the log-likelihood ratio of the evidence, given the available hypotheses (i.e., the logarithm of the ratio of the height of f_stim1 and f_stim2 at a point on the evidence dimension). Assuming no response bias, the decision rule states that the observer selects the response with greater value in the ratio, resulting in a decision boundary along the evidence dimension. Under the usual distributional assumptions of normalcy and equal variance, the measure of observer sensitivity generated by the model, d', is the difference (or distance) between the means of f_stim1 and f_stim2 (it is also closely related to Fechner's own measure of sensitivity; Link, 1994). Architecturally, this model requires an internal stimulus representation along with a decision process that determines choice behavior given the information made explicit by the representation.

LDA is a technique for transforming a space to maximize between class variance (Duda et al., 2001). In the simplest case, a 2D-space is replaced with a single discriminant axis onto which each data point is projected. Importantly, it is assumed that the distributions for the classes along each dimension (or “feature”) are normal and of equal variance. If we further assume the dimensions are independent, then we have a Naïve Bayes classifier, which guesses based on the summed ratios of the posterior probabilities for each class along each dimension. If we take the logarithm of the ratio, and assume equal prior probabilities, then the classifier uses a decision rule applied to the log-likelihood ratio (Pereira et al., 2009). When a 2D space is projected to a single discriminant axis, the architectures of the classifier and the SDT observer are identical. In the multi-dimensional case, the classifier is akin to the decision boundary observer under the multi-dimensional generalization of SDT, when dimensions are independent (Ashby and Townsend, 1986).

An initial implication of this equivalence is that one may be able, in principle, to achieve close correspondence between classifier and observer performance. For example, consider the results of Philiastides and Sajda (2006) who observed similar psychometric and “neurometric” functions for human and classifier performance, when using an LDA classifier applied to EEG data. Not only do their results take on a new theoretical significance in light of the above equivalence, but methodologically their application of a sensitivity measure to classifier performance seems even more appropriate since the measure presupposes the very architecture that the classifier possesses (Tanner, 1956).

A further implication of the equivalence relates to reaction time (RT) and the speed of transforming representations of a stimulus into a decision. A simple feature of perceptual decision-making, as first characterized by SDT, is that the quality of evidence for an observer varies in its uncertainty (Tanner and Swets, 1954). In particular, evidence close to the observers decision boundary, or criterion, is more ambiguous, reflecting the greater likelihood of the evidence under the alternative hypotheses. In contrast, evidence far from the boundary is less ambiguous, reflecting greater likelihood of the evidence under one of the hypotheses about the source of the stimulus. Thus, relative to some decision boundary, evidence quality tends to vary with distance. RT also tends to vary with the quality of evidence: lower quality evidence results in longer decision times compared to high-quality evidence. A simple consequence of this familiar picture from decision boundary models (e.g., SDT), as developed with distance-to-bound models of choice and RT, is that distance from a decision boundary will negatively correlate with RT (Pike, 1973; Ashby and Maddox, 1994).

LDA classifiers learn to discriminate between activity patterns by positioning a decision boundary along a discriminant axis. If an activation space provides the evidence being utilized by an observer (Figure 1B), then one possibility is that distance from a classifier boundary will predict RT (Figure 1C). Such a result would suggest that an activation space implements an explicit representation of stimulus information that is used by the observer in a psychologically plausible manner. When decision boundary models like SDT were first developed, it was presumed that the evidence utilized by an observer was some unknown state of neural activity (Swets et al., 1961; Werner and Mountcastle, 1963). The neural distance-to-bound approach we have described provides a means of making good on this presumption. The approach is potentially quite general and is simple to implement as it relies on familiar MVPA and behavioral methods. We have also conducted experiments to test the approach.

We believe neural distance-to-bound has considerable potential as an (easy to apply) approach for developing representational linking hypotheses. Still, we stress that it is just one possible approach for furthering the study of inner psychophysics. In some domains of perception and cognition, it might not be viable at all. For example, it is unclear how well it will fair in a domain like neurolinguistics, where there is considerable difficulty in linking the primitives of linguistic theory to the brain (Poeppel, 2012). Furthermore, other approaches that may be superior at modeling RT, such as exemplar models (Nosofsky and Stanton, 2005), could provide a better connection between activation spaces and evidence accumulation.

More fundamentally, failure to observe a negative RT-distance correlation does not necessarily entail that the information in neural activity is unused. Instead, we might have the wrong model for how it is used. For instance, crucially our approach assumes linear separability, as using nonlinear classifiers for decoding is typically discouraged on the grounds that they are overpowered and lack biological plausibility (Kamitani and Tong, 2005; DiCarlo and Cox, 2007). However, many quantitative models of categorization do not share this assumption. For example, exemplar models have often been tested using stimulus sets that do not allow for linear separation in perceptual space (e.g., Nosofsky, 1986). Thus, the existence of psychologically plausible nonlinear categorization models may warrant revisiting the use of nonlinear classifiers in MVPA.

The ultimate import of our approach, then, is that it suggests more sophisticated representational linking hypotheses are possible. Recognizing this possibility is an important step along the path to an inner psychophysics, and the realization of Fechner's dream.

JR and TC contributed equally to developing the ideas presented in the paper. JR wrote, and TC helped edit, the manuscript.

This research was supported by an Australian Research Council Future Fellowship [FT120100816] to TC. The funders had no role in the preparation or decision to publish the manuscript.