The seminal investigations of Hubel and Wiesel () established that the receptive field properties of single neurons in V1 emerge from the integration of neurons in the previous processing stage. Since then, it is commonly believed that visual information is processed in a hierarchical fashion, consisting predominantly of feedforward feature integration and convergence. Distinct streams of processing are initially kept anatomically separate, only to come together in higher order areas by specialized cells (Barlow, 1972) – or, alternatively, to be organized functionally, in time through synchronized activity (Von Der Malsburg, 1994; Singer and Gray, 1995).

In this perspective, the earlier stimulus representations are characterized by segregation. The segregation hypothesis postulates that different attributes of visual stimulation are being received and processed by distinct populations of neurons. (Zeki, 1978; Hubel and Livingstone, 1987; Livingstone and Hubel, 1988) and contrasts with integrality, which permits populations of neurons to have mixed selectivities.

More recent studies have taken issue with the notion of segregation (see Lennie, 1998; Gegenfurtner and Kiper, 2003; Shapley and Hawken, 2011 for reviews) as the most viable option for understanding visual representation. It has been shown, for instance, that motion and disparity are encoded jointly in certain subpopulations of neurons (Roy et al., 1992; DeAngelis et al., 1998; Anzai et al., 2001; Pack et al., 2003; Grunewald and Skoumbourdis, 2004). Such results suggest that it may be time to reconsider the segregation model.

Here we will review the status of the evidence for segregation of color and form. This is of special interest because color and form are most likely processed along the ventral cortical pathway (Ungerleider and Mishkin, 1983) and yet, this pair has been regarded as extremely segregated, not only represented in separate neurons, but also in distinct brain regions. For this reason, segregation of color and form is more controversial than that of color or form versus motion. Dubner and Zeki (1971) first observed a region in the superior temporal sulcus of the macaque that was selective for direction of motion but unresponsive to color or orientation. Zeki (1973, 1974, 1977) showed evidence that areas in extrastriate cortex functionally differ from each other; he proposed that area V4 is specialized for color and V5 (MT) for motion. In later studies, Hubel and Livingstone (1987; Livingstone and Hubel, 1988) observed that color and form are processed in distinct regions within both the primary (V1) and prestriate (V2) cortex. Motion is considered to be processed along the dorsal stream. Lennie (1998) argued that motion could be a special case and “the separation of motion signals from signals about other dimensions of image variation means the analysis they subserve is self-contained.” We will not contest this here.

We concentrate on the ventral stream, and investigate if any further subdivisions are necessary. We do not wish to claim, however, that visual object information is exclusively processed in the ventral stream. There is evidence that object representations exist in parallel in both dorsal and ventral streams (Konen and Kastner, 2008). The dorsal information pathway is thought to be involved in the encoding of spatial relationship of objects (Mishkin et al., 1983). It could thus be the case that dimensions related to spatial relationships among objects are anatomically separate from the ones that define an object.

In our present review of color and form segregation we argue, first, that the segregation view does not square well with behavioral findings, including those on attentional feature integration that have traditionally been interpreted in a segregation-friendly framework (Treisman and Gelade, 1980). Next, we re-examine some of the classical neurophysiological studies demonstrating segregated processing of color and form, along with more recent evidence that may undermine the necessity of segregation as the best possible explanation. In areas predominantly representing color or form, weak selectivities for the other feature also exist. The predominance of studies in single neurons has hitherto obscured the role of weak selectivities in distributed coding. Weak selectivities can have a strong collective effect in a neuronal population. Their presence, and generally that of mixed selectivities in single neurons, enables neuronal population codes with flexible and context sensitive feature representations, properties that have been shown to exist in early visual cortex. We discuss how neuronal population codes could be used in perceptual integration of color and form via feedforward, and feedback and horizontal (recurrent) perceptual mechanisms.

Early, influential studies on cortical processing have shown evidence of spatially separate populations of neurons being sensitive to different features of a visual scene. For one, studies in V1 and V2 of the primate cortex have indicated regions with distinct anatomical characteristics. Staining with mitochondrial enzyme cytochrome oxidase (CO), revealed alternating dark and light regions in layers 2/3 of V1, a result that indicates high and low concentrations of CO respectively in V1 (Humphrey and Hendrickson, 1980; Horton and Hubel, 1981). The darker stains in sections tangential to the cortical surface were coined blobs, in accordance with their three dimensional, oval shapes, and the lighter stained regions were called interblobs. V2 shows a different, but equally interesting pattern of patches when stained for CO. Instead of oval shapes, tangential sections show an alternation of dark stripe and light interstripe regions; the dark stripes are of two types; thick and thin ones (Livingstone and Hubel, 1982; Tootell et al., 1983). From tracer injections, Livingstone and Hubel (1984, 1987a) showed evidence that the thin stripes are connected to the blobs, the interstripes to the interblobs, and the thick stripes to layer 4B of V1.

Livingstone and Hubel (1984, 1988) proposed that the anatomically segregated regions in early visual cortex have distinct functional properties. They suggested a link between the CO regions and the magnocellular (M) and parvocellular (P) retino-geniculo-cortical pathways. Whereas the M pathway projects from layer 4B in V1 to the thick stripes in V2 and is selective for depth and motion the P pathway is subdivided into two streams; one passing through the blobs in V1 and the thin stripes in V2 that mediates color and another one passing through the interblobs in V1 and the interstripes in V2 that mediates form (Figure 1). The authors concluded that double opponent cells (cells exhibiting both color and spatial opponency) in V1 blobs are not orientation selective and have low spatial acuity. Edges, they suggested, are signaled by cells in the interblob area in V1. While these cells are orientation selective, they are not color opponent; they can respond to a luminance or color edge regardless of its color but cannot code the color information of the edge (Hubel and Livingstone, 1987; Livingstone and Hubel, 1988). Additional physiological studies have supported the idea that within area V2, separate anatomical regions have distinct functional properties (Hubel and Livingstone, 1985, 1987; Shipp and Zeki, 1985; Tootell and Hamilton, 1989; Ts’o et al., 1990; Malach et al., 1994; Roe and Ts’o, 1995; Moutoussis and Zeki, 2002).

Since then a number of electrophysiological studies have challenged this segregated view on V1 and V2. Lennie et al. (1990) measured the responses of cells in layers 2/3 of V1 and found that cells inside and outside blobs did not have different chromatic properties. Friedman et al. (2003) measured, in layers 2/3 of V1 and in V2, the selectivity of cells for color, orientation and border position from alert macaque monkeys. They found no correlation between any of the selectivities. Leventhal et al. (1995) recorded cells from layers 2/3 and 4 in V1 and also found no correlation between orientation and color selectivity. Clearly, based on the segregation view, a negative correlation would have been predicted. Similarly to Lennie et al. (1990), there was no difference observed in the response properties between cells outside and inside the V1 blobs (Leventhal et al., 1995). Using implanted 100 electrode arrays in V1, Economides et al. (2011) found very subtle differences in orientation tuning between neurons in blobs and interblobs; the mean orientation bandwidth of cells in blobs was 28.4 and in interblobs 25.8°. The most pronounced difference was in activity: blob cells had 49% higher firing rates than interblob cells.

A CO blob system has also been found in primates with no color vision (Condo and Casagrande, 1990). O’Keefe et al. (1998) measured the response of V1 neurons in the nocturnal, New World monkey (a species containing only a single cone type). They found no difference in orientation tuning, eye dominance, temporal frequency tuning and contrast response for neurons in blobs and interblobs. The repeating anatomical patterns found in the visual cortex and other parts of the brain, Purves et al. (1992) argued, do not reflect a fundamental functional principle but rather are byproducts of developmental requirements.

Johnson et al. (2001) divided the neurons from which they recorded in V1 into three groups, depending on their sensitivity to color and spatial patterns of luminance. Most of the neurons strongly preferred luminance patterns compared to colored ones (60% luminance cells); fewer neurons showed strong color selectivity (11% color cells). Interestingly enough, a considerable percentage of neurons were selective to both luminance and color patterns (29% color-luminance cells). Color-luminance cells did not respond or responded poorly to patterns of low spatial frequency (<0.5 cycles per degree); instead they showed a band-pass tuning similar to luminance cells. Most color cells were low pass in their spatial frequency tuning. In a later study Johnson et al. (2004) concluded that color-luminance cells are double opponent. In contrast to the Hubel and Livingstone studies, Johnson et al. (2008) found that most double opponent color-luminance cells are also orientation selective.

In reviewing the functional segregation of early visual areas, Gegenfurtner (2003) collected results from six studies in which cells from the distinct CO compartments in V2 (thin stripes, thick stripes, interstripes) were examined (DeYoe and Van Essen, 1985; Peterhans and von der Heydt, 1993; Levitt et al., 1994; Roe and Ts’o, 1995; Gegenfurtner et al., 1996; Kiper et al., 1997). According to the segregation perspective, cells in the interstripes are selective to form and cells in the thin stripes are selective to color. The averages from these studies confirmed that cells in the thin stripes are most selective for color, cells in the thick stripes and interstripes are most selective for orientation and cells in the thick stripes are most selective for direction of movement (Figure 2). Nevertheless, cells in each compartment were selective for other features as well. The results show that to a considerable extent, the selectivities within both the interstripe and thin stripe regions are mixed, especially for color and form. Around 30% of cells in the interstripes are selective for color and around 40% of cells in the thin stripes are selective for form.

Several studies discussed in this section have questioned the hypothesis that neurons in different CO compartments process separate dimensions of the visual scene. If, on the one hand, there is some degree of anatomical and functional specialization in the brain, why are there these mixed selectivities in the different CO compartments? Or why are there neurons with more than one selectivity in the early visual cortex? If, on the other hand there is no anatomical and functional segregation in the brain why is there a bias for certain features in different CO compartments?

All of the above-mentioned electrophysiological studies analyzed neural activity as if each neuron acted as an independent computational unit, i.e., without considering the possible role of interactions between neurons. Individual neurons with broad selectivities to color or to orientation were categorized as non-selective to color or to form, respectively. Yet, perhaps, perceptually significant information does not arise at the single neuron level, but from a population of neurons. The combination of responses from a population of neurons may reveal robust decoding for conditions where individual neurons show broad selectivity. Neurons, as we discuss in this section, could have mixed selectivities with unequal tuning widths for different features; however, a population code consisting of inputs from neurons like that may show sharp tuning for all features. In the next section we discuss studies that examine possible ways a population of neurons can encode information and what are the attributes of neurons that make encoding of information optimal.

Early influential studies on color and form processing promoted the view that perceptually significant information happens at the single neuron level (Zeki, 1978; Livingstone and Hubel, 1988). An analysis adhering to this view can overlook the possibility that weak selectivities at the single neuron level encode information at the population level. Evidence supports the notion that the brain processes information by combining signals from neuronal populations. Firstly, repeated presentations of the same stimulus evoke considerably variable responses from a single neuron (Tolhurst et al., 1983; Vogels and Orban, 1990). If the activity of single neurons represented perceptually significant activity we would expect less variability in the response of a neuron after repeated presentations of the same stimulus. This leads us to the next point; single neurons in the visual cortex have weak correlations with behavioral decisions (Britten et al., 1996; Shadlen et al., 1996). Finally, the structural features of neurons suggest the formation of distributed circuits with long range connectivity (Alexander and van Leeuwen, 2010; Yuste, 2011).

Contrasting with the single neuron viewpoint, the response of a single neuron gives an ambiguous response by itself and can only provide sufficient information if considered in conjunction with the responses of the rest of the neurons forming a network. In line with this perspective, Lehky and Sejnowski (1988) showed that selectivity of single neural units could give misleading information on the function of a neural network. Population coding analysis examines how information is represented from the pattern activity in a group of neurons. In an influential study on population coding, Georgopoulos et al. (1986) represented the activity of each neuron recorded in the arm area of the primate motor cortex as a vector pointing in a specific direction in 3-D space. The vector associated with each cell was weighted according to the activity of that cell, and then all the vectors were summed. The direction of the vector sum was in close approximation to the direction of the arm movement of the monkey despite the broad tuning of single cells.

Wachtler et al. (2003) examined whether the activity of a population of neurons in macaque V1 can represent color perception. Population responses were expressed as vectors, with each element of the vectors representing the activity of a single neuron. The authors found that distributed neural response changes with different backgrounds corresponded with induction effects in color perception (shown in a follow-up experiment with human participants). An example of the authors’ analysis is shown in Figure 3. In Figure 3A, color patches (c) and (b) are physically identical but appear different because they are displayed on different backgrounds. Furthermore, color patches (a) and (b) appear similar even though they are physically different. In Figure 3B, the pattern of responses of four neurons of patch (b) is more similar to that for patch (a) than for patch (c), indicating that the population response of neurons in V1 correlates with color perception. Note that decoding in this study is represented by a vector with the activities of all the neurons. In the Georgopoulos et al. (1986) study each neuron was represented by a position vector pointing at the preferred direction of that neuron; the decoded direction was given as the weighted average of all the vectors. Thus, perhaps, the rules for information processing from a population of neurons depend on the nature of the target feature.

Our understanding of how a population of neurons could represent information has been facilitated by studies that link machine learning principles with neural processing (Buonomano and Maass, 2009; Rigotti et al., 2013). In this framework, neurons that have small responses to a particular feature or to a combination of features can be crucial in the encoding of distributed information, whereas these weak selectivities would be hard to interpret in single neuron analysis. To date, this issue has predominantly been investigated in the prefrontal cortex. In the remainder of this section we focus on neurons in prefrontal cortex. We argue in the next section that if mixed selectivity is a property of neurons throughout the cortex, a simple assumption of non-linearity will enable us to explain the often conflicting results on the selectivity of neurons in early visual cortex discussed in the previous section.

In a recent study, Rigotti et al. (2013) analyzed activity of neural populations in prefrontal cortex (PFC) while monkeys performed a memory task. The authors showed that the dimensionality of the population code is higher when single neurons are tuned to a non-linear mixture of conditions compared to when they respond exclusively to one condition or a linear mixture of conditions. The concept of dimensionality generally refers to the minimum number of coordinates that are needed to fully specify all the points of a set of vectors. For example, two vectors that are linearly dependent (one is a multiple of the other) are one-dimensional (they lie on a line); if they are linearly independent they are two-dimensional (they lie in a plane). Higher dimensionality leads to a more versatile code since the number of possible classifications of a linear classifier between two conditions grows exponentially with dimensionality. This means that a population of neurons that represent information in a high dimensional space has the capacity to perform complex tasks. Figure 4 shows neurons with different selectivities and their effect in the dimensionality of a neural population code. Neurons 1 and 2 show pure selectivity to feature a and b of some stimuli, respectively, neuron 3 shows linearly mixed selectivity to both features and neuron 4 is non-linearly selective to both features (Figure 4A). In Figure 4B, the representation of the stimuli by the pure and linearly mixed neurons is low dimensional (it is on a line). However, in Figure 4C we see that if we substitute one of the neurons with a non-linearly mixed one, then the representation will be on a higher dimensional space (on a plane).

Can we lose information about the selectivity of a neural population by averaging its responses? To test this hypothesis, Rigotti et al. (2013) removed the classical selectivity from a population of neurons from a set of conditions and then tested whether the conditions could still be predicted from the response of the neurons. Classical selectivity refers to the average differences between conditions. To remove classical selectivity the authors added noise that eventually makes the average responses between conditions equal. The population responses could still predict at an above chance level the condition. Thus, on the one hand, the average responses of a group of neurons showed no significant differences between conditions; on the other hand, population coding could successfully differentiate between them. This result indicates that comparison of the average responses between conditions is not sufficient for the characterization of neuron responses and that these neurons have non-linearly mixed selectivities.

As we discuss in the following section, neurons as early as in V1 show complex selectivity and thus use neural code that is of higher dimension than initially thought. Therefore, it is plausible that these neurons are non-linearly mixed. Simply averaging a population of neurons can then hide some of their selectivities. This could explain the studies with conflicting results discussed in the previous section.

From perceptron theory, it is well-known what is required for non-linear combination of selectivities. A single layer neural network can only solve linearly separable problems, and thus map similar inputs to similar outputs. Such networks cannot solve for instance the exclusive OR (XOR) problem. This problem can only be solved with the addition of a layer in the network. Barak et al. (2013) showed in a theoretical paper that starting from the extreme case of totally segregated (or linearly mixed) representations, the dimensionality of the code can increase with an intermediate layer of randomly connected neurons. Thus even if pre-cortical neurons code for single features, it is feasible for neurons as early as in V1 to have non-linearly mixed selectivities, from connections either within V1 or from higher cortical areas.

Can a neural population represent separated signals if these signals are intermixed at the single neuron level? Mante et al. (2013) recorded from the PFC of macaque monkeys while they performed a color or a motion discrimination task on the same stimuli. The authors found that the representation of the color and motion features, and of the choice the monkeys made were separable at the population level but intermixed at the single neuron level. Separation of function at the neural population level but not at the single neuron level in PFC has been shown in other studies as well (Sigala et al., 2008; Machens, 2010; Machens et al., 2010; Stokes et al., 2013). As discussed previously, analysis of response patterns in V1, V2, and V4 showed that neurons can have multiplexed but separable selectivities to color and form (McClurkin and Optican, 1996; McClurkin et al., 1996).

In light of the results on high level cognitive areas can we make any inferences on neural representations in early visual cortex? A prominent feature of non-linearly selective PFC cells is the complexity of their selectivity. Note that if the neural representation is high dimensional, a linear decoder can implement many input-output combinations; an attribute that is necessary for a population of neurons that perform complex tasks. As discussed earlier (Rigotti et al., 2013), neural populations that can perform complex tasks are suggestive of neurons with non-linearly mixed selectivities. In light of a number of studies showing that neurons in V1 can also show complex response properties previously attributed to higher order areas, we discuss in the next section the possibility that non-linearly mixed neurons are pervasive in the cortex.

Neurons in the early stages of visual processing respond to visual stimuli within a local region in space, the classical receptive field. However, the responses of neurons to stimuli within their classical receptive field do not fully encompass their properties. Stimuli outside the neurons’ classical receptive field do not elicit a response, but can modulate the response of the neurons to stimuli within their receptive field (Gilbert et al., 1996; Angelucci and Bressloff, 2006). An example of this modulation is surround suppression where, after a certain stimulus diameter, as the size of a stimulus centered within the receptive field of a neuron increases, the rate of firing of the neuron decreases (Hubel and Wiesel, 1965; Blakemore and Tobin, 1972; Nelson and Frost, 1978; Knierim and Van Essen, 1992; DeAngelis et al., 1994; Levitt and Lund, 1997; Adesnik et al., 2012). The properties of neurons in higher cortical areas are much more complex than the well-established classical and extra-classical receptive field properties of neurons in early visual cortex. Other studies, however, have indicated even more complex properties of cells in early visual cortex that may suggest that these cortical areas are more than just a relay to higher visual areas. As discussed in the previous section, complex selectivity of a population of neurons is indicative of non-linearly mixed selectivities at the single neuron level. Thus, complex selectivity in early visual cortex could suggest population of neurons that are responsive to several features.

Recent experiments confirm that activity in V1 can be driven or modulated by prior expectations. In an fMRI study, Kok et al. (2013) used a forward model to predict the direction of random dot motion patterns from activity in the early visual areas. Their results indicated that experimental priors can change the contents of the neural representation in early sensory cortex. Keller et al. (2012) showed that a subset of cells in the primary visual cortex of mice responded only when there was a mismatch between what the mouse was expecting to see and what it actually saw while it was running. Interestingly enough, the cells that showed the strongest responses could also encode the degree of mismatch between expectation and actual visual feedback. McManus et al. (2011) recorded from monkeys performing a contour detection task and found that V1 neurons were selective to complex forms and that this selectivity could be modulated by the monkeys’ expectation of the form.

Evidence from electrophysiological studies in monkeys has indicated that attention enhances the response of neurons with receptive fields that are within the focus of attention in all of the cortical areas along the ventral stream, including V1 (Moran and Desimone, 1985; Spitzer et al., 1988; Chelazzi et al., 1993; Luck et al., 1997; McAdams and Maunsell, 1999). Furthermore, it has been shown that V1 neurons have complex perceptual grouping properties previously assigned to higher areas. Lamme (1995) found that V1 neurons play a critical role in figure-ground segregation since they show response enhancement for stimuli presented in the figure compared to stimuli presented in the ground area. In a binocular disparity study, Sugita (1999) showed that some orientation selective neurons in V1 had a diminished response when bars were occluded by a patch, but restored their response when the patch had crossed disparity and thus appeared to be in front of the bars. The studies by Lamme (1995) and Sugita (1999) along with the study by Wachtler et al. (2003) discussed in the previous section suggest that the modulation of neurons depends on global context.

The studies, we discussed in this section, indicate that neurons in early visual cortex are highly context dependent. The proposition, that there exists a context dependent population of neurons which at the same time processes segregated features of the visual scene seems contradictory to us. Furthermore, the complex properties of cells discussed here suggest that the neural population code in early visual cortex is high dimensional. High dimensional codes arise from non-linearly mixed selectivities at the single neuron level. Non-linearly mixed selectivities at the single neuron level may also explain the often conflicting neurophysiological results in the early visual cortex on color and form processing. Population average response can hide some of the selectivities of a neural population. However, as the study by Wachtler et al. (2003) showed, consideration of the pattern of responses from a population of neurons can reveal the complex behavior of the neurons.

The hierarchical model of vision represents V1 cells as localized spatial filters that extract low level visual features to transfer this information to the higher levels. However, V1 cells’ complex perceptual grouping properties suggest strong influences from horizontal and feedback connections from higher visual areas. In the next sections we examine the feedforward and recurrent modes of processing and possible ways they can modulate color and form selectivity of early visual cortical areas.

Thorpe et al. (1996) have shown that in a categorization task of complex natural images, ERP activity starts to differentiate for different visual targets approximately 150 ms after stimulus onset. Furthermore, a simple weighted summation of spike counts on a population of neurons in IT taken between 100–150 ms after an object is presented can decode object identity even with moderate changes in the object’s position, scale, illumination, pose and clutter (DiCarlo et al., 2012). These results along with other electrophysiological (Keysers et al., 2001) and psychophysical (Potter et al., 2014) studies suggest that processing in the visual system can be fast and use mainly if not exclusively feedforward circuits.

However, in the brain, we also find feedback connections from higher order areas to lower ones (Kennedy and Bullier, 1985; Shipp and Zeki, 1989; Felleman and Van Essen, 1991). Feedback connections enable the receptive field properties of neurons to change dynamically, in order to adapt to differences in behavioral state, contextual influences or expectations (Gilbert and Li, 2013). They can also contribute to the disambiguation of noisy scenes (DiCarlo et al., 2012).

As discussed earlier, there are cells as early as in V1 that code for both color and orientation (Leventhal et al., 1995; Friedman et al., 2003). A feedforward model similar to the one proposed by Hubel and Wiesel (1962) could explain this kind of tuning; a V1 cell has oriented, color selective regions in its receptive field because it receives synaptic input from center surround, color opponent LGN cells. This model is physiologically plausible. However, it is unlikely that all processing occurs this way.

Incremental grouping theory proposes a link between different perceptual grouping mechanisms and feedforward and recurrent processing (Roelfsema, 2006; Roelfsema and Houtkamp, 2011). It distinguishes perceptual grouping mediated by base and incremental grouping mechanisms. The base grouping mechanism groups feature conjunctions/objects that are coded by individual cells. The base grouping mechanism can code features of different complexities; it includes cells in V1 that code for both color and orientation and in medial temporal lobe that are selective to specific individuals (Kreiman et al., 2002; Quiroga et al., 2005). It is fast, feedforward and happens in parallel. However, it would lead to a combinatorial explosion if single neurons coded for every possible combination of objects/features. The incremental grouping mechanism is used for objects/features that are not coded by single specialized neurons. It increases the response of a population of neurons that encode the features to be grouped via feedback and horizontal connections. This process is slow since the spread of neural enhancement resulting in perceptual grouping happens gradually. There has been neurophysiological evidence in support of a distinction between base and incremental processing (Roelfsema et al., 1998, 2004; Pooresmaeili et al., 2010). A recent study showed that in macaque, V1 integration spreads out in an approximately 300 ms period from the focus of attention, following perceptual grouping criteria (Wannig et al., 2011). Thus depending on the situation at hand, visual processing can operate in two modes: the feedforward one which is specific, strong and fast and the feedback one which is diffuse, weak and slow.

In incremental grouping, color and orientation are jointly coded during feedforward processing. Recurrent processing, however, can possibly change dynamically the weights of color and orientation selectivity in early visual areas. Neurons as early as in V1 can slowly enhance their response via recurrent influences to signal the association of features to an object (Roelfsema et al., 1998). This may depend on contextual influences, including task or prior expectations.

Predictive coding theory also suggests a mode of visual processing different from the segregated one. The theory states that perceptual, cognitive and action-oriented processing follow a single general strategy, which uses top-down predictions to minimize prediction errors (Clark, 2012). This approach suggests that neuronal selectivity to a feature is not an intrinsic property but the result of interactions across levels of a processing hierarchy (Friston, 2003). Sensory neurons, rather than features per se, encode an error signal, i.e., they feed forward to hierarchically higher areas the discrepancy between the actual input and the top-down expectation (Egner et al., 2010). According to the predictive coding model, predictions are relayed via feedback connections, whereas prediction errors are conveyed via feedforward connections (Rao and Ballard, 1999). Hosoya et al. (2005) showed that retinal ganglion cells’ spatio-temporal receptive fields change dynamically with the visual scene; this result is in line with the view that the raw signal, carried by the receptors, is transformed as early as in the retina by the first interneurons which encode deviations from predicted temporal and spatial structures (Srinivasan et al., 1982). Recent fMRI studies have also shown evidence in support of predictive coding in the visual cortex (Summerfield et al., 2008; Kok et al., 2012). For instance, Kok et al. (2012) found that the amplitude of the fMRI signal in early visual cortex was smaller when the stimulus was expected; typically when we see something that we expect the prediction error encoded in the brain is smaller compared to when we see something unexpected. This mode of processing, however, appears to be at odds with several electrophysiological studies (see Koch and Poggio, 1999 for a commentary).

In the predictive coding framework, context determines whether sensory neurons perform segregated or integrated processing. For example, if the color of some colorful shape is unexpected the visual system generates the prediction error related to the color only. As a result the neural response will indicate color selectivity which is segregated from form. However, if both the color and shape are unexpected, the prediction error will have information about both features, and the neural response will reflect integrated processing.

Early studies on visual processing indicated that different regions in the brain show biases in their selectivity for color and form. These results suggest that color and form are processed by distinct modules in the visual cortex. All of these studies assumed in their analyses that each neuron is an independent computational unit; thus weak selectivities at the single neuron level were disregarded. Meanwhile, these results were found to be at odds with some psychophysical and electrophysiological observations which suggested integrated processing of color and form in early visual cortex.

Studies on higher cortical areas have shown that visual representations and complex task conditions are represented by the distributed activity of a population of neurons. Here selectivities to a feature that appear weak at the single neuron level may encode that same feature robustly at the population level. Populations of neurons that perform complex tasks in PFC were shown to have non-linearly mixed selectivities at the single neuron level.

Recent studies have showed that early visual areas are not just passive relays of local information, but rather complex processing stages that incorporate global context and prior information. This behavior arises from the flow of information from horizontal and feedback connections that can dynamically adapt the selectivities of single neurons to the situation at hand. The increasing evidence that the early visual areas show this kind of complex selectivity suggests that population codes operate in a high dimensional space; this property makes it likely that single neurons have non-linearly mixed selectivities. Examining these selectivities at the single neuron level can be misleading. Based on the above evidence, we argue that color and form features not only are continuously interacting in our visual experience, but are also integrated rather than segregated in the visual cortex.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.