An interesting feature of the human mind is how it tricks us into thinking complex tasks are simple. One semi-apocryphal illustration of this was Marvin Minsky asking an undergraduate to program a computer to “describe what it saw” through a camera, over the course of a single summer. Minsky’s wild underestimate of how hard this would be stemmed from an intuition that it can’t be terribly hard to do something so effortless. Similarly, our innate understanding of time as the inescapable dimension along which life proceeds seems effortless. Surely, the mechanistic underpinnings of this ability should be easy to describe? Not quite. It is not even easy to define what “time” is. Though multiple research groups have made important theoretical and experimental contributions to understanding time in the brain (Mauk and Buonomano, 2004; Buonomano and Laje, 2010; Allman et al., 2014; Eichenbaum, 2014; Merchant et al., 2014, 2015; Tucci et al., 2014; Finnerty et al., 2015; Petter et al., 2018; Wang et al., 2018; Balasubramaniam et al., 2021), neuroscience provides few satisfying answers to the big questions of how time is explicitly represented in cortex, how temporal relationships are stored in memories, and how memories of temporal relationships are used to make predictions.

The ability to make accurate predictions confers clear competitive advantages. Accurately extrapolating the trajectory of a moving object to predict its future state, for example, is very useful for both prey capture and predator evasion. An interesting idea is that prediction might emerge as a natural consequence of resource optimization, thus providing dual benefits for adaptive behavior and energy efficiency. The concept of efficient coding from information theory describes how to achieve such resource optimization for data storage or transmission (Shannon, 1948; Cover and Thomas, 2006b; Stone, 2018). When data are extended in space or time, efficient coding suggests that a predictive coding scheme can be used to compress information and save energy (Elias, 1955). Though these ideas were inspired by problems in telecommunications engineering, they have found significant application in biology.

Efficient coding was first introduced to neuroscience by Attneave (1954) and Barlow (1961), who argued that retinal circuits use efficient coding to transform light patterns into the neural code transmitted through the optic nerve. The basic concept has evolved through the years and been used to explain a wide variety of experimental results across the visual system (Srinivasan et al., 1982; Atick and Redlich, 1992; Dong and Atick, 1995; Dan et al., 1996; Olshausen and Field, 1996; Meister and Berry, 1999; Doi et al., 2012; Pitkow and Meister, 2012). Many basic functional properties of the retina are likely the result of efficient coding, for example the unequal distribution of ON and OFF ganglion cell types (Balasubramanian and Sterling, 2009). Srinivasan et al. (1982) were the first to show that retinal ganglion cells (RGCs) effectively act as linear predictive coders. Extending this notion, Ocko et al. (2018) provided a plausible explanation for the existence of multiple ganglion cell subtypes. Efficient coding may also provide a framework to explain how the evolution and plasticity of cortical circuitry is shaped by natural environmental statistics (Olshausen and Field, 1996; Bell and Sejnowski, 1997; Rao and Ballard, 1999; Wiskott and Sejnowski, 2002; Jehee et al., 2006; Creutzig and Sprekeler, 2008). For example, Olshausen and Field (1996, 1997) created a spatial efficient coding model that mimics primary visual cortex (V1) receptive fields when trained on natural scenes. Rao and Ballard (1999) famously introduced a hierarchical predictive coding model to explain the classical and extra-classical receptive field properties of V1 neurons. Related models have since been suggested to provide a general framework for understanding cortical function (Friston, 2005; Bastos et al., 2012; Spratling, 2017; Whittington and Bogacz, 2019; Millidge et al., 2021).

Overall, there is a rich literature on both efficient and predictive coding. In the visual system, however, much of the experimental and theoretical focus has been in the spatial domain. This may reflect the inherent “spatial-ness” of vision as a sensory modality, but we argue that time is also fundamental, even beyond motion processing. In addition, the notion of predictive coding has become somewhat restrictive in its potential instantiations in neural circuitry, e.g., prediction and predictive coding may still be relevant to understanding cortical function even if hierarchical predictive coding is the inappropriate model. To that end, this review begins with a primer on efficient coding wherein we explicitly derive the close relationship between efficient and predictive coding. Next, we review evidence for efficient coding in the retina and dorsal lateral geniculate nucleus (dLGN). We then move to later visual areas, with particular emphasis on how efficient coding principles in visual cortex may underlie a variety of time-dependent computational tasks, including visual flow processing, spatiotemporal sequence learning, and adaptation. In the end, we hope to clarify the sometimes confusing relationship between efficient and predictive coding, and discuss ways in which these theories may guide experiments and provide clues about how the nervous system codes temporal relationships to make predictions.

Efficient coding is a concept from information theory describing how data can be transmitted or stored with minimal use of energy, time, and resources. Pioneered in the 1920s–1950s by Harry Nyquist, Ralph Hartley, and Claude Shannon of Bell Telephone Labs, information theory provided a quantitative framework to analyze then emergent telecommunications technology and help their employer save money on telegram and telephone transmission. The goal was to design a system capable of reliable message transmission and storage.

As a concrete example, consider constructing a message from a four-letter alphabet, θ = {A, B, C, D}. There are 4¹⁰ possible 10-letter messages, a typical example of which might be BACAAABDAA. If we assume that each letter appears independently within a message according to the following probabilities:

then certain messages are much more likely to occur than others (DDDDDDDDDD is very unlikely, for example). It makes intuitive sense to choose an encoding scheme that takes advantage of this non-uniformity. With this in mind, Shannon introduced the idea of entropy to quantify the average number of symbols required to store or send any such message (Shannon, 1948; Cover and Thomas, 2006c). In his formulation, information is simply I (X) = −log₂ (P(X)) bits, which is the number of binary digits required to store a message that occurs with probability P (X). Due to its inverse relationship with probability, information is a measure of epistemic surprise relative to expectation, and entropy is just the expectation of the information:

The average total amount of information in a message of N symbols is simply NH(θ). In our example,

Shannon’s source coding theorem proves that the entropy defines the minimum possible number of bits that can be used to represent one symbol from our messages without losing information. For this value of entropy, the average information in a 10-symbol message would be 17.3 bits. If we had assumed that all letters were equally likely, then entropy would be 2 bits and the average 10-symbol message would be 20 bits. As this example demonstrates, the source coding theorem implies that data from different distributions can be stored with differing amounts of information. The question is how to design an encoding scheme that takes advantage of this theorem, and minimizes the number of bits utilized.

According to the source coding theorem, an encoding scheme is efficient if messages are, on average, transmitted with a number of bits approaching the entropy of the source (Shannon, 1948; Barlow, 1961; Atick, 1992; Cover and Thomas, 2006a; Sterling and Laughlin, 2015). Efficient codes specify messages using the minimum possible number of bits and do so by removing predictable information.

To get a sense of what this means in practice, consider representing our symbol alphabet using a simple binary encoding scheme:

Transmitting a letter/symbol with this scheme requires 2 bits, which is more than the theoretical minimum of 1.73 bits implied by the source coding theorem. What exactly makes this code inefficient? The answer is redundancy: messages transmitted with this code are, on average, predictable. To see this, consider transmitting a 0 for the first bit. Since P(A)=12 and P(B)=14, the probability that the second bit will also be a zero is 2/3. A more efficient code would reduce such redundancies.

Formally, Shannon redundancy is given by Atick (1992):

where C = 2 bits is the average message length in our encoding scheme. The redundancy is always between 0 and 1, with a perfectly efficient code having H (θ) = C and R = 0. In this case, C = 2 and H (θ) = 1.73, so R = 0.135.

Consider an alternative 3-bit encoding scheme:

Intuition might suggest that adding an extra bit will decrease efficiency, but this is incorrect when we consider the underlying message-generating process. While C and D both require 3 bits to transmit, this relative increase in message length might be compensated by the fact that the most common symbol, A, requires only 1 bit. For the entire 3-bit scheme:

The redundancy is now R = 0.011, an order of magnitude smaller than the 2-bit scheme. As long as the statistics remain stationary, this 3-bit scheme represents a very efficient code to transmit our messages and illustrates the fundamental principle of efficient coding: use relatively fewer symbols to encode prevalent/expected messages and relatively more symbols to encode rare/unexpected messages (Cover and Thomas, 2006a).

Our example also demonstrates the principle that deviations from uniformity decrease entropy (1.73 bits is less than the 2 bits of a uniform distribution). As more complex statistical dependences are added to the source, entropy often drops even more. Consider for example a Markov chain with the following transition probability matrix, T:

The transition matrix provides conditional probabilities such as P (B | A) = 0.7. This Markov chain generates sequences that tend to repeat the pattern ABCD, so neighboring message elements are no longer statistically independent (P (AB) ≠ P (A) P (B)). According to the mathematics of Markov chains, the asymptotic probability of element occurrence, π, solves the equation π = πT. Thus, π is the left eigenvector of the transition matrix, whose corresponding eigenvalue is 1. In this case,

This is known as the stationary distribution of the Markov chain, representing how often we expect each symbol to occur regardless of its position within the message. For Markov chains, the so-called entropy rate becomes (Cover and Thomas, 2006d):

For the present example:

This represents a significant reduction in entropy compared to the 2 bits of a 4-symbol uniform distribution, brought about by the statistical dependence between neighboring message elements. A is now predictive of B, and B of C, etc. despite each letter being equally likely overall.

An efficient code for this Markov distribution might look like this:

If the previous symbol was an A and we receive a 0, the encoding matrix tells us that the second symbol must be B. After that, we get another 0, so C. Here is an example sequence generated by this Markov process and its corresponding binary representation (assuming all messages start with A):

In the absence of noise, we can perfectly reconstruct the original message from its binary counterpart. The average, long-run per-symbol message length is:

Redundancy is therefore R = 0.096. There is still some room for improvement, but this predictive coding scheme is much more efficient than schemes that ignore the statistical dependence between neighboring elements (Elias, 1955; Huang and Rao, 2011; Spratling, 2017). As before, an efficient code uses more information to represent unexpected events, but inclusion of an ordinal (or temporal) relationship changes our interpretation. Now, our efficient code uses more information (more bits) to represent prediction errors relative to expectation: when D comes after C, we expect that and use only 1 bit. We do not expect A to follow C and use correspondingly more bits (3) when this occurs. This is precisely the predictive coding algorithm as originally proposed for efficient transmission of telecommunications data (Elias, 1955). Figure 1 shows how predictive coding might be used to efficiently transfer information, including in a neural environment.

To conclude, it is worth noting that efficient coding is perfectly valid in the presence of internal noise, such as when bits randomly flip during transmission. If the original encoding scheme is optimal but bits flip randomly during transmission, then information is lost at the receiving end. This is no longer an efficient code by definition, so to account for noise, efficient codes ultimately represent a trade-off between minimizing the average message length and preserving redundancy to be robust to noise (Atick, 1992; Atick and Redlich, 1992; van Hateren, 1992; Dong and Atick, 1995; Simoncelli, 2003). Efficient coding is therefore typically formalized as a maximization of mutual information between sensory inputs and neural responses, rather than a minimization of redundancy (Doi et al., 2012). This encourages neural codes that have low redundancy, but also high discriminability of different stimuli with low trial-to-trial variability for the same stimulus. Many authors include an additional energy or metabolic constraint to encourage some form of sparseness in the ultimate solution (Olshausen and Field, 1996; Chalk et al., 2018; Ocko et al., 2018).

From an information-theoretic perspective, neural coding in retinal photoreceptors is inefficient because the statistics of natural visual scenes create activity patterns that are highly correlated in space and time. Any code that simply recapitulates this structure would be inefficient in space, energy, and resource utilization (Landauer, 1976; Balasubramanian and Sterling, 2009; Sterling and Laughlin, 2015). Such inefficiency would then be exacerbated by spiking RGCs, because spikes are particularly expensive in terms of energy consumption, and higher average firing rates require increasingly greater axonal volumes (Laughlin et al., 1998; Attwell and Laughlin, 2001; Balasubramanian et al., 2001; Laughlin, 2001; Perge et al., 2009). Space leaving the retina is very limited (in humans, there are two orders of magnitude fewer axons in the optic nerve, 10⁶, than there are photoreceptors, 10⁸), so a more efficient representation is advantageous. Attneave (1954) and Barlow (1961, 1989) were the first to recognize this and apply Shannon’s theory to neuroscience. Barlow proposed that RGCs encode and transmit visual information using a simple efficient coding heuristic: reduce redundancy by generating fewer action potentials for expected visual inputs and more spikes for unexpected ones. The efficient code is actually created via filtering operations in retinal circuits, which throttle firing rates and decrease redundancy by removing many of the input correlations imparted by natural scene statistics (Atick and Redlich, 1992; Meister and Berry, 1999; Pitkow and Meister, 2012).

Barlow’s hypothesis appears to be approximately correct (see Meister and Berry, 1999; Balasubramanian and Sterling, 2009; Huang and Rao, 2011 for reviews). RGCs are effectively linear predictive coders (Elias, 1955; Srinivasan et al., 1982; Meister and Berry, 1999; Huang and Rao, 2011; Doi et al., 2012) with receptive fields resembling those predicted by efficient coding models (Atick and Redlich, 1992; Doi et al., 2012; Ocko et al., 2018). In the temporal domain, linear RGC receptive field filters are biphasic and compute the difference between recent past and present. Spatially, RGCs have a difference-of-Gaussians organization that compares luminance between center and surround regions. In both cases, RGCs can be understood to predict correlations, responding minimally when they are present (e.g., when luminance patterns are constant in time or uniform across the spatial extent of the receptive field) and responding maximally to expectation violations (e.g., when luminance changes rapidly in space or time). These retinal filters have even been shown to adapt over rapid timescales to more efficiently encode visual information from novel distributions (Hosoya et al., 2005). Beyond the retina, relay neurons in the dLGN continue this process, especially in the temporal domain (Saul and Humphrey, 1990; Hartveit, 1992; Dong and Atick, 1995; Dan et al., 1996) where whitening occurs in a manner consistent with the efficient coding hypothesis (Dong and Atick, 1995; Dan et al., 1996). For example, the efficient coding model of Dong and Atick explains the existence of lagged and non-lagged dLGN relay neurons, as observed in physiological data (Hartveit, 1992).

Despite this evidence, it is not clear that the same principles are sufficient to explain the early visual system in all its complexity. One common critique is that information theory derives from the general principle that all information is created equal. In the real world, some sources of information are more relevant to an organism than others. Frogs, for example, are better served by a visual system evolutionarily tuned to detect and locate flies than by an abstract requirement to efficiently compress all visual information equally.

Several groups have proposed variations on efficient coding that call for more nuanced perspectives by taking messy biological imperatives and constraints into account. Balasubramanian and Sterling (2009) and Sterling and Laughlin (2015), for example, argue that it is advantageous to minimize per-bit computational costs, even while acknowledging that some information sources may be relatively privileged due to their ethological importance. These researchers and others have gone to great lengths to measure the computational cost of information transmission in terms of quantities like axonal volume and ATP consumption. Their more holistic approach leads to a variety of predictions regarding the structure and function of the retina that are well-supported by empirical data (Borghuis et al., 2008; Balasubramanian and Sterling, 2009; Perge et al., 2009; Gjorgjieva et al., 2014).

Another variation, proposed by Zhao et al. (2012); Teulière et al. (2015), Lelais et al. (2019), and Eckmann et al. (2020) takes the name of active efficient coding and is built around an interesting observation: since environmental statistics are partially governed by the animal’s own behavior, changing behavior can make a neural code more or less efficient. This idea leads to a variety of empirical predictions regarding the relationship between sensation and action. Teulière et al. (2015) have shown, for example, that both vergence and smooth pursuit eye movements can be learned de novo in artificial systems that optimize coding efficiency by simultaneously adjusting both the neural representation and eye movements (Zhao et al., 2012; Lelais et al., 2019).

Finally, and particularly relevant to our focus on time, is a hypothesis proposed by Bialek et al. (2007). Efficient codes compress data down to Shannon’s source coding limit. Beyond that limit, rate-distortion theory provides a principled way to select certain information for deletion. Inspired by this idea and a related framework known as the information bottleneck (Tishby et al., 2000), Bialek et al. suggested that sensory systems preferentially delete information about the past. They reasoned that predictive information (about the future) is uniquely useful for action and decision-making, and should therefore be prioritized. Such predictive information, which is inconsistent with traditional models of visual processing, has been observed in the early visual system of various species (Palmer et al., 2015; Salisbury and Palmer, 2015; Sachdeva et al., 2021; Wang et al., 2021).

While the retina and dLGN provide some of the best support for the idea that neural circuits are shaped by environmental statistics to efficiently encode information, there is evidence that similar considerations may hold in other visual areas as well.

Information sources that exhibit statistical dependences across space and time can be compressed by efficient encoding schemes. In the visual system, part of this process occurs in the retina and dLGN, which perform spatial and temporal decorrelation relative to the statistical structure of natural visual inputs (Srinivasan et al., 1982; Dong and Atick, 1995; Dan et al., 1996; Hosoya et al., 2005; Huang and Rao, 2011; Pitkow and Meister, 2012). However, temporal decorrelation appears limited to brief timescales, on the order of ∼30–300 ms, and processing is largely linear, subtracting the mean and removing pairwise correlations (though see, for example, Palmer et al., 2015 for more complex processing in the retina). Longer-timescale and higher-order correlations present in natural visual inputs survive the initial processing stages. This suggests that information reaching V1 is still inefficiently encoded. Regardless of the ultimate computational goal or task, an inefficient representation is in general more energetically costly and more difficult for downstream regions to process, as argued by Barlow (1990). V1 may therefore construct a more efficient representation, acting on longer timescales and reducing higher-order correlations. Examples of higher-order correlations in natural vision are edges in the spatial domain (representing correlations between spatially adjacent center-surround receptive fields) and brief trajectories in the temporal domain [which V1 can rapidly learn to predict (Xu et al., 2012)]. Canonical V1 receptive fields can be understood to operate on both forms of correlation in an efficient coding sense. “Edge detectors” in V1 eliminate spatial correlations found in natural scenes by integrating information across multiple dLGN inputs (Olshausen and Field, 1996; Bell and Sejnowski, 1997). V1 neurons with “space-time inseparable,” or direction-selective receptive fields, similarly eliminate higher-order correlations associated with motion trajectories.

Predictive coding models consistent with efficient coding have been used to explain both classical and extra-classical receptive field properties of V1 neurons, especially in the spatial domain (Spratling, 2010; Huang and Rao, 2011; Keller and Mrsic-Flogel, 2018). The well-known Rao and Ballard (1999) hierarchical predictive coding model learns V1-like receptive fields when trained on natural scenes, showing both classical Gabor-like spatial structure and extra-classical effects like end stopping (neural firing evoked by elongated bars increases with bar length up to some critical length, beyond which firing rapidly decreases). Cells with this property were originally called hypercomplex by Hubel and Wiesel (1965), but are now termed end stopped (Gilbert, 1977). Such contextual effects are often consistent with efficient coding models but difficult to reconcile with strictly feedforward models of visual processing (Schwartz et al., 2007; Huang and Rao, 2011; Carandini and Heeger, 2012; but see Priebe and Ferster, 2012 for an alternative explanation of some contextual effects).

In the following sections, we review evidence for temporal efficient coding in V1, especially focusing on data from rodents.

Though efficient coding and information theory are clearly relevant to neural computation, there is much debate regarding the extent to which these ideas explain what we see in the nervous system, especially beyond the retina and dLGN. We will walk through a point-counterpoint debate that emphasizes three prominent criticisms of efficient and predictive coding as theories of cortical function (see Simoncelli, 2003 for a complementary perspective). (1) The massive expansion in the number of neurons from dLGN to V1 would appear to increase redundancy, contrary to the efficient coding hypothesis. (2) There is functional evidence that contradicts predictive coding theory. (3) Efficient coding makes very precise, testable predictions in certain contexts, but, in general, information-theoretic measures are very difficult, if not impossible, to estimate (Paninski, 2003) making the overall utility of applying efficient coding to the cortex unclear.

The first criticism is suggestive of the inherent difficulty in translating information-theoretic ideas to the nervous system. Unlike a telephone system, where engineers need only concern themselves with transferring data efficiently, cortical circuits must both encode information and operate on it. For computer hard drives, efficiency is defined by minimizing the average number of bits utilized while preserving information from known sources of noise. By comparison, efficiency in the nervous system might mean minimizing the average number of spikes per second while preserving information, or maximizing the average number of bits per molecule of ATP, or maximizing bits per volume of axon (Laughlin et al., 1998; Balasubramanian and Sterling, 2009; Sterling and Laughlin, 2015; Stone, 2018). In the retina, Balasubramanian and Sterling (2009) proposed the following instantiation of efficient coding theory: “Given the information required for behavior, the retina minimizes its computational costs.” By precisely measuring the metabolic and computational costs of information processing under certain conditions, these researchers and others (Laughlin et al., 1998) have found that each additional bit of information transmitted along the optic nerve requires correspondingly more neural resources, space, and energy, creating a dramatic law of diminishing returns (Perge et al., 2009). This explains many of the functional properties of the retina, including its differentiation into multiple parallel processing streams (Borghuis et al., 2008; Balasubramanian and Sterling, 2009; Gjorgjieva et al., 2014; Sterling and Laughlin, 2015; Ocko et al., 2018). Overall, such considerations reflect the subtlety of the problem and the need for very precise specifications of the theory. In V1, the fact that we find more neurons than in dLGN does constitute an increase in physical resources such as space and protein molecules, but it could well cause a decrement in the average number of spikes or in the redundancy of messages transmitted beyond V1. The expansion might also reflect a requirement for additional neural resources in the cortex as V1 integrates information across multiple modalities, or reflect cells that are being used for other cortical functions.

Regarding the second criticism, there are many examples in the literature that seem to contradict efficient and predictive coding. One such example comes from a study by Benucci et al. (2009), where the authors studied neural responses to sequences of oriented gratings in anesthetized cat visual cortex. Membrane potentials measured in response to the sequences were highly predictable from a simple linear model of the responses to individual gratings. Thus, the temporal context in which the gratings were displayed was irrelevant. The authors therefore concluded “spatial and temporal codes in area V1 operate largely independently.” A more recent study by Solomon et al. looked at neural responses in awake macaque and human visual cortex in response to rapidly flashed sequences of sinusoidal gratings (Solomon et al., 2021). They showed standard sequences on 80% of trials and deviant sequences on 20%, expecting to observe prediction errors in response to deviant stimuli. Instead, they found minimal evidence for prediction errors, with the responses to deviant and standard stimuli being almost identical.

Both studies establish crucial limitations to efficient and predictive coding, but do not invalidate the theories. Importantly, the studies presented sequences of non-natural sinusoidal grating stimuli within individual recording sessions, leaving little time for the neural system to learn the new statistics. Efficient and predictive coding both suggest that expectations are either evolved or learned relative to the natural visual environment, so a random sequence of sinusoidal gratings is always novel/unexpected from that perspective (regardless of whether it came from a standard or deviant set). There is no a priori reason to expect the visual system would learn to differentiate standard from deviant stimuli within a recording session. The Solomon et al. result therefore might suggest a limitation to predictive coding: monkeys and humans do not seem to learn non-natural sequences of stimuli on a timescale of minutes to hours (though see also Ekman et al., 2017, which demonstrated anticipatory cue-evoked pre-play of expected visual trajectories in human V1 after a brief period of training). Given more exposure time, they may or may not learn such sequences, depending on the ability of the visual system to flexibly adapt and modify its internal expectations. The use of anesthetized cats in Benucci et al. result is particularly problematic from an interpretive standpoint, since anesthesia seems to be preferentially disrupt cortical processing (Voss et al., 2019). The absence of contextual modulation may reveal little about how the awake brain exploits temporal relationships to make predictions.

The third criticism is perhaps the most difficult, as demonstrated by a simple thought experiment. Suppose we hypothesize that V1 compresses information arriving from dLGN before sending it to V2 and that we want to test this hypothesis in the spatial domain. We might devise an experiment to measure the entropy of dLGN and V1 projection neurons in response to natural scenes. The summed entropies of all V1-projecting dLGN neurons is the average “message length” of that population, likewise for the summed entropies of V1 neurons transmitting to V2. Formally, our hypothesis would be:

where H(L_i) is the entropy of the i-th dLGN projection neuron (of N total), H(V_j) is the entropy of the j-th V1 projection neuron, and I(S; L) is the mutual information between sensory inputs and the population response in dLGN. Input information is preserved at the output of V1 but in a compressed form. In theory, we would need to record from a very large number of neurons for a very long time to test this hypothesis. Accurate estimation of the individual entropies is tractable under certain assumptions, but estimation of the mutual information would be impossible in any realistic neuroscience experiment due to the curse of dimensionality (Paninski, 2003). Were we to include the temporal domain as well, by showing natural movies for example, neural responses at different timepoints would no longer be independent. Entropies and mutual information become exponentially more difficult to estimate. When you begin to consider the complexity of the circuit, rife with feedback and interconnections, the problem becomes even more difficult to specify. Therefore, while it is fairly straightforward to devise hypotheses around efficient coding, it is difficult to see how they will ever be tested. This is a fair criticism, but it is not unique to this specific theory: there is rarely an easy way to compare neural data to theory. Further, there are many ways to test ideas of efficient coding. Most are inconclusive or incomplete individually, but cumulative evidence supporting the theory can still accumulate over time.

One of the things that makes it so difficult to understand the brain is the mechanistic overlap between computation and representation. The conscious percept of a particular thought or idea or image somehow emerges from the combined activity of populations of neurons, and it is probably correct to say that the population activity defines the neural code representing the idea. This same population, however, participates in the input-output transformations responsible for computation. In a real sense, computation and representation are inseparable aspects of neural activity and there are interpretive dangers in focusing exclusively on either. Given these challenges, it is natural to question the extent to which a mathematical framework developed to help optimize data transfer and storage in engineered telecommunication systems can provide insight into brain function. This review has highlighted some of the difficulties in applying information theory to the visual cortex, and it seems unlikely that this (or perhaps any) theory will fully explain the brain’s complex neurobiology.

That said, information theory provides a useful framework to understand how evolutionary pressure toward efficient resource utilization can create predictive coding schemes with an intrinsic role for time. The complex, spatiotemporal distribution of visual information means that if the brain uses an efficient coding strategy anywhere, visual areas are an ideal candidate. Natural visual scenes exhibit autocorrelations that are useful for implying causality and predicting the future or reconstructing the past. A key insight is that a drive toward efficiency encourages temporal relationships to be represented in the neural code. Efficient coding theory also implies that there ought to be selective pressure to learn approximate space-time distributions over natural visual inputs and provides an account of how sensory data ought to be encoded. In particular, neuroscientists may expect to find evidence that data is compressed by removal of predictable spatial and temporal information, thus displaying a degree of spatial and temporal invariance.

The theory also implies that unexpected or unpredictable patterns ought to elicit error signals that would most likely be coded by increased firing rates at either the individual neuron or population level (e.g., an unexpected stimulus could also increase the size of the response population). Neurons in the retina, dLGN, and V1 all show functional properties consistent with this hypothesis (as reviewed above) and higher visual areas may be consistent with this theory as well (Jehee et al., 2006; Beyeler et al., 2016; Piasini et al., 2021). An important thing to note is that predictions are based on the environmental statistics responsible for creating the internal model, and it is not clear what to expect when the system is challenged by inputs with different statistics. This implies that experiments testing neural coding in the visual system should either use stimuli with naturalistic statistics or incorporate a period of training sufficient to encode new statistics into the neural circuits before looking for evidence of predictive processing.

Based on the current state of the field, there are many open questions for future research. In our opinion, one of the first steps should be to more fully characterize the visual system’s model of the visual environment. To what extent does that model incorporate the temporal dimension? On what timescales? Does the system predict the sensory consequences of the animal’s behavior (i.e., is it a joint model of inputs and outputs)? To what extent is the model capable of incorporating new statistics? It is generally a good idea to dissociate characterizations of the modeled environmental distribution from determinations of whether that distribution is efficiently encoded (not least because there are multiple possible dimensions along which efficiency could be measured). Most experimental stimuli include spatiotemporal content that is distinct from the animal’s normal perceptual experience. If the visual system cares about both space and time, and is flexible enough to learn, then presentation of these novel stimuli ought to induce a learning process. Another important experimental goal is therefore to characterize the extent to which the visual system can learn novel distributions, the timescale over which this learning occurs, and the overlap with known plasticity mechanisms.

We have focused this review largely on work in the early rodent visual system, but there is a larger body of literature relevant to this discussion in other brain regions and model systems, and other experimental paradigms, that could be adapted to address the issue; for example, the long-standing hypothesis that “what” and “where” information are processed in parallel ventral and dorsal pathways in primates (Ungerleider and Mishkin, 1982; Goodale and Milner, 1992; Milner and Goodale, 2008). A similar division seems to exist in the mouse visual system as well (Marshel et al., 2011; Garrett et al., 2014; Bakhtiari et al., 2021; Siegle et al., 2021). This functional segregation of space-like and time-like pathways leads to the assumption that while time is explicitly required in the dorsal stream to process motion, in the ventral stream it is useful only to integrate over noisy sensory data. Consequently, many visual processing models work only on static images. This is especially true for object recognition but also models of efficient and predictive coding (Olshausen and Field, 1996; Bell and Sejnowski, 1997; Rao and Ballard, 1999; Riesenhuber and Poggio, 1999; Carandini et al., 2005; Yamins et al., 2014; Cadena et al., 2019; Sanchez-Giraldo et al., 2019). There are theoretical arguments, though, that time could be explicitly used for computations in the ventral pathway. For example, temporal information can be explicitly useful for object recognition (Stone, 1998, 1999), or to identify kinetic borders when camouflaged objects move through the visual environment (Cavanagh and Mather, 1989; Layton and Yazdanbakhsh, 2015). This suggests that experiments manipulating temporal expectation could be used in both the ventral and dorsal streams to determine the extent to which object recognition, localization, border assignment, etc. rely on efficient spatiotemporal coding principles. Given the approximate homogeneity of cortical circuits in visual and non-visual areas, it is likely that the principles used to encode visual information will be useful to understand general cortical processing algorithms as well.

BP and JG wrote the manuscript. Both authors contributed to the article and approved the submitted version.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.