Many of the engineering marvels around us have, as a keystone of their mathematical foundations, a change of basis (Kreyszig, 2007). A technical definition of a mathematical basis is a set of linearly independent vectors within a space that, in combination, can span the entirety of that space. For example, the Cartesian basis for three-dimensional real space (aka R3) is a set of three orthogonal (perpendicular) unit (length of one) vectors, pointing along the x, y, and z axes respectively. In vector notation the orthonormal Cartesian R3 vectors are [1 0 0], [0 1 0] and [0 0 1]. We say these vectors span R3, as any point in R3, for which we have the coordinates [x y z], can be reached from the origin using the vectors [x 0 0] + [0 y 0] + [0 0 z]. The Cartesian R3 basis is, in other words, the way we might account for spatial activity using rulers or graph paper. Add a fourth dimension for time, to span spatio-temporal activity, and the same rules apply for any vector [x y z t].

This spatio-temporal Cartesian basis is our most intuitive approach to representing the world around us, but also a very poor representation for solving many engineering problems. One of the most commonly used changes of basis is the family of Fourier or frequency-domain transforms, in which a function is represented on a basis of sinusoidal periodic functions rather than units of Cartesian distance. In its discrete form, a signal is transformed from a series of consecutive sampled values into a combination of sinusoids of different amplitudes and frequencies.

While conceptually cumbersome at first, Fourier transformation has many advantages for not only the analysis, but the storing and compression, of many kids of data. Take, for example, a sample of a single musical note, vibrating at a particular frequency, that would appear on an oscilloscope as a complex periodic waveform. In the time domain, this signal will be dense, that is, it will contain few if any zeros and most of the signal will be required for its reconstruction as specified by the the Nyquist-Shannon sampling theorem (Shannon, 1949). If, however, the signal is like most signals coming from a musical instrument—a combination of a fundamental frequency and a small number of overtone frequencies—then it can be represented in the Fourier domain with a small number of values, one for each component frequency, leaving the rest of the basis vectors at zero magnitude. The signal vector thus meets the mathematical definition of sparse—most of its coefficients are zero—and its representation can be efficiently compressed, requiring far less data for its representation than the Nyquist theorem specifies. Figure 1A illustrates the relationship between a complex periodic waveform and its sparse Fourier transformation.

Mathematically, a signal and its Fourier transform are one-to-one mappings. The frequency-domain representation of the signal is often much more efficient, however, in the sense that far more of the signal information is packed into a small subset of the vectors that span the basis. JPEG (Skodras et al., 2001) and MPEG (Le Gall, 1991) compression schemes, for example, discard well over 90% of the information within a signal in part by transforming that image into the frequency domain (DCT in the case of MPEG and Daubechies wavelet in the case of JPEG 2000) and eliminating the many frequency bands of near-zero magnitude. The resulting compressed data formats still retain enough of the significant information to have become the lingua franca of images and music, respectively. An example of data compression in the spatial frequency domain is seen in Figure 1B.

Frequency-domain transform is hard-wired into the anatomy of the cochlea, whose hair cells of varying stiffness resonate with stimuli of specific frequencies, triggering action potentials via auditory transduction. The inner ear thus performs a frequency transform of incoming auditory information across a small temporal window, known in its simplest form as a short-time Fourier transform (STFT), though actual observed performance resembles a somewhat more complex transform known as time-frequency reassignment (Auger et al., 2013).

If sufficient information about the signal can be deduced from a small portion of a signal via a mathematical transform, then the benefits to the actor are obvious. Both computationally and metabolically, the organism that can reduce processing demands by such a large amount can expect to reap benefits. If frequency-domain and similar bases yield such improvements in information coding efficiency, the key question for modeling neural coding is to ask how that information might be coded to its optimum.

The optimal basis for a signal in a least-squares sense is its Singular Value Decomposition (SVD) (Strang, 2007). A comparison of spatio-temporal, frequency-domain and SVD data compression is shown in Figure 1B. The frequency-domain images show many more features at each level of compression than the Cartesian (nearest-neighbor) compression, while the SVD images show substantially more than either.

However, the SVD of a single signal is not necessarily the sparsest representation of that signal in the context of a set of signals such as that encoded in neural memory. Much greater compression can be attained through the re-use of common basis vectors to transform many signals. In this approach the process of neural memory is modeled as manipulation of a set of learned basis vectors known as a “dictionary,” in which incoming signals are decomposed in the sparsest possible way using the atomic vectors or “atoms” that make up the dictionary (Rubinstein et al., 2010). This operation is non-linear but many efficient algorithms have been developed for sparse dictionary coding, primarily through L1-norm minimization (Donoho and Elad, 2003). The most efficient dictionary systems are found to be “sparse overcomplete,” that is, they consist of many more basis vectors than necessary for the set of signals, but have great flexibility to maximize the sparsity with which an incoming signal is encoded (Bruckstein et al., 2009; Rubinstein et al., 2010).

Finally the atoms of the dictionary must adapt to the new signals in accordance with the principles of Hebbian and Bayesian learning. Efficient algorithms have been discovered for this process as well, whether the classic K-SVD (Aharon et al., 2006) or more recent parametric or multiscale dictionary updating algorithms (Rubinstein et al., 2010).

Perhaps unsurprisingly, there is abundant experimental evidence for such sparse coding in human and animal brains (Olshausen and Field, 2004). Evidence supporting a sparse coding model has been found in studies of visual (Olshausen and Field, 1997; Vinje and Gallant, 2000), auditory (Hromádka et al., 2008), olfactory (Ito et al., 2008; Poo and Isaacson, 2009), haptic (Jadhav et al., 2009; Crochet et al., 2011) and motor (Hahnloser et al., 2002) processing. Sparse coding models relate to the neuroanatomical observation that progressive stages in signal processing have increasingly redundant amounts of neurons that each fire increasingly rarely (Olshausen and Field, 2004). This is no longer projected to lead to signal-specific “grandmother cells” but rather to a maximally sparse and overcomplete representation of the world given metabolic constraints.

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.