Layered sparse associative network for soft pattern classification and contextual pattern completion
-
1
University of California at Berkeley, Redwood Center for Theoretical Neuroscience, United States
Traditional models of associative networks [1, 5] have impressive pattern completion capabilities. They feature high storage capacity and can reconstruct incomplete sensory input from stored memory in a context-sensitive fashion. However, associative networks are not useful as pattern classifiers beyond simple toy examples where the classes form well-separated clusters in pattern space. Conversely, multi-layer feed-forward neural networks [4] can be trained to solve challenging classification tasks, but they cannot perform pattern completion. Here we tested the ability of sparse two-layer associative networks to learn to 1) recognize patterns in real world data and 2) use memory content to reconstruct noisy input in a context-sensitive fashion. Specifically, we used the memory models with supervised learning on handwritten characters (NIST database [3]) and assessed how well the memory could predict the class of unknown input and reconstruct the input pattern. We started with the Kanerva memory model [2], which has a two-layer structure. The first neural layer has fixed random synapses and contains many more neurons than input fibers, which are sparsely activated. This stage maps the input space into a high-dimensional space of sparse patterns. The synapses of second layer neurons store associations between the high-dimensional sparse patterns and desired output patterns via Hebbian plasticity. We trained the model with the NIST data by storing for each input pattern the given digit interpretation as well as the input pattern in autoassociative fashion. When cross-validated with (noisy) inputs, the model frequently performed pattern recognition and reconstruction successfully. However, ambiguous inputs could not be classified, and the reconstruction was a mixture pattern not corresponding to a valid handwritten digit. To overcome these limitations, we constructed a new model with a two-layer structure similar to the described Kanerva model but with two important new features, enabling soft classification and pattern completion in two subsequent phases. One new feature is labeling first layer neurons during supervised training according to how their activity is correlated with the occurrence of certain classes. Thus, in the classification phase when a new input drives first layer neurons, estimates of class membership are encoded in active populations of first layer cells. The second feature is a phase of selective pattern completion performed after classification. In this phase first layer neurons are also influenced by lateral interactions and feedback from the second layer. This competitive dynamical process accomplishes pattern completion that is contingent on a specific interpretation rather than creating a mixture of the entire memory content, even if the input is ambiguous. We compare the model to current methods of classifying the NIST data. We found that the model exhibits competitive classification performance and adds a valuable explanatory component not provided by ordinary classification algorithms. Finally, we discuss how the proposed memory architecture would map onto dentate gyrus and CA3 of the hippocampus.
References
1. Hopfield, J., PNAS, 1982. 9:p.2554.
2. Kanerva, P., Sparse Distributed Memory. MIT Press. 1988.
3. LeCun, Y., C. Cortes, http://yann.lecun.com/exdb/mnist.
4. Rumelhart, D.E., et al., Nature, 1986. 323:p.533.
5. Willshaw, D.J., et al., Nature, 1969. 222:p.960.
Conference:
Computational and Systems Neuroscience 2010, Salt Lake City, UT, United States, 25 Feb - 2 Mar, 2010.
Presentation Type:
Poster Presentation
Topic:
Poster session III
Citation:
Ehrenberg
E,
Kanerva
P and
Sommer
F
(2010). Layered sparse associative network for soft pattern classification and contextual pattern completion.
Front. Neurosci.
Conference Abstract:
Computational and Systems Neuroscience 2010.
doi: 10.3389/conf.fnins.2010.03.00190
Copyright:
The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers.
They are made available through the Frontiers publishing platform as a service to conference organizers and presenters.
The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated.
Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed.
For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions.
Received:
03 Mar 2010;
Published Online:
03 Mar 2010.
*
Correspondence:
Evan Ehrenberg, University of California at Berkeley, Redwood Center for Theoretical Neuroscience, Berkeley, United States, evanehrenberg@berkeley.edu