Timescales of learning in the basal ganglia and the hippocampus
- 1Neurobehavioral Laboratory, Department of Behavior Analysis, University of North Texas, Denton, TX, USA
- 2Robotics, Autonomous Systems, and Controls Laboratory, University of California Davis, Davis, CA, USA
A commentary on
A role for the medial temporal lobe in feedback-driven learning: evidence from amnesia
by Foerde, K., Race, E., Verfaellie, M., and Shohamy, D. (2013). J. Neurosci. 33, 5698–5704.
Historically there has been a distinction between basal ganglia dependent and hippocampus dependent memory systems (e.g., Squire, 1992). It is commonly understood that whilst the striatum—located in the basal ganglia—supports learning of stimulus-response associations through environmental feedback, the hippocampus supports a distinct long-term episodic memory system. Recent theoretical and experimental considerations have, however, led to the hypothesis that hippocampal function is, similarly to the striatum, dependent on signals from subcortical nuclei that are sensitive to environmental forms of feedback such as reinforcing stimulation (e.g., Gaffan, 2002; Martig and Mizumori, 2011). Compelling evidence of the role of reinforcing stimulation on hippocampal mediated learning also comes from experiments analyzing the theta rhythm (5–12 Hz) in the hippocampus of the behaving rat (for a review see Vertes, 2005). When the rat explores an environment theta oscillations lead to a weak (protein synthesis independent) form of long term potentiation (LTP), determining relatively short (1–3 h) changes in the involved synaptic efficacies. If, however, theta oscillations during exploratory behavior are paired with presentation of reinforcers, the modulatory pathways converging to the hippocampus (Gasbarri et al., 1994; Lisman and Grace, 2005; Martig and Mizumori, 2011) allow long term retention of learning. In other words, learning is facilitated on one level by active responding (e.g., locomotion)—correlated with theta rhythm—and is further enhanced by presentation of reinforcing stimulation which triggers neuromodulation of hippocampal activity. Importantly, recent research carried out on human participants using intracranial EEG (Lega et al., 2012) suggests that the human theta rhythm measured during learning may be equivalent to the theta oscillations described in rats during spatial learning.
A newly published study by Foerde et al. (2013) also supports the idea that environmental feedback plays an important role with respect to both basal ganglia and hippocampus based learning. Specifically Foerde et al. examined behavioral differences across clinical groups whose pathologies were known to selectively impair the neural structures of interest. They compared the behavioral performance of participants affected by hippocampal lesions, due to anoxia or herpes encephalitis, to the performance of patients affected by Parkinson's, which damage involves the striatum. Foerde et al. clearly demonstrated that both the striatum and the hippocampus are sensitive to feedback learning—although at a different timescale. Parkinsonian patients, with basal ganglia damage, showed preserved feedback learning when the response-feedback interval was long (7 s) but not when the response-feedback interval was short (1 s). Conversely patients with hippocampal damage were impaired at feedback learning at short but not long intervals.
The failure to learn from delayed feedback in patients with hippocampal lesions could be related to the loss of function of area CA1, since CA1 receives afferents from areas known to respond phasically to reinforcing stimulation (Gasbarri et al., 1994; Martig and Mizumori, 2011) and appears to be critically involved in temporal analysis when there are long delays among relevant stimuli (e.g., Hunsaker and Kesner, 2008; Farovik et al., 2010). Because CA1 is involved in temporal analysis in presence of long delays and receives a learning signal from areas that respond phasically to reinforcers such as the ventral tegmental area (VTA), it might serve a function in keeping the reinforcer effective during the 7 s delay in the experiment by Foerde et al. (2013). For patients affected by Parkinson's, however, the ability to learn from delayed feedback might be preserved, as area CA1 within the hippocampus is still functional and receiving “environmental” feedback from the VTA, while fast learning mediated by the basal ganglia may be impaired because of the striatal damage caused by the dopaminergic depletion.
Overall, the results described by Foerde et al. (2013) provide evidence that both the hippocampus and the striatum are involved in reinforcement learning. The sensitivity to feedback at different timescales, provided by the striatum and the hippocampus, guarantees the ability to behave adaptively within natural settings. Non-experimental environments are in fact necessarily characterized by variability in the latency of consequences to individual behavior. Accordingly, different neural structures have been shown to be sensitive to different timescales (for a review, see Buhusi and Meck, 2005). The sensitivity to different timescales may be an interesting case of degeneracy (e.g., Edelman and Gally, 2001), in which different neural structures share similar, and perhaps overlapping, functions. From this perspective a significant question that remains to be answered, as Foerde et al. point out, regards the inability of the hippocampus to mediate reinforcement based learning with short delays. Is this inability due to an intrinsic sensitivity of the hippocampus to increasing delays (Picchioni et al., 2007), or is hippocampus-mediated reinforcement at short delays inhibited by prefrontal cortex mechanisms? Prefrontal areas have been in fact found to be involved in inhibitory functions that might regulate a “competition” between learning systems (Poldrack and Rodriguez, 2004). A more precise measurement of the two structures' sensitivities to delay would help answering such a question. In future research, the duration of the response-feedback interval might therefore be manipulated parametrically (e.g., from 1 to 10 s at 1 s intervals), to assess if the timescale in which the striatum and the hippocampus operate are entirely distinct (or if there is a degree of overlap in sensitivity to feedback delay). It seems, however, that at least some of the operations carried out by the hippocampus could not be carried out by the striatum. Degeneracy across the hippocampus and the striatum due to common sensitivity to feedback learning may therefore only be partial as confirmed by other neuropsychological studies emphasizing important differences between the two learning systems (e.g., Myers et al., 2003). In spite of these clear functional differences between the two learning systems, the general picture that emerges from Foerde et al. and the related literature suggests that the basal ganglia and the hippocampus share overlapping sensitivity to reinforcement signals, although at a different timescale.
References
Buhusi, C. V., and Meck, W. H. (2005). What makes us tick. Functional and neural mechanisms of interval timing. Nat. Rev. Neurosci. 6, 755–765. doi: 10.1038/nrn1764
Edelman, G. M., and Gally, J. A. (2001). Degeneracy and complexity in biological systems. Proc. Natl. Acad. Sci. U.S.A. 98, 13763–13768. doi: 10.1073/pnas.231499798
Farovik, A., Dupont, L., and Eichenbaum, H. (2010). Distinct roles for dorsal CA1 and CA3 in memory for sequential nonspatial events. Learn. Mem. 17, 12–17. doi: 10.1101/lm.1616209
Foerde, K., Race, E., Verfaellie, M., and Shohamy, D. (2013). A role for the medial temporal lobe in feedback-driven learning: evidence from amnesia. J. Neurosci. 33, 5698–5704. doi: 10.1523/JNEUROSCI.5217-12.2013
Gaffan, D. (2002). Against memory systems. Philos. Trans. R. Soc. Lond. B Biol. Sci. 357, 1111–1121. doi: 10.1098/rstb.2002.1110
Gasbarri, A., Packard, M. G., Campana, E., and Pacitti, C. (1994). Anterograde and retrograde tracing of projections from the ventral tegmental area to the hippocampal formation in the rat. Brain Res. Bull. 33, 445–452. doi: 10.1016/0361-9230(94)90288-7
Hunsaker, M. R., and Kesner, R. P. (2008). Evaluating the differential roles of the dorsal dentate gyrus, dorsal CA3, and dorsal CA1 during a temporal ordering for spatial locations task. Hippocampus 18, 955–964. doi: 10.1002/hipo.20455
Lega, B. C., Jacobs, J., and Kahana, M. (2012). Human hippocampal theta oscillations and the formation of episodic memories. Hippocampus 22, 748–761. doi: 10.1002/hipo.20937
Lisman, J. E., and Grace, A. A. (2005). The hippocampal-VTA loop: controlling the entry of information into long-term memory. Neuron 46, 703–713. doi: 10.1016/j.neuron.2005.05.002
Martig, A. K., and Mizumori, S. J. (2011). Ventral tegmental area disruption selectively affects CA1/CA2 but not CA3 place fields during a differential reward working memory task. Hippocampus 21, 172–184. doi: 10.1002/hipo.20734
Myers, C. E., Shohamy, D., Gluck, M. A., Grossman, S., Kluger, A., Ferris, S., et al. (2003). Dissociating hippocampal versus basal ganglia contributions to learning and transfer. J. Cogn. Neurosci. 15, 185–193. doi: 10.1162/089892903321208123
Picchioni, M., Matthiasson, P., Broome, M., Giampietro, V., Brammer, M., Mathes, B., et al. (2007). Medial temporal lobe activity at recognition increases with the duration of mnemonic delay during an object working memory task. Hum. Brain Mapp. 28, 1235–1250. doi: 10.1002/hbm.20357
Poldrack, R. A., and Rodriguez, P. (2004). How do memory systems interact. Evidence from human classification learning. Neurobiol. Learn. Mem. 82, 324–332. doi: 10.1016/j.nlm.2004.05.003
Squire, L. R. (1992). Declarative and nondeclarative memory: multiple brain systems supporting learning and memory. J. Cogn. Neurosci. 4, 232–243. doi: 10.1162/jocn.1992.4.3.232
Citation: Ortu D, Skavhaug IM and Vaidya M (2013) Timescales of learning in the basal ganglia and the hippocampus. Front. Behav. Neurosci. 7:98. doi: 10.3389/fnbeh.2013.00098
Received: 21 June 2013; Accepted: 16 July 2013;
Published online: 01 August 2013.
Edited by:
Nuno Sousa, University of Minho, PortugalReviewed by:
João J. Cerqueira, University of Minho, PortugalSrinivasa Chakravarthy, Indian Institute of Technology Madras, India
Copyright © 2013 Ortu, Skavhaug and Vaidya. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: daniele.ortu@unt.edu