Mini Review ARTICLE
Morphological elucidation of basal ganglia circuits contributing reward prediction
- 1Laboratory of Neural Circuitry, Department of Systems Neuroscience, Graduate School of Brain Science, Doshisha University, Kyoto, Japan
- 2Core Research for Evolutional Science and Technology, Japan Science and Technology Agency, Tokyo, Japan
Electrophysiological studies in monkeys have shown that dopaminergic neurons respond to the reward prediction error. In addition, striatal neurons alter their responsiveness to cortical or thalamic inputs in response to the dopamine signal, via the mechanism of dopamine-regulated synaptic plasticity. These findings have led to the hypothesis that the striatum exhibits synaptic plasticity under the influence of the reward prediction error and conduct reinforcement learning throughout the basal ganglia circuits. The reinforcement learning model is useful; however, the mechanism by which such a process emerges in the basal ganglia needs to be anatomically explained. The actor–critic model has been previously proposed and extended by the existence of role sharing within the striatum, focusing on the striosome/matrix compartments. However, this hypothesis has been difficult to confirm morphologically, partly because of the complex structure of the striosome/matrix compartments. Here, we review recent morphological studies that elucidate the input/output organization of the striatal compartments.
Reinforcement learning mechanisms have been recently proposed to be based in circuits of the basal ganglia, assuming that the dopamine nigrostriatal projection acts as a reinforcement signal pathway (Sutton, 1988; Schultz et al., 1997, 1998; Sutton and Barto, 1995; Bayer and Glimcher, 2005; Cohen et al., 2012; Hart et al., 2014). Further, Reynolds et al. (2001) have reported that synaptic potentiation in striatal neurons receiving dopamine projections depends on the input from the cerebral cortex and on the dopaminergic input from the substantia nigra. These reports have led to the hypothesis that synaptic plasticity in the striatum is under the influence of the reward prediction error, and that the striatum conducts reinforcement learning throughout the basal ganglia circuits (Barto, 1995; Montague et al., 1996; Doya, 1999, 2000a,b; Crittenden and Graybiel, 2011; Takahashi et al., 2011).
In the actor–critic models, the actor chooses actions according to some policy of behavior, and the critic offers immediate feedback that notifies the actor whether the selected action was good or bad for obtaining rewards in the long run (Barto et al., 1983; for review, see Takahashi et al., 2011). Houk et al. (1995) proposed the existence of role sharing within the striatum in reinforcement learning, focusing on the striosome/matrix compartments. According to the model, the matrix performs action selection through the basal ganglia output nuclei [the internal segment of globus pallidus (GPi)/substantia nigra pars reticulata (SNr)] (actor), whereas the striosomes perform reward prediction (critic). The projection that targets dopaminergic neurons calculates the reward prediction errors, and the actor–critic learning is processed by the dopaminergic projections to the striatum.
However, this hypothesis has been difficult to test, partly because of the complex structure of the striosome/matrix compartments in the striatum. In particular, because this structure is highly irregular and cannot be visualized without processing, such as immunostaining, identification of the exact input and output pathways is difficult. We recently elucidated the input/output organization of the striosome/matrix structure using single neuron tracing by a viral tracer with a membrane translocation signal and immunohistochemistry for vesicular glutamate transporters. In this manuscript, we review recent progress in understanding the anatomical basis of basal ganglia networks in terms of reinforcement learning models, particularly the actor-critic model.
Striatal Mosaic Organization and Actor–Critic Model
Neurons in the striatum, unlike those in the cerebral cortex and cerebellum, do not form a layered or columnar structure. Although they appear to be randomly distributed, they are actually scattered in two embryologically different compartments called striosomes (often referred to as “patches” in rodents) and the matrix. The striosome compartment is embryologically older and the “dopamine island,” observed only during development, corresponds to patch/striosome. The matrix develops later and eventually accounts for approximately 85% of the entire striatum (Johnston et al., 1990; Nakamura et al., 2009). The matrix compartment is densely stained with acetylcholinesterase, and calbindin and somatostatin are expressed at relatively high levels (Graybiel and Ragsdale, 1978; Gerfen and Young, 1988; Gerfen, 1992). The striosome compartment is rich in μ-opioid receptors (Delfs et al., 1994; Mansour et al., 1994, 1995; Minami et al., 1994; Arvidsson et al., 1995; Kaneko et al., 1995; Ding et al., 1996; Nakamura et al., 2009).
Cortical Input and Striosome/Matrix Structure
The striatum receives glutamatergic inputs from the cerebral cortex and thalamus (Smith and Bolam, 1990), dopaminergic inputs from the substantia nigra pars compacta (SNc), and serotonergic and noradrenergic inputs from the raphe and locus coeruleus of the brain stem. The cortical input arises from almost all areas of the cerebral cortex with a local responsiveness. For example, sensory and motor cortical areas project to the part of the putamen posterior to the anterior commissure, whereas inputs from the frontal–parietal–temporal association area project to the putamen anterior from the anterior commissure and the major part of the caudate nucleus. The limbic cortex projects to the caudate nucleus, to the anterior ventral part of the putamen, and to the nucleus accumbens (Albin et al., 1989; Alexander and Crutcher, 1990).
The cortical input to the striosomes primarily arises from the limbic cortex, specifically from the orbitofrontal cortex and insula. However, it is generally considered that the input to the matrix in the rat (Gerfen, 1984, 1989; Donoghue and Herkenham, 1986), cat (Malach and Graybiel, 1986; Ragsdale and Graybiel, 1991), and monkey (Flaherty and Graybiel, 1994) arises from a wide area of the neocortex, including the motor cortex, somatosensory area, and parietal lobe. A more distinct characteristic is the layer structure of the cerebral cortex. In rats, it has been reported that cortical layers III and Va project to the matrix and layers Vb and VI project to the striosomes and that subregions of the striatum, having both striosome and matrix compartments, are innervated by the related cortical regions (Kincaid and Wilson, 1996). Thus, the striosome and matrix compartments receive the “specific” but “related” information from the cortex to contribute to their putative roles as “actor” and “critic,” respectively. Striatal interneurons with dendrites that cross compartmental borders may be a key for sorting and integrating the corticostriatal projections.
Thalamic Input and Striosome/Matrix Structure
Projections from the thalamus primarily arise from the intralaminar nuclei (particularly the centromedian and parafascicular nucleus), from motor relay nuclei (the anterior ventral and ventral lateral nucleus), and from the posterior thalamus (the posterior lateral nucleus and pulvinar). The role of thalamostriatal projections in learning, particularly in the learning and memory of movement has been recently verified (Kimura et al., 2004; Kato et al., 2011).
With respect to the striatal mosaic organization, we have reported that there is approximately three times as much thalamic input to the matrix as to the striosomes (Figure 1; Fujiyama et al., 2006), using immunohistochemistry of vesicular glutamate transporters (Fujiyama et al., 2001, 2004; Kaneko and Fujiyama, 2002; Kaneko et al., 2002). The intralaminar thalamic nuclei are thought to be the major source of thalamostriatal terminals. Axons from the caudal part of the intralaminar nuclei chiefly project to the matrix compartment in monkeys (Sadikot et al., 1990, 1992), cats (Beckstead, 1984; Ragsdale and Graybiel, 1991), and rats (Herkenham and Pert, 1981; Gerfen, 1984, 1985, 1989, 2004; Deschênes et al., 1996). Conversely, the midline thalamic nuclei, including the paraventricular and rhomboid nuclei, mainly project to the neostriatal striosome compartment and ventral striatum in cat (Ragsdale and Graybiel, 1991). These results indicate that the striosome and matrix compartments not only receive different densities of thalamic inputs but also admit inputs from different thalamic nuclei. The midline nuclei receive inputs from limbic regions (Cornwall and Phillipson, 1988). Limbic afferents through the midline nuclei as well as those from limbic cortical areas are used in striosome compartment to calculate the “state value,” which is possibly a key variable in the process of reinforcement learning (for review, see Doya, 2000a,b; Doya et al., 2002). Thus, the distinct networks for the matrix and striosome compartments involve thalamic and cortical afferents.
Figure 1. Cortical–thalamic input and striosome–matrix structure (from Fujiyama et al., 2006 with modifications). (A) Double immunofluorescence images for MOR and either VGluT1 or VGluT2 in the neostriatum. Intense MOR-positive patch compartment corresponded to areas that were weak in VGluT2 immunoreactivity (D–F). In contrast, VGluT1 immunoreactivity distributed evenly in the neostriatum (A–C). (B) Immunoelectron micrographs for MOR and either VGluT1 or VGluT2 in the neostriatum. MOR, mu-opioid receptor; VGLUT, vesicular glutamate transporter.
Differences exist in the synaptic organization of thalamostriatal neurons between the striosome and matrix compartments (Fujiyama et al., 2006; Raju et al., 2006). Our quantitative analysis of ultrastructural images revealed that in striosomes, 84% of thalamostriatal synapses were made on dendritic spines, whereas in the matrix compartment, 70% were made on dendritic shafts. Contrastingly, corticostriatal terminals preferentially synapsed onto dendritic spines (~80%) in both compartments (Figure 1; Fujiyama et al., 2006). Furthermore, thalamostriatal axospinous synapses in striosomes were larger than corticostriatal axospinous synapses in either compartments (Fujiyama et al., 2006). Excitatory axospinous synapses, including corticostriatal axospinous synapses, often display a high degree of synaptic plasticity (Calabresi et al., 2000). Moreover, dendritic spines are known to rapidly and frequently change their form, presumably reflecting their plastic characteristics (Yuste and Bonhoeffer, 2001). These findings suggest that the thalamostriatal synapses on dendritic shafts in the matrix, contributing to the “Actor,” are less plastic than those on dendritic spines in the striosome compartment, contributing the “Critic” and also less plastic than corticostriatal axospinous synapses.
Nigrostriatal Dopaminergic Input
Beside the excitatory glutamatergic inputs from the cortex and thalamus to the striatum, there are other important striatopetal projections, such as dopaminergic inputs from the SNc. The dopaminergic neurons in the midbrain are known to respond to the reward prediction error (Schultz et al., 1993, 1997, 1998; Schultz, 2007a,b). As described above, dopamine likely modulates synaptic plasticity between the corticostriatal afferents and striatal projection neurons (Calabresi et al., 2000, 2007; Reynolds et al., 2001; Surmeier et al., 2007; Shen et al., 2008).
Motivational value and motivational salience signals of dopaminergic neurons are distributed in an anatomical gradient across the substantia nigra and ventral tegmental area (VTA) (Bromberg-Martin et al., 2010). Anatomically, dopaminergic neurons in SNc are divided into the calbindin-positive dorsal tier and calbindin-negative ventral tier. Conventionally, the nigrostriatal projection in the rat brain has been reported to have the organization such that dopaminergic neurons in dorsal SNc chiefly project to the matrix, whereas those in ventral SNc mainly project to the striosomes (Gerfen et al., 1987). A similar segregation of nigrostriatal projections to striosomes and matrix compartments has been reported in cats and primates (Jimenez-Castellanos and Graybiel, 1987; Langer and Graybiel, 1989). However, this segregated organization was only partly supported by the results from our single neuron tracing study. We found that all single dopaminergic neurons innervated both striosome and matrix compartments, although projections from dorsal SNc neurons favored the matrix compartment and those from ventral SNc neurons favored the striosome compartment (Matsuda et al., 2009). Single dopaminergic neurons in the dorsal and ventral SNc innervated both striosome and matrix compartments is important, suggesting that identical temporal difference (TD) signals are simultaneously sent to a large number of striosome and matrix neurons.
However, how dopaminergic nigrostriatal projection processes specific reward-related learning remains unknown. One possibility is that phasically released dopamine modifies excitatory synapses. Its principal action will thus be at those cortical and thalamic synapses that are “active,” aiding the “selection” of striatal neurons to be fired (see Bolam et al., 2006; Arbuthnott and Wickens, 2007).
How do Basal Ganglia Mediate Motor and Learning?
New Aspects of Direct/Indirect Pathways
Projection neurons in the striatum are classified into two groups, depending on their neurochemical properties and projection targets, which in turn transmit information via different routes to output nuclei, such as GPi and SNr. It is believed that the first projection group corresponds to a direct pathway, wherein the neurons containing both GABA and substance P directly project to the output nuclei, whereas the second one involves an indirect pathway, wherein the neurons containing GABA and enkephalin project to the output nuclei via the external segment of globus pallidus (GPe) and subthalamic nucleus (Albin et al., 1989; Alexander and Crutcher, 1990; Graybiel, 1990). Because these output nuclei contain GABAergic inhibitory neurons that discharge at a high rate, the projection targets in the thalamus and superior colliculus are usually in an inhibited state. Striatal projection neurons are GABAergic; therefore, excitation of these neurons by cortical inputs may lead to temporary inhibition of the output nuclei via the direct pathway and to disinhibition of the target regions (the thalamus and cerebral cortex), allowing selected movements to occur (Nambu et al., 2002). However, when the indirect pathway is activated, the target regions are further inhibited because projection neurons from GPe to the subthalamic nucleus are also GABAergic and those from the subthalamic nucleus to the output nuclei are glutamatergic. Therefore, while the direct pathway allows the expression of required movement via disinhibition during the necessary time period, the indirect pathway may be suppressing unnecessary movement and thus highlighting the outcome from the direct pathway. These findings suggest a “center–surround” model of basal ganglia function, comprising focused selection of an appropriate motor program and inhibition of competing motor programs (Mink and Thach, 1993; Mink, 1996; Hikosaka et al., 2000; Nambu et al., 2002).
Dopaminergic projections from SNc produce excitatory modulation of direct pathway neurons by an action at dopamine D1 receptors (D1Rs) and inhibitory modulation of indirect pathway neurons by an action at dopamine D2 receptors (D2Rs), effectively eliciting opposite effects in the direct and indirect pathways (Hong and Costa, 1978; Hong et al., 1978; Gerfen et al., 1990). This conceptualization has been widely accepted because it can explain clinical findings and therapeutic effects in disorders, such as Parkinson's disease.
However, it has been reported that a majority of striatal neurons are activated during movement (DeLong, 1990; Costa et al., 2004) and that both pathways are co-activated during movement initiation (Cui et al., 2013; Isomura et al., 2013). Further, recent optogenetic studies showed that both pathways were concomitantly active during sequence initiation but behaved differently during sequence performance (Jin et al., 2014). Single neuron tracing studies have revealed that almost all direct pathway neurons projected to GPe, a relay nucleus of the indirect pathway (Kawaguchi et al., 1990; Lévesque and Parent, 2005; Fujiyama et al., 2011), indicating that direct pathway neurons drive both direct and indirect pathways. Other projection systems have also been reported, such as the hyperdirect pathway (Nambu et al., 2002), cortico-dopaminergic projections (Watabe-Uchida et al., 2012), and differential cortical innervation of D1R- and D2R-positive striatal neurons (Wall et al., 2013). Furthermore, Kravitz et al. (2012) reported that positive reinforcement caused by direct pathway stimulation persists for long durations in mice, whereas punishment caused by indirect pathway stimulation was transient (Kravitz and Kreitzer, 2012; Kravitz et al., 2012). Jin et al. (2014) also showed that the basal ganglia contribute to behavior during learning rather than simple motor control (for review, see Friend and Kravitz, 2014). Thus, the original conceptualization of the direct and indirect pathways is likely to be modified through further studies, particularly those using behavioral experiments.
Direct/Indirect Pathways and Striosome/Matrix Structure
The output from striosome/matrix compartments has been difficult to examine with anterograde tracers because of the irregularities in the striatal structure. Using a single neuron tracing technique, we found that striosomes also include indirect pathway neurons projecting to GPe (Figure 2; Fujiyama et al., 2011). Further, unlike the matrix, direct pathway neurons in striosomes project not only to GPi/SNr but also directly to SNc, where dopaminergic neurons are present (Figure 2; Gerfen, 1984; Lévesque and Parent, 2005; Fujiyama et al., 2011; Watabe-Uchida et al., 2012).
Figure 2. A simplified diagram of output from the striosome–matrix structure (from Fujiyama et al., 2011 with modifications) GPe, external segment of globus pallidus; GPi, internal segment of globus pallidus; PPT, pedunculopontine nucleus; SNc, substantia nigra pars compacta; SNr, substantia nigra pars reticulata; STH, subthalamus.
The actor–critic model was supported by the finding that neurons responding to action and state values were distributed in the neostriatum (Samejima et al., 2005; Kawato and Samejima, 2007; Lau and Glimcher, 2008; Wang et al., 2013). Furthermore, dopamine signals contain information about reward and state value (Schultz et al., 1998; Schultz, 2007a,b); striosomal neurons, which directly control dopaminergic neurons in SNc, may provide the dopaminergic neurons with state-based signals. Dopaminergic neurons receive monosynaptic inhibitory input from SNr and polysynaptic disinihibitory input from the GPe (Grofova et al., 1982; Saitoh et al., 2004; Tepper and Lee, 2007; Brazhnik et al., 2008), and SNr and GPe are innervated by striosomal neurons. Hence, dopaminergic neurons may receive disinhibitory input through the SNr, and polysynaptic inhibitory input through the GPe, subthalamic nucleus, and SNr, from the striosomes. Dopaminergic neurons are also considered to receive a stochastic reward signal from the pedunculopontine tegmental nucleus and other structures (Figure 2; Okada et al., 2009). Our recent study revealed that the axonal arbors of most dopamine neurons covered a single large oval volume, occupying at most 5.7% of the total neostriatal volume. Furthermore, all single dopamine neurons we traced innervated striosomes and matrix compartments with or without volume transmission (Matsuda et al., 2009). Dopamine signals may thereby change the response properties of striatal neurons (Calabresi et al., 2000; Reynolds et al., 2001; Surmeier et al., 2007; Shen et al., 2008). The striosomal control of the dopaminergic neurons responsible for reward prediction errors is particularly relevant to its potential role as “critic;” it may serve this function by calculating the state value and sending it to dopaminergic neurons (Barto, 1995; Houk et al., 1995).
Ventral Striatum and Limbic Loop
In the basal ganglia, VTA–nucleus accumbens dopaminergic projection system has been considered along with the nigrostriatal dopaminergic projection system (see Groenewegen et al., 1999; Zahm, 1999). The ventral part of the striatum centered on the nucleus accumbens is called the limbic or ventral striatum; this structure is divided into a central portion (core) and periphery (shell) (Herkenham et al., 1984). GABAergic output from the shell projects either directly or through the ventromedial part of the ventral pallidum to the dorsomedial thalamic nucleus, the lateral hypothalamus, VTA, dopaminergic neurons in the substantia nigra, and is part of the mesolimbic dopaminergic projection system. The core involves a system that projects via the dorsolateral portion of the ventral pallidum to the subthalamic nucleus, SNr, and GPi, and subsequently enters the motor loop (see Groenewegen et al., 1999; Zahm, 1999). Therefore, it may help switch the input from the emotion system to the movement system. Dopamine projections from VTA to the nucleus accumbens are involved in selecting the environmental contexts leading to reward (O'Doherty et al., 2004; Canales, 2005; Goto and Grace, 2008; Humphries and Prescott, 2010; Glimcher, 2011; Morita et al., 2012, 2013; Hart et al., 2014). Takahashi et al. (2008); Takahashi et al. (2011) reported that information about task structure is represented in the orbitofrontal cortex and that it influences the computation of reward prediction error in VTA dopaminergic neurons. This presumably occurs via the ventral striatum, where the state value would be computed. However, a recent optogenetic study reported that the striatal medium spiny neurons, including striosomal neurons, synapse onto dopamine neurons only very weakly and instead strongly synapse onto GABAergic neurons in VTA (Chuhma et al., 2011), which in turn project to cholinergic neurons in the accumbens (Brown et al., 2012). Furthermore, up to 17% of accumbens shell neurons co-express D1R and D2R (Rashid et al., 2006; Bertran-Gonzalez et al., 2008; Ng et al., 2010); therefore, ventral striatal pathways are not well adapted to the direct/indirect schema of the dorsal striatum. Hence, further anatomical and physiological studies are necessary to test the synaptic connections of VTA–nucleus accumbens dopaminergic projection system.
Although both involve the striatum, the direct/indirect pathways and striosome/matrix compartments have been investigated separately. Two functions of the basal ganglia— motor control and reinforcement learning—have also been traditionally discussed and understood separately. However, recent studies have shown that direct pathway neurons mediate movement, reinforcement, and reward, whereas indirect pathway neurons inhibit movement and mediate punishment and aversion (see Kravitz and Kreitzer, 2012). Recent morphological studies have shown how the striatal mosaic organization and direct/indirect pathway neurons coexist in the striatum. This suggests that striosomes are involved in controlling dopaminergic neurons responsible for reward prediction errors and for direct and indirect pathways and is of a particular relevance to both motor control and reinforcement learning. The dual anatomical and physiological pathways in the striatum may hold the key to why the basal ganglia have two functions. Further studies should examine pathway- and compartment-specific activity simultaneously in different contexts to clarify how the structure of the basal ganglia contributes to behavioral learning.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
This study was supported by Grants-in-Aid from The Ministry of Education, Culture, Sports, Science, and Technology (MEXT) for Scientific Research (25282247); and for Scientific Researches on Innovative Areas “Adaptive Circuit Shift” (26112001) and “Prediction and Decision Making” (26120725).
Alexander, G. E., and Crutcher, M. D. (1990). Functional architecture of basal ganglia circuits: neural substrates of parallel processing. Trends Neurosci. 13, 266–271. doi: 10.1016/0166-2236(90)90107-L
Arvidsson, U., Riedl, M., Chakrabarti, S., Lee, J. H., Nakano, A. H., Dado, R. J., et al. (1995). Distribution and targeting of a mu-opioid receptor (MOR1) in brain and spinal cord. J. Neurosci. 15, 3328–3341.
Barto, A. G., Sutton, R. S., and Anderson, C. W. (1983). Neuron-like adaptive elements that can solve difficult learning control problems. IEEE Trans. Syst. Man Cybern. 13, 834–846. doi: 10.1109/TSMC.1983.6313077
Bertran-Gonzalez, J. B., Bosch, C., Maroteaux, M., Herve, D., Valjent, E., and Girault, J. A. (2008). Opposing patterns of signaling activation in dopamine D1 and D2 receptor-expressing striatal neurons in response to cocaine and haloperidol. J. Neurosci. 28, 5671–5685. doi: 10.1523/JNEUROSCI.1039-08.2008
Bolam, J. P., Bergman, H., Graybiel, A., Kimura, M., Plenz, D., Seung, H. S., et al. (2006). “Molecules, microcircuits and motivated behaviour: microcircuits in the striatum,” in Microcircuits: The Interface between Neurons and Global Brain Function. Dahlem Workshop Report 93, eds S. Grillner and A. Graybiel (Cambridge, MA: The MIT Press) 165–190.
Brazhnik, E., Shah, F., and Tepper, J. M. (2008). GABAergic afferents activate both GABAA and GABAB receptors in mouse substantia nigra dopaminergic neurons in vivo. J. Neurosci. 28, 10386–10398. doi: 10.1523/JNEUROSCI.2387-08.2008
Brown, M. T. G., Tan, K. R., O'Connor, E. C., Nikonenko, I., Muller, D., and Lüscher, C. (2012). Ventral tegmental area GABA projections pause accumbal cholinergic interneurons to enhance associative learning. Nature 492, 452–456. doi: 10.1038/nature11657
Canales, J. J. (2005). Stimulant-induced adaptations in neostriatal matrix and striosome systems: transiting from instrumental responding to habitual behavior in drug addiction. Neurobiol. Learn. Mem. 83, 93–103. doi: 10.1016/j.nlm.2004.10.006
Cohen, J. Y., Haesler, S., Vong, L., Lowell, B. B., and Uchida, N. (2012). Neuron-type specific signals for reward and punishment in the ventral tegmental area. Nature 482, 85–88. doi: 10.1038/nature10754
Cornwall, J., and Phillipson, O. T. (1988). Afferent projections to the dorsal thalamus of the rat as shown by retrograde lectin transport. II. The midline nuclei. Brain Res. Bull. 21, 147–161. doi: 10.1016/0361-9230(88)90227-4
Costa, R. M., Cohen, D., and Nicolelis, M. A. L. (2004). Differential corticostriatal plasticity during fast and slow motor skill learnig in mice. Curr. Biol. 14, 1124–1134. doi: 10.1016/j.cub.2004.06.053
Crittenden, J. R., and Graybiel, A. M. (2011). Basal Ganglia disorders associated with imbalances in the striatal striosome and matrix compartments. Front. Neuroanat. 5:59. doi: 10.3389/fnana.2011.00059
Cui, G., Jun, S. B., Jin, X., Pham, M. D., Vogel, S. S., Lovinger, D. M., et al. (2013). Concurrent activation of striatal direct and indirect pathways during action initiation. Nature 494, 238–242. doi: 10.1038/nature11846
Delfs, J. M., Kong, H., Mestek, A., Chen, Y., Yu, L., Reisine, T., et al. (1994). Expression of mu opioid receptor mRNA in rat brain: an in situ hybridization study at the single cell level. J. Comp. Neurol. 345, 46–68. doi: 10.1002/cne.903450104
Deschênes, M., Bourassa, J., Doan, V. D., and Parent, A. (1996). A single-cell study of the axonal projections arising from the posterior intralaminar thalamic nuclei in the rat. Eur. J. Neurosci. 8, 329–343. doi: 10.1111/j.1460-9568.1996.tb01217.x
Donoghue, J. P., and Herkenham, M. (1986). Neostriatal projections from individual cortical fields conform to histochemically distinct striatal compartments in the rat. Brain Res. 365, 397–403. doi: 10.1016/0006-8993(86)91658-6
Fujiyama, F., Furuta, T., and Kaneko, T. (2001). Immunocytochemical localization of candidates for vesicular glutamate transporters in the rat cerebral cortex. J. Comp. Neurol. 435, 379–387. doi: 10.1002/cne.1037
Fujiyama, F., Kuramoto, E., Okamoto, K., Hioki, H., Furuta, T., Zhou, L., et al. (2004). Presynaptic localization of an AMPA-type glutamate receptor in corticostriatal and thalamostriatal axon terminals. Eur. J. Neurosci. 20, 3322–3330. doi: 10.1111/j.1460-9568.2004.03807.x
Fujiyama, F., Sohn, J., Nakano, T., Furuta, T., Nakamura, K. C., Matsuda, W., et al. (2011). Exclusive and common targets of neostriatofugal projections of rat striosome neurons: a single neuron-tracing study using a viral vector. Eur. J. Neurosci. 33, 668–677. doi: 10.1111/j.1460-9568.2010.07564.x
Fujiyama, F., Unzai, T., Nakamura, K., Nomura, S., and Kaneko, T. (2006). Difference in organization of corticostriatal and thalamostriatal synapses between patch and matrix compartments of rat neostriatum. Eur. J. Neurosci. 24, 2813–2824. doi: 10.1111/j.1460-9568.2006.05177.x
Gerfen, C. R. (1985). The neostriatal mosaic. I. Compartmental organization of projections from the striatum to the substantia nigra in the rat. J. Comp. Neurol. 236, 454–476. doi: 10.1002/cne.902360404
Gerfen, C. R., Engber, T. M., Mahan, L. C., Susel, Z., Chase, T. N., Monsma, F. J., et al. (1990). D1 and D2 dopamine receptor regulated gene expression of striatonigral and striatopallidal neurons. Science 250, 1429–1432. doi: 10.1126/science.2147780
Gerfen, C. R., and Young, W. S. (1988). Distribution of striatonigral and striatopallidal peptidergic neurons in both patch and matrix compartments: an in situ hybridization histochemistry and fluorescent retrograde study. Brain Res. 460, 161–167. doi: 10.1016/0006-8993(88)91217-6
Glimcher, P. W. (2011). Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis. Proc. Natl. Acad. Sci. U.S.A. 108, 15647–15654. doi: 10.1073/pnas.1014269108
Graybiel, A. M., and Ragsdale, C. W. Jr. (1978). Histochemically distinct compartments in the striatum of human, monkeys, and cat demonstrated by acetylthiocholinesterase staining. Proc. Natl. Acad. Sci. U.S.A. 75, 5723–5726. doi: 10.1073/pnas.75.11.5723
Groenewegen, H. J., Wright, C. I., Beijer, A. V. J., and Voorn, P. (1999). Convergence and segregation of ventral striatal inputs and outputs. Ann. N.Y. Acad. Sci. 877, 49–63. doi: 10.1111/j.1749-6632.1999.tb09260.x
Grofova, I., Deniau, J. M., and Kitai, S. T. (1982). Morphology of the substantia nigra pars reticulata projection neurons intracellularly labeled with HRP. J. Comp. Neurol. 208, 352–368. doi: 10.1002/cne.902080406
Hart, A. S., Rutledge, R. B., Glimcher, P. W., and Phillips, P. E. M. (2014). Phasic dopamine release in the rat nucleus accumbens symmetrically encodes a reward prediction error term. J. Neurosci. 34, 698–704. doi: 10.1523/JNEUROSCI.2489-13.2014
Herkenham, M., Edley, S. M., and Stuart, J. (1984). Cell clusters in the nucleus accumbens of the rat, and the mosaic relationship of opiate receptors, acetylcolinesterase and subcortical afferent terminations. Neuroscience 11, 561–593. doi: 10.1016/0306-4522(84)90045-9
Hikosaka, O., Takikawa, Y., and Kawagoe, R. (2000). Role of the basal ganglia in the control of purposive saccadic eye movements. Physiol. Rev. 80, 953–978. Available online at: http://physrev.physiology.org/content/physrev/80/3/953.full.pdf#pdfjs.action=download
Hong, J. S., Yang, H. Y., Fratta, W., and Costa, E. (1978). Rat striatal methionine-enkephalin content after chronic treatment with cataleptogenic and noncataleptogenic antischizophrenic drugs. J. Pharmacol. Exp. Ther. 205, 141–147.
Houk, J. C., Adams, J. L., and Barto, A. G. (1995). “A model of how the basal ganglia generate and use neural signals that predict reinforcement,” in Models of Information Processing in the Basal Ganglia, eds J. C. Houk, J. Davis, and D. Beiser (Cambridge, MA: The MIT Press), 215–232.
Humphries, M. D., and Prescott, T. J. (2010). The ventral basal ganglia, a selection mechanism at the crossroads of space, strategy, and reward. Prog. Neurobiol. 90, 385–417. doi: 10.1016/j.pneurobio.2009.11.003
Isomura, Y., Takekawa, T., Harukuni, R., Handa, T., Aizawa, H., Takada, M., et al. (2013). Reward-modulated motor information in identified striatum neurons. J. Neurosci. 33, 10209–10220. doi: 10.1523/JNEUROSCI.0381-13.2013
Jimenez-Castellanos, J., and Graybiel, A. M. (1987). Subdivisions of the dopamine-containing A8-A9-A10 complex identified by their differential mesostriatal innervation of striosomes and extrastriosomal matrix. Neuroscience 23, 223–242. doi: 10.1016/0306-4522(87)90285-5
Johnston, J. G., Gerfen, C. R., Haber, S. N., and van der Kooy, D. (1990). Mechanisms of striatal pattern formation: conservation of mammalian compartmentalization. Brain Res. Dev. Brain Res. 57, 93–102. doi: 10.1016/0165-3806(90)90189-6
Kaneko, T., Fujiyama, F., and Hioki, H. (2002). Immunohistochemical localization of candidates for vesicular glutamate transporters in the rat brain. J. Comp. Neurol. 444, 39–62. doi: 10.1002/cne.10129
Kaneko, T., Minami, M., Satoh, M., and Mizuno, N. (1995). Immunocytochemical localization of mu-opioid receptor in the rat caudate-putamen. Neurosci. Lett. 184, 149–152. doi: 10.1016/0304-3940(94)11192-L
Kato, S., Kuramochi, M., Kobayashi, K., Fukabori, R., Okada, K., Uchigashima, M., et al. (2011). Selective neural pathway targeting reveals key roles of thalamostriatal projection in the control of visual discrimination. J. Neurosci. 31, 17169–17179. doi: 10.1523/JNEUROSCI.4005-11.2011
Kimura, M., Minamimoto, T., Matsumoto, N., and Hori, Y. (2004). Monitoring and switching of cortico-basal ganglia loop functions by the thalamo-striatal system. Neurosci. Res. 48, 355–360. doi: 10.1016/j.neures.2003.12.002
Langer, L. F., and Graybiel, A. M. (1989). Distinct nigrostriatal projection systems innervate striosomes and matrix in the primate striatum. Brain Res. 498, 344–350. doi: 10.1016/0006-8993(89)91114-1
Lévesque, M., and Parent, A. (2005). The striatofugal fiber system in primates: a reevaluation of its organization based on single-axon tracing studies. Proc. Natl. Acad. Sci. U.S.A. 102, 11888–11893. doi: 10.1073/pnas.0502710102
Mansour, A., Fox, C. A., Burke, S., Akil, H., and Watson, S. J. (1995). Immunohistochemical localization of the cloned mu opioid receptor in the rat CNS. J. Chem. Neuroanat. 8, 283–305. doi: 10.1016/0891-0618(95)00055-C
Mansour, A., Fox, C. A., Burke, S., Meng, F., Thompson, R. C., Akil, H., et al. (1994). Mu, delta, and kappa opioid receptor mRNA expression in the rat CNS: an in situ hybridization study. J. Comp. Neurol. 350, 412–438. doi: 10.1002/cne.903500307
Matsuda, W., Furuta, T., Nakamura, K. C., Hioki, H., Fujiyama, F., Arai, R., et al. (2009). Single nigrostriatal dopaminergic neurons form widely spread and highly dense axonal arborizations in the neostriatum. J. Neurosci. 29, 444–453. doi: 10.1523/JNEUROSCI.4029-08.2009
Minami, M., Onogi, T., Toya, T., Katao, Y., Hosoi, Y., Maekawa, K., et al. (1994). Molecular cloning and in situ hybridization histochemistry for rat mu-opioid receptor. Neurosci. Res. 18, 315–322. doi: 10.1016/0168-0102(94)90167-8
Morita, K., Morishima, M., Sakai, K., and Kawaguchi, Y. (2012). Reinforcement learning: computing the temporal difference of values via distinct corticostriatal pathways. Trends Neurosci. 35, 457–467. doi: 10.1016/j.tins.2012.04.009
Morita, K., Morishima, M., Sakai, K., and Kawaguchi, Y. (2013). Dopaminergic control of motivation and reinforcement learning: a closed-circuit account for reward-oriented behavior. J. Neurosci. 33, 8866–8890. doi: 10.1523/JNEUROSCI.4614-12.2013
Nakamura, K. C., Fujiyama, F., Furuta, T., Hioki, H., and Kaneko, T. (2009). Afferent islands are larger than μ-opioid receptor patch in striatum of rat pups. Neuroreport 20, 584–588. doi: 10.1097/WNR.0b013e328329cbf9
Ng, J., Rashid, A. J., So, C. H., O'Dowd, B. F., and George, S. R. (2010). Activation of calcium/calmodulin-dependent protein kinase llalpha in the striatum by the heteromeric D1-D2 dopamine receptor complex. Neuroscience 165, 535–541. doi: 10.1016/j.neuroscience.2009.10.017
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., and Dolan, R. (2004). Dissociable roles of dorsal and ventral striatum in instrumental conditioning. Science 304, 452–454. doi: 10.1126/science.1094285
Okada, K., Toyama, K., Inoue, Y., Isa, T., and Kobayashi, Y. (2009). Different pedunculopontine tegmental neurons signal predicted and actual task rewards. J. Neurosci. 29, 4858–4870. doi: 10.1523/JNEUROSCI.4415-08.2009
Raju, D. V., Shah, D. J., Wright, T. M., Hall, R. A., and Smith, Y. (2006). Differential synaptology of vGluT2-containing thalamostriatal afferents between the patch and matrix compartments in rats. J. Comp. Neurol. 499, 231–243. doi: 10.1002/cne.21099
Rashid, A. J., So, C. H., Kong, M. M., Furtak, T., El-Ghundil, M., Cheng, R., et al. (2006). D1-D2 dopaminereceptor heterooligomers with unique pharmacology are coupled to rapid activation of Gq/11 in the striatum. Proc. Natl. Acad. Sci. U.S.A. 104, 654–659. doi: 10.1073/pnas.0604049104
Sadikot, A. F., Parent, A., and Francois, C. (1990). The centre median and parafascicular thalamic nuclei project respectively to the sensorimotor and associative-limbic striatal territories in the squirrel monkey. Brain Res. 510, 161–165. doi: 10.1016/0006-8993(90)90746-X
Sadikot, A. F., Parent, A., Smith, Y., and Bolam, J. P. (1992). Efferent connections of the centromedian and parafascicular thalamic nuclei in the squirrel monkey: a light and electron microscopic study of the thalamostriatal projection in relation to striatal heterogeneity. J. Comp. Neurol. 320, 228–242. doi: 10.1002/cne.903200207
Schultz, W., Apicella, P., and Ljungberg, T. (1993). Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 13, 900–913.
Smith, A. D., and Bolam, J. P. (1990). The neural network of the basal ganglia as revealed by the study of synaptic connections of identified neurones. Trends Neurosci. 13, 259–265. doi: 10.1016/0166-2236(90)90106-K
Surmeier, D. J., Ding, J., Day, M., Wang, Z., and Shen, W. (2007). D1 and D2 dopamine-receptor modulation of striatal glutamatergic signaling in striatal medium spiny neurons. Trends Neurosci. 30, 228–235. doi: 10.1016/j.tins.2007.03.008
Takahashi, Y. K., Roesch, M. R., Wilson, R. C., Toreson, K., O'Donnell, P., Niv, Y., et al. (2011). Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex. Nat. Neurosci. 14, 1590–1597. doi: 10.1038/nn.2957
Takahashi, Y., Schoenbaum, G., and Niv, Y. (2008). Silencing the critics: understanding the effects of cocaine sensitization on dorsolateral and ventral striatum in the context of an Actor/Critic model. Front. Neurosci. 2:86–99. doi: 10.3389/neuro.01.014.2008
Wall, N. R., Parra, M. D. L., Callaway, E. M., and Kreitzer, A. C. (2013). Differential innervation of direct- and indirect-pathway striatal projection neurons. Neuron 79, 1–14. doi: 10.1016/j.neuron.2013.05.014
Keywords: reinforcement learning, dopamine, striatum, actor-critic, striosome/matrix compartments, cortex, thalamus
Citation: Fujiyama F, Takahashi S and Karube F (2015) Morphological elucidation of basal ganglia circuits contributing reward prediction. Front. Neurosci. 9:6. doi: 10.3389/fnins.2015.00006
Received: 10 October 2014; Accepted: 08 January 2015;
Published online: 05 February 2015.
Edited by:Mitsuhiro Okada, Keio University, Japan
Reviewed by:Naoshige Uchida, Harvard University, USA
V. S. Chandrasekhar Pammi, University of Allahabad, India
Copyright © 2015 Fujiyama, Takahashi and Karube. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Fumino Fujiyama, Laboratory of Neural Circuitry, Department of Systems Neuroscience, Graduate School of Brain Science, Doshisha University, Kaihu-kan, Gakken Toshi Campus, 4-1-1 Kizugawa-dai, Kizugawa, Kyoto 619–0225, Japan e-mail: email@example.com