Abstract
The hypothesis that the basal-ganglia direct and indirect pathways represent goodness (or benefit) and badness (or cost) of options, respectively, explains a wide range of phenomena. However, this hypothesis, named the Opponent Actor Learning (OpAL), still has limitations. Structurally, the OpAL model does not incorporate differentiation of the two types of cortical inputs to the basal-ganglia pathways received from intratelencephalic (IT) and pyramidal-tract (PT) neurons. Functionally, the OpAL model does not describe the temporal-difference (TD)-type reward-prediction-error (RPE), nor explains how RPE is calculated in the circuitry connecting to the DA neurons. In fact, there is a different hypothesis on the basal-ganglia pathways and DA, named the Cortico-Striatal-Temporal-Difference (CS-TD) model. The CS-TD model differentiates the IT and PT inputs, describes the TD-type RPE, and explains how TD-RPE is calculated. However, a critical difficulty in this model lies in its assumption that DA induces the same direction of plasticity in both direct and indirect pathways, which apparently contradicts the experimentally observed opposite effects of DA on these pathways. Here, we propose a new hypothesis that integrates the OpAL and CS-TD models. Specifically, we propose that the IT-basal-ganglia pathways represent goodness/badness of current options while the PT-indirect pathway represents the overall value of the previously chosen option, and both of these have influence on the DA neurons, through the basal-ganglia output, so that a variant of TD-RPE is calculated. A key assumption is that opposite directions of plasticity are induced upon phasic activation of DA neurons in the IT-indirect pathway and PT-indirect pathway because of different profiles of IT and PT inputs. Specifically, at PT→indirect-pathway-medium-spiny-neuron (iMSN) synapses, sustained glutamatergic inputs generate rich adenosine, which allosterically prevents DA-D2 receptor signaling and instead favors adenosine-A2A receptor signaling. Then, phasic DA-induced phasic adenosine, which reflects TD-RPE, causes long-term synaptic potentiation. In contrast, at IT→iMSN synapses where adenosine is scarce, phasic DA causes long-term synaptic depression via D2 receptor signaling. This new Opponency and Temporal-Difference (OTD) model provides unique predictions, part of which is potentially in line with recently reported activity patterns of neurons in the globus pallidus externus on the indirect pathway.
Existing Hypotheses: the OpAL Model and the CS-TD Model
The cortico-basal ganglia circuits have been suggested to be crucially involved in value-related cognitive and affective processes. A prevailing hypothesis, named the Opponent Actor Learning (OpAL) model () (Figure 1A), posits that the direct and indirect pathways of the basal ganglia encode the goodness (or benefit) and badness (or cost) of options, respectively. This model, rooted in previous models (; ), is based on the experimental findings indicating that the striatal direct and indirect-pathway medium spiny neurons (dMSNs and iMSNs) are positively and negatively modulated by dopamine (DA), respectively, in terms of both instantaneous responsiveness and long-term synaptic plasticity () (Figure 1A right, red and blue dashed ovals). The OpAL model explains both choice-related phenomena, such as why stimulation of dMSNs or iMSNs causes appetitive or aversive response, respectively (), and motivation/effort-related phenomena, such as why DA depletion causes a shift in the preference from high-cost-high-benefit to low-cost-low-benefit options () (i.e., according to the OpAL model, it is because dMSN’s benefit representation is weakened while iMSN’s cost representation is exaggerated) (). A recent study () found that visually responsive neurons in the globus pallidus externus (GPe), in the middle of the indirect pathway, were largely more inhibited by objects that were stably associated with bad outcomes than by objects associated with good outcomes, suggesting that the indirect pathway signals the badness of stimuli. More recent work has further revealed that iMSNs tend to show higher activity following the presentation of lower-value conditional stimulus () or in response to lower-value outcome-instructing stimulus () than the case of higher-value stimulus. The OpAL model appears to be in line with these findings.
FIGURE 1
While having the strong explanatory power, however, the OpAL model still has limitations, both structurally and functionally. Specifically, at the structural level, the OpAL model, as well as most previous models, does not incorporate differentiation of two types of cortical inputs to the basal-ganglia pathways received from two types of corticostriatal pyramidal cells, namely, intratelencephalic (IT) and pyramidal-tract (PT) neurons (; ; ). At the functional level, the OpAL model assumes that DA represents reward prediction error (RPE) (; ) and induces plasticity () so as to implement value-update, but does not describe how the DA neurons calculate RPE. Also, the RPE assumed in the OpAL model takes a simple form: R(ti+1) -V(ti), where R(ti+1) is the obtained reward and V(ti) is the expected reward, whereas the experimental results have suggested that DA generally represents a more complex form of RPE called the temporal difference (TD) RPE: R(ti+1) + V(ti+1) -V(ti), where the additional term V(ti+1) represents the future reward(s) expected as outcome of the current/upcoming state or action, which explains the famous DA response to reward-predicting stimuli (; ) (see for the difference between these two forms of RPE). Accordingly, the OpAL model does not describe fine temporal patterns of DA signals or MSN activity. Moreover, how the weights of synapses on dMSNs and iMSNs can converge to values corresponding to the goodness and badness of one single option (action) has actually not been shown, as pointed out by recent work ().
In fact, there is a different hypothesis on the cortico-basal ganglia circuit functions named the Cortico-Striatal-Temporal-Difference (CS-TD) model (, ; ; ) (Figure 1B), which posits that the direct and indirect pathways of the basal ganglia encode the value of the current and previous states/actions, respectively, and positively and negatively impact the DA neurons so that the temporal difference of values, i.e., V(ti+1) -V(ti) which constitutes the TD-RPE, can be calculated. This model is based on the experimental findings that (i) dMSNs and iMSNs are predominantly targeted by the different types of corticostriatal neurons, specifically, the IT and PT neurons, respectively (; ; ), (ii) IT neurons uni-directionally project to PT neurons (), which have strong facilitatory recurrent excitation () that might enable sustained activity, and (iii) the output nucleus of the basal ganglia has strong inhibitory influence on the DA neurons (; ). Although the anatomically suggested preferences in the corticostriatal connections were not supported by physiological () and optogenetic () studies, they were supported by model fitting of short-term plasticity data (), which suggested facilitatory IT→dMSN and PT→iMSN connections and depressive IT→iMSN and PT→dMSN connections.
However, the CS-TD model has a critical drawback. Specifically, although there are experimental results suggesting that DA modulates synaptic plasticity to the opposite directions in dMSNs and iMSNs (; ) as the OpAL model assumes, the CS-TD model assumes the same direction of plasticity induction in dMSNs and iMSNs (Figure 1B right, red dashed ovals). As a result, the stronger inhibition of GPe neurons by bad objects (), as well as the higher activity of iMSNs in the case of lower-value stimulus (; ), cannot be explained by the CS-TD model.
A New Hypothesis That Integrates the OpAL and CS-TD Models: the OTD Model
At first glance, these two models are mutually exclusive, because they made such contrasting assumptions on the synaptic plasticity on iMSNs. However, given that there exist two populations of corticostriatal neurons, i.e., IT and PT neurons, those assumptions might not be mutually exclusive. Specifically, if the iMSN synapses considered in the OpAL model are those targeted by IT neurons while the iMSN synapses considered in the CS-TD model are, as originally assumed, primarily PT neuron-targeting synapses, the two assumptions could go together (Figure 2A).
FIGURE 2
Crucially, the IT→iMSN connections and PT→iMSN connections are expected to have different activation profiles. In particular, because PT neurons receive uni-directional projections from IT neurons () and excite each other via strong excitatory synapses exhibiting short-term facilitation (), activation of PT→iMSN synapses is expected to be delayed from, and more sustained and intense than, activation of IT→iMSN synapses (schematically illustrated by spike trains of IT and PT inputs in Figures 2Ba,b). The suggestion from model fitting () that IT→iMSN synapses and PT→iMSN synapses entail short-term depression and facilitation, respectively, can also contribute to this differentiation. At PT→iMSN synapses, such sustained intense (and facilitatory) PT inputs might generate high concentration of adenosine around the synapses, because adenosine is suggested to be released depending on glutamate receptor activation in the striatum (). Then, given the suggested allosteric inhibition of DA signaling by adenosine at A2A-D2 receptors-heteromer (; ), phasic DA representing positive RPE is expected not to be able to induce long-term depression (LTD) through D2 receptor (D2R) signaling. Moreover, given that DA is suggested to cause adenosine release through activations of D1 receptors (D1Rs) and NMDA receptors in the nucleus accumbens (; ), we assume that the RPE-representing phasic DA induces phasic adenosine that also reflects RPE: since adenosine causes vasodilation () presumably on a sub-second time scale (), such RPE-reflecting phasic adenosine may cause oxygen changes that could underlie the widely reported striatal fMRI-BOLD signals correlated with RPE (; ). The positive RPE-representing phasic adenosine is then expected to induce long-term potentiation (LTP) of PT→iMSN synapses through A2A receptor signaling (c.f., ) (Figure 2B). In contrast, at IT→iMSN synapses where adenosine is scarce, phasic DA representing positive RPE is assumed to cause LTD via D2R signaling, which could then inhibit A2A receptor signaling through the suggested canonical antagonistic interaction at the level of adenylyl cyclase (; ; ; ).
Figure 2A shows the integrated Opponency and Temporal-Difference (OTD) model. At time ti (Figure 2A, left), action A(ti) is represented by a population of cortical IT neurons, and its goodness (benefit) and badness (cost) [Good(A(ti)) and Bad(A(ti))] are represented by dMSNs and iMSNs, respectively, so that the utility of the action, i.e., Good(A(ti)) -Bad(A(ti)) is computed in the downstream. When there are multiple action candidates, one action is selected based on the utility in a soft-max manner. The selected action is represented by the cortical PT neurons, which are driven by the IT neurons and the basal ganglia output, and executed through the pyramidal tract. At time ti+1 (Figure 2A, right), a population of dMSNs and a population of iMSNs represent the goodness (benefit) and badness (cost) of the upcoming action [Good(A(ti+1)) and Bad(A(ti+1))], respectively, while a different population of iMSNs represents the value of the executed action [V(A(ti))]. The dMSN population and iMSN populations positively and negatively modulate the DA neurons via the basal ganglia output, respectively, so that the DA neurons compute a form of TD-RPE: δ(ti+1) = R(ti+1) + {Good(A(ti+1)) -Bad(A(ti+1))}-V(A(ti)). When the TD-RPE is positive, the IT-dMSN connections are potentiated (red dashed oval in Figure 2A right) whereas the IT-iMSN connections are depressed (blue dashed oval), and the PT-iMSN connections are potentiated (red dashed oval). Figure 3A shows the operation of the OTD model in more detail, illustrating different populations of neurons corresponding to different actions. Notably, the IT/PT-iMSN connections corresponding to the previous action that constitutes a cause of the TD-RPE (action “A1” in the figure) are plastically changed whereas the IT/PT-iMSN connections corresponding to the current action (“A3” in the figure) are not, ensuring the causality; this could be achieved through mechanisms for creating a delayed time window for plasticity, such as those revealed for the synapses on dMSNs (). As shown in Figures 2A and 3A, the OTD model literally has functions of both OpAL and CS-TD models. Specifically, the direct and indirect pathways serve for good-bad(benefit-cost)-analysis of current states/actions/options, and simultaneously perform the calculation of TD-RPE, which is used for updating the value of previous states/actions/options. This is enabled by the duality of the role of iMSNs: initially representing the badness (cost) of a state/action/option and later representing the value (≈ goodness – badness) of the same state/action/option (Figure 3B).
FIGURE 3
Predictions, Limitations, and Perspectives
The OTD model provides testable predictions, a few of which will be described below. First, since iMSNs are assumed to initially represent the badness and later represent the overall value as mentioned just above, a reversal of the valence in the coding of the indirect pathway is predicted to be likely to occur (Figure 3B). This is potentially in line with a result reported in a recent study, which examined the response of visually responsive GPe neurons, on the indirect pathway, to objects that were stably associated with good or bad outcomes (). These GPe neurons are largely more inhibited by the presentation of bad objects, consistent with the iMSN’s coding of badness assumed in the OpAL or OTD models. But later on, the value-coding responses were reversed, on average, so that these neurons became more inhibited, albeit slightly, by good objects (Figure 4C of ). This is potentially in line with the OTD model’s operation, although the observed reversal could instead reflect a similar reversal in the DA neuronal activity (Figure 3E bottom of ) via modulations of iMSNs’ activity by DA. The predicted reversal of the valence of value-coding in the indirect pathway in the OTD model could also explain why good-preferring neurons outnumbered bad-preferring neurons in the striatum () while dMSNs and iMSNs are roughly equinumerous, a point raised in a recent review (). The second prediction of the OTD model is that the activity of IT→dMSN/IT→iMSN pathways representing the goodness/badness not only biases current choice but also contributes to DA signal representing TD-RPE used for updating the value of previous state/action and thereby biases future choices. This is potentially in line with the recently suggested role of iMSNs in lose-switch, i.e., choice switching following bad outcomes (). Moreover, if these pathways entail differential short-term plasticity as predicted by model-fitting (), i.e., facilitation at IT→dMSN and depression at IT→iMSN, DA neurons could receive biphasic impacts, i.e., initially negative impact via the indirect pathway and subsequently positive impact via the direct pathway. Then, a recently proposed mechanism () might enable TD (higher-order) learning of both goodness and badness of one single option (action).
The OTD model also has limitations. The model’s key assumption lies in the plasticity of corticostriatal synapses depending on DA and adenosine. Regarding this topic, recent work () has shown that, in both putative dMSNs and iMSNs, repetition of “pre-post” activity paring followed by reward-predicting sensory inputs causes potentiation of response to contralateral cortical stimulation, which presumably activates IT axons (because IT cells, but not PT cells, project to the contralateral cortex/striatum; ). This is apparently not in line with any of the OTD, OpAL, or CS-TD models. However, they have also shown results indicating that blockade of adenosine A2A receptors changes potentiation in iMSNs into depression. Considering this, a conceivable possibility is that, in their experiment, electrical stimulation of IT axons resulted in richer adenosine around IT→iMSN synapses than the natural condition (i.e., to the level comparable to, or even beyond, the PT→iMSN synapses in the natural condition), leading to potentiation of IT→iMSN synapses that would naturally undergo depression. It should also be noted that the authors () described that in their protocol ”adenosine signaling is also likely to be coincident with light flash evoked dopamine signaling (p. 10)”; our assumption that phasic DA induces phasic adenosine would be consistent with this argument.
Another recent work () examined the effects of transient (rather than tonic) DA inputs, with or without tonic adenosine (agonist) inputs, on the intracellular signaling in both D1 and D2R-expressing cells (presumably dMSNs and iMSNs, respectively) by using DA uncaging. It found () that, under the presence of tonic adenosine input in D2-MSNs, transient DA input causes a reduction in cAMP, but its efficacy is similar to the efficacy of DA-dependent cAMP increase in D1-MSNs, challenging the traditional notion that D2R signaling is much more effective than D1R signaling. Moreover, at the downstream of cAMP, transient DA (with tonic adenosine) hardly decreased the level of PKA-dependent phosphorylation (). Counteraction of D2R signaling by A2AR stimulation has also been shown in previous studies with bath application of D2R agonist (; ). These could potentially support the OTD model’s impaired D2R signaling at adenosine-rich PT→iMSN synapses, although the authors of the abovementioned recent study () suggested that allosteric inhibition of D2R signaling by adenosine may not be included, different from our assumption. The same study () further indicated, through mathematical modeling based on the previous work (), that D2-MSNs would also have a different, “tone-sensing” mode, in which phasic DA reduction effectively causes PKA-dependent phosphorylation. This mode was achieved by assuming high tonic DA in their simulations, but the authors discussed that the switch between the different modes may also result from changes in adenosine. The OTD model’s adenosine-level-dependent differential plasticity between IT→iMSN and PT→iMSN synapses is potentially in line with their discussion.
Yet another important experimental result regarding adenosine is that striatum-specific knockout of A2A receptors caused selective impairment of habit formation (). This is also hard to explain by the OTD, OpAL, or CS-TD models. One possibility is that there exist several (or many) mechanisms for TD-RPE calculation and the OTD model is just one of them specifically operating in the dorsal striatum, where adenosine release evoked by stimulation was robustly detected (), while other mechanisms, e.g., those involving striosomes, operate in more ventral parts of the striatum. Existence of multiple mechanisms for TD-RPE calculation seems to be in line with the observed distributed RPE-related information in the regions projecting to DA neurons (). Then, knockout of A2A receptors might particularly impair the learning function of the dorsal striatum, which, or more specifically the dorsolateral striatum, is suggested to be crucial for habit formation (; ). In addition to the issues so far described, there are important issues that need to be addressed so as to validate, deny, or elaborate the OTD model (Box 1).
BOX 1. Outstanding issues.
Differences Between IT→iMSN Synapses and PT→iMSN Synapses
– The OTD model assumes that sustained intense PT inputs generate rich adenosine so that the local baseline adenosine concentration around PT→iMSN synapses is higher than the concentration around IT→iMSN synapses. Does such local regional variation indeed exist?
– It has been shown that PT-type axospinous synaptic terminals on MSNs are typically larger than IT-type axospinous synaptic terminals (; ), although IT axospinous terminals on iMSNs are larger than those on dMSNs (). Does the size difference between IT and PT axospinous terminals also relate to the hypothesized differential basal adenosine levels and/or plasticity inductions between IT→iMSN synapses and PT→iMSN synapses?
– Do the A2A receptors exist at/around IT→iMSN synapses and PT→iMSN synapses equally or differentially? Ultrastructural immunohistochemical study examining rat striatum () observed A2AR immunoreactivity primarily at asymmetric (putative excitatory) synapses and less frequently at symmetric (putative inhibitory) synapses, but whether A2ARs are differentially distributed among different types of excitatory synapses receiving IT, PT, and thalamic inputs remains to be seen.
DA-Dependent Adenosine Release
– DA-dependent adenosine release was indicated in the nucleus accumbens in vitro (; ). Does similar release occur also in the dorsal striatum in vivo? What are the time and spatial scales of the DA-dependent adenosine release? Looking at Fig. 5B of (), it seems that the effect of D1R agonist SKF38393 on the paired-pulse ratio of cortico-D1-MSN transmission, which was suggested to be mediated by adenosine, began to appear soon after the application of agonist, although the exact latency is difficult to read out. It thus seems not impossible that DA-dependent adenosine release occurs in a fast time scale, but this issue, as well as the spatial spread of released adenosine (in particular, whether it can affect synaptic plasticity in iMSNs), needs to be experimentally examined with high temporal/spatial resolutions.
– If adenosine release is indeed induced by phasic DA that signals TD-RPE, can the concentration of adenosine also reflect TD-RPE? Reward-related oxygen changes in the rat nucleus accumbens have been observed and suggested to be consistent with RPE-representing fMRI-BOLD signals in humans (). Given that adenosine causes vasodilation (; ), it seems conceivable that DA-dependent release of adenosine contributes to such oxygen changes, and this would be interesting to examine.
Plasticity
– Do the hypothesized differential DA and adenosine-dependent plasticity inductions at IT→iMSN and PT→iMSN synapses indeed occur? Since experimental validation would not be straightforward, it would be desired to construct mathematical models, based on previous models of the signaling cascades in MSNs (; ; ). Known properties of adenosine (; ; ), time course of phasic DA release (; ; ), and also dendritic morphology () and spines () are desired to be incorporated. Moreover, because adenosine, as well as DA, has been shown to modulate not only synaptic plasticity but also synaptic transmission (), such effects are also desired to be incorporated in future models.
– We assumed that, at IT→iMSN synapses, phasic DA representing positive TD-RPE causes LTD via D2R signaling in iMSNs. However, recent work conducting cell-type-specific removal of D2R () has shown, using high-frequency stimulation for LTD induction (), that D2R signaling in iMSNs only weakly modulates LTD in iMSNs while D2R signaling in cholinergic interneurons strongly modulates LTD in both dMSNs and iMSNs. Given this, the assumed positive TD-RPE-dependent LTD at IT→iMSN synapses might actually occur through D2R signaling not in iMSNs but in cholinergic interneurons, while the same LTD induction at PT→iMSN synapses could be masked by adenosine-dependent LTP. Instead, decay/forgetting (c.f., ; ) and/or homeostatic plasticity could operate as a functional alternative to LTD.
– What occurs when TD-RPE is negative? Negative TD-RPE-representing phasic decrease in DA would drastically shift the balance of D2R/A2AR signaling to the A2AR side so as to induce LTP. For the OTD model to hold also when TD-RPE is negative, however, it would be desired that whereas IT→iMSN synapses undergo LTP, PT→iMSN synapses do not (and rather undergo LTD). Whether and how such differentiation between IT→iMSN synapses and PT→iMSN synapses can arise remain to be examined. There is a recent finding that is possibly related to this. Specifically, impairment in LTP induction in A2R-expressing MSNs (i.e., iMSNs) was observed in Rhes (a GTPase enriched in MSNs) knockout female mice, and it was indicated to be associated to excessive phasic cAMP/PKA signaling (). In light of this result, we speculate that when TD-RPE is negative and DA phasically decreases, at IT→iMSN synapses, moderate A2AR/cAMP signaling leads to LTP induction, whereas at PT→iMSN synapses where PT inputs generate high baseline adenosine, excessive A2AR/cAMP signaling prevents LTP induction.
– At the algorithm level, what plasticity rules can ensure that the weights of IT-dMSN synapses, IT-iMSN synapses, and PT-iMSN synapses converge to the goodness, badness, and action-value, respectively?
Circuit Connectivity
– Whereas the CS-TD model assumed preferential IT→dMSN and PT→iMSN transmissions, the OTD model no longer assumes IT→dMSN preference given that the IT→iMSN connections are now assumed to encode the badness of current option. However, the situation remains elusive for PT→dMSN/iMSN connections. One possibility, extending the OTD model, is that the PT→iMSN connections and PT→dMSN connections represent the goodness and badness of the executed action, respectively.
– The OTD (or CS-TD) model assumes that activation of dMSNs and iMSNs has net positive and negative impacts on the activity of DA neurons (or DA release), respectively. Potentially in line with this, stimulation of the terminals of nucleus-accumbens D1R-MSNs led to disinhibition of DA neurons in the ventral tegmental area (; ). Also, stimulation of caudate tail caused a phasic increase of activity in a population of DA neurons, possibly through the substantia nigra pars reticulata (SNr) (). Regarding the indirect pathway, chemical excitation of rat GP (homologous to primate GPe) resulted in an elevation in neostriatal DA levels presumably disynaptically via SNr (). However, this last study indicated that the increase in DA release was due to an increase in burst firing rather than in firing rate. Whether changes in firing rate can occur remains to be seen, while extension of the OTD model to incorporate temporal coding beyond firing rate will also be an important future direction.
Consistency With In Vivo Experimental Results
– The OTD (or CS-TD) model assumes that PT neurons can sustain activity via strong facilitatory recurrent excitation (). This point has been challenged by a recent study () showing that extratelencephalic (ET) pyramidal cells, which would largely overlap with PT neurons, exhibit post-spike suppression (i.e., suppression of the generation of a next spike with a short duration) in vivo and arguing that it would interrupt sustained activity. Although this is an important argument, if successive PT→PT inputs with short durations cause synaptic short-term depression, post-spike suppression could actually be beneficial for its prevention. Also related to this point, recent studies have shown that sustained activity is maintained through cortico-thalamic interactions (; ; ). Because PT neurons, but not IT neurons, innervate thalamus, PT neurons may sustain activity through the interaction with thalamus.
– It has been shown that dMSNs and iMSNs are concurrently activated during action initiation (). Such concurrent activation can be in line with the OpAL or OTD model, but seems difficult to explain by the CS-TD model. The OTD (or CS-TD) model, however, also predicts sustained activity of iMSNs representing previous value, which was not shown in the experiments (). This potential discrepancy could be resolved in multiple ways. First, if goodness (benefit) and badness (cost) of an action are nearly comparable, overall value (≈ benefit – cost) is expected to be small and can be difficult to detect. Second, in the OTD model, representation of goodness and badness is transiently done for all the action candidates/options (A1 and A2 at ti in the case of Figure 3A) whereas sustained representation of previous value is done only for the single action that was actually chosen/executed (A1 at ti+1 in Figure 3A), and therefore the latter can be more difficult to detect than the former. Third, the goodness/badness representation and the previous-value representation could be done with different firing patterns, in particular, bursty and nonbursty firings, respectively. If so, the former can generate larger calcium transients that are easier to detect. These explanations are, however, all speculations, and direct experimental test of whether previous value is represented in iMSNs is desired.
Statements
Author contributions
KM conceived of the hypothesis, and elaborated it through discussion with YK.
Funding
This work was supported by Grant-in-Aid for Scientific Research Nos. 15H05876 and 17H06311 of The Ministry of Education, Culture, Sports, Science and Technology in Japan to KM and YK, respectively.
Acknowledgments
The authors thank Dr. Arvind Kumar for his comments on the draft. The original version of this work has been deposited as a preprint in PsyArXiv (https://psyarxiv.com/5y7su/).
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
References
1
AugustinS. M.ChanceyJ. H.LovingerD. M. (2018). Dual dopaminergic regulation of corticostriatal plasticity by cholinergic interneurons and indirect pathway medium spiny neurons.Cell Rep.242883–2893. 10.1016/j.celrep.2018.08.042
2
AzdadK.GallD.WoodsA. S.LedentC.FerréS.SchiffmannS. N. (2009). Dopamine D2 and adenosine A2A receptors regulate NMDA-mediated excitation in accumbens neurons through A2A-D2 receptor heteromerization.Neuropsychopharmacology34972–986. 10.1038/npp.2008.144
3
BallionB.MalletN.BézardE.LanciegoJ. L.GononF. (2008). Intratelencephalic corticostriatal neurons equally excite striatonigral and striatopallidal neurons and their discharge activity is selectively reduced in experimental parkinsonism.Eur. J. Neurosci.272313–2321. 10.1111/j.1460-9568.2008.06192.x
4
BlackwellK. T.SalinasA. G.TewatiaP.EnglishB.Hellgren KotaleskiJ.LovingerD. M. (2018). Molecular mechanisms underlying striatal synaptic plasticity: relevance to chronic alcohol consumption and seeking.Eur. J. Neurosci.10.1111/ejn.13919 [Epub ahead of print].
5
BocklischC.PascoliV.WongJ. C.HouseD. R.YvonC.de RooM.et al (2013). Cocaine disinhibits dopamine neurons by potentiation of GABA transmission in the ventral tegmental area.Science3411521–1525. 10.1126/science.1237059
6
BogaczR. (2017). Theory of reinforcement learning and motivation in the basal ganglia.bioRxiv10.1101/174524
7
BolkanS. S.StujenskeJ. M.ParnaudeauS.SpellmanT. J.RauffenbartC.AbbasA. I.et al (2017). Thalamic projections sustain prefrontal activity during working memory maintenance.Nat. Neurosci.20987–996. 10.1038/nn.4568
8
BurtonA. C.NakamuraK.RoeschM. R. (2015). From ventral-medial to dorsal-lateral striatum: neural correlates of reward-guided decision-making.Neurobiol. Learn. Mem.11751–59. 10.1016/j.nlm.2014.05.003
9
CalabresiP.MajR.PisaniA.MercuriN. B.BernardiG. (1992). Long-term synaptic depression in the striatum: physiological and pharmacological characterization.J. Neurosci.124224–4233. 10.1523/JNEUROSCI.12-11-04224.1992
10
CollinsA. G.FrankM. J. (2014). Opponent actor learning (OpAL): modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive.Psychol. Rev.121337–366. 10.1037/a0037015
11
CowanR. L.WilsonC. J. (1994). Spontaneous firing patterns and axonal projections of single corticostriatal neurons in the rat medial agranular cortex.J. Neurophysiol.7117–32. 10.1152/jn.1994.71.1.17
12
CuiG.JunS. B.JinX.PhamM. D.VogelS. S.LovingerD. M.et al (2013). Concurrent activation of striatal direct and indirect pathways during action initiation.Nature494238–242. 10.1038/nature11846
13
DayJ. J.RoitmanM. F.WightmanR. M.CarelliR. M. (2007). Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens.Nat. Neurosci.101020–1028. 10.1038/nn1923
14
DengY.LanciegoJ. L.Kerkerian-Le GoffL.CoulonP.SalinP.KachidianP.et al (2015). Differential organization of cortical inputs to striatal projection neurons of the matrix compartment in rats.Front. Syst. Neurosci.9:51. 10.3389/fnsys.2015.00051
15
EverittB. J.RobbinsT. W. (2005). Neural systems of reinforcement for drug addiction: from actions to habits to compulsion.Nat. Neurosci.81481–1489. 10.1038/nn1579
16
FerréS.BonaventuraJ.ZhuW.Hatcher-SolisC.TauraJ.QuirozC.et al (2018). Essential control of the function of the striatopallidal neuron by pre-coupled complexes of adenosine A.Front. Pharmacol.9:243. 10.3389/fphar.2018.00243
17
FerreS.von EulerG.JohanssonB.FredholmB. B.FuxeK. (1991). Stimulation of high-affinity adenosine A2 receptors decreases the affinity of dopamine D2 receptors in rat striatal membranes.Proc. Natl. Acad. Sci. U.S.A.887238–7241. 10.1073/pnas.88.16.7238
18
FisherS. D.RobertsonP. B.BlackM. J.RedgraveP.SagarM. A.AbrahamW. C.et al (2017). Reinforcement determines the timing dependence of corticostriatal synaptic plasticity in vivo.Nat. Commun.8:334. 10.1038/s41467-017-00394-x
19
FrancoisJ.ConwayM. W.LowryJ. P.TricklebankM. D.GilmourG. (2012). Changes in reward-related signals in the rat nucleus accumbens measured by in vivo oxygen amperometry are consistent with fMRI BOLD responses in man.Neuroimage602169–2181. 10.1016/j.neuroimage.2012.02.024
20
FrankM. J. (2005). Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism.J. Cogn. Neurosci.1751–72. 10.1162/0898929052880093
21
FrankM. J.SeebergerL. C.O’reillyR. C. (2004). By carrot or by stick: cognitive reinforcement learning in parkinsonism.Science3061940–1943. 10.1126/science.1102941
22
GerfenC. R.SurmeierD. J. (2011). Modulation of striatal projection systems by dopamine.Annu. Rev. Neurosci.34441–466. 10.1146/annurev-neuro-061010-113641
23
GhiglieriV.NapolitanoF.PelosiB.SchepisiC.MigliariniS.Di MaioA.et al (2015). Rhes influences striatal cAMP/PKA-dependent signaling and synaptic plasticity in a gender-sensitive fashion.Sci. Rep.5:10933. 10.1038/srep10933
24
GuoZ. V.InagakiH. K.DaieK.DruckmannS.GerfenC. R.SvobodaK. (2017). Maintenance of persistent activity in a frontal thalamocortical loop.Nature545181–186. 10.1038/nature22324
25
HarveyJ.LaceyM. G. (1997). A postsynaptic interaction between dopamine D1 and NMDA receptors promotes presynaptic inhibition in the rat nucleus accumbens via adenosine release.J. Neurosci.175271–5280. 10.1523/JNEUROSCI.17-14-05271.1997
26
HettingerB. D.LeeA.LindenJ.RosinD. L. (2001). Ultrastructural localization of adenosine A2A receptors suggests multiple cellular sites for modulation of GABAergic neurons in rat striatum.J. Comp. Neurol.431331–346. 10.1002/1096-9861(20010312)431:3<331::AID-CNE1074>3.0.CO;2-W
27
HigleyM. J.SabatiniB. L. (2010). Competitive regulation of synaptic Ca2+ influx by D2 dopamine and A2A adenosine receptors.Nat. Neurosci.13958–966. 10.1038/nn.2592
28
HikosakaO.KimH. F.AmitaH.YasudaM.IsodaM.TachibanaY.et al (2018). Direct and indirect pathways for choosing objects and actions.Eur. J. Neurosci.10.1111/ejn.13876 [Epub ahead of print].
29
HillionJ.CanalsM.TorvinenM.CasadoV.ScottR.TerasmaaA.et al (2002). Coaggregation, cointernalization, and codesensitization of adenosine A2A receptors and dopamine D2 receptors.J. Biol. Chem.27718091–18097. 10.1074/jbc.M107731200
30
KatoA.MoritaK. (2016). Forgetting in reinforcement learning links sustained dopamine signals to motivation.PLoS Comput. Biol.12:e1005145. 10.1371/journal.pcbi.1005145
31
KeiflinR.JanakP. H. (2015). Dopamine prediction errors in reward learning and addiction: from theory to neural circuitry.Neuron88247–263. 10.1016/j.neuron.2015.08.037
32
KimH. F.AmitaH.HikosakaO. (2017). Indirect pathway of caudal basal ganglia for rejection of valueless visual objects.Neuron94920–930.e3. 10.1016/j.neuron.2017.04.033
33
KimH. F.GhazizadehA.HikosakaO. (2015). Dopamine neurons encoding long-term memory of object value for habitual behavior.Cell1631165–1175. 10.1016/j.cell.2015.10.063
34
KimH. F.HikosakaO. (2013). Distinct basal ganglia circuits controlling behaviors guided by flexible and stable values.Neuron791001–1010. 10.1016/j.neuron.2013.06.044
35
KravitzA. V.TyeL. D.KreitzerA. C. (2012). Distinct roles for direct and indirect pathway striatal neurons in reinforcement.Nat. Neurosci.15816–818. 10.1038/nn.3100
36
KressG. J.YamawakiN.WokosinD. L.WickershamI. R.ShepherdG. M.SurmeierD. J. (2013). Convergent cortical innervation of striatal projection neurons.Nat. Neurosci.16665–667. 10.1038/nn.3397
37
KullB.FerréS.ArslanG.SvenningssonP.FuxeK.OwmanC.et al (1999). Reciprocal interactions between adenosine A2A and dopamine D2 receptors in Chinese hamster ovary cells co-transfected with the two receptors.Biochem. Pharmacol.581035–1045. 10.1016/S0006-2952(99)00184-7
38
LeeC. R.AbercrombieE. D.TepperJ. M. (2004). Pallidal control of substantia nigra dopaminergic neuron firing pattern and its relation to extracellular neostriatal dopamine levels.Neuroscience129481–489. 10.1016/j.neuroscience.2004.07.034
39
LeiW.JiaoY.Del MarN.ReinerA. (2004). Evidence for differential cortical input to direct pathway versus indirect pathway striatal projection neurons in rats.J. Neurosci.248289–8299. 10.1523/JNEUROSCI.1990-04.2004
40
LindroosR.DorstM. C.DuK.FilipovićM.KellerD.KetzefM.et al (2018). Basal ganglia neuromodulation over multiple temporal and structural scales-simulations of direct pathway msns investigate the fast onset of dopaminergic effects and predict the role of Kv4.2.Front. Neural Circuits12:3. 10.3389/fncir.2018.00003
41
LindskogM.KimM.WikströmM. A.BlackwellK. T.KotaleskiJ. H. (2006). Transient calcium and dopamine increase PKA activity and DARPP-32 phosphorylation.PLoS Comput. Biol.2:e119. 10.1371/journal.pcbi.0020119
42
McClureS. M.BernsG. S.MontagueP. R. (2003). Temporal prediction errors in a passive learning task activate human striatum.Neuron38339–346. 10.1016/S0896-6273(03)00154-5
43
MontagueP. R.DayanP.SejnowskiT. J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning.J. Neurosci.161936–1947. 10.1523/JNEUROSCI.16-05-01936.1996
44
MorishimaM.KawaguchiY. (2006). Recurrent connection patterns of corticostriatal pyramidal cells in frontal cortex.J. Neurosci.264394–4405. 10.1523/JNEUROSCI.0252-06.2006
45
MorishimaM.MoritaK.KubotaY.KawaguchiY. (2011). Highly differentiated projection-specific cortical subnetworks.J. Neurosci.3110380–10391. 10.1523/JNEUROSCI.0772-11.2011
46
MoritaK. (2014). Differential cortical activation of the striatal direct and indirect pathway cells: reconciling the anatomical and optogenetic results by using a computational method.J. Neurophysiol.112120–146. 10.1152/jn.00625.2013
47
MoritaK.KatoA. (2014). Striatal dopamine ramping may indicate flexible reinforcement learning with forgetting in the cortico-basal ganglia circuits.Front. Neural Circuits8:36. 10.3389/fncir.2014.00036
48
MoritaK.KawaguchiY. (2015). Computing reward-prediction error: an integrated account of cortical timing and basal-ganglia pathways for appetitive and aversive learning.Eur. J. Neurosci.422003–2021. 10.1111/ejn.12994
49
MoritaK.MorishimaM.SakaiK.KawaguchiY. (2012). Reinforcement learning: computing the temporal difference of values via distinct corticostriatal pathways.Trends Neurosci.35457–467. 10.1016/j.tins.2012.04.009
50
MoritaK.MorishimaM.SakaiK.KawaguchiY. (2013). Dopaminergic control of motivation and reinforcement learning: a closed-circuit account for reward-oriented behavior.J. Neurosci.338866–8890. 10.1523/JNEUROSCI.4614-12.2013
51
NairA. G.BhallaU. S.Hellgren KotaleskiJ. (2016). Role of DARPP-32 and ARPP-21 in the emergence of temporal constraints on striatal calcium and dopamine integration.PLoS Comput. Biol.12:e1005080. 10.1371/journal.pcbi.1005080
52
NairA. G.Gutierrez-ArenasO.ErikssonO.VincentP.Hellgren KotaleskiJ. (2015). Sensing positive versus negative reward signals through Adenylyl Cyclase-Coupled GPCRs in direct and indirect pathway striatal medium spiny neurons.J. Neurosci.3514017–14030. 10.1523/JNEUROSCI.0730-15.2015
53
NakanoT.DoiT.YoshimotoJ.DoyaK. (2010). A kinetic model of dopamine- and calcium-dependent striatal synaptic plasticity.PLoS Comput. Biol.6:e1000670. 10.1371/journal.pcbi.1000670
54
NavarroG.AguinagaD.MorenoE.HradskyJ.ReddyP. P.CortésA.et al (2014). Intracellular calcium levels determine differential modulation of allosteric interactions within G protein-coupled receptor heteromers.Chem. Biol.211546–1556. 10.1016/j.chembiol.2014.10.004
55
NivY.SchoenbaumG. (2008). Dialogues on prediction errors.Trends Cogn. Sci.12265–272. 10.1016/j.tics.2008.03.006
56
NonomuraS.NishizawaK.SakaiY.KawaguchiY.KatoS.UchigashimaM.et al (2018). Monitoring and updating of action selection for goal-directed behavior through the striatal direct and indirect pathways.Neuron991302–1314. 10.1016/j.neuron.2018.08.002
57
O’DohertyJ. P.DayanP.FristonK.CritchleyH.DolanR. J. (2003). Temporal difference models and reward-related learning in the human brain.Neuron38329–337. 10.1016/S0896-6273(03)00169-7
58
PajskiM. L.VentonB. J. (2010). Adenosine release evoked by short electrical stimulations in striatal brain slices is primarily activity dependent.ACS Chem. Neurosci.1775–787. 10.1021/cn100037d
59
PhillisJ. W. (1989). Adenosine in the control of the cerebral circulation.Cerebrovasc. Brain Metab. Rev.126–54.
60
ReinerA.HartN. M.LeiW.DengY. (2010). Corticostriatal projection neurons - dichotomous types and dichotomous functions.Front. Neuroanat.4:142. 10.3389/fnana.2010.00142
61
ReinerA.JiaoY.Del MarN.LaverghettaA. V.LeiW. L. (2003). Differential morphology of pyramidal tract-type and intratelencephalically projecting-type corticostriatal neurons and their intrastriatal terminals in rats.J. Comp. Neurol.457420–440. 10.1002/cne.10541
62
ReynoldsJ. N.HylandB. I.WickensJ. R. (2001). A cellular mechanism of reward-related learning.Nature41367–70. 10.1038/35092560
63
SaikiA.SakaiY.FukaboriR.SomaS.YoshidaJ.KawabataM.et al (2018). In vivo spiking dynamics of intra- and extratelencephalic projection neurons in rat motor cortex.Cereb. Cortex281024–1038. 10.1093/cercor/bhx012
64
SalamoneJ. D.CorreaM. (2002). Motivational views of reinforcement: implications for understanding the behavioral functions of nucleus accumbens dopamine.Behav. Brain Res.1373–25. 10.1016/S0166-4328(02)00282-6
65
SchiffmannS. N.FisoneG.MorescoR.CunhaR. A.FerréS. (2007). Adenosine A2A receptors and basal ganglia physiology.Prog. Neurobiol.83277–292. 10.1016/j.pneurobio.2007.05.001
66
SchmittL. I.WimmerR. D.NakajimaM.HappM.MofakhamS.HalassaM. M. (2017). Thalamic amplification of cortical connectivity sustains attentional control.Nature545219–223. 10.1038/nature22073
67
SchultzW.DayanP.MontagueP. R. (1997). A neural substrate of prediction and reward.Science2751593–1599. 10.1126/science.275.5306.1593
68
ShenW.FlajoletM.GreengardP.SurmeierD. J. (2008). Dichotomous dopaminergic control of striatal synaptic plasticity.Science321848–851. 10.1126/science.1160575
69
ShepherdG. M. (2013). Corticostriatal connectivity and its role in disease.Nat. Rev. Neurosci.14278–291. 10.1038/nrn3469
70
ShinJ. H.KimD.JungM. W. (2018). Differential coding of reward and movement information in the dorsomedial striatal direct and indirect pathways.Nat. Commun.9:404. 10.1038/s41467-017-02817-1
71
ShindouT.ArbuthnottG. W.WickensJ. R. (2008). Actions of adenosine A 2A receptors on synaptic connections of spiny projection neurons in the neostriatal inhibitory network.J. Neurophysiol.991884–1889. 10.1152/jn.01259.2007
72
TepperJ. M.LeeC. R. (2007). GABAergic control of substantia nigra dopaminergic neurons.Prog. Brain Res.160189–208. 10.1016/S0079-6123(06)60011-3
73
TepperJ. M.MartinL. P.AndersonD. R. (1995). GABAA receptor-mediated inhibition of rat substantia nigra dopaminergic neurons by pars reticulata projection neurons.J. Neurosci.153092–3103. 10.1523/JNEUROSCI.15-04-03092.1995
74
TianJ.HuangR.CohenJ. Y.OsakadaF.KobakD.MachensC. K.et al (2016). Distributed and mixed information in monosynaptic inputs to dopamine neurons.Neuron911374–1389. 10.1016/j.neuron.2016.08.018
75
WallM.DaleN. (2008). Activity-dependent release of adenosine: a critical re-evaluation of mechanism.Curr. Neuropharmacol.6329–337. 10.2174/157015908787386087
76
WangW.DeverD.LoweJ.StoreyG. P.BhansaliA.EckE. K.et al (2012). Regulation of prefrontal excitatory neurotransmission by dopamine in the nucleus accumbens core.J. Physiol.5903743–3769. 10.1113/jphysiol.2012.235200
77
WangY.VentonB. J. (2017). Correlation of transient adenosine release and oxygen changes in the caudate-putamen.J. Neurochem.14013–23. 10.1111/jnc.13705
78
YagishitaS.Hayashi-TakagiA.Ellis-DaviesG. C.UrakuboH.IshiiS.KasaiH. (2014). A critical time window for dopamine actions on the structural plasticity of dendritic spines.Science3451616–1620. 10.1126/science.1255514
79
YapoC.NairA. G.ClementL.CastroL. R.Hellgren KotaleskiJ.VincentP. (2017). Detection of phasic dopamine by D1 and D2 striatal medium spiny neurons.J. Physiol.5957451–7475. 10.1113/JP274475
80
YuC.GuptaJ.ChenJ. F.YinH. H. (2009). Genetic deletion of A2A adenosine receptors in the striatum selectively impairs habit formation.J. Neurosci.2915100–15103. 10.1523/JNEUROSCI.4215-09.2009
Summary
Keywords
reinforcement learning, reward prediction error, cost, basal ganglia, dopamine, adenosine
Citation
Morita K and Kawaguchi Y (2019) A Dual Role Hypothesis of the Cortico-Basal-Ganglia Pathways: Opponency and Temporal Difference Through Dopamine and Adenosine. Front. Neural Circuits 12:111. doi: 10.3389/fncir.2018.00111
Received
31 August 2018
Accepted
29 November 2018
Published
07 January 2019
Volume
12 - 2018
Edited by
Anita Disney, Vanderbilt University, United States
Reviewed by
Veronica Ghiglieri, University of Perugia, Italy; Jeanette Hellgren Kotaleski, Karolinska Institutet (KI), Sweden
Updates
Copyright
© 2019 Morita and Kawaguchi.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Kenji Morita, morita@p.u-tokyo.ac.jp
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.