Dopamine and Effort-Based Decision Making

Motivational theories of choice focus on the influence of goal values and strength of reinforcement to explain behavior. By contrast relatively little is known concerning how the cost of an action, such as effort expended, contributes to a decision to act. Effort-based decision making addresses how we make an action choice based on an integration of action and goal values. Here we review behavioral and neurobiological data regarding the representation of effort as action cost, and how this impacts on decision making. Although organisms expend effort to obtain a desired reward there is a striking sensitivity to the amount of effort required, such that the net preference for an action decreases as effort cost increases. We discuss the contribution of the neurotransmitter dopamine (DA) toward overcoming response costs and in enhancing an animal's motivation toward effortful actions. We also consider the contribution of brain structures, including the basal ganglia and anterior cingulate cortex, in the internal generation of action involving a translation of reward expectation into effortful action.

2005). Thus, ducks choose between walking or flying depending on optimal solution of net gain between energy requirements in walking or flying and the food gained (Bautista et al., 2001).
In what follows we discuss a literature that has endeavored to understand the neural mechanisms of effort and reward integration, including the involvement of dopamine (DA) in effort-based behavior. This literature points to the basal ganglia, particularly dorsal and ventral striatum, and anterior cingulate cortex (ACC) as the principal substrates in both representing and integrating effort and action implementation. Finally, we suggest that pathologies in effort expenditure, the paradigmatic instance being the clinical phenomenon of apathy, can be characterized behaviorally as impairment in representing action-outcome association and neurally as a disruption of core cortico-subcortical circuitry.

The regulaTory role of dopamine in efforT
Over the past three decades, theories concerning the role of midbrain DA on behavior have changed dramatically. The hedonic hypothesis of DA (Wise, 1980) is now challenged by empirical evidence revealing that global DA depletion (including within the accumbens, a major recipient for DA) does not impair hedonic ('liking') responses to primary rewards such as orofacial reactions, the preference for sucrose over water, or discrimination among reinforcement (Berridge et al., 1989;Cousins and Salamone, 1994;Cannon and Palmiter, 2003). On the other hand the same lesions profoundly impair performance of instrumental tasks necessary to obtain rewards that are liked (Berridge and Robinson, 1998). These observations have led to a formulation that the contribution of DA includes an effect on motivated behaviors toward desired goals, a concept referred to as "wanting" (Berridge and Robinson, 1998). "Wanting" ( Table 1) can be expressed in simple instrumental responses, such as button or lever presses or in a more expanded form of behaviors which require an agent to overcome action costs. As demonstrated unequivocally by Salamone and colleagues (Salamone and Correa, 2002;Salamone "So she was considering in her own mind […], whether the pleasure of making a daisy-chain would be worth the trouble of getting up and picking the daisies…" Alice in "Alice's Adventures in Wonderland, " Carroll, (1865, p. 11) Effort is commonly experienced as a burden, and yet we readily expend effort to reach a desired goal. Many classical and contemporary studies have assessed the effect of effort expenditure on response rates, by varying experimental parameters such as the weight of a lever press, the height of a barrier to scale, or the number of handle turns needed to generate a unit of reward (Lewis, 1964;Collier and Levitsky, 1968;Kanarek and Collier, 1973;Collier et al., 1975;Walton et al., 2006;Kool et al., 2010). There is general agreement that animals, including humans, are disposed to avoid effortful actions. It is paradoxical then that effort is not always treated as a nuisance, and there are instances where its expenditure enhances outcome value as observed in food palatability (Johnson and Gallagher, 2010), likability (Aronson, 1961), and indeed the propensity to choose a previously effortful option (Friedrich and Zentall, 2004). What is most surprising is the observation that effort often biases future choice toward effortful actions (Eisenberger et al., 1989).
Laboratory results show that if reward magnitude is held constant then high-effort tasks tend to be avoided (Kool et al., 2010). Yet, in daily life most organisms seem superficially indifferent to the varying costs of action and readily choose challenging tasks to achieve a desired goal (Duckworth et al., 2007). Such observations point to the presence of a mechanism that integrates effort costs with benefits in order to implement desired actions (see Floresco et al., 2008b for review on various cost-benefit analyses and Salamone et al., 2007 for an earlier review on dopamine and effort). This perspective has been addressed by optimal foraging theory. It is known that animals will strive to maximize gain whilst minimizing energy expenditure (Bautista et al., 2001;Stevens et al., reinforcement learning (Sutton and Barto, 1998). Reinforcement learning offers ways to formalize the process of reward maximization through learned choices and has a close resonance with the neuroscience of decision making (Montague et al., 1996;Montague and Berns, 2002;Niv et al., 2005;Daw and Doya, 2006). In particular, phasic responses of macaque and rodent midbrain dopaminergic neurons to rewards, and reward-associated stimuli, are akin to a reward prediction error signal within reinforcement learning algorithms, responding to unexpected rewards and stimuli that predict rewards but not to fully predicted rewards (Schultz et al., 1997;Bayer and Glimcher, 2005;Morris et al., 2006;Roesch et al., 2007). Moreover, functional magnetic resonance imaging (fMRI) studies report that the BOLD signal in the striatum, a major target of the dopaminergic system, correlates with the prediction error signals derived from fitting subject's behavior to a reinforcement learning model (Mcclure et al., 2003;O'Doherty et al., 2003O'Doherty et al., , 2004. In support of such a role for dopamine in reinforcement learning processes, stimulation of the substantia nigra (using intracranial self-stimulation paradigm) has been shown to induce a potentiation within corticostriatal synapses at the site where nigral output cells terminate, with these effects in turn being blocked by systemic administration of a DA D1/D5 antagonist (Reynolds et al., 2001). Importantly, the magnitude of potentiation is negatively correlated with the time taken by an animal to learn the self-stimulation paradigm.
Dopamine is also proposed to signal stimulus salience, as opposed to reward prediction error (Redgrave et al., 1999). Redgrave et al. (1999) have discussed the stereotypical latency and duration of phasic bursts of nigral dopaminergic neurons, as well as the connectivity Salamone, 2007), accumbens DA depletion disrupts instrumental responding if the responses require an energetic cost such as climbing a barrier , but leaves reward preference intact when effort is minimal. This has led to an hypothesis that DA plays a role in overcoming "costs" (Salamone and Correa, 2002;Phillips et al., 2007).

alTernaTive views on The role of dopamine in decision making
There are several alternative views to dopamine which we summarize in Figure 1. Aside from a role in the expression of motivated behavior, DA is also involved in its acquisition through learning. An influential view on how dopamine influences behavior comes from Clockwise from top left corner: a wealth of evidence shows that DA acts to invigorate an agent's effortful action, integrating ideas about overcoming effort costs, agents' choice for high-effort options as well as modeling work on vigor. Another influential view pertains to DA acting as a signal for a prediction in reward as exemplified by its role in reinforcement learning. An alternative view interprets this signal as a saliency signal which allows agents to implement associative learning. Finally, dopamine may facilitate flexible switching and re-engagement in relation to reward-driven behavior. The summarized perspectives are not mutually exclusive, nor do they represent the entire literature on dopamine, but are useful in understanding what determines motivated behavior. Wanting often entails actions such as lever pressing in rodents or non-human primates to obtain a goal object. One influential hypothesis regarding dopamine function highlights a role in mediating wanting, but not liking.

Cost-benefit integration
The process of deriving a value of an action based on a combination between potential utility in attaining and disutility incurred in so doing.
There is evidence that this type of integration takes place when one is judging whether an action is worth taking, although the mechanisms by which costs and benefits are integrated remain unclear.
Apathy A mental or behavioral state devoid of motivation with a core feature of lack of self-initiated actions.

Invigoration
To vitalize or increase strength. One hypothesis regarding the role of dopamine formalizes its role as facilitating motivated behavior by invigorating an organism when faced with increasing demands of effort. This is supported by studies that highlight the effects of a dopaminergic manipulation on effort expenditure. of cortico-subcortical networks for cost-benefit decision making as highlighted in depletion effects Aberman and Salamone, 1999;Salamone et al., 2007). In these experiments, rats are trained on a T-maze that requires choosing between two actions; one yields high reward (four pellets of food) but requires higher effort (climb a 30-cm barrier or higher lever press fixed-ratio schedule), the other yields low reward (two pellets of food) but requires less effort. DA depletion in the NAc changes a rat's preference away from the high-effort/high reward option, but does not impact on reward preference when it is readily available, nor does it alter response selection based on reward alone Salamone et al., 1994). This finding has been replicated in other laboratories with a variety of depletion methods (Denk et al., 2005;Floresco et al., 2008a), where some studies point to a stronger effect from depletion in the core as opposed to shell of the NAc (Ghods-sharifi and Floresco, 2010;Nicola, 2010). The impact of DA elevation on effort is much less conclusive. Enhancing DA function is commonly realized through injection of amphetamine, an indirect DA agonist that increases synaptic DA levels (but also that of other neuromodulators). Floresco et al. (2008a) revealed a dose-dependent effect of amphetamine such that low-doses of amphetamine increased effortful choice, but high dose decreased it. This dose-dependent effect is difficult to interpret. First of all, it is unclear what the precise effect of a high dose of amphetamine is on DA concentration level since amphetamine also results in increased extracellular serotonin and noradrenaline (Salomon et al., 2006). Moreover, it is unclear whether a low dose of amphetamine acts by increasing the value of the reward, decreasing the cost of an action, modifying the integration of both, or by affecting other components of behavioral control such as impulsivity (see Pine et al., 2010 in relation to the latter). Nevertheless, the data suggest that increasing DA levels per se does not invariably enhance preference for a high reward/high-effort option, ruling out a simple monotonic relationship between DA and effort.
Another study showed an interactive effect of haloperidol, a DA receptor blocker, and amphetamine. While an injection of haloperidol 48 h before treatment, followed by saline 10 min before test, significantly reduced preference for high reward/high-effort arm, giving the same haloperidol injection followed by amphetamine 10 min before testing blunted the effect of haloperidol, and completely recovered preference for high reward/high-effort arm (Bardgett et al., 2009). Evidence therefore points to amphetamine's ability to overcome the effects of DA blockade induced by haloperidol. However, as indicated amphetamine also increases the levels of serotonin and noradrenaline as well as DA levels, making it difficult to completely outrule a possibility that the effect might relate to elevations of other amines aside from DA. We also know that amphetamine increases locomotor activity (Salomon et al., 2006) and it is impossible to dismiss the possibility that a recovered preference for the high-effort arm found might be due to enhanced locomotion.
Recent advances in neurochemical assay techniques, particularly in vivo fast scan cyclic voltammetry, allow detection of DA transients with a temporal resolution of milliseconds in awake behaving animals (Robinson et al., 2003;Roitman et al., 2004). Gan et al. (2010) performed in vivo voltammetry while rats selected between two options in a task where there was an independent manipulation between nigral dopaminergic neurons and sensory subcortical structures such as the superior colliculus. They argue that activity of dopamine neurons can be interpreted as reporting biological salient events, either due to novelty or unpredictability. From this perspective, salient events generate short-latency bursts of dopaminergic activity that reinforce actions occurring immediately preceding the unpredictable event. This signal allows an agent to learn that an action caused the salient event (see Redgrave and Gurney, 2006 for an elegant discussion on signal transmission in tecto-nigral and corticosubcortical pathways for learning of action-outcome associations). According to this view, unpredictable rewarding events are just one among many exemplars of a salient event.
Finally, Nicola (2010) recently suggested that DA is required to flexibly initiate goal-directed instrumental responses. This view is based on observations that the effects of DA depletion in the rat nucleus accumbens (NAc) are dependent on inter-trial interval, such that when this is short instrumental responses are not affected but disruption increases as a function of increasing time between responses. Detailed behavioral analysis shows these effects of time are explained by the fact that as the duration between responses increases animals tend to engage with behaviors different from the required instrumental response, with depleted animals unable to flexibly reinitiate execution of the instrumental responses. On the other hand, depleted rats can perform complex sequences of behavior in situations where these are not interrupted. Such findings suggest that rather than impairing lever presses, dopamine depletion disrupts an animal's ability to flexibly re-engage with a task after engaging in a task-irrelevant behavior.

exTending reinforcemenT learning To accounT for dopamine involvemenT in efforT
The most compelling attempt to link the known role of DA in reward learning to effort is that of Niv et al. (2007) who have developed a model that specifies the vigor (defined as the inverse latency) of action. This model realizes a trade-off between two costs: one stemming from the harder work assumed necessary to emit faster actions and the other from the opportunity cost inherent in acting more slowly. The latter arises out of the ensuing delay to the next, and indeed to all subsequent, rewards. Niv et al. (2007) suggested that agents should choose latencies (and actions) to maximize the rate of accumulated reward per unit time, and showed that the resulting optimal latencies would be inversely proportional to the average reward rate. Based on a review of experimental evidence, Niv et al. (2007) proposed that tonic levels of DA report the average rate of reward, thus tying together prediction error (Montague et al., 1996;Schultz et al., 1997;Mcclure et al., 2003), incentive salience (Berridge and Robinson, 1998), and invigoration (Salamone and Correa, 2002) theories of DA. As defined by Niv et al. (2007), vigor can be thought of as a specific manifestation of effort expenditure in the time domain. Future work might usefully extend this temporal computational concept of vigor into other aspects of physical effort.

dopamine and iTs role in overcoming efforT cosTs
Considerable evidence points to midbrain DA depletion discouraging animals from choosing effortful actions Aberman and Salamone, 1999;Denk et al., 2005;Phillips et al., 2007). A series of experiments in rats has pointed to the crucial role BG output nuclei (Alexander and Crutcher, 1990;Frank et al., 2004;Frank, 2005;Frank and Fossella, 2011). Under basal conditions, the output nuclei of the BG have a high level of firing and maintain thalamic inhibition that serve to dampen activity in corticostriatal loops (Frank, 2005). The distinct connectivity of direct and indirect pathways (see Figure 2) results in opposite effects: the direct pathway promotes inhibition of BG output nuclei and release of inhibition in thalamic activity whereas the indirect pathway promotes excitation of BG output structure and drives thalamic inhibition.
Neurons within the direct pathway express D1 dopamine receptors, promoting cell activity and long-term potentiation (LTP), whereas neurons of the indirect pathway mainly express D2 dopamine receptors which promote cell inhibition (when activated) and long-term depression (LTD). This scheme means that an increase in striatal DA tends to promote activity in the direct pathway and inhibition of the indirect pathway, resulting in net disinhibition of the thalamo-cortical connections and the generation of behavioral output. On the other hand a decrease in striatal DA promotes excitation of the indirect pathway resulting in dampening of activity in thalamo-cortical loops and behavioral inhibition. A segregation between striatal neurons that form the direct and indirect pathway means increases in dopamine potentiate the direct pathway while inhibiting the indirect pathway, thus facilitating thalamic output and corticostriatal flow. On the other hand a decrease in dopamine promotes inhibition of the direct pathway and an excitation of the indirect pathway, dampening thalamic output and shutting down corticostriatal information flow. Finally, to complete a picture of the BG circuitry we need to include reference to an hyperdirect pathway from inferior frontal cortex to the STN, a circuit that sends excitatory projections to output nuclei of the BG (Frank, 2006). of the amount of reward and effort. These authors found that rats had the expected preference for higher magnitude of reward when costs were held constant and higher preference for options which require less effort when reward magnitude was constant. This study also included a separate set of trials which offered rats either option, while measuring the amount of DA released in the core of the NAc elicited by cues predicting reward and effort. By having this set of non-choice trials the authors ensured that the dopaminergic response was not confounded by the presentation of the second option. Whereas DA release reliably reflected the magnitude of the reward available in these trials, the amount of effort required to obtain the goal was not coded in the amount of DA released in the core of the NAc. This lack of evidence for an integration between costs and benefits in the dopaminergic signal was surprising given the extent of prior evidence (discussed above) pointing to a link between DA and the expenditure of effort in overcoming costs.
Overall, there is evidence that DA is required to overcome costs when high levels of effort are necessary to obtain a desired goal. However, the precise mechanism by which DA supports a cost-overcoming function, and how effort is integrated into a dopaminergic modulation of the striatum and prefrontal cortex, is much less clear. In addition, dopamine depleted animals can engage in high-effort responding given a limited, inflexible set of possible responses but exhibit difficulties and are slower in re-engaging with simple onelever presses where multiple responses are allowed (Nicola, 2010). Whilst dopamine may be key to the computation and execution of highly effortful tasks, its role in strategic flexibility (Nicola, 2010) suggests it exerts a more subtle contribution to the complex relationship between task demands and the integration of task-relevant and task-irrelevant behavior.
We next consider the likely contribution of basal ganglia (BG) and ACC, and the formation of action-outcome association necessary for motivated behavior.

Basal ganglia: anaTomy and physiology
The basal ganglia are a set of subcortical nuclei comprising dorsal (putamen and caudate nucleus) and ventral aspects (often synonymous with NAc), the internal (GPi) and external (GPe) segments of globus pallidus, substantia nigra pars compacta (SNc), and reticulata (SNr) as well as the subthalamic nucleus (STN). The BG receives afferents from almost all cortical areas, especially the frontal lobe. Information processed within the BG network is sent via output nuclei (the internal segment of the globus pallidus and substantia nigra pars reticulata) to the thalamus, which eventually feeds back to frontal cortex (Alexander and Crutcher, 1990;Bolam et al., 2002). This basic circuitry is reproduced in different parallel and integrative corticostriatal loops, with their origin in different frontal domains, and is held to play a critical role in cognitive functions that span motor generation to more cognitive aspects of causal learning, executive function and working memory (Frank et al., 2001;Frank, 2005;Haber and Knutson, 2010;Vitay and Hamker, 2010).
Neurons in the striatum project either to output nuclei of the BG (GPi and SNr) or to an intermediate relay involving GPe neurons which ultimately project to BG output nuclei. These two populations provide the origin of BG direct and indirect pathways which funnel information, conveyed in parallel to striatum by cortical afferents, to pared to the rest of striatum. First, it is involved in unconditioned responding in the appetitive and aversive domains, spanning feeding (Kelley et al., 2005), and maternal behavior (Li and Fleming, 2003) to defensive treading (Reynolds and Berridge, 2002). Moreover, the NAc shell is involved in invigorating effects of dopamine on conditioned behaviors controlled by the NAc core (Parkinson et al., 1999). Second, the shell is the only striatal subdivision projecting to lateral hypothalamus (Pennartz et al., 1994, and reviewed by Humphries and Prescott, 2010), a key structure in an "actionarousal" network. Note lateral hypothalamus also exerts an influence over autonomic function and contains orexin-producing cells which influence arousal and energy balance control (see Ikemoto, 2007 for a comprehensive review). Third, whereas amygdala has extensive projections to both the core and shell (Humphries and Prescott, 2010), the NAc shell is the only recipient of hippocampal afferents within the striatal complex (Wickens et al., 2007;Haber and Knutson, 2010). This restricted projection from hippocampus has generated extensive discussion concerning the unique role of ventral BG in spatial navigation, fear-modulated free-feeding, and acquisition of stimulus value through stimulus-outcome pairings (Humphries and Prescott, 2010). These lines of evidence point to the shell as critical in forming linkages between an object/event in the environment and the agent's natural response toward it.
An alternative interpretation of the anatomical and physiological organization of the BG is a selection and control model (Gurney et al., 2001). In this model inputs for selection and control are received separately by striatal D1 receptors and D2-like receptors, respectively. D1 transmission is then projected as inhibition to GPi/ SNr which acts as an action selection output, whereas D2 transmission inhibits GPe which acts as an output layer for a control mechanism. The control output layer, in turn modulates action selection: GPe inhibits activity GPi/SNr output nuclei. Akin to inhibitory mechanisms described in the direct/indirect BG model, this selection/control BG model also describes inhibitory relationships between nuclei in BG. It is not clear what the thalamic inhibitory/excitatory impacts are on movement. Nevertheless this model highlights an important role for BG in action selection and control. More recently, Nicola (2007) has discussed the potential role of NAc in such a model, particularly in disinhibiting motor efferents for one action and inhibiting motor efferents for another, thereby allowing action selection.

Basal ganglia and efforT-relaTed processes
To facilitate execution of motivated behavior, one needs to internally represent action costs and benefits. Using fMRI, Croxson et al. (2009) investigated where in the human brain effort and reward are represented. Participants saw a discriminative stimulus signaling an action with a particular cost and benefit and then completed a series of finger movements using a computer mouse, to gain secondary reinforcers. The cost, in terms of effort and time, increased as more finger movements were completed, whilst the benefit increased as the secondary reinforcer was larger. When anticipating these actions, striatum activity correlated with both anticipated costs and anticipated reward of effortful actions.
More recent fMRI studies have replicated an involvement of striatum in effort-related processes, reporting higher dorsolateral striatal activity for choosing low compared to high-effort options The contribution of this pathway to behavioral control has been discussed extensively (Aron and Poldrack, 2006;Frank, 2006) and is beyond the scope of the present review.

anaTomical and funcTional gradienTs in The sTriaTum
The functional organization of BG along the direct and indirect pathways, as described above, applies to the full extent of the striatum, forming an integral re-iterated processing matrix which performs common operations across different subdivisions (Wickens et al., 2007). Although there are suggestions of a dorsal-ventral segregation, the consensus favors a dorsolateral-ventromedial gradient (Voorn et al., 2004) with no sharp anatomical distinction between dorsal-ventral areas. Indeed, based on the cytology of spiny projection neurons, dopaminergic inputs, and dopamine-modulated plasticity and inhibition, dorsal and ventral striatum are strikingly similar (Wickens et al., 2007). However, there is evidence for a functional segregation such that dorsolateral striatum, receiving sensorimotor afferents, supports habitual, stimulus-reward associations. This contrasts with ventromedial striatum, receiving afferents from orbito and medial prefrontal cortex, hippocampus, and amygdala, which supports formation of stimulus-action-reward associations (Voorn et al., 2004;Haber and Knutson, 2010).
A functional gradient in dopamine signaling is also described in BG (Wickens et al., 2007). DA release is determined by density of DA innervation (densities reduce the distance between release and receptor sites), such that higher innervation densities are necessary for rapid DA signaling. DA clearance is regulated by density of DA transporters (DAT), hence affecting distance and time course of volume transmission. Wickens et al. (2007) has documented greater DA innervation and higher DAT densities in dorsolateral striatum with these densities decreasing along a ventromedial gradient (also Haber and Knutson, 2010). High densities of release sites and DAT result in fast clearance in dorsolateral striatum, which may be related to encoding of discrete events involving reinforced responding, or even automatized and habitualized behaviors. Ventromedially, lower densities of DA innervation and DAT result in slow clearance in NAc core, and even slower clearance in NAc shell, which may be related to slower time course of action-outcome evaluation (Wickens et al., 2007;Humphries and Prescott, 2010).
Moreover, it is noteworthy that within the ventromedial subdivisions of the striatum, the NAc has interesting particularities. The NAc is subdivided, on the basis of anatomical and histochemical features, into the core and the shell, with the latter more medial and ventral in location than the former (Voorn et al., 2004;Ikemoto, 2007;Humphries and Prescott, 2010). This core/shell distinction is particularly important when considering the role of BG in motivated behavior.
The NAc core is similar to dorsal striatum (Humphries andPrescott, 2010, but see Nicola, 2007 on role of dorsal-ventral striatum in temporal predictability). Functionally, NAc core seems critical in the translation of raw, unconditioned stimulus value, into a conditioned response. Thus, NAc core plays an important role in conditioned behavior (Ikemoto, 2007), such as autoshaping in classical conditioning paradigms and conditioned reinforced responses in instrumental learning paradigms. On the other hand, the NAc shell, the most ventromedial aspect of striatum, has unique features com-visual discrimination task, but instead was specific to an inability to utilize reward-action association to make the correct response. In a different experiment, monkeys with ACC lesion were impaired in selecting a set of response when the correct responses were determined by an integration across past contingencies between action and reward . In addition, using fMRI, human ACC was found to be most active when participants had to simultaneously internally generate a sequence of actions whilst monitoring the outcome of their actions (Walton et al., 2004).
Lesions studies with rodents using the T-maze consistently show impairments in effort-based decision making following removal of ACC. As with dopamine depletion experiments, these lesions result in a shift of preference away from an option with a larger food reward that requires scaling a high barrier, thus requiring more effort. This reduced preference for larger/effortful arm was not due to lethargy or immobility as it is immediately restored when both arms have equal effort costs (Denk et al., 2005;Floresco and Ghods-Sharifi, 2007;Walton et al., 2009, but see Floresco et al., 2008b for a discussion the extent to which ACC plays a role in effort-based tasks).
Human ACC lesions provide a more subtle interpretation for the role of ACC in effort processing. Naccache et al. (2005) tested a patient with a large lesion to left mesial frontal region including the left ACC using a, cognitively demanding, Stroop task (Stroop, 1935). This patient could not verbally recognize nor express discriminatory skin conductance responses in difficult trials where greater mental effort was required, but could perform as well as healthy controls. This case study suggests dissociability of objective cognitive performance from a physiological response and from the subjective appraisal of mental effortfulness (Naccache et al., 2005, but see McGuire and Botvinick, 2010 for the involvement of lateral prefrontal cortex, instead of ACC, in a closer inspection of subjective experience of mental effort through intentional and behavioral avoidance from mentally challenging tasks).
The ACC is implicated in a host of cognitive processes, ranging from cognitive control to suppression of prepotent responses such as in Stroop or go-nogo tasks, tasks that induce negative emotions, and tasks that predict delivery of painful stimuli (see a review by Shackman et al., 2011). Shackman et al. (2011) discussed a challenge in advancing knowledge of its functional organization being the complexity of its anatomical organization and variability across individuals. For example, a tertiary sulcus in dorsal ACC, the paracingulate sulcus, is present in one-third of the population, and its presence causes location change of architectonic Brodmann area 32′, and a volumetric reduction of Brodmann areas 24a′ and 24b′. Consequently, spatially normalized cingulate premotor regions differ across subjects, and an unmodeled cingulate sulcal variability may inflate the spread of activation clusters found across studies, rendering complex a clear functional dissociation within ACC. Bush et al. (2000) proposed the rostro-ventral cingulate could be functionally segregated into cognitive and affective components located to dorsal and ventral ACC, respectively. This segregation seems too broad. Shackman et al. (2011) using a sample of almost 200 neuroimaging experiments that included negative affect, pain, and cognitive control reported strongly overlapping activation clusters in dorsal ACC, or what they termed as middle cingulate in a physical effort task (Kurniawan et al., 2010) and higher ventral striatal activity in a low cognitive demand block compared to a high cognitive demand block in a mental effort task (Botvinick et al., 2009). Whilst, it is still unclear whether physical and cognitive mechanisms of effortful actions reflect similar psychological and neural processes, together these studies provide support for the importance of striatum in effort-related processes. In the following section, we assess the type of association formed when an organism performs a motivated goal-directed behavior.

encoding acTion and iTs ouTcomes
Linking a chosen action to its outcome is central for optimal goaldirected behavior. When a monkey travels a distance to forage for food, not only does it need to link contextual cues to food consumption, for example associating a tree full of ripe fruits with eating fruits, it also needs to associate the action (climbing a tree) with the consequences of the action, namely the energetic cost of climbing. Neurons in primate dorsal striatum, can be categorized into those that encode the action made by the monkey (direction of saccade made) and neurons sensitive to the outcome of the monkey's choice (reward/unrewarded; Lau and Glimcher, 2007). However, these neurons do not appear to support the kind of action-outcome association required for goal-directed behavior.
Using reinforcement learning models, similar to those used to characterize activity in DA neurons, Samejima et al. (2005) reported neurons in the striatum whose activity correlated with the value of an action. These action value neurons are important because they track the value of say, a left handle turn in a probabilistic twochoice task, independent of whether the monkey ultimately selects the action, and thus provide input information for action selection. Furthermore, in a subsequent study, Lau and Glimcher (2008) found action value neurons, including neurons which traced the value of the chosen action, in the striatum. These chosen value neurons show enhanced activity when the tracked action has a higher value and, on this basis, was subsequently chosen. Using similar reinforcement learning models, human fMRI studies also report that BOLD signal in the dorsal striatum correlate with the relative advantage of taking one action over an alternative (O'Doherty et al., 2004).

efforTful Behavior, acTion and iTs ouTcomes: implicaTion of The acc
These action and chosen value representations in the striatum are precisely the kind of association between action and outcome required for goal-directed behavior. However, the unanswered question is where does the information needed for this computation come from? One possibility is ACC, a region suggested to represent this action-outcome association (Rushworth et al., 2007). For example, Hadland et al. (2003) trained macaque monkeys to pull a joystick upward after receiving a type of food, say a peanut, in order to obtain a second peanut and to turn a joystick to the side after obtaining a different food type, say a raisin, to receive a second raisin. They found that while control monkeys could select an action based on this reward-response association, monkeys with a lesion to ACC were impaired in selecting the correct response. Interestingly, the impairment was not due to an inability to make an association between visual cues and reward as tested in a second severely impaired in execution of real-life events such as holding a job, although various measures of logical reasoning, general knowledge, planning, and social and moral judgments proved intact. The authors discussed how the lesion did not impact on pure action execution, but on the analysis and integration of the costs and benefits pertaining to real-life situations.

apaThy and The execuTion of efforTful responding
Several distinct types of brain insult are associated with apathy in humans. For example, bilateral ACC lesions can present with akinetic mutism, a wakeful state characterized by prominent apathy, indifference to painful stimulation, lack of motor and psychological initiative (Tekin and Cummings, 2002). Apathy is also often present in patients with subcortical brain lesions (involving BG), but is more commonly found in those with prefrontal, mainly ACC, lesions (van Reekum et al., 2005). In this section we draw upon findings with apathy to understand cost benefit integration and implementation of effortful choices.
Effort is a salient variable in individuals with apathy who lack the ability to initiate simple day-to-day activities (van Reekum et al., 2005;Levy and Dubois, 2006). This lack of internally generated actions may stem from impaired incentive motivation: the ability to convert basic valuation of reward into action execution (Schmidt et al., 2008). Patients with auto-activation deficit (AAD), the most severe form of apathy, are characterized by lack of self-initiated action (van Reekum et al., 2005) or a quantitative reduction in selfgenerated voluntary behaviors (Levy and Dubois, 2006). Thus, the key feature in AAD is an inability to internally generate goal-based actions, a deficit that may variously reflect an ability to (1) encode the consequence of an action as pleasurable or as having hedonic value (e.g., to attain reward, "liking"); (2) execute the action; and (3) represent the association between action and reward. We now discuss a proposal that the behavioral and neural mechanisms underlying AAD are mostly intimately linked to the third sub-process.
Auto-activation deficit is not associated with impaired "liking" as patients with AAD have a normal skin conductance response to receipt of rewards and verbally distinguish between different magnitudes of monetary reward (Schmidt et al., 2008). In addition, the most prominent damage in AAD pertains to BG and the dopaminergic system. Secondly, AAD is probably not linked to specific impairments of action execution. Schmidt et al. (2008) tested patients with bilateral BG lesions with the history of AAD and found that, compared to normal and Parkinson's disease control groups, patients with AAD are worse when generating voluntary vigorous actions based on contingent reward, but are equally able to generate the same motor response if based on external instructions. This provides evidence against AAD being explicable in terms of an impairment in pure motor action execution.
We suggest that AAD reflects an impairment in linking reward anticipation to action. Damage to BG in AAD most commonly involves a focal bilateral insult to the internal portion of pallidum (Levy and Dubois, 2006). Pessiglione et al. (2007) investigated the role of ventral pallidum in incentive motivation employing a task where individuals voluntarily squeezed a handgrip device in response to different reward magnitudes. Notably, the amount of voluntary force during squeezing was proportional to reward magnitude, suggesting that participants were able to identify a cortex (MCC), challenging a strict segregationist view of ACC (see Figure 3). These authors also pointed to evidence that the dorsal ACC might be involved in affective control, including autonomic regulation  and pain processing, suggesting these findings may reflect an agent's need for behavioral control when habitual responses are not sufficient under uncertain actionoutcome contingencies.
Anatomically, the ACC projects to striatum, particularly the caudate nucleus and portions of ventral striatum (Haber and Knutson, 2010). Moreover, ACC has bilateral connections to motor and prefrontal cortex fulfilling a role as a hub where action and outcome associations might be represented. In human and nonhuman primates, the ventral cingulate have strong interconnections with ventral striatum including the NAc, whilst the dorsal cingulate connects more strongly to dorsal striatum including putamen and caudate (Kunishio and Haber, 1994;Beckman et al., 2009), potentially facilitating transmission of reward-related information. Furthermore, dorsal ACC is interconnected with premotor cortex and a more posterior part constitutes the cingulate motor area (Beckman et al., 2009) implicated in action selection (Picard and Strick, 2001). Shima and Tanji (1998) reported that cingulate motor areas in monkeys respond to selection of voluntary movement based on reward, supporting a role in linking internally generated action to reward. Indeed, a working hypothesis is that ACC could support adaptive control, integrating aversive, biologically relevant information in order to bias motor regions toward a contextually appropriate action (Shackman et al., 2011).
This wide-ranging anatomical connectivity between BG, ACC, and other cortical regions provide a neuroanatomical foundation for establishing action and outcome representations, of a type needed for motivated behavior. Normal function of this circuitry can be inferred to facilitate willingness to execute effortful actions. On the other hand, disruption of this circuitry, as in people with apathy, would discourage execution of such actions. This account has a resemblance to phenomena in a case study of a patient with a lesion to mesial prefrontal cortex (which included ACC) that led to profound apathy (Eslinger and Damasio, 1985). This patient was Figure 3 | Views on the psychological function of ACC. Left: ACC function has been suggested as anatomically segregated into a dorsal cognitive division and a ventral affective division (Bush et al., 2000). Right: more than a decade later, a meta-analysis on almost 200 fMRI experiments suggested a strong overlap in clusters of activation in studies of cognitive control, negative affect, and pain (Shackman et al., 2011). Figures adapted from Shackman et al. (2011).

conclusion
We provide evidence for an intimate interplay between ACC, BG, and dopaminergic pathways in enabling animals, including humans, to choose and execute effortful action. We suggest that effort may act as a discounting factor for action value, and that integrative mechanisms between cost and benefit facilitate a willingness to incur costs. Our review of reinforcement learning, empirical findings on the relationship between dopaminergic coding and costbenefit parameters of an action, and the organization of BG and ACC point to these latter structures as critical in linking a stimulus to an action and the consequences of that action. Notably, patients with apathy often manifest a pathology that disrupts this ACC-BG network. This fractures a link between action and outcomes resulting in lack of drive to execute potentially valuable actions.
Our review highlights the psychological and neural mechanisms through which an organism is willing and capable of executing an effortful act to attain a goal. The core process appears to involve coding of specific action requirements, an analysis and integration of costs and benefits, and a decision to expend effort and to implement an action. We do not dissect a potentially important distinction between cognitive and physical types of effort (Kool et al., 2010;Kurniawan et al., 2010). Future research might usefully endeavor to examine how one makes a trade-off between both effort types and examine how we determine when investing in one type of effort (mental) is more appropriate than investing in the other (physical).

acknowledgmenTs
This work was supported by a Wellcome Trust Programme Grant to Ray J. Dolan. Marc Guitart-Masip is funded by a Max Plank Society Award to Ray J. Dolan. The authors would like to thank Fred H. Hamker and Julien Vitay for their helpful suggestions. reward context where it was advantageous to produce more physical effort. Furthermore, ventral pallidal activity correlated with outcome context, providing a neural basis for enhanced effort as a response to increased payoff. Similarly, damage to BG in AAD may have caused a failure to recognize an advantageous context to make an adaptive action (Walton et al., 2004;Levy and Dubois, 2006). These data suggest that bilateral BG damage, at least in AAD, produces a syndrome that arises out of a deficit in translating reward cues into appropriate action selection and execution.
In light of Schmidt et al.'s (2008) findings that AAD patients were mostly impaired in the execution of actions, when an internal link between a reward and action is required, it is noteworthy that AAD may cause impairments beyond simple abstract action-reward association. In other words, AAD may cause impairments in the actual execution of reward-based actions. This highlights the importance of BG in energizing individuals to persevere with acting, a deficit commonly found in patients with Parkinson's disease (which is largely associated with a dysfunction in BG). Schneider (2007) tested Parkinson's disease patients in solving a difficult cognitive task, and found that the patients were making significantly fewer attempts to solve the task than normal controls, pointing to a deficit in mental persistence in such patients. It may well be that persistence is linked to a higher tendency to generate internal motivation or arousal which then energizes individuals to persevere (Gusnard et al., 2003), or perhaps lessens a tendency to distraction (see Nicola, 2010).
Taken together, apathy, as a manifestation of impaired motivation to overcome the cost of an action, is associated with damage to a cortico-subcortical network (either lesions in the ACC or BG) that generates internal association between action and its consequences. This highlights a key involvement of the ACC and BG in the anticipation and execution of effortful actions.