Abstract
The tendency to make unhealthy choices is hypothesized to be related to an individual's temporal discount rate, the theoretical rate at which they devalue delayed rewards. Furthermore, a particular form of temporal discounting, hyperbolic discounting, has been proposed to explain why unhealthy behavior can occur despite healthy intentions. We examine these two hypotheses in turn. We first systematically review studies which investigate whether discount rates can predict unhealthy behavior. These studies reveal that high discount rates for money (and in some instances food or drug rewards) are associated with several unhealthy behaviors and markers of health status, establishing discounting as a promising predictive measure. We secondly examine whether intention-incongruent unhealthy actions are consistent with hyperbolic discounting. We conclude that intention-incongruent actions are often triggered by environmental cues or changes in motivational state, whose effects are not parameterized by hyperbolic discounting. We propose a framework for understanding these state-based effects in terms of the interplay of two distinct reinforcement learning mechanisms: a “model-based” (or goal-directed) system and a “model-free” (or habitual) system. Under this framework, while discounting of delayed health may contribute to the initiation of unhealthy behavior, with repetition, many unhealthy behaviors become habitual; if health goals then change, habitual behavior can still arise in response to environmental cues. We propose that the burgeoning development of computational models of these processes will permit further identification of health decision-making phenotypes.
Introduction
Unhealthy behaviors, such as tobacco smoking, excessive alcohol intake, physical inactivity and substance misuse, account for significant morbidity and mortality worldwide (Smith et al., 2012). For example tobacco smoking was estimated to be responsible for a fifth of all deaths in the US in 2005 (Danaei et al., 2009). In the developed world these behavioral risk factors can be viewed as a direct expression of personal choice, and research has therefore been directed toward establishing their psychological determinants (Conner and Norman, 2005). Unhealthy behavior frequently has a delayed effect on health, leading researchers to hypothesize that an individual's tendency to make unhealthy choices is related to their temporal discount rate, the theoretical rate at which they devalue delayed outcomes (Grossman, 1973; Bickel et al., 2012). In recent years a large number of studies have addressed this hypothesis, forming part of a growing endeavor to identify decision-making phenotypes which correlate with maladaptive behavior (Montague et al., 2012).
Given a choice, people tend to prefer immediate rewards to those available after a delay, for example preferring to receive $10 today over $15 in a month, implying that they discount the value of delayed rewards relative to more immediate ones. A function which describes the pattern of discounting can be estimated by observing choices between delayed outcomes. Economic theories of rational behavior posit that goods ought to be discounted exponentially with delay (Samuelson, 1937). Formally, an outcome which has utility A if received immediately (t = 0) is worth A· δt if delayed t periods into the future. The present-time value, V, of receiving A at time t is thus given by:
Here the discount rate, δ, represents the constant proportional decrease in value with each added time period of delay. However both humans and animals violate the exponential assumption of a constant proportional discount factor per unit time, appearing rather to discount rewards occurring in the immediate future more steeply than those in the distant future. In other words, delaying an immediate reward 1 day into the future decreases its implied value proportionally more than does delaying an already distant reward by 1 day (Strotz, 1957; Chung and Herrnstein, 1967; Ainslie, 1974, 1975; Mazur, 1987; Benzion et al., 1989; Green et al., 1994). The discount function estimated from observed choices is better accounted for by a hyperbolic function, written in its simplest form as follows, where k represents the hyperbolic discount rate:
A proposed alternative is the “quasi-hyperbolic” approximation to hyperbolic discounting (Phelps and Pollak, 1968; Laibson, 1997; Angeletos et al., 2001; McClure et al., 2004, 2007), which is formalized as exponential discounting, with an additional preference for immediate rewards, expressed as a second discount factor, β, applied to all time-periods except the first:
The hyperbolic and quasi-hyperbolic forms qualitatively capture the observation that individuals make far-sighted plans when outcomes are distant, but reverse their choices in favor of short-term rewards when the future is reached (see Kalenscher and Pennartz, 2008 for a review). Hyperbolic or quasi-hyperbolic discounting therefore putatively imparts a psychological explanation for why people intend to perform actions they subsequently fail to carry out. In economics this is referred to as myopic preference reversal (Kirby and Herrnstein, 1995), and bears considerable relevance to health choice, where there is a marked discrepancy between health intentions and health behavior (Conner and Norman, 2005).
This article examines the relationship between temporal discounting and health behavior at two levels of analysis: empirical and mechanistic (Rescigno et al., 1987). A distinction is often made in psychology and neuroscience between empirical and mechanistic classes of model. Empirical, high-level, or descriptive models seek to capture structure in observed data, allowing predictions to be made, but lack psychological content (Lewandowsky and Farrell, 2011). Thus temporal discounting might predict markers of health behavior, without being upheld as a psychological process underlying unhealthy choice (see Frederick et al., 2002 for an historical perspective on discounting as a descriptive model). For example, discounting measures might simply be correlated with environmental factors influencing health behavior, such as peer influence. By contrast, mechanistic, low-level or explanatory models specify psychological processes at an abstract level, or seek to identify how neural systems solve computational problems (Farrell and Lewandowsky, 2010; Lewandowsky and Farrell, 2011; Montague et al., 2012). Some authors propose that hyperbolic discounting is a mechanistic model, i.e., a fundamental principle for choosing between delayed outcomes (for example Ainslie, 2001). This notion leads to the hypothesis that unhealthy choices directly result from hyperbolic discounting of their distant health consequences. The latter is consistent with the recent proposal that discounting is a trans-disease process underlying several impulsive pathologies (Bickel et al., 2012). To address the first, empirical, hypothesis we perform systematic review of studies which examine relationships between temporal discount rates and health behavior or health status. To address the plausibility of the second, mechanistic, hypothesis we examine whether both health behavior in the field and laboratory choice behavior fulfill the predictions of hyperbolic discounting. Predicting unhealthy choice has potential utility in guiding health behavior change interventions (Conner and Norman, 2005), while understanding the processes involved in unhealthy choice may assist in designing novel interventions.
Can measured discount rates predict health behavior?
Differences between discounting for health and money
Measured discount rates are not equal for all commodities, for example, people discount primary reinforcers such as food and water more steeply than money (Odum, 2011). This raises an important question: which outcome modality is the best candidate predictor of health behavior? With regard to real-world health choices the relevant delayed outcome is a health state. Indeed numerous studies have attempted to measure discount rates for hypothetical health states (Asenso-Boadi et al., 2008). Many such studies require participants to imagine a hypothetical illness, usually described in generic terms, and to trade-off the severity or duration of the illness against when they would prefer the illness to occur (for example Van Der Pol and Cairns, 2001). Individuals who are willing to accept a more severe illness occurring after a delay over a less severe immediate illness are said to discount future illness. Alternatively, the health state may be described as an improvement in health from a baseline of illness (for example Ganiats et al., 2000), where individuals who prefer immediate over delayed health improvement, and are willing to accept a small improvement in health occurring sooner, over a larger improvement at a delay are said to discount future health.
Studies of hypothetical health discounting, for both health improvements and illness, demonstrate that some important properties of monetary discounting are conserved in the health domain. Several studies find that many people indeed discount future health improvements in the conventional sense, also referred to as positive time preference for health (Lipscomb, 1989; Olsen, 1993; Bleichrodt and Johannesson, 2001). Also consistent with monetary discounting there is robust evidence for decreasing health discount rates over increasing delay (Bleichrodt and Johannesson, 2001). Furthermore, Van Der Pol and Cairns (2002) asked participants to make choices relating to a generic illness, which could be delayed into the future by means of an imaginary treatment, showing that hyperbolic discount functions fitted choices better than exponential ones (see also Van Der Pol and Cairns, 2011).
Problematically however temporal discounting for health, unlike monetary, outcomes is far from universal. A proportion of people in fact prefer to advance the timing of illness (Cairns, 1992; Redelmeier and Heller, 1993; Chapman, 1996; Chapman and Coups, 1999; Van Der Pol and Cairns, 2000, 2002, 2011; Chapman et al., 2001) or to delay health improvements (Olsen, 1993; Chapman and Elstein, 1995; Dolan and Gudex, 1995; Chapman, 1996). This is the opposite pattern to conventional (positive) discounting, and is referred to as negative time preference or negative discounting. Across the majority of studies the proportion of people displaying negative time preference ranges from 3 to 10%. In addition, a proportion of people do not discount hypothetical future health outcomes at all, preferring to experience better health, irrespective of delay, which is termed zero time preference (Cairns, 1992; Olsen, 1993; Redelmeier and Heller, 1993; Chapman and Elstein, 1995; Dolan and Gudex, 1995; Chapman, 1996; Chapman and Coups, 1999; Van Der Pol and Cairns, 2000; Chapman et al., 2001; Van Der Pol and Cairns, 2002, 2011).
Notably, negative and zero time preference are also observed for aversive outcomes. In choices between genuine delayed painful events, many people prefer to expedite inevitable pain (negative time preference) (Loewenstein, 1987; Berns et al., 2006; Story et al., 2013), or simply to receive less pain, irrespective of its timing (zero time preference) (Story et al., 2013). Negative time preference can be attributed to the fact that anticipating delayed pain is itself aversive, termed dread (Loewenstein, 1987), a quantity which can be minimized by choosing to “get the pain out of the way” (Loewenstein, 1987; Loewenstein and Prelec, 1991; Berns et al., 2006; Story et al., 2013). A related observation is that monetary losses tend to be positively discounted at a lower rate than monetary gains, referred to as the sign effect (e.g., Thaler, 1981; Benzion et al., 1989), which could be explained by a degree of dread for monetary losses (Loewenstein, 1987) (although could also result from differences in the shape of the instantaneous utility function for losses and gains; Loewenstein and Prelec, 1991). There is also evidence for a sign effect in the aversive domain, whereby describing delayed painful outcomes in terms of relief from a more severe pain reduced the overall degree of negative time preference (Story et al., 2013). Chapman (1996) found a sign-effect for health outcomes, whereby illness was discounted less than health improvement, to the extent that discount rates for illness were uncorrelated with discount rates for health improvement. MacKeigan et al. (1993) report a similar result for preferences over hypothetical improvements or decrements in health framed as a scenario of arthritis, finding positive discounting for health improvement and for long periods of illness, but negative discounting for fleeting illnesses.
Thus discounting behavior for hypothetical illness and health improvement closely reflects that for pain and relief. Indeed illness and health improvement are intuitively more analogous to pain and relief than to appetitive rewards and losses. Furthermore, consistent with respondents exhibiting dread for delayed illness, in an illness discounting task, individuals who perceived the illness as more severe were more likely to have negative discount rates (Van Der Pol and Cairns, 2000). Zero time preference for health and pain-related outcomes may result from the experiential nature of these outcomes; unlike money, health is an experiential commodity which cannot be saved or invested (Chapman and Elstein, 1995).
Commensurate with the prevalence of negative and zero time preference for health, discount rates for health and money are poorly correlated across individuals (Cairns, 1992; Chapman and Elstein, 1995; Chapman, 1996, 2002; Lazaro et al., 2001; Petry, 2003), and remain so even when health and monetary outcomes are utility matched (Chapman, 1996). For example, Chapman and Elstein (1995), who also included a third domain of vacations of varying duration, report an overall mean Spearman coefficient of 0.25 for the correlation between discount rates of different domains. Chapman (1996) further demonstrated that low correlations between health and monetary domains persist even when the outcomes are matched for sign as well as utility. Correlations between the two domains overall showed a trend toward being higher when matched for sign, although all correlations remained weak (Spearman r 0.06–0.28). In other words discount rates for illness showed marginally stronger correlations with monetary losses than with monetary gains, and discount rates for health improvement showed marginally stronger correlations with monetary gains than with monetary losses, though all such correlations are weak and most do not reach conventional levels of significance.
The low correlations between health and monetary discounting as well as the prevalence of negative and zero time preference for health and aversive outcomes demonstrate that discounting cannot be considered a universal mechanism by which all delayed events are evaluated, and appear to question any mechanistic hypothesis whereby discounting of delayed health causes unhealthy choice. It remains possible however that people represent health outcomes differently when making real health choices as opposed to responding in hypothetical health discounting tasks. In addition, discounting for either health or money may nevertheless be capable of predicting real-world health behavior, through a correlation with processes underlying health choice.
Systematic review methods
To address the question of whether measured discount rates are capable of empirically predicting health behavior we performed systematic literature review. In order to identify studies comparing discount rates with health behavior or health status, we searched the PubMed database for full-text articles published up to January 2014, using keywords relevant to discounting together with the words “Health,” “Illness,” “BMI,” “Obesity,” “Alcohol,” “Drinking,” “Smoking,” “Drug” or “Behavior.” This search strategy yielded 104 studies. The abstracts (and where needed full text) of these were then reviewed for suitability: studies were included if they compared the results of any delay discounting paradigm to an observed health-related measure or behavior. In total 34 suitable full text articles were identified from this search. The references and citation lists of these studies were also reviewed for inclusion, yielding a further 78 suitable studies, yielding a total of 112 suitable studies (summarized in Supplementary Tables 1–5). Several of these studies have been reviewed elsewhere, chiefly in the context of addiction (Reynolds, 2006b; MacKillop et al., 2011; Bickel et al., 2012, 2014; Koffarnus et al., 2013). Here a broader range of health outcomes are reviewed. The studies are discussed below, organized by first by the modality of discounting (hypothetical health versus money or other appetitive rewards) and subsequently by the nature of the health outcomes (tobacco smoking, alcohol use, illicit substance misuse, obesity and eating behavior, preventive health behavior, risky sexual behavior and drug-taking practices or multiple health behaviors).
Can discount rates for hypothetical health outcomes predict health behavior?
The finding that a significant proportion of individuals exhibit negative and zero discount rates for health casts doubt on the suggestion that health discount rates can predict unhealthy behavior. Indeed the few studies to have compared health discount rates with observed health behavior find weak or absent correlations. Four studies in our search sample examined relationships between health discounting and unhealthy behavior. Two of these tested relationships between health discounting and cigarette smoking status. Baker et al. (2003) compared discount rates for health in current and never-before cigarette smokers, finding that current smokers discounted both hypothetical improvements and decrements in health at a marginally higher rate than the never-before smokers, but this difference did not reach statistical significance. For monetary gains both hypothetical and real, current smokers delay discounted at a significantly higher rate than the never before smokers. Khwaja et al. (2007) found no significant differences between smokers and non-smokers in health discounting for health outcomes (however neither did this study find a relationship between smoking status and monetary discounting). A single study (Petry, 2003) examined discount rates for health outcomes in substance misusers, finding significantly higher discount rates for hypothetical health, money and freedom (from time spent in jail) in a group of current or previous heroin and/or cocaine users compared with a group of controls with no history of substance misuse in their lifetime. Finally, Chapman and Coups (1999) asked whether discount rates for monetary losses, described in terms of a parking fine, and for a flu-like illness, could explain uptake of free-of-charge influenza vaccinations. The respondents were future oriented, with 85% not significantly discounting the flu-like illness (i.e., showing zero time preference) and 83% not significantly discounting parking fines. Monetary time preferences were related to vaccine uptake: 45% of those with no discounting accepted the vaccine, compared with 29% of those who discounted money in the conventional manner. However, health discount rates were unrelated to vaccine acceptance. These findings suggest that hypothetical health discounting measures are an unreliable predictor of health behavior. The observed higher health discount rates in substance misusers compared with controls (Petry, 2003) and the trend toward higher health discounting in current cigarette smokers (Baker et al., 2003), taken together with the larger between-group differences in monetary discounting, suggest that health discounting does exhibit a weak relationship with health behavior, but is a less sensitive predictor than monetary discounting.
Can discount rates for monetary or appetitive outcomes predict health behavior?
Discounting for hypothetical health appears to be an unreliable predictor of observed health behavior. Nevertheless many studies in our search sample demonstrate that discount rates for money or other appetitive outcomes such as food or drug rewards, correlate with health behavior or health status. The key findings are discussed below, grouped by health outcome.
Discount rates and self-reported health
Self-reported health is perhaps the most general health outcome measure and correlates with life-expectancy in the developed world (Idler and Benyamini, 1997). An early study (Fuchs, 1982), surveying approximately 500 adults in the US, and a more recent household survey in the Netherlands (N = 2300) (Van Der Pol, 2011) both found weak negative correlations between monetary discount rates and self-reported health status. A further study in a South African population (Chao, 2009) found evidence for a U-shaped relationship between subjective health and monetary discount rates, whereby those who reported “average” health had lower discount rates than those who are either very healthy or very sick. The authors suggested that this might be due to the fact that those with very poor health were in more urgent need of money to fund medical care, whereas those in excellent health were able to enjoy the benefits of immediate economic consumption, highlighting the considerable difficulties in establishing a casual pathway between discounting and health.
Discount rates and cigarette smoking
Monetary discount rates have consistently been shown to be higher in people who currently smoke tobacco than in non-smokers (Bickel et al., 1999; Mitchell, 1999; Odum et al., 2002; Reynolds et al., 2003, 2004, 2007a,b, 2009; Reynolds, 2006a; Fields et al., 2009; Rezvanfard et al., 2010; Stillwell and Tunney, 2012; Wing et al., 2012). A recent meta-analysis of studies comparing discount rates with addictive behaviors, (MacKillop et al., 2011) estimated a moderate and highly significant effect (Cohen's d = 0.57, p < 0.0001) across all studies comparing discount rates in smokers versus non-smokers. Monetary discount rates also correlate with smoking frequency (Epstein et al., 2003; Ohmura et al., 2005; Kang and Ikeda, 2013). In keeping with this infrequent smokers exhibit monetary discount rates intermediate between heavy smokers and non-smokers (Heyman and Gibb, 2006; Reynolds and Fields, 2012; Stillwell and Tunney, 2012N = 9454; however see also Reynolds et al., 2003 and Johnson et al., 2007b for negative findings) and both smoking frequency and monetary discount rates were found to be higher in a group of young-adult smokers than in a group of adolescent smokers (Reynolds, 2004). The relationship between smoking frequency and discounting does not appear to be mediated by the acute effects of nicotine, since acute nicotine administration to non-smokers has recently been shown to have no effect on intertemporal choice behavior (Kobiella et al., 2013). However the relationship may be related to the level of nicotine dependence (Sweitzer et al., 2008), consistent with discounting being a state-based marker of addiction severity.
Interestingly those who have previously smoked and those who have never smoked do not significantly differ in their discounting behavior for money (Bickel et al., 1999). Furthermore in a prospective study of smoking cessation, participants were separated into a group who received an intervention program directed at reducing smoking and a control group who continued to smoke as usual. The two groups did not differ in their discounting behavior at baseline. Whilst the control group showed no changes in discounting over time, the intervention group (who overall reduced their smoking frequency) showed a significant decrease in discounting for both money and cigarettes after only 5 days into the program (Yi et al., 2008). These results strongly suggest that the state of nicotine addiction acts reversibly to increase discount rates. However there is also evidence to support the idea that discounting operates as an antecedent vulnerability marker, since monetary discount rates in smokers predict rates of smoking adoption (Audrain-McGovern et al., 2009). Monetary discount rates in smokers also predict rates of relapse within smoking cessation programs (Krishnan-Sarin et al., 2007; Yoon et al., 2007; MacKillop and Kahler, 2009; Sheffer et al., 2012; Brown and Adams, 2013), and the ability to abstain from smoking under laboratory conditions (Dallery and Raiff, 2007; Mueller et al., 2009). It seems reasonable to conclude therefore that relationships between discounting and smoking behavior are subject to both state- and trait-based influences (Odum, 2011).
Finally short-term abstinence from cigarettes increases discount rates in addicted smokers (Mitchell, 2004; Field et al., 2006; Yi and Landes, 2012). For example, Field et al. (2006) measured discount rates for hypothetical gains of money or cigarettes in a group of 30 smokers: one randomized group performed the procedures following their usual cigarette consumption, the other following a minimum of 13 h of abstinence from cigarettes. Implied discount rates for both money and cigarettes were significantly higher in the abstinence group.
In summary, monetary discount rates predict many features of smoking behavior: monetary discount rates are higher in current smokers, correlate with smoking frequency and prospectively predict the adoption of smoking and abstinence from smoking. The upward effect of nicotine cravings on discount rates and the decrease in discounting concomitant with reductions in smoking indicate that discounting is influenced by state-based environmental and motivational factors. Taken together these results indicate that relationships between discounting and smoking have both state- and trait-based components; a consideration which most likely also applies to relationships between discounting and other addictive behaviors (de Wit, 2008; Odum, 2011).
Discount rates and alcohol use
Monetary discount rates exhibit consistent relationships with alcohol use. A recent meta-analysis of studies comparing discount rates in persons meeting clinical criteria of an alcohol dependence syndrome with controls (MacKillop et al., 2011) demonstrated a moderate, highly significant effect (Cohen's d = 0.50, p < 0.0001). Monetary discount rates are higher in currently abstinent alcohol dependent individuals compared with non-dependent controls (Bjork et al., 2004; Mitchell et al., 2005; Boettiger et al., 2007), in early-onset as opposed to late-onset alcohol dependence (Dom et al., 2006), and correlate with the severity of alcohol dependence (Mitchell et al., 2005), as well as symptoms of an alcohol abuse disorder (MacKillop et al., 2010). Monetary discount rates have also been shown to be higher in a group with a previous lifetime diagnosis of an alcohol abuse disorder as compared with those without a lifetime history of alcohol abuse (Bobova et al., 2009).
Several studies have further linked higher discounting with relatively moderate levels of alcohol consumption. Vuchinich and Simpson (1998) demonstrated higher monetary discount rates in “problem drinkers” and also heavy social drinkers, compared with light social drinkers, suggesting a relationship between alcohol intake and discount rates even amongst those designated as social drinkers. Similarly, Field et al. (2007) found that delay discounting for alcohol positively correlated with weekly alcohol consumption (Pearson r = 0.31) amongst adolescents, where those in highest tertile of alcohol use had a mean weekly consumption of 23 units, while those in the lowest tertile had a mean of 3 units. Yankelevitz et al. (2012) found that implied discounting for both money and alcohol was moderately correlated with levels of alcohol use in female students (Pearson r = 0.43 for money and 0.41 for alcohol discounting), though no correlation was found in male students.
Amongst students, monetary discount rates appear related to adverse consequences of alcohol use (Kollins, 2003; Rossow, 2008; Dennhardt and Murphy, 2011). Kollins (2003) observed that monetary discount rates were negatively correlated with age at first using alcohol and showed a strong positive correlation with the number of times that students had “passed out” as a result of alcohol use (Pearson r = 0.73, P < 0.01) and Rossow (2008), studying a sample of 17,413 adolescents in Norway, demonstrated that high monetary discounters became intoxicated more frequently and were more likely to vomit or “pass out” as a result of drinking. Paralleling findings in previous smokers of cigarettes, previously addicted users of alcohol who have achieved long-term abstinence have discount rates intermediate between current users and controls (Petry, 2001). Finally, in an elegant field study, the discount rates of male social-drinkers on their entry to a bar prospectively predicted increases in blood alcohol level on their exit (Moore and Cusens, 2010), such that those that had steeper discounting on entry showed greater increases in alcohol level. Discount rates were not confounded by baseline intoxication, since blood alcohol level at entry to the bar did not significantly predict the baseline discount rates.
In summary, despite a minority of studies reporting negative findings (Kirby and Petry, 2004; MacKillop et al., 2007; Fernie et al., 2010), monetary discount rates (and in some studies discount rates for alcohol) show robust relationships with alcohol intake over a wide range of usage, being higher in currently dependent individuals, where they correlate with the degree of dependence, and predicting use in non-dependent individuals.
Discounting and illicit substance misuse
Amongst health behaviors, illicit substance misuse exhibits the most consistent relationships with discount rates. In an early study, heroin dependent individuals exhibited monetary discount rates twice those of non-drug-using controls (Kirby et al., 1999). Several other studies have also demonstrated significantly higher monetary discount rates in opioid-dependent individuals compared with controls (Madden et al., 1997; Kirby and Petry, 2004). Monetary discount rates are also higher in users of stimulant drugs such as cocaine and methamphetamine than in non-drug-using controls (Moeller et al., 2002; Coffey et al., 2003; Kirby and Petry, 2004; Heil et al., 2006; Hoffman et al., 2006; Monterosso et al., 2007; Johnson, 2012), with one study finding significantly higher monetary discount rates among individuals primarily using crack cocaine than among those primarily using heroin (Bornovalova et al., 2005). Indeed, MacKillop et al. (2011) estimated a large and highly significant aggregate effect (Cohen's d = 0.87, p < 0.0001) across studies comparing discount rates in dependent users of stimulant drugs versus controls and a moderate highly significant effect across studies comparing discount rates in opiate dependent individuals versus controls (Cohen's d = 0.76, p < 0.0001).
Consistent with nicotine abstinence studies, mild opioid deprivation in opioid dependent individuals increases discounting for money as well as heroin (Giordano et al., 2002), whereas those who have achieved longer term abstinence from heroin have lower discount rates than those currently addicted (Bretteville-Jensen, 1999; Kirby and Petry, 2004). By contrast abstinent formerly dependent cocaine users do not differ in discounting behavior from current users (Kirby and Petry, 2004; Heil et al., 2006) while Hoffman et al. (2006) found no relationship between length of abstinence and monetary discount rates in amphetamine dependent individuals. These findings suggest that either discount rates do not predict abstinence from stimulants, or that addiction to stimulants can have an irreversible effect to increase discount rates. Evidence against the former suggestion is that baseline discounting has been shown to predict the duration of abstinence from cocaine under a contingency management intervention (with low-incentives but not with high-incentives) (Washio et al., 2011).
Monetary discounting has not been consistently associated with concurrent cannabis use. Johnson et al. (2010) found that discount rates for hypothetical money in a group of marijuana dependent individuals did not differ from non-drug using controls, despite their study being adequately powered to detect any such difference (also see Stea et al., 2011). Similarly Heinz et al. (2013) found that monetary discounting did not correlate significantly with frequency of cannabis use over a 90 day period, although higher discounting was associated with younger age at first cannabis use. A recent study has shown that discount rates for hypothetical large monetary amounts ($1000) prospectively predicted abstinence outcomes amongst adolescents undergoing treatment for marijuana dependence (Stanger et al., 2012), more recent studies have (Heinz et al., 2013; Peters et al., 2013) found that discount rates did not predict response to a similar intervention in adults.
Finally studies have demonstrated an additive effect of smoking and alcohol use on discounting (Moallem and Ray, 2012; see also Andrade et al., 2013) but not of smoking and other forms of substance misuse (Businelle et al., 2010), and the combination of gambling problems and substance misuse appears highly predictive of impulsive choice (Petry and Casarella, 1999; Petry, 2001; however see Ledgerwood et al., 2009). In summary, with the exception of cannabis use, monetary discount rates consistently show strong correlations with the use of illicit substances.
Discounting, obesity, and eating behavior
Researchers have examined relationships between obesity and discounting for both food and money outcomes, citing similarities between eating behavior and addiction. Obese children have been shown to choose immediate over delayed edible rewards more often than normal weight children, though the effects were small and not found for non-food rewards (Johnson et al., 1978; Bonato and Boland, 1983). Notably the ability to delay gratification for food rewards at aged four predicts the likelihood of being overweight at aged 11 (Seeyave et al., 2009). In addition, cross-sectional studies have examined links between measures of monetary discount and Body Mass Index (BMI) in adults, with mixed findings (Epstein et al., 2003; Borghans and Golsteyn, 2006; Nederkoorn et al., 2006; Reimers et al., 2009; Ikeda et al., 2010). In a large sample from the Netherlands financial proxies for the discount rate, for example reported under-saving or excessive expenditure, were significantly correlated with BMI, however measured discount rates themselves were not (Borghans and Golsteyn, 2006). Similarly, Ikeda et al. (2010) found that BMI was positively correlated with a survey measure of procrastination, but showed no correlation with monetary discount rates in a sample of 2987 Japanese adults. In group comparisons, obese women have been shown to exhibit significantly higher discount rates than healthy weight women (Weller et al., 2008), and people who smoke cigarettes who are also obese to exhibit higher rates than non-obese smokers (Fields et al., 2011). Davis et al. (2010) found that obese women with a binge-eating disorder, but not obese women without binge-eating disorder, had significantly higher monetary discount rates than normal weight women. It has been suggested that sensitivity to food rewards interacts with delay discounting, in support of which high discount rates predict palatable food intake amongst normal weight women who find palatable foods highly rewarding (Rollins et al., 2010), an effect which has been replicated in obese and overweight women (Appelhans et al., 2011). More recently Kulendran and colleagues found significantly higher monetary discount rates in obese adolescents compared with normal-weight adolescents (Kulendran et al., 2013a), and demonstrated that monetary discount rates in obese adolescents decreased over the course of a residential obesity intervention (Kulendran et al., 2013b). Taken together these studies suggest an emerging relationship between discounting and weight status, although further work is clearly required to establish whether particular aspects of eating behavior, such as caloric intake, or eating frequency show relationships with discounting.
Discount rates and preventive health behavior
While some studies find relationships between discounting and preventive health behaviors, the findings are less consistent than for addictive behaviors. As described above, Chapman and Coups (1999) asked whether discount rates for monetary losses, as well as for a flu-like illness, could explain uptake of influenza vaccinations, with the finding that time preferences for money, but not illness, were related to vaccine uptake. In a later study (Chapman et al., 2001) monetary discounting showed an absent or very weak correlation with compliance with anti-hypertensive or cholesterol lowering medication. Similarly, a meta-analysis (Chapman, 2005) of 16 existing studies, including those described above, found no significant correlation between discounting and preventive health behavior (Mean Pearson r = 0.04, 95% CI = −0.01, 0.09). These studies suggest that in the population as whole preventive health behaviors show little or no relationship with discounting. However, two subsequent studies suggest that a subset of the highest discounters diverge from the rest of the population in their patterns of preventive health behavior. Firstly, Axon et al. (2009), studying 422 hypertensive adults, found that those in the highest quintile of monetary discount rates reported that they would be less likely to alter their diet and exercise plans to improve their future health. The highest discounters were not however significantly less likely to check their blood pressure or to follow their doctors' plans, as assessed by self-report. Secondly, Bradford (2010), analyzing discounting in 978 adults, found that for high discounting women the implied probability of attending mammography was reduced by 15.30% over the preceding 2 years and high discounting men had significantly lower rates of prostate examination (probability reduction 8.31%). The influence of discounting on attendance for cervical cancer screening was marginally significant. Across gender, high discounters were significantly less likely to have attended the dentist (probability reduction of 24.8%) or to have had any cholesterol testing (probability reduction 12.38%) or any influenza vaccination (probability reduction 11.05%) over the preceding 2 years. Additionally, high discounters were significantly less likely to be non-smokers or to have undertaken weekly vigorous activity. These studies suggest that monetary discount rates might be a useful tool for identifying groups at risk of failing to engage in preventive health practices.
Discount rates and risky sexual behavior or drug-taking practices
Convergent evidence associates high monetary discount rates with behaviors that increase the risk of contracting sexually-transmitted or blood-borne viral infections. Individuals infected with hepatitis C exhibit higher rates of discounting than controls (Huckans et al., 2010), although the direction of causality cannot be established from this study. Higher discount rates are associated with needle-sharing amongst heroin users (Odum et al., 2000). Dierst-Daviese t al. (2011) found that a sample of homeless, men who abused substances and had sex with men, had higher discount rates than a control sample of men, deemed to be at lower risk of HIV, who had sex with men however had stable housing and did not abuse substances. Finally, Chesson et al. (2006) found relationships between monetary discounting and a spectrum of sexual behaviors and outcomes in a combined sample of university students and adolescents attending clinics (N = 1042). For example, adolescents with higher discount rates were more likely to have had sexual intercourse before age 16 years, to have contracted gonorrhea or chlamydia, or to have become pregnant.
Discount rates and multiple health behaviors
Two studies in our search sample compared discount rates with a broad range of health behavior. Firstly Daugherty and Brase (2010), collected data from 467 undergraduates on an inventory of health behaviors, namely tobacco, alcohol and drug use, number of visits to a doctor or dentist in the past year, exercise frequency, eating breakfast, seat-belt use when in a vehicle, motorbike or bicycle helmet use, and the use of sunscreen. They found that, in a two-step hierarchical regression analysis, a combination of delay discounting for hypothetical money and survey measures of time perspective explained a significant proportion of the overall variance in health behavior over and above the combination of the respondents' gender and their personality type (Costa and McCrae, 1990). At the level of predicting individual behaviors, the improvement in model fit achieved by adding the time preference measures at the second step was small (the largest improvement in R2 was 0.05) but significant for all the behaviors above except helmet-wearing. Secondly Melanko and Larkin (2013) analyzed data from 72 young adults who performed both a discounting task with real monetary rewards and a hypothetical monetary discounting task as well as completing a Healthy Lifestyle Questionnaire (Corbin et al., 2006), assessing a variety of health behaviors, including smoking, alcohol use, substance misuse, physical activity, nutrition, avoiding destructive habits or practicing safe sex. In a hierarchical multiple regression, implied monetary discount rates for real rewards emerged as a significant predictor of the overall variance in health behavior. With regard to specific health behaviors, discounting for real monetary rewards emerged as a significant predictor of only smoking and nutrition scores. Notably however neither Daugherty and Brase (2010) nor Melanko and Larkin (2013) separated individuals by their level cigarette, alcohol or drug use. As a result the observed relationships between discounting and other behaviors may have been confounded by the effects of these addictive behaviors to increase impulsivity in other domains.
Conclusions: monetary discounting predicts unhealthy behavior
Taken as a whole, the studies reviewed here support the hypothesis that high discount rates for money, and in specific instances food or drug rewards, are associated with many unhealthy behaviors. Furthermore the effect sizes reported compare favorably to existing social cognitive models of health behavior (Conner and Norman, 2005), establishing high discounting as a reliable correlate of unhealthy choice.
The majority of studies reviewed above are cross-sectional and are therefore indeterminate as to whether high discounting antecedes unhealthy behavior, or vice versa. Two longitudinal studies reviewed here demonstrate that monetary discounting can prospectively predict onset of unhealthy behavior or relapse after health behavior change (Yoon et al., 2007; Audrain-McGovern et al., 2009). However, there is also considerable evidence that discounting is influenced by state-based factors (Koffarnus et al., 2013). In addition, for addictive behaviors, the severity of addiction is positively correlated with discounting (for example Mitchell et al., 2005; Sweitzer et al., 2008; MacKillop et al., 2010), and discount rates have been shown to decrease following behavior change (Landes et al., 2012). These observations combine to suggest that discounting can be viewed as a concurrent marker of the extent of unhealthy behavior, rather than exclusively as an anteceding risk factor. Consistent with either interpretation, a growing number of studies have shown that monetary discounting predicts response to behavior-change interventions (Dallery and Raiff, 2007; Krishnan-Sarin et al., 2007; MacKillop and Kahler, 2009; Mueller et al., 2009; Washio et al., 2011; Sheffer et al., 2012; Stanger et al., 2012; Brown and Adams, 2013). Thus discounting has clear predictive utility and may allow health-behavior change interventions to be tailored to benefit at-risk groups. In addition, interventions may be targeted at modifying the cognitive mechanisms associated with discounting, which are assumed to contribute to unhealthy behavior. For example working memory training has been shown to both reduce discount rates and modify addictive behavior (Bickel et al., 2011).
However, in order for discounting to be upheld as a mechanism for unhealthy choice, the features of unhealthy behavior must also be consistent with the predictions of a particular model of discounting. We test this by examining the proposal that hyperbolic discounting can explain goal-incongruent unhealthy action. We conclude that hyperbolic discounting on its own has explanatory shortcomings, and might be usefully supplemented by a broader conceptual framework.
Does hyperbolic discounting explain goal-incongruent unhealthy actions?
The observation that hyperbolic functions consistently provide excellent fits to choices between delayed rewards has led to the suggestion that hyperbolic discounting may be a guiding computational principle of intertemporal choice (Ainslie, 1975, 2001). In particular, since the curvature of the hyperbolic function predicts myopic preference reversals (Figure 1), hyperbolic discounting has been widely proposed as a an explanation for impulsive reward-seeking at the expense of long-term plans (for example Laibson, 1997; Ainslie, 2001; Angeletos et al., 2001; Bickel et al., 2007, 2012). Taken together with the observation that hyperbolic discount rates correlate with many forms of unhealthy behavior (Supplementary Tables 1–5 indicate which of the above studies measured hyperbolic rates), hyperbolic discounting proffers to explain goal-incongruent unhealthy action. In this section we examine this hypothesis in more detail.
Figure 1
Evidence for and against hyperbolic discounting
A key prediction of hyperbolic discounting is that, where a smaller-sooner reward is preferred over a larger-later reward, adding sufficient delay to both sooner and later options ought to shift preference toward the larger-later reward (see Figure 1). Several studies have demonstrated evidence for this in conventional monetary discounting tasks (for example Green et al., 1994; Kirby and Herrnstein, 1995). Such preference reversals have also been demonstrated in choices with health-relevant outcomes. For example, Read and Van Leeuwen (1998) asked people whether they would prefer to receive in 1 week's time either a healthy snack (such as a piece of fruit) or an unhealthy snack (such as a chocolate bar). The same individuals were followed up and 1 week later they were offered an immediate choice between a healthy and an unhealthy snack. Respondents chose healthy snacks more frequently when the choice was made in advance compared to when the snacks were immediately available. If healthy behavior is considered to carry larger-later health rewards, this result is consistent with the hyperbolic discounting of delayed health.
Despite the above findings, subsequent studies have challenged the hyperbolic model of preference reversal (Read, 2001; Read and Roelofsma, 2003; Sayman and Öncüler, 2009; Kable and Glimcher, 2010; Read et al., 2012). For example Kable and Glimcher (2010) found that discount functions based on a choice set in which all options were delayed by a fixed amount had the same hyperbolic curvature as those based on the same choice set in which the sooner option always occurred immediately: in contrast to conventional hyperbolic discounting, in which all outcomes are evaluated relative to the present time, this “as-soon-as-possible” function does not predict impulsive preference reversals. Additionally, several longitudinal studies have tested the predictions of hyperbolic discounting in real-time using monetary outcomes, with mixed findings (Ainslie and Haendel, 1983; Sayman and Öncüler, 2009; Read et al., 2012). The earliest of these studies found support for the preference reversals predicted by hyperbolic discounting. In this study Ainslie and Haendel (1983) asked participants on a Monday to choose between smaller amount of (hypothetical) money on to be received on Friday and larger amount to be received the following Monday. Participants were offered the choice again on the Friday, this time between receiving the smaller amount (for real) immediately or the delayed amount on the coming Monday. Consistent with hyperbolic discounting, the most common pattern was a preference for the larger-later amount when choices were made in advance, but for the smaller sooner amount when this was immediate. However subsequent studies have not replicated this finding. (Sayman and Öncüler, 2009) found the opposite result using a design similar to that of (Ainslie and Haendel, 1983). Furthermore a recent study performed over several weeks using real monetary rewards showed that preference reversals toward choosing smaller-sooner amounts (that is, in the direction predicted by hyperbolic discounting) were not significantly more common than those in the opposite direction (Read et al., 2012). Importantly this was the case despite the participants displaying hyperbolic discounting in conventional “cross-sectional” choices.
In summary, the preference reversals of the form predicted by conventional hyperbolic discounting have hitherto not been consistently demonstrated with monetary outcomes. This suggests that the preference reversals underlying health-related choices (such as those in Read and Van Leeuwen, 1998) may result from peoples' inability to predict in advance the impact of motivational and environmental states on their future decision-making.
Goal-incongruent actions often result from state changes
The suggestion that intention-action discrepancies in health choice result exclusively from hyperbolic discounting (Ainslie, 2001) can also be questioned. Everyday experience suggests that people often abandon long-term plans in favor of immediate reward in response to environmental cues or changes in internal motivational state; for example, one might plan to abstain from eating dessert as part of a diet plan, but find it harder to resist when presented with a piece of cake (see for example Allan et al., 2010). Loewenstein (1996) has proposed that motivational drives and cues which elicit them, rather than hyperbolic discounting, are responsible for impulsive preference reversal. This idea is supported by evidence; for example, relapses in drug-taking behavior following abstinence commonly occur after exposure to a previous drug-taking environment (O'Brien et al., 1998). Hyperbolic discounting, to the extent that it applies as a model for preference reversal, may in some instances be sufficient to explain these cue-triggered behaviors, since cues provide information about the timing of outcomes, thereby signaling that reward is at hand. However, hyperbolic discounting does not appear necessary to explain these state-dependent influences. For example, in a study of analgesic preferences for childbirth (Christensen-Szalanski, 1984), women asked roughly 1 month in advance of labor preferred to avoid invasive spinal anesthesia in favor of less invasive but less effective pain relief methods, however during active labor women frequently reversed preference and opted for anesthesia. These findings are easily explained by an increase in the marginal utility for anesthesia during the painful state, which was not accurately predicted in advance, without reference to hyperbolic discounting.
In summary therefore not all forms of health-related preference reversal are consistent with hyperbolic discounting, and many preference reversals occur in response to changes in motivation or environmental states. Taken together with the lack of clear longitudinal evidence for the myopic preference reversals predicted by hyperbolic discounting, this suggests that any generative model of intertemporal health choice ought to be expanded beyond hyperbolic discounting alone to account for the effects of environmental cues. At best hyperbolic discounting alone provides no explicit framework either for incorporating the motivational information provided by environmental cues, or for how this information becomes associated with cues through learning. Existing models for these influences have not aimed to provide a mechanistic level of interpretation (Loewenstein, 1996). The following discussion advances a mechanistic framework based on the principles of reinforcement learning for understanding the effects of environmental cues on intertemporal health choice. Key to this account is the notion that cues previously associated with rewarding actions can trigger goal-incongruent habits, leading to preference reversal even in the absence of hyperbolic discounting. A full exploration of learning is beyond the scope of this review and we therefore restrict ourselves to the effects of cues after learning has taken place.
A reinforcement learning approach to cue-triggered preference reversal
Reinforcement learning provides an approach to understanding intertemporal choice. Models of reinforcement learning posit that action control proceeds by estimating the expected value of ensuing reward over series of temporally connected future states, encapsulated in a state-action value function (Sutton and Barto, 1998); such models are therefore well placed to incorporate changes in state on choice behavior. Attempting to optimize value in changing environments can be considered a trade-off between flexibility in rapidly incorporating new information and the efficient use of past experience (Daw et al., 2005). This trade-off is embodied by two methods of learning action-value: a rather rigid, but computationally lean method, referred to as model-free, and a flexible, planning method capable of simulating future possible outcomes, often referred to as model-based (Gläscher et al., 2010; Daw et al., 2011; McDannald et al., 2011; Daw, 2012; Wunderlich et al., 2012; Dolan and Dayan, 2013). These systems reflect an established distinction in psychology between deliberative and automatic processes (Evans and Stanovich, 2013), but endow this with a normative and explicitly computational basis (Daw et al., 2005).
A model-based decision-maker is generally assumed to search through the possible future states consequent on each action. Model-based decision-making corresponds to the definition of “goal-directed” behavior in animal learning experiments as rapidly sensitive to changes in outcome value or the contingency between response and outcome (Colwill and Rescorla, 1986; Dickinson and Balleine, 1994; Balleine and Dickinson, 1998; Domjan, 2003). A model-free decision-maker, by contrast, through a gradual integration of outcome values encountered through experience, assigns a scalar estimate of long-run future value to taking an action in a particular state, without explicitly representing the corresponding future state of the world. The resulting cached action values are relatively insensitive to immediate changes in the outcomes. Instrumental behavior is initially goal-directed (model-based), but becomes increasingly model-free with learning, such that actions eventually become insensitive to changes in the value of the outcome, acquiring the characteristic of habits (Dickinson et al., 1995; Ouellette, 1998; Neal, 2006). This bears direct analogy to economic models of habit formation, which modify the instantaneous utility function to depend on past consumption (Becker and Murphy, 1988).
The differential engagement of these two systems has the potential to explain the environmental dependence of the preference reversals which underlie many forms of unhealthy behavior. While steep temporal discounting over the model-based valuation of future health would be expected to encourage the initiation of unhealthy behavior, with repetition, unhealthy behavior is likely to become increasingly model-free, or habitual. At this stage, even if the decision-maker re-evaluates their goals in favor of making healthy choices, cached action values will continue to encourage unhealthy choice in response to relevant environmental cues, leading to apparently impulsive preference reversals (intention-action discrepancies).
Environmental cues can trigger goal-incongruent habits
Once a person has initiated an unhealthy behavior, they may later change their goals and form the intention to abstain from that behavior. For example, a person who smokes might decide to quit after being diagnosed with lung disease. However, if sufficient learning has taken place, the behavior might nevertheless be maintained as a stimulus-response habit under the dominant influence of cached action values. Thus, the smoker might find it particularly hard to resist when he or she spies the cigarette packet. As we outline below, the goal-incongruent influence of cached (habitual) action values can produce preference reversal, without invoking hyperbolic discounting. Furthermore, preference reversal can result even if each system in isolation exhibits exponential discounting and discounts the future at the same rate, a crucial distinction from dual-process models of quasi-hyperbolic discounting (Laibson, 1997; Angeletos et al., 2001; McClure et al., 2004; Bickel et al., 2012; Koffarnus et al., 2013). To demonstrate this formally, consider a decision-making agent for whom overall action value is a weighted average of the value from each controller, where both systems discount the future exponentially with a per period rate, γ (γ is the conventional symbol for the discount rate in reinforcement learning approaches; its meaning is equivalent to that of δ in Equation 1). Say, for example, the agent is a person following a diet plan who is choosing whether or not to consume a calorific biscuit when faced with a cue, the biscuit tin. A simplified (semi-Markov) state space for this decision is depicted in Figure 2A. State B represents the presence of the biscuit tin. Consuming biscuits leads after a short delay, dc, to state C, which carries reward, Rc, and after a longer delay, dh, to maintaining current weight, for simplicity here assigned a reward value of zero. Abstaining from biscuits leads, via the unrewarded state, A, to a health benefit in the form of weight loss, Rh, after delay dh. Notably this is a radical simplification of reality. It is assumed that, while the model-based system is capable of making such simplifications based on declarative knowledge, the model-free system cannot, and has never experienced the health consequences. As a result, the model-free system has learned the values of each action in state B (termed “Q-values”) based solely on the reward previously provided from consumption (Figure 2B). It is assumed that the model-based system is initially naive to these cached values. Consider that the agent, after learning, is asked to make their decision when situated in state P, at some time delay, p in advance of state B, where cached values have no influence, and that here they are indifferent between indulging and abstaining, that is to say that the model-based value, QMB, of consumption is equal to that of abstention:
Figure 2
Given by:
Which simplifies to:
On reaching B, the presentation of the biscuit tin, cached action values are also “brought online,” incrementing the benefit of indulging, such that:
Given by:
And therefore, by (3c):
Predicting a preference for indulging for ω < 1 (Figure 2C). Therefore the presentation of the biscuit tin brings about a preference for sooner consumption. In economic terms, environmental cues such as the biscuit tin might be viewed as updating the utility of the immediately available option, by providing (previously inaccessible) information from prior experience.
The interplay between model-based and model-free systems in the account above bears some similarity to existing dual-systems models of intertemporal choice, which posit a deliberative planning system in opposition with an impulsive system. However, while the former are often mapped onto quasi-hyperbolic models of discounting (McClure et al., 2007, 2004), which combine two exponential discount functions with differing rates, here the two systems may share the same discount rate. Dynamic inconsistency can then result from the different sources of information available to either controller (also see Dayan et al., 2006). In particular, the state-dependent valuations of the cached system can explain why real-world preference reversals occur in response to learned cues and, unlike existing quasi-hyperbolic accounts, why these preference reversals become more prominent with the formation of habits. In addition, unlike existing dual-process accounts, reinforcement learning models can explicitly model the learning process generating incongruent preferences. (A direct treatment of learning is beyond the scope of this review).
Exponential discounting is used here to illustrate that hyperbolic discounting is not necessary to predict preference reversals, although clearly hyperbolic discounting is more consistent with cross-sectional intertemporal choice data than exponential discounting. The framework above could readily incorporate hyperbolic discounting, and several authors have demonstrated reinforcement learning models which produce hyperbolic discounting (Daw and Touretzky, 2000; Tsitsiklis and Van Roy, 2002; Kalenscher and Pennartz, 2008; Kurth-Nelson and Redish, 2009; Alexander and Brown, 2010).
Model-based and model-free interactions in addiction
The relative contributions of model-based and model-free strategies might in part explain why discount rates correlate particularly strongly with addictive behaviors (Keramati et al., 2012; Lucantonio et al., 2014). Steep discount rates would putatively favor initial goal-directed drug-seeking behavior (as with other forms of unhealthy behavior). The high rewards provided by substances of abuse might then lead to rapid habitization of drug taking behavior by comparison with other repeated behaviors (Everitt and Robbins, 2005; Everitt et al., 2008; Lucantonio et al., 2014), effectively binding impulsive individuals to their initial choices. In addition, repeated choice of immediately available rewards by individuals with high discount rates would be expected to lead to these individuals acquiring habits more rapidly (by more reinforced choices). In support of this, animal studies of addiction demonstrate that rats bred to exhibit steeper delay discounting more rapidly acquire compulsive self-administration of cocaine than their low discounting counterparts (Belin et al., 2008). Finally, chronic addiction may further shift responding toward model-free control (Keramati et al., 2012), in part by damaging frontal cortical areas on which model-based valuations are thought to depend (Rogers and Robbins, 2001; Gläscher et al., 2010; Camchong et al., 2011; Smittenaar et al., 2013), further decreasing the capacity to exert model-based control over goal-incongruent habits. Although these mechanisms most likely play in a role in addiction, there is an ongoing debate as to their precise contribution, and in particular the interplay between habitual mechanisms and classical (Pavlovian) conditioning in addictive disorders (Everitt and Robbins, 2005).
Future directions
The studies reviewed here indicate that discounting is a promising predictor of health behavior, however hyperbolic discounting is challenged as an explanation for the discrepancy between intentions and actions in health choice, and a framework based on the trade-off between model-free and model-based action control appears better placed to incorporate the influences of environment and learning. Nevertheless the study of myopic health-related decision-making remains nascent. Further work is required firmly establish discounting as a predictive tool, to extend the measurement framework and to develop novel interventions capable of reducing goal-incongruent health choice.
Discounting as an individual difference measure
The endeavor to predict and understand health behavior through comparison with discounting measures forms part of a wider paradigm to characterize individual differences in field behavior using decision-making tasks (Montague et al., 2012). A question relevant to this endeavor is the extent to which discounting can be considered as a either a personality trait or a state variable (de Wit, 2008; Odum, 2011; Bickel et al., 2012). Personality traits are defined as stable and enduring characteristics, reflecting a general tendency to respond in a given manner under given circumstances and can be seen to represent persistent patterns of internal states (see Costa and McCrae, 1990). State variables by contrast vary over a shorter time scale, such that their rank ordering between individuals may be altered with changes in the motivational state of the respondents and/or the elicitation conditions (Kraemer et al., 1993).
Several pieces of evidence reviewed here demonstrate that discounting has a state-based component. Firstly studies comparing the discounting of hypothetical health with that of money find that discounting varies with the domain and valence of the outcome, in a manner that changes the rank ordering of discount rates between individuals. Secondly discount rates amongst substance misusers are greater in a state of drug-craving than in a drug-sated state. Thirdly, even under conditions of drug-satiety, addiction appears to be associated with a reversible increase in discounting. These findings are supported by a wealth of additional evidence showing that discounting can be manipulated through contextual framing (see Koffarnus et al., 2013 for a review). Indeed from a normative perspective it is sensible for agents to adjust their tolerance of delay to match environmental conditions; for example steep discounting is adaptive in an environment where delayed rewards are highly uncertain to be received (see Lahav et al., 2011).
However there is also evidence that discounting has attributes of trait variable. The test-retest reliability of monetary discounting is substantial at intervals of up to 1 year (Pearson r = 0.71; Kirby, 2009) and across different elicitation methods (Odum, 2011). Furthermore, whilst monetary discounting is poorly correlated with hypothetical health discounting across individuals monetary discount rates are strongly and significantly correlated with other forms of appetitive outcome, such as the discounting of cigarettes for cigarette smokers, the discounting of heroin for opioid-dependent outpatients and the discounting of food amongst college students (Odum, 2011; Pearson r = 0.93; p = 0.0007 for money versus the mean of all other outcomes). There is also evidence that discounting is heritable (see MacKillop, 2013 for a review). A recent longitudinal twin study estimated the heritability of delay discounting in adolescence at up to 50% (Anokhin et al., 2011), rats and mice can be bred to exhibit greater degrees of delay discounting (e.g., Anderson and Woolverton, 2005; Belin et al., 2008), and steeper discounting in humans is associated with specific polymorphisms related to dopamine signaling (Eisenberg et al., 2007). Also commensurate with discounting as an enduring trait, steeper discounting is associated with lower socio-economic status (e.g., Bradford, 2010; Anokhin et al., 2011). In summary, discounting for appetitive outcomes has features of a trait marker.
Trait-level differences in discounting can be viewed as long-term adaptations to prevailing environmental conditions, shaped either through learning or inheritance (this notion is consistent with a branch of evolutionary theory termed Life History Theory; Del Giudice et al., 2013; Del Giudice and Ellis, 2014). An important direction for future research will be to examine the relative contributions of genes and childrearing environment to discounting. The study of self-regulation in developmental psychology has adopted this approach; for example children who experience emotionally close, sensitive, and responsive caregiving have been found to exhibit higher levels of self-regulation (Belsky et al., 2007), and Berry et al. (2013) find evidence that self-regulation ability appears to be more sensitive to early childcare experiences in a group with a particular dopamine receptor polymorphism. Furthermore low childhood self-regulation has been prospectively related to poorer health outcomes later in life (for example Francis and Susman, 2009; Seeyave et al., 2009; Moffitt et al., 2011). Future research into delay discounting would benefit from a similar developmental perspective to better understand the origins of trait-level individual differences.
In conclusion discount rates are far from immutable, and are sensitive to environmental and motivational conditions. However, discounting for appetitive outcomes is stable across individuals when measured under similar conditions, is partly heritable and is associated with a range of similar constructs, and as a result has the potential to provide an endophenotype which mediates between genetic influences, more fundamental neuro-computational processes and maladaptive patterns of impulsive behavior in the real-world (MacKillop, 2013). Further work is required to more completely characterize the relationships between these levels of analysis. We have proposed that discounting is best considered within a broader framework for understanding choice between temporally extended outcomes, based on the theory of reinforcement learning. We have shown how the interaction between model-based and model-free value estimates may contribute to real-world instances of goal-incongruent unhealthy choice. However several important questions remain largely unanswered. For example, can the balance of model-based versus model-free control be measured, and can such measures be used to predict health-related behavior? Is there a trait component to this balance? What is the relationship between model-based control and measured discount rates, or related metrics of self-regulation? We briefly address these questions in turn below.
Measuring model-based and model-free interactions
One approach to measuring the interaction between model-based and model-free decision-making is to directly observe the acquisition of habitual behavior through repeated training on a given laboratory task. Here, the rate of acquisition of habitual responding may offer a novel measure for predicting field behavior. Using this approach, outcome-insensitive habits have been demonstrated in humans (Tricomi et al., 2009), providing a behavioral counterpart to studies of habit formation which measure subjective automaticity (Lally et al., 2010). An important aim for future studies will be to examine habitual or cue-triggered preference reversals in real-time. Along these lines, subjective measures of habitual automaticity in relation to smoking behavior have been shown to predict goal-incongruent smoking-related responses (Orbell and Verplanken, 2010). However, observing habit learning directly is time-consuming. Recent human studies (Gläscher et al., 2010; Daw et al., 2011; Eppinger et al., 2013; Smittenaar et al., 2013) have used a paradigm with a probabilistic tree structure which separates model-free and model-based control, before habitization has taken place, depending on whether respondents incorporate the transition structure of the task into their learning (model-based) or learn solely based on the reinforcement obtained in each discrete state (model-free). Humans performing this task generally exhibit some combination of the two modes of control, and the relative contribution of the two strategies may provide a novel behavioral marker. Further studies are required to establish the longitudinal stability of these measures, and whether they have a trait component, as well as to examine their relationship with habitual behavior in the field.
Relationships between model-based control, discounting and self-regulation
Responding on discounting paradigms cannot easily be considered habitual, and most likely requires model-based processes. Nevertheless, we propose that directly representing outcomes during choices on discounting paradigms, rather than relying on a low-level tradeoff between amount and delay, is likely to be associated with more future-oriented responses. In line with this suggestion, mentally simulating future outcomes decreases measured discount rates (Peters and Büchel, 2010) and lesioning neural structures on which this simulation process depends, such as the hippocampus (Hassabis et al., 2007; Johnson et al., 2007a; Schacter et al., 2008) increases discounting (Mariano et al., 2009). Furthermore, existing studies suggest that the choice of delayed rewards, model-based control and working memory engage overlapping neural substrates: neuroimaging studies have found that the dorsolateral prefrontal cortex (dlPFC) is activated in both model-based learning (Gläscher et al., 2010), and in choosing delayed rewards on intertemporal choice paradigms (McClure et al., 2004, 2007), while disrupting this area (using either transcranial magnetic stimulation or transcranial direct current stimulation) both decreases model-based behavior (Smittenaar et al., 2013) and increases temporal discounting (Hecht et al., 2013). A recent study has also demonstrated that in younger adults, but not in older adults, a greater degree of model-based behavior is associated with higher working memory capacity (Eppinger et al., 2013). The finding that working memory training decreases discounting among substance misusers (Bickel et al., 2011) is especially interesting in this regard. A possible unifying interpretation is that explicitly representing the future consequences of action, a process associated with model-based decision-making, produces more future-oriented choice and hence lowers discount rates (see also Peters and Büchel, 2010) and that this process is also limited by working memory capacity. Notably the exercise of model-based control is similar to existing definitions of self-regulation, as “the largely (but not exclusively) volitional act of managing attention and arousal in a manner that facilitates goal-directed behavior” (Berry et al., 2013, p. 2). An advantage of the reinforcement learning approach is its ability to formalize such behavior within a normative computational framework. It is important to reiterate here that, whilst we view model-based valuations as supporting future-oriented choice, we do not identify the model-free controller with an “impulsive system.” In our view both controllers share the same fundamental goals and it is the relative inflexibility of model-free decision-making which gives its responses their short-sighted character (Dayan et al., 2006).
Novel behavioral predictors and interventions
Several additional approaches may yield novel behavioral markers of unhealthy choice. Unlike the naïve decision maker described above, people often demonstrate that they can predict their future tendencies, termed sophistication, for instance by choosing paths that remove their opportunity to make myopic choices, an activity referred to as pre-commitment (Ainslie, 2001; Ariely and Wertenbroch, 2002; Prelec and Bodner, 2003). For instance, a person attempting to abstain from smoking might throw away their cigarette packets. Pre-commitment would be expected to obscure real-world relationships between discount rates and myopic behavior, since at least a subset of sophisticated steep discounters would exhibit far–sighted real–world choices. Furthermore, within the model-based versus model-free framework above, we propose that the ability to predict and therefore pre-empt the influence of state changes on one's behavior is a key substrate of self-control. Economic theories of pre-commitment, often based on quasi-hyperbolic discounting, provide a useful conceptual framework for predicting the effects of varying degrees of sophistication on behavior (O'Donoghue and Rabin, 2003), and computational models of these processes (Kurth-Nelson and Redish, 2010) offer the potential to enrich predictions of health behavior.
An additional important influence not considered above is the effect of internal motivational states. The effects of motivational state on habitual responding are complex, having immediate effects on the vigor of responding, while having effects on choice by altering the utility of outcomes (see Niv et al., 2007). Models which formalize these effects remain the subject of ongoing theoretical work, though may eventually provide a valuable substrate for applied health behavior research.
Although not discussed in detail here, since many health-promoting behaviors are to a degree aversive, measures of dread (Berns et al., 2006; Story et al., 2013) might form a predictor of engagement in such behaviors. A complexity tending to preclude clear a priori predictions in this area is that, if dreading aversive health-promoting behaviors were to promote their avoidance (e.g., Kleinknecht, 1978), then dreading illness would be expected to have the opposite effect, promoting engagement in such behaviors. Perhaps reflecting these competing influences, Chapman and Coups (1999) report that rates of vaccination uptake were not significantly higher in individuals with negative time preference for illness, as compared to individuals with positive time preference for illness.
Finally, the account above focuses on instrumental learning. However, there is also evidence that animals use state-state, as well as state-action-state, associations to guide action. This third mode of learning, embodied by classical conditioning, is referred to as Pavlovian learning (Domjan, 2003). Based on state-state predictions, the Pavlovian controller initiates stereotyped actions directed toward obtaining predicted rewards. Crucially, unlike instrumental control, Pavlovian actions are initiated regardless of whether or not they lead to reward (Williams and Williams, 1969). The precise contributions of instrumental and Pavlovian effects to real-world choices are difficult to distinguish. Nevertheless the mechanism of choice inconsistency proposed above for the case of model-based and model-free interactions would remain largely equivalent for the case of interactions between model-based and Pavlovian decision-making (Dayan et al., 2006). An advantage of the reinforcement learning approach is its ability to generate simulations of these interactions over the course of learning and such models may yield parameters capable of explaining further variance in health behavior.
Novel interventions might be directed at specific constructs within the above framework, and indeed several existing health behavior interventions can be viewed in this manner. For example strategies aimed at making healthy choices habitual are already known to be effective (Lally et al., 2007). There is an urgent requirement for novel interventions capable of reducing goal-incongruent unhealthy choice, since the increasing burden of disease attributable to unhealthy behavior is placing unsustainable demands on existing healthcare systems (Smith et al., 2012). We propose that the identification of health decision-making phenotypes will play an important role in evaluating and optimizing the necessary interventions.
Statements
Author contributions
All authors (Giles W. Story, Ivo Vlaev, Ben Seymour, Raymond J. Dolan, and Ara Darzi) contributed to the conception of the work. GS performed systematic review and drafted the work. All authors (Giles W. Story, Ivo Vlaev, Ben Seymour, Raymond J. Dolan, and Ara Darzi) were involved in revising the work critically for important intellectual content. All authors (Giles W. Story, Ivo Vlaev, Ben Seymour, Raymond J. Dolan, and Ara Darzi) gave final approval to the version of the manuscript being published and agree to be accountable for all aspects of the work.
Acknowledgments
We would like to thank Zeb Kurth-Nelson and Peter Smittenaar for their helpful comments. This work is supported by a Wellcome Trust Senior Investigator Award to Raymond J. Dolan (098362/Z/12/Z). The Wellcome Trust Centre for Neuroimaging is supported by core funding from the Wellcome Trust 091593/Z/10/Z.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Supplementary material
The Supplementary Material for this article can be found online at: http://www.frontiersin.org/journal/10.3389/fnbeh.2014.00076/abstract
References
1
AinslieG. (1974). Impulse control in pigeons. J. Exp. Anal. Behav. 21, 485–489. 10.1901/jeab.1974.21-485
2
AinslieG. (1975). Specious reward: a behavioral theory of impulsiveness and impulse control. Psychol. Bull. 82, 463–496. 10.1037/h0076860
3
AinslieG. (2001). Breakdown of Will. Cambridge: Cambridge University Press. 10.1017/CBO9781139164191
4
AinslieG.HaendelV. (1983). The motives of the will, in Etiologic aspects of alcohol and drug abuse, eds GottheilE.DruleyK. A.SkodolaT. E.WaxmanH. M. (Springfield, IL: Charles C. Thomas), 119–140.
5
AlexanderW. H.BrownJ. W. (2010). Hyperbolically discounted temporal difference learning. Neural Comput. 22, 1511–1527. 10.1162/neco.2010.08-09-1080
6
AllanJ. L.JohnstonM.CampbellN. (2010). Unintentional eating. What determines goal-incongruent chocolate consumption?Appetite54, 422–425. 10.1016/j.appet.2010.01.009
7
AndersonK. G.WoolvertonW. L. (2005). Effects of clomipramine on self-control choice in Lewis and Fischer 344 rats. Pharmacol. Biochem. Behav. 80, 387–393. 10.1016/j.pbb.2004.11.015
8
AndradeL. F.AlessiS. M.PetryN. M. (2013). The effects of alcohol problems and smoking on delay discounting in individuals with gambling problems. J. Psychoactive Drugs45, 241–248. 10.1080/02791072.2013.803645
9
AngeletosG.-M.LaibsonD.RepettoA.TobacmanJ.WeinbergS. (2001). The Hyperbolic consumption model: calibration, simulation, and empirical evaluation. J. Econ. Perspect. 15, 47–68. 10.1257/jep.15.3.47
10
AnokhinA. P.GolosheykinS.GrantJ. D.HeathA. C. (2011). Heritability of delay discounting in adolescence: a longitudinal twin study. Behav. Genetics41, 175–183. 10.1007/s10519-010-9384-7
11
AppelhansB. M.WoolfK.PagotoS. L.SchneiderK. L.WhitedM. C.LiebmanR. (2011). Inhibiting food reward: delay discounting, food reward sensitivity, and palatable food intake in overweight and obese women. Obesity19, 2175–2182. 10.1038/oby.2011.57
12
ArielyD.WertenbrochK. (2002). Procrastination, deadlines, and performance: self-control by precommitment. Psychol. Sci. 13, 219–224. 10.1111/1467-9280.00441
13
Asenso-BoadiF.PetersT. J.CoastJ. (2008). Exploring differences in empirical time preference rates for health: an application of meta-regression. Health Econ. 17, 235–248. 10.1002/hec.1256
14
Audrain-McGovernJ.RodriguezD.EpsteinL. H.CuevasJ.RodgersK.WileytoE. P. (2009). Does delay discounting play an etiological role in smoking or is it a consequence of smoking?Drug Alcohol Depend. 103, 99–106. 10.1016/j.drugalcdep.2008.12.019
15
AxonR. N.BradfordW. D.EganB. M. (2009). The role of individual time preferences in health behaviors among hypertensive adults: a pilot study. J. Am. Soc. Hypertens3, 35–41. 10.1016/j.jash.2008.08.005
16
BakerF.JohnsonM. W.BickelW. K. (2003). Delay discounting in current and never-before cigarette smokers: similarities and differences across commodity, sign, and magnitude. J. Abnorm. Psychol. 112, 382–392. 10.1037/0021-843X.112.3.382
17
BalleineB. W.DickinsonA. (1998). Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology37, 407–419. 10.1016/S0028-3908(98)00033-1
18
BeckerG. S.MurphyK. M. (1988). A theory of rational addiction. J. Political Econ. 96, 675–700. 10.1086/261558
19
BelinD.MarA. C.DalleyJ. W.RobbinsT. W.EverittB. J. (2008). High impulsivity predicts the switch to compulsive cocaine-taking. Science320, 1352–1355. 10.1126/science.1158136
20
BelskyJ.Pasco FearonR. M.BellB. (2007). Parenting, attention and externalizing problems: testing mediation longitudinally, repeatedly and reciprocally. J. Child Psychol. Psychiatry48, 1233–1242. 10.1111/j.1469-7610.2007.01807.x
21
BenzionU.RapoportA.YagilJ. (1989). Discount rates inferred from decisions - an experimental-study. Manag. Sci. 35, 270–284. 10.1287/mnsc.35.3.270
22
BernsG. S.ChappelowJ.CekicM.ZinkC. F.PagnoniG.Martin-SkurskiM. E. (2006). Neurobiological substrates of dread. Science312, 754–758. 10.1126/science.1123721
23
BerryD.McCartneyK.PetrillS.Deater-DeckardK.BlairC. (2013). Gene–environment interaction between DRD4 7-repeat VNTR and early child-care experiences predicts self-regulation abilities in prekindergarten. Dev. Psychobiol. [Epub ahead of print]. 10.1002/dev.21105
24
BickelW. K.JarmolowiczD. P.MuellerE. T.KoffarnusM. N.GatchalianK. M. (2012). Excessive discounting of delayed reinforcers as a trans-disease process contributing to addiction and other disease-related vulnerabilities: emerging evidence. Pharmacol. Ther. 134, 287–297. 10.1016/j.pharmthera.2012.02.004
25
BickelW. K.KoffarnusM. N.MoodyL.WilsonA. G. (2014). The behavioral- and neuro-economic process of temporal discounting: a candidate behavioral marker of addiction. Neuropharmacology76(Part B), 518–527. 10.1016/j.neuropharm.2013.06.013
26
BickelW. K.MillerM. L.YiR.KowalB. P.LindquistD. M.PitcockJ. A. (2007). Behavioral and neuroeconomics of drug addiction: competing neural systems and temporal discounting processes. Drug Alcohol Depend. 90(Suppl. 1), S85–S91. 10.1016/j.drugalcdep.2006.09.016
27
BickelW. K.OdumA. L.MaddenG. J. (1999). Impulsivity and cigarette smoking: delay discounting in current, never, and ex-smokers. Psychopharmacology146, 447–454. 10.1007/PL00005490
28
BickelW. K.YiR.LandesR. D.HillP. F.BaxterC. (2011). Remember the future: working memory training decreases delay discounting among stimulant addicts. Biol. Psychiatry69, 260–265. 10.1016/j.biopsych.2010.08.017
29
BjorkJ. M.HommerD. W.GrantS. J.DanubeC. (2004). Impulsivity in abstinent alcohol-dependent patients: relation to control subjects and type 1–/type 2–like traits. Alcohol34, 133–150. 10.1016/j.alcohol.2004.06.012
30
BleichrodtH.JohannessonM. (2001). Time preference for health: a test of stationarity versus decreasing timing aversion. J. Math. Psychol. 45, 265–282. 10.1006/jmps.2000.1312
31
BobovaL.FinnP. R.RickertM. E.LucasJ. (2009). Disinhibitory psychopathology and delay discounting in alcohol dependence: personality and cognitive correlates. Exp. Clin. Psychopharmacol. 17, 51. 10.1037/a0014503
32
BoettigerC. A.MitchellJ. M.TavaresV. C.RobertsonM.JoslynG.D'espositoM.et al. (2007). Immediate reward bias in humans: fronto-parietal networks and a role for the catechol-O-methyltransferase 158(Val/Val) genotype. J. Neurosci. 27, 14383–14391. 10.1523/JNEUROSCI.2551-07.2007
33
BonatoD. P.BolandF. J. (1983). Delay of gratification in obese children. Addict. Behav. 8, 71–74. 10.1016/0306-4603(83)90059-X
34
BorghansL.GolsteynB. H. (2006). Time discounting and the body mass index. Evidence from the Netherlands. Econ. Hum. Biol. 4, 39–61. 10.1016/j.ehb.2005.10.001
35
BornovalovaM. A.DaughtersS. B.HernandezG. D.RichardsJ. B.LejuezC. (2005). Differences in impulsivity and risk-taking propensity between primary users of crack cocaine and primary users of heroin in a residential substance-use program. Exp. Clin. Psychopharmacol. 13, 311. 10.1037/1064-1297.13.4.311
36
BradfordW. D. (2010). The association between individual time preferences and health maintenance habits. Med. Decis. Making30, 99–112. 10.1177/0272989X09342276
37
Bretteville-JensenA. L. (1999). Addiction and discounting. J. Health Econ. 18, 393–407. 10.1016/S0167-6296(98)00057-5
38
BrownH.AdamsJ. (2013). The role of time preference in smoking cessation: a longitudinal analysis of data from the Household Income and Labour Dynamics of Australia survey, 2001-08. Addiction108, 186–192. 10.1111/j.1360-0443.2012.03997.x
39
BusinelleM. S.McVayM. A.KendzorD.CopelandA. (2010). A comparison of delay discounting among smokers, substance abusers, and non-dependent controls. Drug Alcohol Depend. 112, 247–250. 10.1016/j.drugalcdep.2010.06.010
40
CairnsJ. (1992). Discounting and health benefits: another perspective. Health Econ. 1, 76–79. 10.1002/hec.4730010111
41
CamchongJ.MacDonaldA. W.3rdNelsonB.BellC.MuellerB. A.SpeckerS.et al. (2011). Frontal hyperconnectivity related to discounting and reversal learning in cocaine subjects. Biol. Psychiatry69, 1117–1123. 10.1016/j.biopsych.2011.01.008
42
ChaoL. W. (2009). Time preference and its relationship with age, health, and survival probability. Judgm. Decis. Mak. 4, 1.
43
ChapmanG. B. (1996). Temporal discounting and utility for health and money. J. Exp. Psychol. Learn. Mem. Cogn. 22, 771–791. 10.1037/0278-7393.22.3.771
44
ChapmanG. B. (2002). Your money or your health: time preferences and trading money for health. Med. Decis. Making22, 410–416. 10.1177/027298902236929
45
ChapmanG. B. (2005). Short-term cost for long-term benefit: time preference and cancer control. Health Psychol. 24, S41–S48. 10.1037/0278-6133.24.4.S41
46
ChapmanG. B.BrewerN. T.CoupsE. J.BrownleeS.LeventhalH.LeventhalE. A. (2001). Value for the future and preventive health behavior. J. Exp. Psychol. Appl. 7, 235–250. 10.1037/1076-898X.7.3.235
47
ChapmanG. B.CoupsE. J. (1999). Time preferences and preventive health behavior: acceptance of the influenza vaccine. Med. Decis. Making19, 307–314. 10.1177/0272989X9901900309
48
ChapmanG. B.ElsteinA. S. (1995). Valuing the future: temporal discounting of health and money. Med. Decis. Making15, 373–386. 10.1177/0272989X9501500408
49
ChessonH. W.LeichliterJ. S.ZimetG. D.RosenthalS. L.BernsteinD. I.FifeK. H. (2006). Discount rates and risky sexual behaviors among teenagers and young adults. J. Risk Uncertainty32, 217–230. 10.1007/s11166-006-9520-1
50
Christensen-SzalanskiJ. J. (1984). Discount functions and the measurement of patients' values. Women's decisions during childbirth. Med. Decis. Making4, 47–58. 10.1177/0272989X8400400108
51
ChungS. H.HerrnsteinR. J. (1967). Choice and delay of reinforcement. J. Exp. Anal. Behav. 10, 67–74. 10.1901/jeab.1967.10-67
52
CoffeyS. F.GudleskiG. D.SaladinM. E.BradyK. T. (2003). Impulsivity and rapid discounting of delayed hypothetical rewards in cocaine-dependent individuals. Exp. Clin. Psychopharmacol. 11, 18–25. 10.1037/1064-1297.11.1.18
53
ColwillR. M.RescorlaR. A. (1986). Associative structures in instrumental learning, in Psychology of Learning and Motivation, ed GordonH. B. (Orlando, FL: Academic Press), 55–104.
54
ConnerM.NormanP. (2005). Predicting Health Behaviour. Maidenhead: Open University Press.
55
CorbinC. B.WelkG. J.CorbinW. R.WelkK. A. (2006). Concepts of Fitness and Wellness: A Comprehensive Lifestyle Approach, 6th Edn. New York, NY: McGraw-Hill.
56
CostaP. T.McCraeR. R. (1990). Personality disorders and the five-factor model of personality. J. Pers. Disord. 4, 362–371. 10.1521/pedi.1990.4.4.362
57
DalleryJ.RaiffB. R. (2007). Delay discounting predicts cigarette smoking in a laboratory model of abstinence reinforcement. Psychopharmacology190, 485–496. 10.1007/s00213-006-0627-5
58
DanaeiG.DingE. L.MozaffarianD.TaylorB.RehmJ.MurrayC. J.et al. (2009). The preventable causes of death in the United States: comparative risk assessment of dietary, lifestyle, and metabolic risk factors. PLoS Med. 6:e1000058. 10.1371/journal.pmed.1000058
59
DaughertyJ. R.BraseG. L. (2010). Taking time to be healthy: predicting health behaviors with delay discounting and time perspective. Pers. Individ. Differ. 48, 202–207. 10.1016/j.paid.2009.10.007
60
DavisC.PatteK.CurtisC.ReidC. (2010). Immediate pleasures and future consequences. A neuropsychological study of binge eating and obesity. Appetite54, 208–213. 10.1016/j.appet.2009.11.002
61
DawN. D. (2012). Model-based reinforcement learning as cognitive search: neurocomputational theories, in Cognitive Search: Evolution, Algorithms and the Brain, eds ToddP. M.HillesT. T.RobbinsT. W. (Cambridge, MA: MIT Press), 195–208.
62
DawN. D.GershmanS. J.SeymourB.DayanP.DolanR. J. (2011). Model-based influences on humans' choices and striatal prediction errors. Neuron69, 1204–1215. 10.1016/j.neuron.2011.02.027
63
DawN. D.NivY.DayanP. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704–1711. 10.1038/nn1560
64
DawN. D.TouretzkyD. S. (2000). Behavioral considerations suggest an average reward TD model of the dopamine system. Neurocomputing32–33, 679–684. 10.1016/S0925-2312(00)00232-0
65
DayanP.NivY.SeymourB.DawN. D. (2006). The misbehavior of value and the discipline of the will. Neural Netw. 19, 1153–1160. 10.1016/j.neunet.2006.03.002
66
DennhardtA. A.MurphyJ. G. (2011). Associations between depression, distress tolerance, delay discounting, and alcohol-related problems in European American and African American college students. Psychol. Addict. Behav. 25, 595–604. 10.1037/a0025807
67
de WitH. (2008). Impulsivity as a determinant and consequence of drug use: a review of underlying processes. Addict. Biol. 14, 22–31. 10.1111/j.1369-1600.2008.00129.x
68
Del GiudiceM.EllisB. J.ShirtcliffE. A. (2013). Making sense of stress: an evolutionary—developmental framework, in Adaptive and Maladaptive Aspects of Developmental Stress: Current Topics in Neurotoxicity, Vol 3, eds LaviolaG.MacriS. (New York, NY: Springer), 23–43.
69
Del GiudiceM.EllisB. J. (2014). Evolutionary foundations of developmental psychopathology in Developmental Psychopathology, Vol. 1, Theory and Method, 3rd Edn. ed CicchettiD. (New York, NY: Wiley and Sons).
70
DickinsonA.BalleineB. (1994). Motivational control of goal-directed action. Learn. Behav. 22, 1–18. 10.3758/BF03199951
71
DickinsonA.BalleineB.WattA.GonzalezF.BoakesR. (1995). Motivational control after extended instrumental training. Learn. Behav. 23, 197–206. 10.3758/BF03199935
72
Dierst-DaviesR.RebackC. J.PeckJ. A.NuñoM.KamienJ. B.AmassL. (2011). Delay-discounting among homeless, out-of-treatment, substance-dependent men who have sex with men. Am. J. Drug. Alcohol Abuse37, 93–97.
73
DolanP.GudexC. (1995). Time preference, duration and health state valuations. Health Econ. 4, 289–299. 10.1002/hec.4730040405
74
DolanR.DayanP. (2013). Goals and habits in the brain. Neuron80, 312–325. 10.1016/j.neuron.2013.09.007
75
DomG.D'haeneP.HulstijnW.SabbeB. (2006). Impulsivity in abstinent early- and late-onset alcoholics: differences in self-report measures and a discounting task. Addiction101, 50–59. 10.1111/j.1360-0443.2005.01270.x
76
DomjanM. (2003). The Principles of Learning and Behaviour.. Belmont, CA: Wadsworth.
77
EisenbergD. T.MacKillopJ.ModiM.BeaucheminJ.DangD.LismanS. A.WilsonD. S.et al. (2007). Examining impulsivity as an endophenotype using a behavioral approach: a DRD2 TaqI A and DRD4 48-bp VNTR association study. Behav. Brain Funct. 3, 2. 10.1186/1744-9081-3-2
78
EppingerB.WalterM.HeekerenH. R.LiS.-C. (2013). Of goals and habits: age-related and individual differences in goal-directed decision-making. Front. Neurosci. 7:253. 10.3389/fnins.2013.00253
79
EpsteinL. H.RichardsJ. B.SaadF. G.PaluchR. A.RoemmichJ. N.LermanC. (2003). Comparison between two measures of delay discounting in smokers. Exp. Clin. Psychopharmacol. 11, 131–138. 10.1037/1064-1297.11.2.131
80
EvansJ. S.B.StanovichK. E. (2013). Dual-process theories of higher cognition advancing the debate. Perspect. Psychol. Sci. 8, 223–241. 10.1177/1745691612460685
81
EverittB. J.BelinD.EconomidouD.PellouxY.DalleyJ. W.RobbinsT. W. (2008). Review. Neural mechanisms underlying the vulnerability to develop compulsive drug-seeking habits and addiction. Philos. Trans. R. Soc. Lond. B Biol. Sci. 363, 3125–3135. 10.1098/rstb.2008.0089
82
EverittB. J.RobbinsT. W. (2005). Neural systems of reinforcement for drug addiction: from actions to habits to compulsion. Nat. Neurosci. 8, 1481–1489. 10.1038/nn1579
83
FarrellS.LewandowskyS. (2010). Computational models as aids to better reasoning in psychology. Curr. Dir. Psychol. Sci. 19, 329–335. 10.1177/0963721410386677
84
FernieG.ColeJ. C.GoudieA. J.FieldM. (2010). Risk-taking but not response inhibition or delay discounting predict alcohol consumption in social drinkers. Drug Alcohol Depend. 112, 54–61. 10.1016/j.drugalcdep.2010.05.011
85
FieldM.ChristiansenP.ColeJ.GoudieA. (2007). Delay discounting and the alcohol Stroop in heavy drinking adolescents. Addiction102, 579–586. 10.1111/j.1360-0443.2007.01743.x
86
FieldM.SantarcangeloM.SumnallH.GoudieA.ColeJ. (2006). Delay discounting and the behavioural economics of cigarette purchases in smokers: the effects of nicotine deprivation. Psychopharmacology186, 255–263. 10.1007/s00213-006-0385-4
87
FieldsS.LeraasK.CollinsC.ReynoldsB. (2009). Delay discounting as a mediator of the relationship between perceived stress and cigarette smoking status in adolescents. Behav. Pharmacol. 20, 455–460. 10.1097/FBP.0b013e328330dcff
88
FieldsS.SabetM.PealA.ReynoldsB. (2011). Relationship between weight status and delay discounting in a sample of adolescent cigarette smokers. Behav. Pharmacol. 22, 266. 10.1097/FBP.0b013e328345c855
89
FrancisL. A.SusmanE. J. (2009). Self-regulation and rapid weight gain in children from age 3 to 12 years. Arch. Pediatr. Adolesc. Med. 163, 297–302. 10.1001/archpediatrics.2008.579
90
FrederickS.LoewensteinG.O'donoghueT. (2002). Time discounting and time preference: a critical review. J. Econ. Lit. 40, 351–401.
91
FuchsV. R. (1982). Time preference and health: an exploratory study, in Economic Aspects of Health, ed FuchsV. R. (Chicago: University of Chicago Press), 93–132.
92
GaniatsT. G.CarsonR. T.HammR. M.CantorS. B.SumnerW.SpannS. J.et al. (2000). Population-based time preferences for future health outcomes. Med. Decis. Making20, 263–270. 10.1177/0272989X0002000302
93
GiordanoL. A.BickelW. K.LoewensteinG.JacobsE. A.MarschL.BadgerG. J. (2002). Mild opioid deprivation increases the degree that opioid-dependent outpatients discount delayed heroin and money. Psychopharmacology163, 174–182. 10.1007/s00213-002-1159-2
94
GläscherJ.DawN. D.DayanP.O'DohertyJ. P. (2010). States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron66, 585–595. 10.1016/j.neuron.2010.04.016
95
GreenL.FristoeN.MyersonJ. (1994). Temporal discounting and preference reversals in choice between delayed outcomes. Psychon. Bull. Rev. 1, 383–389. 10.3758/BF03213979
96
GrossmanM. (1973). On the concept of health capital and the demand for health. J. Polit. Econ. 80, 223–255. 10.1086/259880
97
HassabisD.KumaranD.VannS. D.MaguireE. A. (2007). Patients with hippocampal amnesia cannot imagine new experiences. Proc. Natl. Acad. Sci. U.S.A. 104, 1726–1731. 10.1073/pnas.0610561104
98
HechtD.WalshV.LavidorM. (2013). Bi-frontal direct current stimulation affects delay discounting choices. Cogn. Neurosci. 4, 7–11. 10.1080/17588928.2011.638139
99
HeilS. H.JohnsonM. W.HigginsS. T.BickelW. K. (2006). Delay discounting in currently using and currently abstinent cocaine-dependent outpatients and non-drug-using matched controls. Addict. Behav. 31, 1290–1294. 10.1016/j.addbeh.2005.09.005
100
HeinzA. J.PetersE. N.BodenM. T.Bonn-MillerM. O. (2013). A comprehensive examination of delay discounting in a clinical sample of Cannabis-dependent military veterans making a self-guided quit attempt. Exp. Clin. Psychopharmacol. 21, 55–65. 10.1037/a0031192
101
HeymanG. M.GibbS. P. (2006). Delay discounting in college cigarette chippers. Behav. Pharmacol. 17, 669–679. 10.1097/FBP.0b013e3280116cfe
102
HoffmanW. F.MooreM.TemplinR.McFarlandB.HitzemannR. J.MitchellS. H. (2006). Neuropsychological function and delay discounting in methamphetamine-dependent individuals. Psychopharmacology188, 162–170. 10.1007/s00213-006-0494-0
103
HuckansM.SeelyeA.WoodhouseJ.ParcelT.MullL.SchwartzD.et al. (2010). Discounting of delayed rewards and executive dysfunction in individuals infected with hepatitis C. J. Clin. Exp. Neuropsychol. 33, 176–186. 10.1080/13803395.2010.499355
104
IdlerE. L.BenyaminiY. (1997). Self-rated health and mortality: a review of twenty-seven community studies. J. Health Soc. Behav. 38, 21–37. 10.2307/2955359
105
IkedaS.KangM. I.OhtakeF. (2010). Hyperbolic discounting, the sign effect, and the body mass index. J. Health Econ. 29, 268–284. 10.1016/j.jhealeco.2010.01.002
106
JohnsonA.van der MeerM. A.RedishA. D. (2007a). Integrating hippocampus and striatum in decision-making. Curr. Opin. Neurobiol. 17, 692–697. 10.1016/j.conb.2008.01.003
107
JohnsonM. W. (2012). An efficient operant choice procedure for assessing delay discounting in humans: initial validation in cocaine-dependent and control individuals. Exp. Clin. Psychopharmacol. 20, 191–204. 10.1037/a0027088
108
JohnsonM. W.BickelW. K.BakerF. (2007b). Moderate drug use and delay discounting: a comparison of heavy, light, and never smokers. Exp. Clin. Psychopharmacol. 15, 187–194. 10.1037/1064-1297.15.2.187
109
JohnsonM. W.BickelW. K.BakerF.MooreB. A.BadgerG. J.BudneyA. J. (2010). Delay discounting in current and former marijuana-dependent individuals. Exp. Clin. Psychopharmacol. 18, 99–107. 10.1037/a0018333
110
JohnsonW. G.ParryW.DrabmanR. S. (1978). The performance of obese and normal size children on a delay of gratification task. Addict. Behav. 3, 205–208. 10.1016/0306-4603(78)90020-5
111
KableJ. W.GlimcherP. W. (2010). An “as soon as possible” effect in human intertemporal decision making: behavioral evidence and neural mechanisms. J. Neurophysiol. 103, 2513–2531. 10.1152/jn.00177.2009
112
KalenscherT.PennartzC. M. (2008). Is a bird in the hand worth two in the future? The neuroeconomics of intertemporal decision-making. Prog. Neurobiol. 84, 284–315. 10.1016/j.pneurobio.2007.11.004
113
KangM. I.IkedaS. (2013). Time discounting and smoking behavior: evidence from a panel survey. Health Econ. [Epub ahead of print]. 10.1002/hec.2998
114
KeramatiM.DezfouliA.PirayP. (2012). Understanding addiction as a pathological state of multiple decision making processes: a neurocomputational perspective, in Computational Neuroscience of Drug Addiction, eds GutkinB.AhmedS. H. (New York, NY: Springer), 205–233.
115
KhwajaA.SilvermanD.SloanF. (2007). Time preference, time discounting, and smoking decisions. J. Health Econ. 26, 927–949. 10.1016/j.jhealeco.2007.02.004
116
KirbyK. N. (2009). One-year temporal stability of delay-discount rates. Psychon. Bull. Rev. 16, 457–46210.3758/PBR.16.3.457
117
KirbyK. N.HerrnsteinR. J. (1995). Preference reversals due to myopic discounting of delayed reward. Psychcol. Sci. 6, 83–89. 10.1111/j.1467-9280.1995.tb00311.x
118
KirbyK. N.PetryN. M. (2004). Heroin and cocaine abusers have higher discount rates for delayed rewards than alcoholics or non-drug-using controls. Addiction99, 461–471. 10.1111/j.1360-0443.2003.00669.x
119
KirbyK. N.PetryN. M.BickelW. K. (1999). Heroin addicts have higher discount rates for delayed rewards than non-drug-using controls. J. Exp. Psychol. Gen. 128, 78–87. 10.1037/0096-3445.128.1.78
120
KleinknechtR. A. (1978). The assessment of dental fear. Behav. Ther. 9, 626. 10.1016/S0005-7894(78)80138-5
121
KobiellaA.RipkeS.KroemerN. B.VollmertC.Vollstadt-KleinS.UlshoferD. E.et al. (2013). Acute and chronic nicotine effects on behaviour and brain activation during intertemporal decision making. Addict. Biol. [Epub ahead of print]. 10.1111/adb.12057
122
KoffarnusM. N.JarmolowiczD. P.MuellerE. T.BickelW. K. (2013). Changing delay discounting in the light of the competing neurobehavioral decision systems theory: a review. J. Exp. Anal. Behav. 99, 32–57. 10.1002/jeab.2
123
KollinsS. H. (2003). Delay discounting is associated with substance use in college students. Addict. Behav. 28, 1167–1173. 10.1016/S0306-4603(02)00220-4
124
KraemerH. C.GullionC.RushA. J.FrankE.KupferD. J. (1993). Can state and trait variables be disentangled? A methodological framework for psychiatric disorders. Psychiatry Res. 52, 55–69. 10.1016/0165-1781(94)90120-1
125
Krishnan-SarinS.ReynoldsB.DuhigA. M.SmithA.LissT.McFetridgeA.et al. (2007). Behavioral impulsivity predicts treatment outcome in a smoking cessation program for adolescent smokers. Drug Alcohol Depend. 88, 79–82. 10.1016/j.drugalcdep.2006.09.006
126
KulendranM.VlaevI.SugdenC.KingD.AshrafianH.GatelyP.et al. (2013b). Neuropsychological assessment as a predictor of weight loss in obese adolescents. Int. J. Obes. [Epub ahead of print]. 10.1038/ijo.2013.198
127
KulendranM.VlaevI.SugdenC.KingD.HusseinM.Le VayR.et al. (2013a). Inhibitory control and perception of the future health in obese adolescents and normal weight adolescents. Lancet382(Suppl. 3), S55. 10.1016/S0140-6736(13)62480-0
128
Kurth-NelsonZ.RedishA. D. (2009). Temporal-difference reinforcement learning with distributed representations. PLoS ONE4:e7362. 10.1371/journal.pone.0007362
129
Kurth-NelsonZ.RedishA. D. (2010). A reinforcement learning model of precommitment in decision making. Front. Behav. Neurosci. 4:184. 10.3389/fnbeh.2010.00184
130
LaibsonD. (1997). Golden eggs and hyperbolic discounting. Q. J. Econ. 112, 443–478. 10.1162/003355397555253
131
LahavE.BenzionU.ShavitT. (2011). The effect of military service on soldiers' time preference-Evidence from Israel. Judgm. Decis. Mak. 6, 130–138.
132
LallyP.ChipperfieldA.WardleJ. (2007). Healthy habits: efficacy of simple advice on weight control based on a habit-formation model. Int. J. Obes. 32, 700–707. 10.1038/sj.ijo.0803771
133
LallyP.Van JaarsveldC. H.PottsH. W.WardleJ. (2010). How are habits formed: modelling habit formation in the real world. Eur. J. Soc. Psychol. 40, 998–1009. 10.1002/ejsp.674
134
LandesR. D.ChristensenD. R.BickelW. K. (2012). Delay discounting decreases in those completing treatment for opioid dependence. Exp. Clin. Psychopharmcol. 20, 302–309. 10.1037/a0027391
135
LazaroA.BarberanR.RubioE. (2001). Private and social time preferences for health and money: an empirical estimation. Health Econ. 10, 351–356. 10.1002/hec.599
136
LedgerwoodD. M.AlessiS. M.PhoenixN.PetryN. M. (2009). Behavioral assessment of impulsivity in pathological gamblers with and without substance use disorder histories versus healthy controls. Drug Alcohol Depend. 105, 89–96. 10.1016/j.drugalcdep.2009.06.011
137
LewandowskyS.FarrellS. (2011). Computational Modeling in Cognition: Principles and Practice. Thousand Oaks, CA: Sage.
138
LipscombJ. (1989). Time preference for health in cost-effectiveness analysis. Med. Care27, S233–S253. 10.1097/00005650-198903001-00019
139
LoewensteinG. (1987). Anticipation and the valuation of delayed consumption. Econ. J. 97, 666–684. 10.2307/2232929
140
LoewensteinG. (1996). Out of control: visceral influences on behavior. Organ. Behav. Hum. Decis. Process. 65, 272–292. 10.1006/obhd.1996.0028
141
LoewensteinG.PrelecD. (1991). Negative time preference. Am. Econ. Rev. 81, 347–352.
142
LucantonioF.CaprioliD.SchoenbaumG. (2014). Transition from ‘model-based’ to ‘model-free’ behavioral control in addiction: involvement of the orbitofrontal cortex and dorsolateral striatum. Neuropharmacology76(Part B), 407–415. 10.1016/j.neuropharm.2013.05.033
143
MacKeiganL. D.LarsonL. N.DrauglisJ. R.BootmanJ. L.BurnsL. R. (1993). Time preference for health gains versus health losses. Pharmacoeconomics3, 374–386. 10.2165/00019053-199303050-00005
144
MacKillopJ. (2013). Integrating behavioral economics and behavioral genetics: delayed reward discounting as an endophenotype for addictive disorders. J. Exp. Anal. Behav. 99, 14–31. 10.1002/jeab.4
145
MacKillopJ.AmlungM. T.FewL. R.RayL. A.SweetL. H.MunafoM. R. (2011). Delayed reward discounting and addictive behavior: a meta-analysis. Psychopharmacology216, 305–321. 10.1007/s00213-011-2229-0
146
MacKillopJ.KahlerC. W. (2009). Delayed reward discounting predicts treatment response for heavy drinkers receiving smoking cessation treatment. Drug Alcohol Depend. 104, 197–203. 10.1016/j.drugalcdep.2009.04.020
147
MacKillopJ.MattsonR. E.AndersonE. J.CasteldaB. A.DonovickP. J. (2007). Multidimensional assessment of impulsivity in undergraduate hazardous drinkers and controls. J. Stud. Alcohol Drugs68, 785–788.
148
MacKillopJ.MirandaR.Jr.MontiP. M.RayL. A.MurphyJ. G.RohsenowD. J.et al. (2010). Alcohol demand, delayed reward discounting, and craving in relation to drinking and alcohol use disorders. J. Abnorm. Psychol. 119, 106–114. 10.1037/a0017513
149
MaddenG. J.PetryN. M.BadgerG. J.BickelW. K. (1997). Impulsive and self-control choices in opioid-dependent patients and non-drug-using control patients: drug and monetary rewards. Exp. Clin. Psychopharmcol. 5, 256–262. 10.1037/1064-1297.5.3.256
150
MarianoT. Y.BannermanD. M.McHughS. B.PrestonT. J.RudebeckP. H.RudebeckS. R.et al. (2009). Impulsive choice in hippocampal but not orbitofrontal cortex−lesioned rats on a nonspatial decision−making maze task. Eur. J. Neurosci. 30, 472–484. 10.1111/j.1460-9568.2009.06837.x
151
MazurJ. (1987). An adjusting procedure for studying delayed reinforcement, in Quantitative Analysis of Behavior, eds CommonsM. L.MazurJ. E.NevinJ. A.RachlinH. (Hillsdale, NJ: Erlbaum), 55–73.
152
McClureS. M.EricsonK. M.LaibsonD. I.LoewensteinG.CohenJ. D. (2007). Time discounting for primary rewards. J. Neurosci. 27, 5796–5804. 10.1523/JNEUROSCI.4246-06.2007
153
McClureS. M.LaibsonD. I.LoewensteinG.CohenJ. D. (2004). Separate neural systems value immediate and delayed monetary rewards. Science306, 503–507. 10.1126/science.1100907
154
McDannaldM. A.LucantonioF.BurkeK. A.NivY.SchoenbaumG. (2011). Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning. J. Neurosci. 31, 2700–2705. 10.1523/JNEUROSCI.5499-10.2011
155
MelankoS.LarkinK. T. (2013). Preference for immediate reinforcement over delayed reinforcement: relation between delay discounting and health behavior. J. Behav. Med. 36, 34–43. 10.1007/s10865-012-9399-z
156
MitchellJ. M.FieldsH. L.D'espositoM.BoettigerC. A. (2005). Impulsive responding in alcoholics. Alcohol Clin. Exp. Res. 29, 2158–2169. 10.1097/01.alc.0000191755.63639.4a
157
MitchellS. H. (1999). Measures of impulsivity in cigarette smokers and non-smokers. Psychopharmacology146, 455–464. 10.1007/PL00005491
158
MitchellS. H. (2004). Effects of short-term nicotine deprivation on decision-making: delay, uncertainty and effort discounting. Nicotine Tob. Res. 6, 819–828. 10.1080/14622200412331296002
159
MoallemN. R.RayL. A. (2012). Dimensions of impulsivity among heavy drinkers, smokers, and heavy drinking smokers: singular and combined effects. Addict. Behav. 37, 871–874. 10.1016/j.addbeh.2012.03.002
160
MoellerG. F.DoughertyD. M.BarrattE. S.OderindeV.MathiasC. W.HarperR. A.et al. (2002). Increased impulsivity in cocaine dependent subjects independent of antisocial personality disorder and aggression. Drug Alcohol Depend. 68, 105–111. 10.1016/S0376-8716(02)00106-0
161
MoffittT. E.ArseneaultL.BelskyD.DicksonN.HancoxR. J.HarringtonH.et al. (2011). A gradient of childhood self-control predicts health, wealth, and public safety. Proc. Natl. Acad. Sci. U.S.A. 108, 2693–2698. 10.1073/pnas.1010076108
162
MontagueP. R.DolanR. J.FristonK. J.DayanP. (2012). Computational psychiatry. Trends Cogn. Sci. 16, 72–80. 10.1016/j.tics.2011.11.018
163
MonterossoJ. R.AinslieG.XuJ.CordovaX.DomierC. P.LondonE. D. (2007). Frontoparietal cortical activity of methamphetamine-dependent and comparison subjects performing a delay discounting task. Hum. Brain Mapp. 28, 383–393. 10.1002/hbm.20281
164
MooreS. C.CusensB. (2010). Delay discounting predicts increase in blood alcohol level in social drinkers. Psychiatry Res. 179, 324–327. 10.1016/j.psychres.2008.07.024
165
MuellerE. T.LandesR. D.KowalB. P.YiR.StitzerM. L.BurnettC. A.et al. (2009). Delay of smoking gratification as a laboratory model of relapse: effects of incentives for not smoking, and relationship with measures of executive function. Behav. Pharmacol. 20, 461–473. 10.1097/FBP.0b013e3283305ec7
166
NealD. T. (2006). Habits—A repeat performance. Curr. Dir. Psychol. Sci. 15, 198. 10.1111/j.1467-8721.2006.00435.x
167
NederkoornC.SmuldersF. T.HavermansR. C.RoefsA.JansenA. (2006). Impulsivity in obese women. Appetite47, 253–256. 10.1016/j.appet.2006.05.008
168
NivY.DawN. D.JoelD.DayanP. (2007). Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology191, 507–520. 10.1007/s00213-006-0502-4
169
O'BrienC. P.ChildressA. R.EhrmanR.RobbinsS. J. (1998). Conditioning factors in drug abuse: can they explain compulsion?J. Psychopharmacol. 12, 15–22. 10.1177/026988119801200103
170
O'DonoghueT.RabinM. (2003). Self Awareness and Self Control. Time and Decision: Economic and Psychological Perspectives on Intertemporal Choice. New York, NY: Russell Sage Foundation.
171
OdumA. L. (2011). Delay discounting: trait variable?Behav. Processes87, 1–9. 10.1016/j.beproc.2011.02.007
172
OdumA. L.MaddenG. J.BadgerG. J.BickelW. K. (2000). Needle sharing in opioid-dependent outpatients: psychological processes underlying risk. Drug Alcohol Depend. 60, 259–266. 10.1016/S0376-8716(00)00111-3
173
OdumA. L.MaddenG. J.BickelW. K. (2002). Discounting of delayed health gains and losses by current, never- and ex-smokers of cigarettes. Nicotine Tob. Res. 4, 295–303. 10.1080/14622200210141257
174
OhmuraY.TakahashiT.KitamuraN. (2005). Discounting delayed and probabilistic monetary gains and losses by smokers of cigarettes. Psychopharmacology (Berl.)182, 508–515. 10.1007/s00213-005-0110-8
175
OlsenJ. A. (1993). Time preferences for health gains: an empirical investigation. Health Econ. 2, 257–265. 10.1002/hec.4730020309
176
OrbellS.VerplankenB. (2010). The automatic component of habit in health behavior: habit as cue-contingent automaticity. Health Psychol. 29, 374. 10.1037/a0019596
177
OuelletteJ. A. (1998). Habit and intention in everyday life: the multiple processes by which past behavior predicts future behavior. Psychol. Bull. 124, 54. 10.1037/0033-2909.124.1.54
178
PetersJ.BüchelC. (2010). Episodic future thinking reduces reward delay discounting through an enhancement of prefrontal-mediotemporal interactions. Neuron66, 138–148. 10.1016/j.neuron.2010.03.026
179
PetersE. N.PetryN. M.LapagliaD. M.ReynoldsB.CarrollK. M. (2013). Delay discounting in adults receiving treatment for marijuana dependence. Exp. Clin. Psychopharmcol. 21, 46–54. 10.1037/a0030943
180
PetryN. M. (2001). Delay discounting of money and alcohol in actively using alcoholics, currently abstinent alcoholics, and controls. Psychopharmacology154, 243–250. 10.1007/s002130000638
181
PetryN. M. (2003). Discounting of money, health, and freedom in substance abusers and controls. Drug Alcohol Depend. 71, 133–141. 10.1016/S0376-8716(03)00090-5
182
PetryN. M.CasarellaT. (1999). Excessive discounting of delayed rewards in substance abusers with gambling problems. Drug Alcohol Depend. 56, 25–32. 10.1016/S0376-8716(99)00010-1
183
PhelpsE. S.PollakR. A. (1968). On second-best national saving and game-equilibrium growth. Rev. Econ. Stud. 35, 185–199. 10.2307/2296547
184
PrelecD.BodnerR. (2003). Self-signaling and self-control, in Time and Decision, eds LoewensteinG.ReadD.BaumeisterR. F. (New York, NY: Russel Sage Foundation), 277–298.
185
ReadD. (2001). Is time-discounting hyperbolic or subadditive?J. Risk Uncertainty23, 5–32. 10.1023/A:1011198414683
186
ReadD.FrederickS.AiroldiM. (2012). Four days later in Cincinnati: longitudinal tests of hyperbolic discounting. Acta Psychol. 140, 177–185. 10.1016/j.actpsy.2012.02.010
187
ReadD.RoelofsmaP. H. (2003). Subadditive versus hyperbolic discounting: a comparison of choice and matching. Organ. Behav. Hum. Decis. Process. 91, 140–153. 10.1016/S0749-5978(03)00060-8
188
ReadD.Van LeeuwenB. (1998). Predicting hunger: the effects of appetite and delay on choice. Organ. Behav. Hum. Decis. Process. 76, 189–205. 10.1006/obhd.1998.2803
189
RedelmeierD. A.HellerD. N. (1993). Time preference in medical decision making and cost-effectiveness analysis. Med. Decis. Making13, 212–217. 10.1177/0272989X9301300306
190
ReimersS.MaylorE. A.StewartN.ChaterN. (2009). Associations between a one-shot delay discounting measure and age, income, education and real-world impulsive behavior. Pers. Indivd. Differ. 47, 973–978. 10.1016/j.paid.2009.07.026
191
RescignoA.BeckJ. S.ThakurA. K. (1987). The use and abuse of models. J. Pharmacokinet. Pharmacodyn. 15, 327–340. 10.1007/BF01066325
192
ReynoldsB. (2004). Do high rates of cigarette consumption increase delay discounting?: A cross-sectional comparison of adolescent smokers and young-adult smokers and nonsmokers. Behav. Processes67, 545–549. 10.1016/j.beproc.2004.08.006
193
ReynoldsB. (2006a). The experiential discounting task is sensitive to cigarette-smoking status and correlates with a measure of delay discounting. Behav. Pharmacol. 17, 133–142. 10.1097/01.fbp.0000190684.77360.c0
194
ReynoldsB. (2006b). A review of delay-discounting research with humans: relations to drug use and gambling. Behav. Pharmacol. 17, 651–667. 10.1097/FBP.0b013e3280115f99
195
ReynoldsB.FieldsS. (2012). Delay discounting by adolescents experimenting with cigarette smoking. Addiction107, 417–424. 10.1111/j.1360-0443.2011.03644.x
196
ReynoldsB.KarrakerK.HornK.RichardsJ. B. (2003). Delay and probability discounting as related to different stages of adolescent smoking and non-smoking. Behav. Processes64, 333–344. 10.1016/S0376-6357(03)00168-2
197
ReynoldsB.LeraasK.CollinsC.MelankoS. (2009). Delay discounting by the children of smokers and nonsmokers. Drug Alcohol Depend. 99, 350–353. 10.1016/j.drugalcdep.2008.07.015
198
ReynoldsB.PatakM.ShroffP. (2007b). Adolescent smokers rate delayed rewards as less certain than adolescent nonsmokers. Drug Alcohol Depend. 90, 301–303. 10.1016/j.drugalcdep.2007.04.008
199
ReynoldsB.PatakM.ShroffP.PenfoldR. B.MelankoS.BuhigA. M. (2007a). Laboratory and self-report assessments of impulsive behavior in adolescent daily smokers and nonsmokers. Exp. Clin. Psychopharmacol. 15, 264–271. 10.1037/1064-1297.15.3.264
200
ReynoldsB.RichardsJ. B.HornK.KarrakerK. (2004). Delay discounting and probability discounting as related to cigarette smoking status in adults. Behav. Processes65, 35–42. 10.1016/S0376-6357(03)00109-8
201
RezvanfardM.HamedE.MokriA.DjavidG. E. (2010). Psychological and behavioral traits in smokers and their relationship with nicotine dependence level. Arch. Iran. Med. 13, 395–405.
202
RogersR. D.RobbinsT. W. (2001). Investigating the neurocognitive deficits associated with chronic drug misuse. Curr. Opin. Neurobiol. 11, 250–257. 10.1016/S0959-4388(00)00204-X
203
RollinsB. Y.DearingK. K.EpsteinL. H. (2010). Delay discounting moderates the effect of food reinforcement on energy intake among non-obese women. Appetite55, 420–425. 10.1016/j.appet.2010.07.014
204
RossowI. (2008). Alcohol consumption and discounting. Addict. Res. Theory16, 572–584. 10.1080/16066350801896248
205
SamuelsonP. A. (1937). A note on the measurement of utility. Rev. Econ. Stud. 4, 155–161. 10.2307/2967612
206
SaymanS.ÖncülerA. (2009). An investigation of time inconsistency. Manag. Sci. 55, 470–482. 10.1287/mnsc.1080.0942
207
SchacterD. L.AddisD. R.BucknerR. L. (2008). Episodic simulation of future events. Ann. N.Y. Acad. Sci. 1124, 39. 10.1196/annals.1440.001
208
SeeyaveD. M.ColemanS.AppuglieseD.CorwynR. F.BradleyR. H.DavidsonN. S.et al. (2009). Ability to delay gratification at age 4 years and risk of overweight at age 11 years. Arch. Pediatr. Adolesc. Med. 163, 303–308. 10.1001/archpediatrics.2009.12
209
ShefferC.MacKillopJ.McGearyJ.LandesR.CarterL.YiR.et al. (2012). Delay discounting, locus of control, and cognitive impulsiveness independently predict tobacco dependence treatment outcomes in a highly dependent, lower socioeconomic group of smokers. Am. J. Addict. 21, 221–232. 10.1111/j.1521-0391.2012.00224.x
210
SmithR.CorriganP.ExeterC. (2012). Countering Non-Communicable Disease Through Innovation. Report of the Non-Communicable Disease Working Group. London: The Global Helath Policy Summit. Available online at: https://workspace.imperial.ac.uk/global-health-innovation/Public/GHPS_NCD_Report.pdf
211
SmittenaarP.FitzgeraldT. H.RomeiV.WrightN. D.DolanR. J. (2013). Disruption of dorsolateral prefrontal cortex decreases model-based in favor of model-free control in humans. Neuron4, 914–919. 10.1016/j.neuron.2013.08.009
212
StangerC.RyanS. R.FuH.LandesR. D.JonesB. A.BickelW. K.et al. (2012). Delay discounting predicts adolescent substance abuse treatment outcome. Exp. Clin. Psychopharmcol. 20, 205–212. 10.1037/a0026543
213
SteaJ. N.HodginsD. C.LambertM. J. (2011). Relations between delay discounting and low to moderate gambling, cannabis, and alcohol problems among university students. Behav. Processes88, 202–205. 10.1016/j.beproc.2011.09.002
214
StillwellD. J.TunneyR. J. (2012). Effects of measurement methods on the relationship between smoking and delay reward discounting. Addiction107, 1003–1012. 10.1111/j.1360-0443.2011.03742.x
215
StoryG.VlaevI.SeymourB.DarziA.DolanR. (2013). Dread and the disvalue of future pain. PLoS Comput. Biol. 9:e1003335. 10.1371/journal.pcbi.1003335
216
StrotzR. H. (1957). Myopia and inconsistency in dynamic utility maximisation. Rev. Econ. Stud. 23, 165–180. 10.2307/2295722
217
SuttonR. S.BartoA. G. (1998). Reinforcement Learning: An Introduction. Cambridge: Cambridge Univ Press.
218
SweitzerM. M.DonnyE. C.DierkerL. C.FloryJ. D.ManuckS. B. (2008). Delay discounting and smoking: association with the Fagerstrom Test for Nicotine Dependence but not cigarettes smoked per day. Nicotine Tob. Res. 10, 1571–1575. 10.1080/14622200802323274
219
ThalerR. (1981). Some empirical evidence on dynamic inconsistency. Econ. Lett. 8, 201–207. 10.1016/0165-1765(81)90067-7
220
TricomiE.BalleineB. W.O'dohertyJ. P. (2009). A specific role for posterior dorsolateral striatum in human habit learning. Eur. J. Neurosci. 29, 2225–2232. 10.1111/j.1460-9568.2009.06796.x
221
TsitsiklisJ. N.Van RoyB. (2002). On average versus discounted reward temporal-difference learning. Mach. Learn. 49, 179–191. 10.1023/A:1017980312899
222
Van Der PolM. (2011). Health, education and time preference. Health Econ. 20, 917–929. 10.1002/hec.1655
223
Van Der PolM.CairnsJ. (2000). Negative and zero time preference for health. Health Econ. 9, 171–175. 10.1002/(SICI)1099-1050(200003)9:2<171::AID-HEC492>3.0.CO;2-Z
224
Van Der PolM.CairnsJ. (2001). Estimating time preferences for health using discrete choice experiments. Soc. Sci. Med. 52, 1459–1470. 10.1016/S0277-9536(00)00256-2
225
Van Der PolM.CairnsJ. (2002). A comparison of the discounted utility model and hyperbolic discounting models in the case of social and private intertemporal preferences for health. J. Econ. Behav. Organ. 49, 79–96. 10.1016/S0167-2681(02)00059-8
226
Van Der PolM.CairnsJ. (2011). Descriptive validity of alternative intertemporal models for health outcomes: an axiomatic test. Health Econ. 20, 770–782. 10.1002/hec.1628
227
VuchinichR. E.SimpsonC. A. (1998). Hyperbolic temporal discounting in social drinkers and problem drinkers. Exp. Clin. Psychopharmcol. 6, 292–305. 10.1037/1064-1297.6.3.292
228
WashioY.HigginsS. T.HeilS. H.McKercharT. L.BadgerG. J.SkellyJ. M.et al. (2011). Delay discounting is associated with treatment response among cocaine-dependent outpatients. Exp. Clin. Psychopharmcol. 19, 243. 10.1037/a0023617
229
WellerR. E.CookE. W.3rdAvsarK. B.CoxJ. E. (2008). Obese women show greater delay discounting than healthy-weight women. Appetite51, 563–569. 10.1016/j.appet.2008.04.010
230
WilliamsD. R.WilliamsH. (1969). Auto-maintenance in the pigeon: sustained pecking despite contingent non-reinforcement. J. Exp. Anal. Behav. 12, 511–520. 10.1901/jeab.1969.12-511
231
WingV. C.MossT. G.RabinR. A.GeorgeT. P. (2012). Effects of cigarette smoking status on delay discounting in schizophrenia and healthy controls. Addict. Behav. 37, 67–72. 10.1016/j.addbeh.2011.08.012
232
WunderlichK.SmittenaarP.DolanR. J. (2012). Dopamine enhances model-based over model-free choice behavior. Neuron75, 418–424. 10.1016/j.neuron.2012.03.042
233
YankelevitzR. L.MitchellS. H.ZhangY. (2012). Gender differences in factors associated with alcohol drinking: delay discounting and perception of others' drinking. Drug Alcohol Depend. 123, 273–276. 10.1016/j.drugalcdep.2011.11.012
234
YiR.JohnsonM. W.GiordanoL. A.LandesR. D.BadgerG. J.BickelW. K. (2008). The effects of reduced cigarette smoking on discounting future rewards: an initial evaluation. Psychol. Rec. 58, 163–174.
235
YiR.LandesR. D. (2012). Temporal and probability discounting by cigarette smokers following acute smoking abstinence. Nicotine Tob. Res. 14, 547–558. 10.1093/ntr/ntr252
236
YoonJ. H.HigginsS. T.HeilS. H.SugarbakerR. J.ThomasC. S.BadgerG. J. (2007). Delay discounting predicts postpartum relapse to cigarette smoking among pregnant women. Exp. Clin. Psychopharmcol. 15, 176–186. 10.1037/1064-1297.15.2.186
Summary
Keywords
discounting, health, addiction, model-based, model-free, habit, preference reversal, hyperbolic
Citation
Story GW, Vlaev I, Seymour B, Darzi A and Dolan RJ (2014) Does temporal discounting explain unhealthy behavior? A systematic review and reinforcement learning perspective. Front. Behav. Neurosci. 8:76. doi: 10.3389/fnbeh.2014.00076
Received
15 January 2014
Accepted
21 February 2014
Published
12 March 2014
Volume
8 - 2014
Edited by
Jack Van Honk, Utrecht University, Netherlands
Reviewed by
Barak Morgan, University of Cape Town, South Africa; Matthew Wanat, University of Texas at San Antonio, USA
Copyright
© 2014 Story, Vlaev, Seymour, Darzi and Dolan.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Giles W. Story, Department of Surgery and Cancer, Centre for Health Policy, Institute of Global Health Innovation, St. Mary's Hospital, Imperial College London, Praed Street, London, W2 1NY, UK e-mail: g.story@imperial.ac.uk
This article was submitted to the journal Frontiers in Behavioral Neuroscience.
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.