Alaina L. Pearce
Bari A. Fuchs
Kathleen L. Keller- 1Social Science Research Institute, Pennsylvania State University, University Park, PA, United States
- 2Department of Nutritional Sciences, Pennsylvania State University, University Park, PA, United States
- 3Department of Food Science, Pennsylvania State University, University Park, PA, United States
The obesogenic food environment includes easy access to highly-palatable, energy-dense, “ultra-processed” foods that are heavily marketed to consumers; therefore, it is critical to understand the neurocognitive processes the underlie overeating in response to environmental food-cues (e.g., food images, food branding/advertisements). Eating habits are learned through reinforcement, which is the process through which environmental food cues become valued and influence behavior. This process is supported by multiple behavioral control systems (e.g., Pavlovian, Habitual, Goal-Directed). Therefore, using neurocognitive frameworks for reinforcement learning and value-based decision-making can improve our understanding of food-choice and eating behaviors. Specifically, the role of reinforcement learning in eating behaviors was considered using the frameworks of (1) Sign-versus Goal-Tracking Phenotypes; (2) Model-Free versus Model-Based; and (3) the Utility or Value-Based Model. The sign-and goal-tracking phenotypes may contribute a mechanistic insight on the role of food-cue incentive salience in two prevailing models of overconsumption–the Extended Behavioral Susceptibility Theory and the Reactivity to Embedded Food Cues in Advertising Model. Similarly, the model-free versus model-based framework may contribute insight to the Extended Behavioral Susceptibility Theory and the Healthy Food Promotion Model. Finally, the value-based model provides a framework for understanding how all three learning systems are integrated to influence food choice. Together, these frameworks can provide mechanistic insight to existing models of food choice and overconsumption and may contribute to the development of future prevention and treatment efforts.
Introduction
Each day we make hundreds of choices about what to eat, many of which occur automatically with little conscious thought (1). While in lay terms, the phrase “food choice” is often limited to the decisions about the composition of a meal (e.g., What’s for dinner?), the current review uses a broader definition that encompasses the behavioral and environmental factors that influence meal initiation, amount consumed, and quality of the food choices (2–4). Food choices are extremely complex because they evolve over varying time scales, have multiple determinants, and occur within various contexts (e.g., celebratory, meals, and snacks) (2, 3, 5). Adding to the complexity is the overwhelming influence of the obesogenic food environment, which makes highly palatable, energy-dense (i.e., “ultra-processed”) foods more affordable and accessible (6). Food choice in the context of an obesogenic environment requires the integration of multiple, often conflicting pieces of information (3). For example, presence of food cues such as McDonald’s “Golden Arches” may trigger wanting for energy-dense foods (e.g., Big Mac and French fries) that are not compatible with goals to maintain a healthy diet (7). With 1 in 5 deaths linked to a poor diet (8, 9) and obesity rates among children continuing to rise (10), it is critically important to understand how food choices are made in response to environmental food cues (e.g., food images, advertising/branding). Understanding the neurocognitive processes those underly food choices in this context is crucial for the development of effective, tailored health interventions.
Environmental food cues influence food choice through three behavioral controllers or systems: Pavlovian, instrumental/habit, and goal-directed (1, 11, 12). The Pavlovian system regulates automatic behavioral responses to cues that are associated with evolutionarily relevant outcomes. The classical example is of Pavlov’s dogs salivating at the sight of food (13). While these responses can be present without learning (i.e., “hard-wired”), the association between a stimulus or cue (e.g., bell sounding) and an evolutionarily significant outcome (e.g., food delivery) can be learned and presumably confers selective advantages to human and non-human animals in their search for edible and nutritious foods (11, 13, 14). For example, approaching a cue that predicts food delivery (1, 11) or consuming all the food available on a plate regardless of hunger would be considered Pavlovian behaviors (11). In contrast to the Pavlovian system where the outcome or reward is delivered regardless of behavior, in instrumental learning, reward delivery is contingent upon the behavior performed in response to the cue (11, 14, 15). Thus, while the Pavlovian system supports stimulus-outcome (S-O) learning the instrumental system supports stimulus-response (S-R) learning. The instrumental system has also been termed the “habit” system because learned actions can occur even when the outcome is not desired, which can lead to habitual behaviors (15). For example, the instrumental system would drive habitual coffee intake at a specific time of day regardless of whether the stimulating effect of caffeine is needed or desired (1, 11, 15). While a habitual behavior may occur regardless of state as in the prior example, the value of food-related actions is also influenced by internal states like hunger (16–18). In contrast to the instrumental system which is driven by previously learned S-R associations, the goal-directed system prospectively evaluates response-outcome (R-O) associations based on the anticipated or predicted outcome for each action (1, 11, 15). For example, the goal-directed behavior of choosing where to eat in a novel city would be driven by the anticipated value for the food at each restaurant. Together, these three systems drive eating behavior and food choice in response to environmental food cues.
While the instrumental and goal-directed systems contribute to value-based decision-making in general, food choice is a unique because it can also be influenced by the Pavlovian system (1). Therefore, applying neurocognitive frameworks to understanding the factors that motivate food choice may elucidate novel behaviors to target in dietary interventions. The current review is intended to provide an overview of three frameworks that encompass these learning systems: (1) sign-and goal-tracking phenotypes; (2) model-based and model-free reinforcement learning; and (3) the utility or value-based model. For each framework we will provide a brief translational review of the theory and its supporting neurobiological substrates, followed by a summary of possible applications to understanding food choice and eating behaviors. Finally, we will consider how these frameworks can be utilized to improve understanding of food-choice and applied to the development of more effective prevention/treatment programs for disordered or dysregulated eating.
Sign-and goal-tracking
The sign-and goal-tracking phenotype is an animal model for motivational control of behavior in response to environmental cues (19–23). These phenotypes are characterized in animals using the Pavlovian Conditioned Approach (PCA) test (24, 25). Pavlovian conditioning occurs when a neutral cue (e.g., lever) becomes a conditioned stimulus (CS) after being repeatedly paired with an unconditioned stimulus (US) like food. In the PCA test (Figure 1A), a lever (neutral) is repeatedly presented prior to food delivery (US) allowing the animal to learn the lever-food (S-O) association (Figure 1A). Once the lever becomes a CS, it is able to elicit conditioned responses (CR) (22, 24, 25). Animals display three patterns of CRs: (1) goal-tracking: approaching the location of food delivery (US); (2) sign-tracking: approaching the lever (CS) itself; and (3) intermediate: switching between the two CRs (20–22, 25). Importantly, all animals are equally able to learn the S-O association regardless of CR displayed (26). The differing patterns of CRs occur due to differences in the attribution of incentive salience or motivational value to the CS (20–22, 25). For sign-trackers, the CS becomes an incentivized stimulus, which has three defining properties: (1) it biases attention; (2) it is desired and the animal will work for it (i.e., is “wanted”); and (3) it can increase motivation to seek reward (20–22, 25, 27). Once the CS becomes desired, sign-trackers will approach and interact with the CS even if it means losing access to the primary reward (e.g., food) (25). Therefore, a key behavioral distinction between these phenotypes is the propensity for environmental cues to take on rewarding properties and motivate wanting.
Figure 1. Methods of assessing sign-and goal-tracking phenotypes. (A) Pavlovian conditioned approach task–this task is used in non-human animal models. Animals learn the stimulus-outcome (S–O) association between the lever and food delivery. The conditioned responses are: (1) sign-trackers–approach the lever; (2) goal-trackers–approach the location of food delivery. (B) Pavlovian conditioning task–this is a simplified schematic of a Pavlovian conditioning task where a conditioned stimulus either predicts reward receipt or no reward. The conditioned stimulus and reward are presented in different locations so the conditioned response of eye-gaze can be measured. The conditioned responses are: (1) sign-trackers–look at the conditioned stimulus more than reward location; (2) goal-trackers–look more at the reward location than conditioned response. (C) Value-Driven Attentional Capture Task–this task includes a training and a test phase. Both phases include a visual search task where participants must locate the location of the horizontal line. During training, one color is associated with high reward (e.g., green) and one color is associated with low reward (e.g., purple). During the test phase, the target is the unique shape and the previously rewarded colors are used as high or low value distractors. No reward is given for correct responses in the test phase. The conditioned responses are: (1) sign-trackers–looking at the previously rewarded cue, resulting in slower reaction times; (2) goal-trackers–not distracted by previously rewarded cues. CS, conditioned stimuli; CR, conditioned response; US, unconditioned stimuli. Gray boxes highlight the definition of sign-and goal-tracking for each task.
Neural pathways that support sign-and goal-tracking phenotypes
The sign-and goal-tracking phenotypes have well-characterized differences in neural engagement during stimulus-reward learning and attribution of incentive salience. Sign-trackers show greater phasic dopaminergic (DA) signaling in ventral striatum, a region integral in stimulus-reward learning, which has been linked to the attribution of incentive salience to the CS (21, 22, 26, 28, 29). Sign-trackers also show a higher firing rate for excitatory signals in response to the CS in ventral pallidum (30), a subcortical region that is important for motivated behaviors and incentive salience (31). While both ventral striatum and pallidum have “hedonic hotspots” that enhance hedonic influence of the CS (31–33), incentive motivation or wanting of the CS (i.e., sign-tracking) seems to be driven by projections from ventral striatum to ventral pallidum (34). Although sign-tracking seems to be driven by these subcortical DA-related signaling differences, there are also important differences in cortical signaling. In particular, sign-trackers show cortical differences in acetylcholine (ACh), a neuromodulator that is important for attentional control and learning. In response to attentional demands, sign-trackers are less able to upregulate ACh which leads to stimulus-driven or bottom-up attention control [for review see (24)]. Therefore, sign-trackers show a pattern of greater signaling in subcortical “hedonic hotspots” in conjunction with a reduced cortical ACh signaling, which limits engagement top-down attentional control.
The pattern of greater bottom-up reward signaling and reduced top-down control signaling in sign-compared to goal-trackers is paralleled by circuit-level differences. Cue-motivated behaviors driven by incentive salience involve widespread circuits including cortical, thalamic pallidum, and striatal loops that converge in the ventral striatum (21, 33, 35). Sign-trackers have greater engagement of ventral and dorsal striatum (i.e., caudate-putamen) during stimulus-reward learning while goal-trackers show greater engagement of prefrontal cortical regions [for review see (21)]. Therefore, it has been hypothesized that cue-motivated behaviors are subserved by subcortical circuits while top-down cortical circuits inhibit the attribution of incentive salience to cues (21). Reduced engagement of cortical regions associated with top-down control may also contribute to greater impulsivity (36, 37) and reduced behavioral flexibility (38) observed in sign-trackers compared to goal-trackers. Together, this suggests neural differences between phenotypes contribute to differences in attribution of incentive salience and may also be related to differences in attentional control and impulsivity (21, 24).
Translation of sign-and goal-tracking phenotypes to humans
In humans, sign-and goal-tracking have been characterized using Pavlovian conditioning tasks (often as part of the Pavlovian instrumental transfer paradigm) and the Value-Drive Attentional Capture (VDAC) task. Using eye-tracking, incentive salience can be measured in Pavlovian conditioning tasks by examining the amount of time looking at the location of the CS compared to the location where reward is delivered (Figure 1B). Much like sign-tracking animals that fixate on the lever rather than location of food delivery, adult humans who spend more time looking at the location of the CS compared to the reward have also been classified as sign-trackers (39, 40). In line with the animal phenotype of sign-tracking, adults classified as sign-trackers during Pavlovian conditioning show greater impulsivity than those classified as goal-trackers (39). Similarly, VDAC tasks (Figure 1C) measure attentional bias toward high-value stimuli, however, these tasks assess this bias when the stimuli are no longer relevant to the task goal and are no longer rewarded (41–43). Continued attentional bias toward previous high-value stimuli–termed attentional capture–reflects the attribution of incentive salience to these stimuli (27, 43–45) and sign-tracking (27, 45). Greater attentional capture on the VDAC has been associated with greater compulsivity (45, 46) and impulsivity (41) as well as risk for substance use disorder (45). Together, this shows that behavioral profiles associated with sign-tracking have similarities in human and non-human animals (e.g., impulsivity, poor attentional control).
Relevance to food choice and eating behaviors
The sign-and goal-tracking phenotype model has high translational potential to inform our understanding of food choice and overconsumption. This is supported by animal studies which have shown that obesity-prone rats display greater attribution of incentive salience compared to obesity-resistant models (47). There is initial evidence that obesity is associated with cue-outcome behavioral responses that are indicative of sign-tracking. In a Pavlovian conditioning task that paired visual cues with receipt of chocolate milkshake, water, or nothing, adults with overweight showed the CR of increased swallowing in response to cues that predicted chocolate milkshake delivery while adults with healthy weight did not (48). This suggests that adults with overweight were more likely to attribute incentive salience to the cues that predicted chocolate milkshake receipt (i.e., sign-track) than those with healthy weight. Additionally, in adolescents, greater caudate and ventral pallidum activity is seen during Pavlovian cue-outcome learning for milkshake compared to water (49, 50) with greater ventral pallidum activity predicting greater increases in BMI 2 years later (49). This finding parallels greater ventral pallidum activity in animal models of sign-tracking (30), suggesting that this may be a common neural pathway for sign-tracking and may be associated with tendency to develop obesity.
The sign-tracking phenotype, in particular, may also play an important role in eating behaviors. While we are not aware of studies examining Pavlovian conditioning, there is one study showing that adults with greater eating restraint were less likely to attribute incentive salience to food cues in a VDAC task (51). This indicates that adults who report a greater tendency to restrict calories are less likely to attribute salience to food cues. There is also a larger literature examining attentional bias to food cues [see reviews (52–54)], which is an indirect measure of incentive salience (27). A recent meta-analysis examining direct [e.g., electroencephalographic (EEG) recordings, eye-tracking] and indirect (e.g., reaction times) measures of food-related attentional bias showed that greater bias was associated with greater hunger, food cravings, and food intake but not body mass index (52). In particular, EEG recordings may be a promising approach for characterizing sign-tracking as late positive event-related potentials (ERPs, e.g., P300 or late positive potentials–LPP) index motivational salience associated with cues (55, 56). In support of this, a recent study used a data-driven approach to cluster adults based on emotional and food-related LLPs with those classified as “sign-trackers” showing larger food-related LLP and higher rates of obesity compared to those classified as “goal-trackers” (57). While late positive ERPs to food-cues is a promising approach for measuring incentive salience and sign-tracking, there is mixed evidence for an association with obesity and binge eating disorder (53). Together, these studies highlight initial evidence that the tendency to attribute incentive salience to food cues (i.e., sign-track) may increase susceptibility to eating behaviors associated with overconsumption [for review of food-cue reactivity beyond incentive salience see (58)].
Based on initial evidence of its role in eating behaviors related to overconsumption and obesity, the sign-tracking phenotype may provide mechanistic insight on the role of food-cue incentive salience in two prevailing models of overconsumption–the Extended Behavioral Susceptibility Theory (59) and the Reactivity to Embedded Food Cues in Advertising Model (REFCAM; Figure 2; 60). The importance of food-cue incentive salience across models highlights its broad potential as a behavioral target for prevention and intervention efforts. For example, cue-exposure therapy aims to reduce food-cue incentive salience by repeatedly exposing participants to a food-cue without the CR of food intake [for review see (12, 61, 62)]. Thus far, cue-exposure therapy has focused on exposures to specific foods, which has been successful in reducing the number of binge eating episodes, number of binge eating days, intake of exposed food, and body weight (63–65) in adults with binge eating disorder and obesity (61–65). While cue-exposure therapy has shown effectiveness for individuals who have already developed food-specific cravings and overconsumption, it is not clear if this approach would be effective for targeting brand or advertising related cues as proposed in the REFCAM model. Additionally, it is not clear if targeting incentive salience would be more efficacious for reducing overconsumption in individuals with sign-tracking compared to goal-tracking phenotypes. Therefore, future work is needed to determine whether targeting individuals based on sign-and goal-tracking phenotypes will contribute to more effective and sustainable weight maintenance.
Figure 2. Adapted models with the sign-and goal-tracking and model-free and model-based learning frameworks incorporated. (A) Boutelle et al.’s (59) Extended Behavioral Susceptibility; (B) Folkvord et al.’s (60) Reactivity to Embedded Food Cues in Advertising Model; (C) Folkvord et al.’s (7) Healthy Food Promotion Model.
Model-free and model-based reinforcement learning
Reinforcement learning is the process through which environmental cues become valued and influence behavior (66). This process is driven by two competing systems–a habitual and a goal-driven system (15, 67–71). The habitual system drives model-free reinforcement learning which relies on stimulus-response (S-R) associations and is a fast, almost automatic, process that requires little cognitive effort (72). For example, stopping for coffee at the same coffee shop on the way to work every day is likely a habitual process. Model-free learning increases the probability of choosing actions that were most recently rewarded, which leads to less accurate and flexible responses. In contrast, the goal-directed system drives model-based learning because it relies on a mental model or cognitive map of the expected value of different responses (i.e., R-O associations) for different “states” or environmental situations. For example, if the coffee shop is closed for maintenance, a goal-directed process is needed to change the morning routine and make coffee at home. Model-based learning leads to more flexible responses; however, it is also more cognitively demanding. These reinforcement learning strategies operate in parallel, with optimal value-based decision-making balancing the need for accuracy with cognitive demand (67, 70, 73, 74).
Neural pathways that support model-free and model-based learning
Reinforcement learning processes rely on neural encoding of prediction errors, which are used to update outcome expectations and improve accuracy. Model-free learning depends on reward prediction errors (RPEs). A RPE is the difference between the expected outcome and the actual outcome. For example, if someone orders their morning coffee and receives a free donut, that would be a positive RPE. In contrast, if someone orders their morning coffee and receives decaffeinated coffee, that would be a negative RPE. RPEs are encoded by phasic DA signaling in the basal ganglia, which includes ventral striatum, caudate-putamen, and dorsal pallidum (29, 75). In contrast, model-based learning relies on a cognitive model of a task or environmental reward structure so learning is driven by state prediction errors (SPEs). A SPE is the difference between the expected “state” and the actual “state” (70). For example, arriving at coffee shop in the morning and finding it closed for maintenance would be a SPE. SPEs are thought to be encoded by lateral prefrontal cortex, intraparietal sulcus, and anterior cingulate (70, 76, 77). While the neural systems supporting RPEs and SPEs are partially distinct, both model-free and model-based learning include value-based signaling associated with ventral striatal activation (68, 70, 73, 78, 79). A recent meta-analysis showed that in addition to ventral striatum, model-free learning specifically engaged dorsal striatum and dorsal pallidum while model-based learning specifically engaged ventral medial prefrontal cortex and anterior cingulate cortex (79). In addition to regions supporting SPEs, model-based learning also involves dorsolateral prefrontal cortex, orbital frontal cortex, posterior parietal cortex, and hippocampus to support the mental model of different states (15, 71, 80). Given these learning strategies likely operate in parallel (67, 70, 73, 74), common neural correlates for these strategies may help to mediate switching between model-free and model-based learning (15, 69, 81).
Characterizing model-free and model-based learning
The advent of computational models for reinforcement learning has propelled our ability to distinguish model-free and model-based learning processes. In particular, the dual-system model incorporates both model-free and model-based algorithms (68, 73) which allows for individual differences in the balance of these systems to be examined. A task structure that leverages the dual-system model is the two-step or serial decision-making task (73, 82). This task involves a series of decisions between two stages. Actions in the first stage lead probabilistically to one of two second-stage states (i.e., high versus low transition probability; Figure 3A). Decisions made in the second-stage then lead to different probabilities of reward, which change or drift slowly throughout the task to encourage learning. The transition structure between stages allows for model-based and model-free strategies to be distinguished. In particular, model-free learners are more likely to repeat an action after a rare or low probability reward due to positive RPE. In contrast, model-based learners will experience a SPE and will be less likely to repeat the action due to the overall low probability of reward. This task has also been adapted to enhance the accuracy-demand tradeoff such that model-based strategies will lead to greater reward (82). In the adapted version, the transitions between the stages are deterministic rather than probabilistic (Figure 3B). Overall, greater use of model-free learning has also been associated with poorer working memory (83, 84), cognitive control (85), and processing speed (86). Therefore, greater reliance on model-free learning during this task is thought to reflect less adaptive reinforcement learning.
Figure 3. Methods of assessing model-based and model-free reinforcement learning using two-step serial decision-making tasks. Two-step tasks have two stages of decisions with the second stage state dependent upon the first stage choice. (A) Daw et al.’s (68) task that uses a probabilistic transition from stage 1 to 2. Gray box highlights the theoretically expected probability of repeating a stage 1 choice for model-free and model-based learning based on previous trial reward and transition probability. (B) Kool et al.’s (82) task that uses a deterministic transition from stage 1 to 2. Gray box highlights the theoretically expected probability of repeating a stage 1 choice for model-free and model-based learning based on previous trial reward and whether the current trial stage 1 state differs from the previous trial’s stage 1 state. (C) Miller et al.’s (80) translation of a two-stage task for non-human animals with probabilistic state transitions. (D) Groman et al.’s (87) translation of a two-stage task for non-human animals with deterministic state transitions.
While two-step tasks were first developed for human studies, translational applications of the task to rodent models [e.g., (76, 80, 87; Figures 3C,D)] has shown similar patterns of behavior as seen in humans [for reviews on other animal models of habit see (88, 89)]. Animals show evidence of both model-free and model-based learning and evidence for switching between strategies (76, 87, 90, 91). An advantage to animal models is that ability to measure reinforcement learning before and after drug exposure. Drug-naïve animals with less model-free learning exhibited greater subsequent drug administration in animals, while use of model-based learning did not predict subsequent drug administration (92). However, after drug self-administration, rodents showed a reduction in both model-free and model-based learning (92). While this study used a computational model that quantified use of model-free and model-based strategies independently, studies in humans tend to look at the relative use of learning strategies (68, 73) and have shown relatively more model-free than model-based learning in drug users (93). Together, this highlights the importance of having translational assays of decision-making frameworks to better understand behavioral and neural mechanisms of reinforcement learning.
Relevance to food choice and eating behaviors
While the advent of the dual-system model and two-stage task has led to a swell of research on individual differences in reinforcement learning, little work has directly tested the role of reinforcement learning in food choice and obesity. Of the two studies we are aware of that have directly tested this association one showed greater reliance on model-free learning in adults with obesity compared to those without (94) and one showed no relationship between weight status and reinforcement learning (93). Additionally, model-free learning has been associated with psychological disorders marked by compulsivity including addiction, gambling disorder, obsessive compulsive disorder, and binge eating disorder (93, 95, 96). Model-free learning has been indirectly implicated in overconsumption (97) due to the contribution of compulsivity in habitual overeating (98, 99). Model-free learning may also contribute insight into the Extended Behavioral Susceptibility model, which proposes that “habitual” or instrumental systems contribute to overconsumption (Figures 2A,B; 59). While the majority of the literature and prevailing theories have focused on overconsumption, the Healthy Food Promotion Model proposes that habit learning can be leveraged to bolster intake of fruit and vegetables (Figure 2C; 100). Together, this suggests that interventions that leverage habit learning strategies may be able to increase healthy eating behaviors, but future studies are needed to test this empirically.
Utility model
Value-based decision-making often involves choosing between multiple actions that could lead to different advantageous outcomes. In utility or value-based decision-making models, every action has an expected value or utility (i.e., action-outcome association) and the action with the highest expected value will be selected. Values associated with different actions are integrated across Pavlovian, instrumental, and goal-directed systems (11) for different consequences of that action, termed attributes. For example, choosing to eat at a restaurant rather than at home could occur because the cumulative value of convenience and taste of food at the restaurant is greater than the value of cost saving and alignment with health goals for eating at home. Thus, the cumulative value of an action integrates both positive and negative value signals across learning systems and attributes. Additionally, the weight of the value signals from different learning systems can be influenced by individual characteristics, such as delay discounting (101, 102). Individuals who value smaller, immediate rewards more than larger, delayed rewards may be more influenced by value signals from the Pavlovian or instrumental systems than the goal-directed system. Further, environmental cues can modulate the weight given to different attributes (e.g., taste, health) (103). For example, an advertisement that draws attention to the palatable aspects of food may increase the value of taste when choosing what to eat. Therefore, value-based decisions are influenced by the subjective value of relevant attributes in addition to self-regulation and environmental contexts.
Neural pathways that support value-based decision-making
Value-based decision-making relies on the integration of multiple value signals across different learning systems. To compare value signals across dissimilar actions (e.g., take a lunch break or continue reading this paper), a “common currency” or value is encoded in the brain (104, 105). Neuroimaging research suggests that this common value signal is encoded in ventromedial prefrontal cortex and medial orbitofrontal cortex, while value signals for distinct attributes are encoded throughout the brain (106, 107). A meta-analysis showed that when executing reward-based decisions, valuations of different types of reward (e.g., food, money) were associated with activation in ventromedial prefrontal cortex, ventral striatum, posterior cingulate cortex, and superior frontal gyrus; however, only ventromedial prefrontal cortex activity was related to valuations for each reward modality separately (108). This suggests ventromedial prefrontal cortex is a key region for encoding subjective value of both primary rewards like food and secondary rewards like money during decisions. Dorsolateral prefrontal cortex has also been shown to modulate ventromedial prefrontal cortex value signaling during self-control (109, 110) and during context-dependent valuation (111) indicating the importance of both regions in goal-directed decisions. In sum, attribute-specific value signals across the brain are integrated in ventromedial prefrontal cortex, which can be modulated by dorsolateral prefrontal cortex when self-control is engaged or environmental context is important.
Characterizing value-based decision-making
Characterizing value-based decision-making can involve assessing overall value of an action or stimuli, assessing how different attributes impact overall value, or assessing how psychological and environmental characteristics impact value-based decisions. To estimate the overall expected value of an action, participants can rate how much they want (e.g., strong yes, yes, no, strong no) or how much they are willing to pay for an item (112). Direct ratings of the value of different attributes (e.g., health or taste) have been shown to relate to real world behaviors such as fruit and vegetable intake (113) and smoking initiation (114). These ratings can also be used to examine how attributes influence value-based decision-making by asking participants to make choices between the items. For example, after rating the health and taste of foods, the influence of these attributes on food choice can be examined by having participants choose between food items that differ in taste and health attributes (109, 115–118). Assessing mouse-tracking during these decisions can provide insight into how attributes impact value-based decisions. For example, mouse-tracking trajectories have been used to measure the cognitive effort required to make healthy choices in children (117) and determine when different attributes impact the decision-making process (115, 118). Computational models of decision-making can also be used to examine individual differences in decision-making processes when choosing among options that vary in value. For example, in the Iowa Gambling Task (119) or its adapted child version the Hungry Donkey Task (120), participants try to accumulate as many rewards as possible by repeatedly choosing between four options associated with different reward and punishment probabilities. Computational models can characterize decision-making processes such as how value is updated, consistency between valuation and choice, loss aversion, and sensitivity to the magnitude of gains and losses (121–126). Together, these approaches can be used to understand how individual differences in valuation or cognitive and psychological process relate to disordered or dysregulated eating behaviors.
Relevance to food choice and eating behaviors
Food choice and eating behaviors require the evaluation of multiple food-related attributes (e.g., taste, health) in addition to personal goals and environmental cues. Taste and health ratings are predictive of food choices in adults (127, 128), however, the impact of these attributes on decisions varies among individuals (109, 128) and can be altered following exposure to taste and health cues (103). These behavioral differences are underpinned by differences in ventromedial and dorsolateral prefrontal cortex activation during decisions (103, 109). In children, taste is more predictive of food choices than health ratings (129, 130), although the temporal dynamics of taste and health attributes on children’s food choices vary by children’s hunger and weight status (115). Additionally, children’s food choices have been shown to be influenced by what they believe their mothers would choose for them (130). For both children and adults, food choices are impacted by many attributes including expectations about the likelihood of feeling satisfied and happy, feeling in control of one’s behavior, eliminating hunger, cost, and convenience (113, 128, 131). This suggests that in addition to food-related attributes, social context, and individual characteristics (132) influence value-based food choices. Understanding the individual characteristics and environmental contexts that influence the value of certain eating behaviors could contribute to interventions that increase the value and selection of foods that optimize health.
Value-based decision-making models complement the other models discussed in this review (Figure 2). For example, a value-based perspective of the Extended Behavioral Susceptibility Theory would suggest that social and environmental factors, genes, and metabolic signals increase the valuation of food cues (i.e., food responsiveness) relative to satiety signals (i.e., satiety responsiveness), contributing to a positive energy balance. Similarly, a value-based perspective of REFCAM would be that food advertisements subconsciously increase the value of food through incentive sensitization, which increases the likelihood of consumption. Correspondingly, interventions that modulate value from social and environmental attributes could lead to changes in food intake. This may include techniques such as cognitive reappraisal and food cue-exposure, which could reduce the value of food cues and increase the relative influence of goal-directed values on food choice. Additionally, manipulations that increase the self-relevancy of goals or influence delay discounting for food may have the potential to influence eating behaviors through their impact on valuation (101). Future research should assess ways to modify food-related value signals across learning systems and attributes and identify who would benefit most from these interventions.
Discussion
This paper presented three neurocognitive frameworks that could help to advance our understanding of the neurocognitive processes that underly food choices, a critical step toward the development of effective, tailored health interventions. These frameworks support and may help provide mechanistic insight to prominent models for food choice and overconsumption such as the Extended Behavioral Susceptibility model, REFCAM, and the Healthy Food Promotion Model. The sign-and goal-tracking framework can help to provide insight in behavioral phenotypes that may be more susceptible to the attribution of incentive salience to food cues, which could increase craving and overconsumption. The model-free versus model-based framework provides computational models that could be used to better understand habitual intake and compulsive overeating. Finally, the utility or value-based decision-making model provides a framework for understanding how value signals from all three learning systems could be integrated to influence food choice.
The primary advantage of utilizing neurocognitive frameworks is the ability to directly probe valuation and reinforcement learning processes that drive food choice and overconsumption. As the frameworks presented here involve but distinct reward-learning processes, it is often not possible to distinguish causal mechanisms without task behavior. For example, while obesity (133–136), future weight gain (49, 137, 138), and greater food intake (139–141) have all been associated with greater food-cue reactivity in ventral striatum [see (58) for review on neural food-cue reactivity], the interpretation of these findings may differ based on which framework is referenced. Under the sign-and goal-tracking framework, this pattern of results could be interpreted as evidence that greater attribution of incentive salience to food cues drives obesity and overconsumption. In contrast, under the model-free and model-based learning framework, this pattern of results would not be sufficient to make a distinction as both strategies engage ventral striatum (79). However, when considered along with consistent evidence that greater prefrontal cortex engagement is associated with healthy weight (133–136) and lower food intake (142, 143), the combined pattern of results may be interpreted as evidence that greater reliance on model-based strategies is associated with lower weight status and food intake. Alternatively, when using the utility or value-based decision-making framework, the combined pattern of findings could be interpreted as evidence that greater relative value for goal-directed than hedonic values when viewing food cues is protective from excess consumption and adiposity. Therefore, future studies need to assess both neural food-cue reactivity and reward learning. In order to determine how these frameworks mechanistically contribute to different aspects of food choice and overconsumption, ingestive behavior needs to be characterized alongside reward learning and neuroimaging.
All three of these frameworks have utility for better understanding food choice and overconsumption; the choice of which framework(s) to reference ultimately depends on the theory of eating behavior and hypotheses being tested. The sign-and goal-tracking framework enables one to test very specific hypotheses related to the attribution of incentive salience to food cues and its role in motivated behavior such as craving. Model-free and model-based reinforcement learning provides a broader framework to examine reinforcement learning and its role in habitual or compulsive overeating. Lastly, the utility or value-based decision-making theory provides a larger framework to understand how valuation and reinforcement learning processes interact across behavioral control systems during food choice. In sum, applying these frameworks to provide mechanistic insight of prominent models of food choice and overconsumption may eventually contribute to more informed prevention and treatment efforts.
Author contributions
AP conceptualized the topic and structure and lead the writing and editing of manuscript. BF and KK helped to write and edit the manuscript. All authors contributed to the article and approved the submitted version.
Funding
This study was funded by National Center for Advancing Translational Sciences KL2 TR002015 and UL1 TR002014; 1F31DK131868-01A1, R01DK126050, and R01DK110060.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
1. Rangel A. Regulation of dietary choice by the decision-making circuitry. Nat Neurosci. (2013) 16:1717–24. doi: 10.1038/nn.3561
2. Doucerain M, Fellows LK. Eating right: linking food-related decision-making concepts from neuroscience, psychology, and education. Mind Brain Educ. (2012) 6:206–19. doi: 10.1111/j.1751-228X.2012.01159.x
3. Sobal J, Bisogni CA, Jastran M. Food choice is multifaceted, contextual, dynamic, multilevel, integrated, and diverse: food choice complexity. Mind Brain Educ. (2014) 8:6–12. doi: 10.1111/mbe.12044
4. Marijn Stok F, Renner B, Allan J, Boeing H, Ensenauer R, Issanchou S, et al. Dietary behavior: an interdisciplinary conceptual analysis and taxonomy. Front Psychol. (2018) 9:1689. doi: 10.3389/fpsyg.2018.01689
5. Enax L, Weber B. Neurobiology of food choices-between energy homeostasis, reward system, and neuroeconomics. E Neuroforum. (2016) 7:13–22. doi: 10.1007/s13295-015-0020-0
6. Birch LL, Anzman SL. Learning to eat in an obesogenic environment: a developmental systems perspective on childhood obesity. Child Dev Perspect. (2010) 4:138–43. doi: 10.1111/j.1750-8606.2010.00132.x
7. Folkvord F, Veling H, Hoeken H. Targeting implicit approach reactions to snack food in children: effects on intake. Health Psychol. (2016) 35:919–22. doi: 10.1037/hea0000365
8. Forouhi NG, Unwin N. Global diet and health: old questions, fresh evidence, and new horizons. Lancet. (2019) 393:1916–8. doi: 10.1016/S0140-6736(19)30500-8
9. Afshin A, Sur PJ, Fay KA, Cornaby L, Ferrara G, Salama JS, et al. Health effects of dietary risks in 195 countries, 1990–2017: a systematic analysis for the Global Burden of Disease Study 2017. Lancet. (2019) 393:1958–72.
10. Hu K, Staiano AE. Trends in obesity prevalence among children and adolescents aged 2 to 19 years in the US from 2011 to 2020. JAMA Pediatr. (2022) 176:1037–9. doi: 10.1001/jamapediatrics.2022.2052
11. Rangel A, Camerer C, Montague PR. A framework for studying the neurobiology of value-based decision making. Nat Rev Neurosci. (2008) 9:545–56. doi: 10.1038/nrn2357
12. Boutelle KN, Bouton ME. Implications of learning theory for developing programs to decrease overeating. Appetite. (2015) 93:62–74. doi: 10.1016/j.appet.2015.05.013
13. Skinner BF. Conditioning and extinction and their relation to drive. J Gen Psychol. (1936) 14:296–317. doi: 10.1080/00221309.1936.9713156
14. Dayan P, Niv Y, Seymour BD, Daw N. The misbehavior of value and the discipline of the will. Neural Netw. (2006) 19:1153–60. doi: 10.1016/j.neunet.2006.03.002
15. O’Doherty JP, Cockburn J, Pauli WM. Learning, reward, and decision making. Annu Rev Psychol. (2017) 68:73–100. doi: 10.1146/annurev-psych-010416-044216
16. Juechems K, Summerfield C. Where does value come from? Trends Cogn Sci. (2019) 23:836–50. doi: 10.1016/j.tics.2019.07.012
17. Simon JJ, Wetzel A, Sinno MH, Skunde M, Bendszus M, Preissl H, et al. Integration of homeostatic signaling and food reward processing in the human brain. JCI Insight. (2017) 2:e92970. doi: 10.1172/jci.insight.92970
18. Dan O, Wertheimer EK, Levy I. A neuroeconomics approach to obesity. Biol Psychiatry. (2021) 91:860–8. doi: 10.1016/j.biopsych.2021.09.019
19. Anselme P. Motivational control of sign-tracking behaviour: a theoretical framework. Neurosci Biobehav Rev. (2016) 65:1–20. doi: 10.1016/j.neubiorev.2016.03.014
20. Robinson TE, Yager LM, Cogan ES, Saunders BT. On the motivational properties of reward cues: individual differences. Neuropharmacology. (2014) 76:450–9. doi: 10.1016/j.neuropharm.2013.05.040
21. Flagel SB, Robinson TE. Neurobiological basis of individual variation in stimulus-reward learning. Curr Opin Behav Sci. (2017) 13:178–85. doi: 10.1016/j.cobeha.2016.12.004
22. Flagel SB, Robinson TE, Clark JJ, Clinton SM, Watson SJ, Seeman P, et al. An animal model of genetic vulnerability to behavioral disinhibition and responsiveness to reward-related cues: implications for addiction. Neuropsychopharmacology. (2010) 35:388–400. doi: 10.1038/npp.2009.142
23. Boakes RA, Poli M, Lockwood MJ, Goodall G. A study of misbehavior: token reinforcement in the rat. J Exp Anal Behav. (1978) 29:115–34. doi: 10.1901/jeab.1978.29-115
24. Sarter M, Phillips KB. The neuroscience of cognitive-motivational styles: sign- and goal-trackers as animal models. Behav Neurosci. (2018) 132:1–12. doi: 10.1037/bne0000226
25. Meyer PJ, Lovic V, Saunders BT, Yager LM, Flagel SB, Morrow JD, et al. Quantifying individual variation in the propensity to attribute incentive salience to reward cues. PLoS One. (2012) 7:e38987. doi: 10.1371/journal.pone.0038987
26. Robinson TE, Flagel SB. Dissociating the predictive and incentive motivational properties of reward-related cues through the study of individual differences. Biol Psychiatry. (2009) 65:869–73. doi: 10.1016/j.biopsych.2008.09.006
27. Colaizzi JM, Flagel SB, Joyner MA, Gearhardt AN, Stewart JL, Paulus MP. Mapping sign-tracking and goal-tracking onto human behaviors. Neurosci Biobehav Rev. (2020) 111:84–94. doi: 10.1016/j.neubiorev.2020.01.018
28. Flagel SB, Clark JJ, Robinson TE, Mayo L, Czuj A, Willuhn I, et al. A selective role for dopamine in stimulus–reward learning. Nature. (2011) 469:53–7. doi: 10.1038/nature09588
29. Schultz W. Dopamine reward prediction-error signalling: a two-component response. Nat Rev Neurosci. (2016) 17:183–95. doi: 10.1038/nrn.2015.26
30. Ahrens AM, Ferguson LM, Robinson TE, Aldridge JW. Dynamic encoding of incentive salience in the ventral pallidum: dependence on the form of the reward cue. Eneuro. (2018) 5:ENEURO.328–317. doi: 10.1523/ENEURO.0328-17.2018
31. Smith KS, Tindell AJ, Aldridge JW, Berridge KC. Ventral pallidum roles in reward and motivation. Behav Brain Res. (2009) 196:155–67. doi: 10.1016/j.bbr.2008.09.038
32. Castro DC, Cole SL, Berridge KC. Lateral hypothalamus, nucleus accumbens, and ventral pallidum roles in eating and hunger: interactions between homeostatic and reward circuitry. Front Syst Neurosci. (2015) 9:90. doi: 10.3389/fnsys.2015.00090
33. Morales I, Berridge KC. ‘Liking’ and ‘wanting’ in eating and food reward: brain mechanisms and clinical implications. Physiol Behav. (2020) 227:113152. doi: 10.1016/j.physbeh.2020.113152
34. Smedley EB, DiLeo A, Smith KS. Circuit directionality for motivation: lateral accumbens-pallidum, but not pallidum-accumbens, connections regulate motivational attraction to reward cues. Neurobiol Learn Mem. (2019) 162:23–35. doi: 10.1016/j.nlm.2019.05.001
35. Haber SN. The place of dopamine in the cortico-basal ganglia circuit. Neuroscience. (2014) 282:248–57. doi: 10.1016/j.neuroscience.2014.10.008
36. Lovic V, Saunders BT, Yager LM, Robinson TE. Rats prone to attribute incentive salience to reward cues are also prone to impulsive action. Behav Brain Res. (2011) 223:255–61. doi: 10.1016/j.bbr.2011.04.006
37. Enkel T, Bartsch D, Bähner F. Sign- and goal-tracking rats show differences in various executive functions. Behav Brain Res. (2019) 371:111979. doi: 10.1016/j.bbr.2019.111979
38. Nasser HM, Chen YW, Fiscella K, Calu DJ. Individual variability in behavioral flexibility predicts sign-tracking tendency. Front Behav Neurosci. (2015) 9:289. doi: 10.3389/fnbeh.2015.00289
39. Garofalo S, di Pellegrino G. Individual differences in the influence of task-irrelevant Pavlovian cues on human behavior. Front Behav Neurosci. (2015) 9:163. doi: 10.3389/fnbeh.2015.00163
40. Schad DJ, Rapp MA, Garbusow M, Nebe S, Sebold M, Obst E, et al. Dissociating neural learning signals in human sign- and goal-trackers. Nat Hum Behav. (2020) 4:201–14. doi: 10.1038/s41562-019-0765-5
41. Anderson BA, Laurent PA, Yantis S. Value-driven attentional capture. Proc Natl Acad Sci U.S.A. (2011) 108:10367–71. doi: 10.1073/pnas.1104047108
42. Le Pelley ME, Pearson D, Griffiths O, Beesley T. When goals conflict with values: counterproductive attentional and oculomotor capture by reward-related stimuli. J Exp Psychol Gen. (2015) 144:158–71. doi: 10.1037/xge0000037
43. Anderson BA, Folk CL, Garrison R, Rogers L. Mechanisms of habitual approach. J Exp Psychol Gen. (2016) 145:796–805. doi: 10.1037/xge0000169
44. Anderson BA. The attention habit: how reward learning shapes attentional selection: the attention habit. Ann N Y Acad Sci. (2016) 1369:24–39. doi: 10.1111/nyas.12957
45. Albertella L, Le Pelley ME, Chamberlain SR, Westbrook F, Fontenelle LF, Segrave R, et al. Reward-related attentional capture is associated with severity of addictive and obsessive–compulsive behaviors. Psychol Addict Behav. (2019) 33:495–502. doi: 10.1037/adb0000484
46. Albertella L, Chamberlain SR, Le Pelley ME, Greenwood LM, Lee RS, Den Ouden L, et al. Compulsivity is measurable across distinct psychiatric symptom domains and is associated with familial risk and reward-related attentional capture. CNS Spectr. (2020) 25:519–26. doi: 10.1017/S1092852919001330
47. Ferrario CR. Why did I eat that? Contributions of individual differences in incentive motivation and nucleus accumbens plasticity to obesity. Physiol Behav. (2020) 227:113114. doi: 10.1016/j.physbeh.2020.113114
48. Meyer MD, Risbrough VB, Liang J, Boutelle KN. Pavlovian conditioning to hedonic food cues in overweight and lean individuals. Appetite. (2015) 87:56–61. doi: 10.1016/j.appet.2014.12.002
49. Burger KS, Stice E. Greater striatopallidal adaptive coding during cue–reward learning and food reward habituation predict future weight gain. Neuroimage. (2014) 99:122–8. doi: 10.1016/j.neuroimage.2014.05.066
50. Sadler JR, Shearrer GE, Papantoni A, Yokum ST, Stice E, Burger KS. Correlates of neural adaptation to food cues and taste: the role of obesity risk factors. Soc Cogn Affect Neurosci. (2021) nsab018. doi: 10.1093/scan/nsab018
51. Watson P, Vasudevan A, Pearson D, Le Pelley ME. Eating restraint is associated with reduced attentional capture by signals of valuable food reward. Appetite. (2021) 159:105050. doi: 10.1016/j.appet.2020.105050
52. Hardman CA, Jones A, Burton S, Duckworth JJ, McGale LS, Mead BR, et al. Food-related attentional bias and its associations with appetitive motivation and body weight: a systematic review and meta-analysis. Appetite. (2021) 157:104986. doi: 10.1016/j.appet.2020.104986
53. Chami R, Cardi V, Lautarescu A, Mallorquí-Bagué N, McLoughlin G. Neural responses to food stimuli among individuals with eating and weight disorders: a systematic review of event-related potentials. Int Rev Psychiatry. (2019) 31:318–31. doi: 10.1080/09540261.2019.1622515
54. Carbine KA, Rodeback R, Modersitzki E, Miner M, LeCheminant JD, Larson MJ. The utility of event-related potentials (ERPs) in understanding food-related cognition: a systematic review and recommendations. Appetite. (2018) 128:58–78. doi: 10.1016/j.appet.2018.05.135
55. Olofsson JK, Nordin S, Sequeira H, Polich J. Affective picture processing: an integrative review of ERP findings. Biol Psychol. (2008) 77:247–65. doi: 10.1016/j.biopsycho.2007.11.006
56. Schupp HT, Flaisch T, Stockburger J, Junghöfer M. Emotion and attention: event-related brain potential studies. Prog Brain Res. (2006) 156:31–51. doi: 10.1016/S0079-6123(06)56002-9
57. Versace F, Kypriotakis G, Basen-Engquist K, Schembre SM. Heterogeneity in brain reactivity to pleasant and food cues: evidence of sign-tracking in humans. Soc Cogn Affect Neurosci. (2016) 11:604–11. doi: 10.1093/scan/nsv143
58. Kanoski SE, Boutelle KN. Food cue reactivity: neurobiological and behavioral underpinnings. Rev Endocr Metab Disord. (2022) 23:683–96. doi: 10.1007/s11154-022-09724-x
59. Boutelle KN, Manzano MA, Eichen DM. Appetitive traits as targets for weight loss: the role of food cue responsiveness and satiety responsiveness. Physiol Behav. (2020) 224:113018. doi: 10.1016/j.physbeh.2020.113018
60. Folkvord F, Anschütz DJ, Boyland E, Kelly B, Buijzen M. Food advertising and eating behavior in children. Curr Opin Behav Sci. (2016) 9:26–31. doi: 10.1016/j.cobeha.2015.11.016
61. van den Akker K, Schyns G, Jansen A. Learned overeating: applying principles of pavlovian conditioning to explain and treat overeating. Curr Addict Rep. (2018) 5:223–31. doi: 10.1007/s40429-018-0207-x
62. Magson NR, Handford CM, Norberg MM. The empirical status of cue exposure and response prevention treatment for binge eating: a systematic review. Behav Ther. (2021) 52:442–54. doi: 10.1016/j.beth.2020.06.005
63. Boutelle KN, Knatz S, Carlson J, Bergmann K, Peterson CB. An open trial targeting food cue reactivity and satiety sensitivity in overweight and obese binge eaters. Cogn Behav Pract. (2017) 24:363–73. doi: 10.1016/j.cbpra.2016.08.003
64. Schyns G, van den Akker K, Roefs A, Houben K, Jansen A. Exposure therapy vs lifestyle intervention to reduce food cue reactivity and binge eating in obesity: a pilot study. J Behav Ther Exp Psychiatry. (2020) 67:101453. doi: 10.1016/j.jbtep.2019.01.005
65. Boutelle KN, Eichen DM, Peterson CB, Strong DR, Kang-Sim DJE, Rock CL, et al. Effect of a novel intervention targeting appetitive traits on body mass index among adults with overweight or obesity: a randomized clinical trial. JAMA Netw Open. (2022) 5:e2212354. doi: 10.1001/jamanetworkopen.2022.12354
66. Pavlov I. Conditioned Reflexes: An Investigation of the Physiological Activity of the Cerebral Cortex. (1927). Available online at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4116985/ (accessed November 11, 2020).
68. Daw ND, Niv Y, Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci. (2005) 8:1704–11. doi: 10.1038/nn1560
69. Balleine BW, O’Doherty JP. Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology. (2010) 35:48–69. doi: 10.1038/npp.2009.131
70. Gläscher J, Daw N, Dayan P, O’Doherty JP. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron. (2010) 66:585–95. doi: 10.1016/j.neuron.2010.04.016
71. Dolan RJ, Dayan P. Goals and habits in the brain. Neuron. (2013) 80:312–25. doi: 10.1016/j.neuron.2013.09.007
72. Thorndike E. Animal Intelligence; Experimental Studies. New York, NY: The Macmillan company (1911). doi: 10.5962/bhl.title.55072
73. Daw ND, Gershman SJ, Seymour B, Dayan P, Dolan RJ. Model-based influences on humans’ choices and striatal prediction errors. Neuron. (2011) 69:1204–15. doi: 10.1016/j.neuron.2011.02.027
74. Kool W, Gershman SJ, Cushman FA. Cost-benefit arbitration between multiple reinforcement-learning systems. Psychol Sci. (2017) 28:1321–33. doi: 10.1177/0956797617708288
75. Glimcher PW. Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis. Proc Natl Acad Sci U.S.A. (2011) 108(Suppl. 3):15647–54. doi: 10.1073/pnas.1014269108
76. Akam T, Rodrigues-Vaz I, Marcelo I, Zhang X, Pereira M, Oliveira RF, et al. The anterior cingulate cortex predicts future states to mediate model-based action selection. Neuron. (2021) 109:149–63.e7. doi: 10.1016/j.neuron.2020.10.013
77. Bartolo R, Averbeck BB. Prefrontal cortex predicts state switches during reversal learning. Neuron. (2020) 106:1044–54.e4. doi: 10.1016/j.neuron.2020.03.024
78. Averbeck B, O’Doherty JP. Reinforcement-learning in fronto-striatal circuits. Neuropsychopharmacology. (2022) 47:147–62. doi: 10.1038/s41386-021-01108-0
79. Huang Y, Yaple ZA, Yu R. Goal-oriented and habitual decisions: neural signatures of model-based and model-free learning. Neuroimage. (2020) 215:116834. doi: 10.1016/j.neuroimage.2020.116834
80. Miller KJ, Botvinick MM, Brody CD. Dorsal hippocampus contributes to model-based planning. Nat Neurosci. (2017) 20:1269–76. doi: 10.1038/nn.4613
81. O’Doherty JP, Lee SW, McNamee D. The structure of reinforcement-learning mechanisms in the human brain. Curr Opin Behav Sci. (2015) 1:94–100. doi: 10.1016/j.cobeha.2014.10.004
82. Kool W, Cushman FA, Gershman SJ. When does model-based control pay off? PLoS Comput Biol. (2016) 12:e1005090. doi: 10.1371/journal.pcbi.1005090
83. Otto AR, Raio CM, Chiang A, Phelps EA, Daw ND. Working-memory capacity protects model-based learning from stress. Proc Natl Acad Sci U.S.A. (2013) 110:20941–6. doi: 10.1073/pnas.1312011110
84. Eppinger B, Walter M, Heekeren HR, Li SC. Of goals and habits: age-related and individual differences in goal-directed decision-making. Front Neurosci. (2013) 7:253. doi: 10.3389/fnins.2013.00253
85. Otto AR, Skatova A, Madlon-Kay S, Daw ND. Cognitive control predicts use of model-based reinforcement learning. J Cogn Neurosci. (2015) 27:319–33. doi: 10.1162/jocn_a_00709
86. Schad DJ, Junger E, Sebold M, Garbusow M, Bernhardt N, Javadi AH, et al. Processing speed enhances model-based over model-free reinforcement learning in the presence of high working memory functioning. Front Psychol. (2014) 5:1450. doi: 10.3389/fpsyg.2014.01450
87. Groman SM, Massi B, Mathias SR, Curry DW, Lee D, Taylor JR. Neurochemical and behavioral dissections of decision-making in a rodent multistage task. J Neurosci. (2019) 39:295–306. doi: 10.1523/JNEUROSCI.2219-18.2018
88. Vandaele Y, Ahmed SH. Habit, choice, and addiction. Neuropsychopharmacology. (2021) 46:689–98. doi: 10.1038/s41386-020-00899-y
89. McKim TH, Shnitko TA, Robinson DL, Boettiger CA. Translational research on habit and alcohol. Curr Addict Rep. (2016) 3:37–49. doi: 10.1007/s40429-016-0089-8
90. Hasz BM, Redish AD. Deliberation and procedural automation on a two-step task for rats. Front Integr Neurosci. (2018) 12:30. doi: 10.3389/fnint.2018.00030
91. Dezfouli A, Balleine BW. Learning the structure of the world: the adaptive nature of state-space and action representations in multi-stage decision-making. PLoS Comput Biol. (2019) 15:e1007334. doi: 10.1371/journal.pcbi.1007334
92. Groman SM, Massi B, Mathias SR, Lee D, Taylor JR. Model-free and model-based influences in addiction-related behaviors. Biol Psychiatry. (2019) 85:936–45. doi: 10.1016/j.biopsych.2018.12.017
93. Voon V, Derbyshire K, Rück C, Irvine MA, Worbe Y, Enander J, et al. Disorders of compulsivity: a common bias towards learning habits. Mol Psychiatry. (2015) 20:345–52. doi: 10.1038/mp.2014.44
94. Janssen LK, Mahner FP, Schlagenhauf F, Deserno L, Horstmann A. Reliance on model-based and model-free control in obesity. PsyArXiv (2019). Available online at: https://osf.io/6s47t (accessed October 22, 2020). doi: 10.31234/osf.io/6s47t
95. Voon V, Reiter A, Sebold M, Groman S. Model-based control in dimensional psychiatry. Biol Psychiatry. (2017) 82:391–400. doi: 10.1016/j.biopsych.2017.04.006
96. Gillan CM, Robbins TW, Sahakian BJ, van den Heuvel OA, van Wingen G. The role of habit in compulsivity. Eur Neuropsychopharmacol. (2016) 26:828–40. doi: 10.1016/j.euroneuro.2015.12.033
97. Kakoschke N, Aarts E, Verdejo-García A. The cognitive drivers of compulsive eating behavior. Front Behav Neurosci. (2019) 12:338. doi: 10.3389/fnbeh.2018.00338
98. Moore CF, Sabino V, Koob GF, Cottone P. Pathological overeating: emerging evidence for a compulsivity construct. Neuropsychopharmacology. (2017) 42:1375–89. doi: 10.1038/npp.2016.269
99. Moore CF, Sabino V, Koob GF, Cottone P. Neuroscience of compulsive eating behavior. Front Neurosci. (2017) 11:469. doi: 10.3389/fnins.2017.00469
100. Folkvord F, Hermans RCJ. Food marketing in an obesogenic environment: a narrative overview of the potential of healthy food promotion to children and adults. Curr Addict Rep. (2020) 7:431–6. doi: 10.1007/s40429-020-00338-4
101. Berkman ET. Value-based choice: an integrative, neuroscience-informed model of health goals. Psychol Health. (2018) 33:40–57. doi: 10.1080/08870446.2017.1316847
102. Koban L, Lee S, Schelski DS, Simon MC, Lerman C, Weber B, et al. An fMRI-based brain marker predicts individual differences in delay discounting. bioRxiv [Preprint]. (2021). Available online at: http://biorxiv.org/lookup/doi/10.1101/2021.03.18.435969 (accessed August 5, 2022). doi: 10.1101/2021.03.18.435969
103. Hare TA, Malmaud J, Rangel A. Focusing attention on the health aspects of foods changes value signals in vmpfc and improves dietary choice. J Neurosci. (2011) 31:11077–87. doi: 10.1523/JNEUROSCI.6383-10.2011
104. Montague PR, Berns GS. Neural economics and the biological substrates of valuation. Neuron. (2002) 36:265–84. doi: 10.1016/S0896-6273(02)00974-1
105. Chib VS, Rangel A, Shimojo S, O’Doherty JP. Evidence for a common representation of decision values for dissimilar goods in human ventromedial prefrontal cortex. J Neurosci. (2009) 29:12315–20. doi: 10.1523/JNEUROSCI.2575-09.2009
106. Hutcherson CA, Bushong B, Rangel AA. Neurocomputational model of altruistic choice and its implications. Neuron. (2015) 87:451–62. doi: 10.1016/j.neuron.2015.06.031
107. Basten U, Biele G, Heekeren HR, Fiebach CJ. How the brain integrates costs and benefits during decision making. Proc Natl Acad Sci U.S.A. (2010) 107:21767–72. doi: 10.1073/pnas.0908104107
108. Clithero JA, Rangel A. Informatic parcellation of the network involved in the computation of subjective value. Soc Cogn Affect Neurosci. (2014) 9:1289–302. doi: 10.1093/scan/nst106
109. Hare TA, Camerer CF, Rangel A. Self-control in decision-making involves modulation of the vmPFC valuation system. Science. (2009) 324:646–8. doi: 10.1126/science.1168450
110. Harris A, Hare T, Rangel A. Temporally dissociable mechanisms of self-control: early attentional filtering versus late value modulation. J Neurosci. (2013) 33:18917–31. doi: 10.1523/JNEUROSCI.5816-12.2013
111. Rudorf S, Hare TA. Interactions between dorsolateral and ventromedial prefrontal cortex underlie context-dependent stimulus valuation in goal-directed choice. J Neurosci. (2014) 34:15988–96. doi: 10.1523/JNEUROSCI.3192-14.2014
112. Becker GM, Degroot MH, Marschak J. Measuring utility by a single-response sequential method. Behav Sci. (1964) 9:226–32. doi: 10.1002/bs.3830090304
113. Hanlon AS, Weiss JW, McMahan S, Chen E. Using the multi-attribute utility model to better understand fruit and vegetable intake among college students. Calif J Health Promot. (2012) 10:40–56. doi: 10.32398/cjhp.v10iSI-Obesity.1470
114. Weiss JW, Weiss DJ, Edwards W. A descriptive multi-attribute utility model for everyday decisions. Theory Decis. (2010) 68:101–14. doi: 10.1007/s11238-009-9155-1
115. Pearce AL, Adise S, Roberts NJ, White C, Geier CF, Keller KL. Individual differences in the influence of taste and health impact successful dietary self-control: a mouse tracking food choice study in children. Physiol Behav. (2020) 223:112990. doi: 10.1016/j.physbeh.2020.112990
116. Lim SL, Penrod MT, Ha OR, Bruce JM, Bruce AS. Calorie labeling promotes dietary self-control by shifting the temporal dynamics of health- and taste-attribute integration in overweight individuals. Psychol Sci. (2018) 29:447–62. doi: 10.1177/0956797617737871
117. Ha OR, Bruce AS, Pruitt SW, Cherry JBC, Smith TR, Burkart D, et al. Healthy eating decisions require efficient dietary self-control in children: a mouse-tracking food decision study. Appetite. (2016) 105:575–81. doi: 10.1016/j.appet.2016.06.027
118. Sullivan N, Hutcherson C, Harris A, Rangel A. Dietary self-control is related to the speed with which attributes of healthfulness and tastiness are processed. Psychol Sci. (2015) 26:122–34. doi: 10.1177/0956797614559543
119. Bechara A, Damasio AR, Damasio H, Anderson SW. Insensitivity to future consequences following damage to human prefrontal cortex. Cognition. (1994) 50:7–15. doi: 10.1016/0010-0277(94)90018-3
120. Crone EA, van der Molen MW. Developmental changes in real life decision making: performance on a gambling task previously shown to depend on the ventromedial prefrontal cortex. Dev Neuropsychol. (2004) 25:251–79. doi: 10.1207/s15326942dn2503_2
121. Ahn WY, Busemeyer JR, Wagenmakers EJ, Stout JC. Comparison of decision learning models using the generalization criterion method. Cogn Sci. (2008) 32:1376–402. doi: 10.1080/03640210802352992
122. Busemeyer JR, Stout JC. A contribution of cognitive decision models to clinical assessment: decomposing performance on the Bechara gambling task. Psychol Assess. (2002) 14:253–62. doi: 10.1037/1040-3590.14.3.253
123. Alabi OO, Fortunato MP, Fuccillo MV. Behavioral paradigms to probe individual mouse differences in value-based decision making. Front Neurosci. (2019) 13:50. doi: 10.3389/fnins.2019.00050
124. Worthy DA, Hawthorne MJ, Otto AR. Heterogeneity of strategy use in the Iowa gambling task: a comparison of win-stay/lose-shift and reinforcement learning models. Psychon Bull Rev. (2013) 20:364–71. doi: 10.3758/s13423-012-0324-9
125. Worthy DA, Pang B, Byrne KA. Decomposing the roles of perseveration and expected value representation in models of the Iowa gambling task. Front Psychol. (2013) 4:640. doi: 10.3389/fpsyg.2013.00640
126. Ahn WY, Krawitz A, Kim W, Busemeyer JR, Brown JW. A model-based fMRI analysis with hierarchical Bayesian parameter estimation. J Neurosci Psychol Econ. (2011) 4:95–110. doi: 10.1037/a0020684
127. Kourouniotis S, Keast RSJ, Riddell LJ, Lacy K, Thorpe MG, Cicerale S. The importance of taste on dietary choice, behaviour and intake in a group of young adults. Appetite. (2016) 103:1–7. doi: 10.1016/j.appet.2016.03.015
128. Glanz K, Basil M, Maibach E, Goldberg J, Snyder D. Why Americans eat what they do: taste, nutrition, cost, convenience, and weight control concerns as influences on food consumption. J Am Diet Assoc. (1998) 98:1118–26. doi: 10.1016/S0002-8223(98)00260-0
129. Nguyen SP, Girgis H, Robinson J. Predictors of children’s food selection: the role of children’s perceptions of the health and taste of foods. Food Qual Prefer. (2015) 40:106–9. doi: 10.1016/j.foodqual.2014.09.009
130. Lim SL, Cherry JBC, Davis AM, Balakrishnan SN, Ha OR, Bruce JM, et al. The child brain computes and utilizes internalized maternal choices. Nat Commun. (2016) 7:11700. doi: 10.1038/ncomms11700
131. Livingstone KM, Lamb KE, Abbott G, Worsley T, McNaughton SA. Ranking of meal preferences and interactions with demographic characteristics: a discrete choice experiment in young adults. Int J Behav Nutr Phys Act. (2020) 17:157. doi: 10.1186/s12966-020-01059-7
132. Berkman ET, Hutcherson CA, Livingston JL, Kahn LE, Inzlicht M. Self-control as value-based choice. Curr Dir Psychol Sci. (2017) 26:422–8. doi: 10.1177/0963721417704394
133. Kanoski SE. Food cue reactivity: neurobiological and behavioral underpinnings. Rev Endocr Metab Disord. (2022) 23:683–96.
134. Devoto F, Zapparoli L, Bonandrini R, Berlingeri M, Ferrulli A, Luzi L, et al. Hungry brains: a meta-analytical review of brain activation imaging studies on food perception and appetite in obese individuals. Neurosci Biobehav Rev. (2018) 94:271–85. doi: 10.1016/j.neubiorev.2018.07.017
135. Meng X, Huang D, Ao H, Wang X, Gao X. Food cue recruits increased reward processing and decreased inhibitory control processing in the obese/overweight: an activation likelihood estimation meta-analysis of fMRI studies. Obes Res Clin Pract. (2020) 14:127–35. doi: 10.1016/j.orcp.2020.02.004
136. van Meer F, van der Laan LN, Adan RAH, Viergever MA, Smeets PAM. What you see is what you eat: an ALE meta-analysis of the neural correlates of food viewing in children and adolescents. Neuroimage. (2015) 104:35–43. doi: 10.1016/j.neuroimage.2014.09.069
137. Demos KE, Heatherton TF, Kelley WM. Individual differences in nucleus accumbens activity to food and sexual images predict weight gain and sexual behavior. J Neurosci. (2012) 32:5549–52. doi: 10.1523/JNEUROSCI.5958-11.2012
138. Yokum S, Gearhardt AN, Harris JL, Brownell KD, Stice E. Individual differences in striatum activity to food commercials predict weight gain in adolescents: individual differences in striatum activity. Obesity. (2014) 22:2544–51. doi: 10.1002/oby.20882
139. Lawrence NS, Hinton EC, Parkinson JA, Lawrence AD. Nucleus accumbens response to food cues predicts subsequent snack consumption in women and increased body mass index in those with reduced self-control. Neuroimage. (2012) 63:415–22. doi: 10.1016/j.neuroimage.2012.06.070
140. Giuliani NR, Merchant JS, Cosme D, Berkman ET. Neural predictors of eating behavior and dietary change: neural predictors of eating behavior. Ann N Y Acad Sci. (2018) 1428:208–20. doi: 10.1111/nyas.13637
141. Gearhardt AN, Yokum S, Harris JL, Epstein LH, Lumeng JC. Neural response to fast food commercials in adolescents predicts intake. Am J Clin Nutr. (2020) 111:493–502. doi: 10.1093/ajcn/nqz305
142. Frankort A, Roefs A, Siep N, Roebroeck A, Havermans R, Jansen A. Neural predictors of chocolate intake following chocolate exposure. Appetite. (2015) 87:98–107. doi: 10.1016/j.appet.2014.12.204
Keywords: food choice, obesity, value-based decision-making, reinforcement learning, model-free vs. model-based learning, sign-and goal-tracking
Citation: Pearce AL, Fuchs BA and Keller KL (2022) The role of reinforcement learning and value-based decision-making frameworks in understanding food choice and eating behaviors. Front. Nutr. 9:1021868. doi: 10.3389/fnut.2022.1021868
Received: 17 August 2022; Accepted: 04 November 2022;
Published: 22 November 2022.
Edited by:
Carol Coricelli, Western University, CanadaReviewed by:
Cassandra Justine Lowe, Western University, CanadaSoyoung Park, German Institute of Human Nutrition Potsdam-Rehbruecke (DIfE), Germany
Copyright © 2022 Pearce, Fuchs and Keller. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Alaina L. Pearce, azp271@psu.edu