Ambiguity Aversion in Rhesus Macaques

People generally prefer risky options, which have fully specified outcome probabilities, to ambiguous options, which have unspecified probabilities. This preference, formalized in economics, is strong enough that people will reliably prefer a risky option to an ambiguous option with a greater expected value. Explanations for ambiguity aversion often invoke uniquely human faculties like language, self-justification, or a desire to avoid public embarrassment. Challenging these ideas, here we demonstrate that a preference for unambiguous options is shared with rhesus macaques. We trained four monkeys to choose between pairs of options that both offered explicitly cued probabilities of large and small juice outcomes. We then introduced occasional trials where one of the options was obscured and examined their resulting preferences; we ran humans in a parallel experiment on a nearly identical task. We found that monkeys reliably preferred risky options to ambiguous ones, even when this bias was costly, closely matching the behavior of humans in the analogous task. Notably, ambiguity aversion varied parametrically with the extent of ambiguity. As expected, ambiguity aversion gradually declined as monkeys learned the underlying probability distribution of rewards. These data indicate that ambiguity aversion reflects fundamental cognitive biases shared with other animals rather than uniquely human factors guiding decisions.

Humans reliably prefer risky options to ambiguous ones in a variety of laboratory and real-world situations, and will pay a premium to avoid ambiguity (Einhorn and Hogarth, 1985;Curley et al., 1986;Fox and Tversky, 1995). Precisely why people avoid ambiguity remains unclear. Several proposed explanations focus on uniquely human factors, including verbally representing probabilities, the need to justify one's decision (Curley et al., 1986), the assumption that the "deck is stacked" by one's opponent (Kühberger and Perner, 2003), and the desire to avoid public embarrassment (Heath and Tversky, 1991). Since ambiguity aversion has never been demonstrated experimentally in any animal, these explanations remain to be fully tested.
To address these issues experimentally, we probed the preferences of monkeys amongst options characterized by different degrees of uncertainty about the probability of obtaining a large reward if chosen. Here we use the terms risky and ambiguous to refer to options whose reward probabilities are either fully specified or obscured, although we note that knowledge about probability more likely varies along a continuum in real life situations. We found that monkeys, like humans, are averse to ambiguity, and that this aversion increases parametrically with degree of uncertainty. The preferences observed in monkeys closely matched those found in humans in a nearly identical task. These findings have important implications for understanding economic decision-making, uncertainty, and learning, as well as the evolution of human cognitive biases.

Materials and Methods
All experiments have been approved by the Duke University IACUC and IRB, and confirm to relevant regulatory standards. Informed consent was obtained from human subjects.

introduction
Risk and ambiguity are two forms of uncertainty distinguished by the amount of uncertainty associated with the likelihoods of their outcomes. Whereas the outcome of a risky choice is drawn from a distribution known by the decision-maker, the outcome of an ambiguous choice is drawn from an unknown distribution. The distinction between the two is often illustrated by the Ellsberg paradox, in which the subject chooses between two urns containing colored balls: the risky urn contains an equal number of blue (high reward) and red (low reward) balls while the ambiguous urn contains an unknown number of each ball. A ball is drawn blindly from the chosen urn and a large or small reward is given, depending on the color of the ball. Even when informed that the ratio of balls from the second urn is selected at random, people consistently prefer known probability distributions to unknown ones -even if the unknown has greater expected value (Ellsberg, 1961).
Economists and psychologists have long recognized that these two forms of uncertainty have dissociable influences on behavior, so a complete explication of decision-making under uncertainty must encompass both (Keynes, 1921;Knight, 1921;Ellsberg, 1961;Gardenfors and Sahlin, 1982;Frisch and Baron, 1988;Camerer and Weber, 1992;Fox and Tversky, 1995;Yu and Dayan, 2005). A similar distinction, between expected and unexpected uncertainty, features prominently in learning theory (Yu and Dayan, 2002;Dayan and Yu, 2006). Far from a mere economic and mathematical curiosity, the distinction between different forms of uncertainty has important implications for business, medicine, microeconomic theory, and neuroscience (Knight, 1921;Epstein and Wang, 1994;Hsu et al., 2005;Huettel et al., 2006). that choosing that bar would yield a large reward ( Figure 1C). The red portion indicated the probability that choosing that bar would yield a small reward. All probability bars were 80 pixels wide and 300 pixels tall. The occluder was either 150, 225, or 300 pixels tall (depending on the level of ambiguity; see below) and always 200 pixels wide.
The monkey had 1 s to inspect these stimuli. Casual observation showed that monkeys reliably looked at both bars during this period. Next, a small yellow fixation point appeared at the center of the monitor. Once eye position was aligned with this point (±0.5°), the monkey was required to maintain fixation for 1 s. The fixation point was then extinguished, and two response targets appeared. These were small yellow squares overlaid on the center of the probability bars. The monkey then had to select one of these bars by shifting gaze to it (±3°). Following the saccade, the gamble was immediately resolved by the delivery of a reward. All task stimuli were then extinguished. The reward probabilities for the ambiguous option were never revealed.
The cyan occluder appeared on all trials. On two-thirds of trials, the occluder appeared at a random location on the screen, and sometimes covered part of the bar without obscuring information about probabilities ( Figure 1D). On these trials, the horizontal position of the occluder was randomly jittered. On one-third of trials (ambiguous trials), the occluder obscured the center of one of the bars at the intersection of the blue and red portions ( Figure 1E). The height of the occluder on the ambiguous option was equally likely to be 150, 225, or 300 pixels; the resulting occluders were called low, medium, and high ambiguity occluders, respectively.

Monkey behavioral techniques
All animal procedures were approved by the Duke University Institutional Animal Care and Use Committee and were designed and conducted in compliance with the Public Health Service's Guide for the Care and Use of Animals. Four male rhesus monkeys (Macaca mulatta) served as subjects. Prior to the beginning of experiments, a small head-holding prosthesis was implanted in all animals using standard surgical techniques to permit high-resolution measurement of eye position and intracerebral neurophysiological recording (not reported here). Six weeks later, animals were habituated to training conditions and then trained to perform oculomotor tasks. To motivate behavior, monkeys were placed on controlled access to fluid outside of experimental sessions and task performance was reinforced with liquid rewards.
Horizontal and vertical eye positions were sampled at 1000 Hz by an infrared eye-monitoring camera system (SR Research, Osgoode, ON, Canada). Stimuli were controlled by a computer running Matlab (Mathworks, Natick, MA, USA) with Psychtoolbox (Brainard, 1997) and Eyelink Toolbox (Cornelissen et al., 2002). Visual stimuli were colored rectangles on a computer monitor placed directly in front of the animal and centered on his eyes. A standard solenoid valve controlled the duration of juice delivery. Reward volume was 67, 200, or 333 μL in all cases.
Every trial began when two bars and one potential occluder appeared ( Figure 1A). Each bar was divided into a blue portion and a red portion, or was completely gray (indicating a 100% probability of a medium sized reward; Figure 1B). For the risky targets, the blue portion, always on top, indicated the probability given. Participants obtained 50, 140, or 230 points; these values corresponded to the small, certain, and large rewards in the monkey task. Rewards were indicated at the end of each trial with text on the monitor for 1 s. Cumulative score was indicated with text on the monitor for 3 s every 10 trials. Participants performed three blocks of 250 trials. Participants were not required to fixate, but pressed the space bar on the keyboard to initiate trials. Participants did not indicate choices with saccades, but by pressing the left and right arrows on the keyboard.
Participants were read the following script before beginning. "On each trial, you will choose between two options. Each option is represented by a bar, and can pay either 50, 140, or 230 points. The colors of the bars give information about the probabilities of these two possibilities. When the options appear, press the space bar and then either the left or right arrows. Your goal is to get the most points you can. This is all I can tell you right now. Good luck." Participants were asked to explain their interpretations of the meaning of the stimuli at the end of the entire session by an undergraduate researcher who was blind to the hypotheses of the experiment. All participants were paid $10.

results
All four monkeys preferred risky options to ambiguous options (Figures 2A,B). Although monkeys were more likely to choose the risky option over the ambiguous option the greater the expected value of the risky option, there was a strong overall bias toward the risky option (Figure 2A). The PSE measures the probability of large reward for the risky option for which the monkey chooses the ambiguous option equally often; a PSE below 50% indicates ambiguity aversion (Deaner et al., 2005;Hayden et al., 2007). Monkeys' average PSE for ambiguous options was a risky option with 30% chance of large reward, meaning that they required a premium of 53 μL of juice to choose the ambiguous option. This amount is substantial -15.9% the size of the large reward, and 80% the size of the small reward. All four monkeys had PSEs less than 50% [mean 29.9%, t(3) = −4.4, p = 0.02, Figure 2B]. These data demonstrate that monkeys distinguish degrees of uncertainty, and, like humans, are reluctant to choose ambiguous gambles. Monkeys showed greater aversion to the high ambiguity option than the medium and low ambiguity options ( Figure 2C, bootstrap t-test, p < 0.001 in both cases). This preference pattern indicates that monkeys were not simply reluctant to choose an obscured bar, which may have appeared to be an entirely novel stimulus, but rather were systematically averse to the level of uncertainty associated with the ambiguous option.
To be certain that our task tapped into traditional ambiguity aversion, we examined behavior of 10 human participants in an analogous task for monetary rewards using the same test stimuli, without providing instructions about the relationship between the bar stimuli and reward probabilities (Figure 3). We found that people, like monkeys, preferred risky options to ambiguous options (population PSE 42, n = 7500 trials, bootstrap t-test p < 0.001). The distribution of individual PSEs was also significantly biased toward ambiguity aversion (mean of individual PSEs = 41.6, p = 0.019). We observed a significant preference for risky options in 5/10 participants and a preference for ambiguous options in two participants (p < 0.05, bootstrap t-test). No significant changes in preferences The probability that the ambiguous option would provide a large reward was drawn from a uniform distribution of the probabilities within the range of those obscured by the occluder, and an outcome was chosen accordingly. This is mathematically equivalent to a 50% probability of a large reward on all ambiguous trials, and the expected value of the ambiguous option was always 200 μL. On a small minority of risky trials, the occluder covered only a portion of the bar; on such trials, the border between the blue and red regions was visible, and these trials were considered risky options in all analyses. Finally, on 10% of trials (chosen randomly), a gray bar appeared instead of one of the two red/blue bars; this option had a 100% probability of a medium sized reward (200 μL) and was considered certain.

the triple bar control task
We were concerned that monkeys would adopt a strategy such as "look for the largest blue bar" that would bias them away from the occluded options. We performed a control experiment to test for this possibility (Figure 1F). In the Triple Bar control, the two bars were each divided into three sections -blue, gray, and redthat independently indicated the probability of a large, medium, or small reward. As in the standard task, the amount of blue and red indicated the probabilities of obtaining the large (333 μL) and small (67 μL) rewards, respectively. In this control task, the size of the gray portion of the bar indicated the probability of obtaining a medium reward (200 μL). As in the standard task, all probabilities were drawn from a uniform distribution. To reduce the possibility that the monkeys would learn this task and treat it differently from the standard task, we recorded behavior on only one short (∼500 trials) session with no training. During the Triple Bar control, the occluder never covered the bars.

analyses
We calculated the point of subjective equivalence (PSE) between risky and ambiguous options (Deaner et al., 2005). We first fit a cumulative Gaussian function to the monkeys' choices of risky over ambiguous options, as a function of the expected value of the risky option. (Because only two outcomes were used, the probability scaled linearly with the expected value, so these are interchangeable in this task.) We used a least-squares minimization method (Matlab Statistics Toolbox) with four free parameters: mean, variance, gain, and bias. To estimate standard errors on PSEs, we calculated the standard error of a distribution of 10 PSEs calculated on 10 subsets randomly selected (without replacement) from the original data. The resulting PSE provides a revealed preference measure of value assigned to the ambiguous option. Any PSE below 50% reflects a preference for the risky option and an aversion to the ambiguous option.

huMan behavior
All procedures were approved by the Duke University Institutional Review Board. Behavior of 10 human participants in a human analog of the task was analyzed. Participants were recruited from the undergraduate and graduate population of Duke University. No participants had extensive psychological, neuroscientific, or economic training. Participants were merely told to maximize the number of points earned on each trial. All parameters of the task were identical with the following exceptions. No juice rewards were they noticed about the task, all participants stated that they quickly came to recognize that the occluder obscured information about outcome probabilities even though they had not been told this fact. The similarity between human and monkey performance suggests that individuals of both species readily interpreted the bar stimuli as reward probability cues and that the cyan rectangle occasionally obscured such information.
On a subset of trials, the occluder partially covered the bar, but did not cover the border between the blue and red sections. On these trials, information about probabilities was available, and so no ambiguity aversion should be observed. Indeed, we observed no difference in monkeys' preferences for these partially covered bars and fully uncovered bars ( Figure 4A, no difference, p > 0.4, bootstrap t-test). We further tested this idea of novelty aversion with a behavioral control performed near the conclusion of data collection. On 50% of trials, the occluder was magenta, making it a novel stimulus compared to the normal cyan occluder. We found that monkeys' preferences for options obscured by the two differently colored occluders were indistinguishable statistically ( Figure 4B, p > 0.5, binomial test).
If monkeys learn about the underlying probabilities of ambiguous options, PSEs should eventually converge on 50%. For two of the monkeys in this study (monkeys E and O), we continued to collect data for several months. PSEs for these two monkeys gradually approached indifference after 40-50 individual behavioral sessions ( Figure 4C). PSEs in the final 10 sessions were significantly higher than those during the first 10 sessions (p < 0.05 for both animals individually, bootstrap t-test) and were not significantly different from 50% (p > 0.3 for both animals; binomial test). Despite this evidence for learning, monkeys never developed an outright preference for the ambiguous option during the course of this experiment.
were observed over the course of the single session (three blocks of trials, one-way ANOVA, p > 0.5). Preferences for certainty were weaker in human participants than in monkeys. This distinction may reflect differences in motivation, familiarity with abstract probabilities or familiarity with the task. To assess our human participants' motivations, an experimenter naïve to the goals of the study debriefed them in detail. When asked to explain what  curve. This plot thus also demonstrates that the monkeys were less likely to choose the inferior (i.e., less probable) option as the penalty grew. We observed no systematic changes in preferences between two risky options over the course of behavioral training.
Second, ambiguity aversion did not simply arise from reinforcement learning mediated associations of the bar stimuli. In principle, monkeys could derive a value for each of the 100 different bar patterns based on an integrated reward history, and choose based on these reward associations (Thorndike, 1911). To address this concern, we analyzed preferences for the bars the first time each target appeared. Because monkeys confused targets that were within 3 pixels of each other (see above), we restricted this analysis to the first time a stimulus or any other stimulus that could be confused with it appeared. These data therefore all originate from the first day of behavioral training with the risky bars. Before this, monkeys had been trained only on all-blue and all-red certain bars. This restriction limited us to 14 or 15 trials total for each monkey, and makes this test very conservative. Nonetheless, we found that even on these early trials, three of the four monkeys expressed a significant preference for the option with greater EV (p < 0.03 in all cases, bootstrap t-test). This analysis provides strong evidence that monkeys generalized the meaning of the blue and red bars to mixed bars.
Third, monkeys did not merely adopt a strategy such as "look for the largest blue bar" that would bias them away from the occluded options (including the ambiguous ones). The fact that monkeys did not avoid partially occluded options that did not fully obscure reward probability information speaks to this issue (Figure 4A). To directly test this possibility, we performed a control experiment in which the two bars were each divided into three sections -blue, gray, and red -which indicated the probability of a large, medium, Although not the main focus of our study, we also examined choices between risky and certain options. All four monkeys significantly preferred risky options to certain options with the same expected value (p < 0.001 in all cases). On average, monkeys sacrificed a potentially larger reward of 71 μL of juice to choose the risky option instead of the certain option, and preferred 50:50 risky options to certain options on 72% of trials. These findings are consistent with previous studies showing that rhesus macaques choose risky options about 75% of the time over certain options in an uncued gambling task with highly familiar probability contingencies (McCoy and Platt, 2005;Hayden et al., 2008a,b). Our task occasionally pitted the ambiguous option against the certain option. Monkeys weakly preferred the ambiguous option to the certain option (they chose the ambiguous option on 58.9% of these trials, binomial test, p = 0.0024). This preference was weaker than the monkeys' preference for the risky option over the certain option (71% of trials). The observed preference for the ambiguous and risky options over the certain option may reflect exploration strategies aimed at learning the underlying probability distributions (Hayden et al., 2008a;Pearson et al., 2009).
Several additional control analyses and experiments confirm that ambiguity aversion does not reflect other processes such as difficulty in distinguishing the stimuli or simple reinforcement learning. First, ambiguity aversion did not arise from poor discrimination of the bars used to symbolically cue reward probability. Monkeys were better than chance at distinguishing similar bars that differed by as few as 4% (12 pixels on the monitor, p = 0.026 for 4%, p < 0.001 for all larger differences, bootstrap t-test, Figure 5A). This figure indicates monkeys' preference behavior when choosing between pairs of risky options, in the absence of an ambiguous one. Preferences for larger differences followed a simple psychometric non-human animals, like humans, are sensitive to the distinction between risk and ambiguity, and prefer options with full information. Because it is impossible to know the content of the monkeys' thoughts, we cannot be completely certain that the monkeys in our task understood the concepts of probabilities and relative degrees of uncertainty. Nonetheless, the monkeys' pattern of performance, and their behavior on the controls, is most straightforwardly interpreted in this context.
Uncertainty is a ubiquitous and inevitable aspect of decisionmaking. The distinction between risk and ambiguity, and between expected and unexpected uncertainty, is a fundamental and natural one (Knight, 1921;Ellsberg, 1961;Becker and Brownson, 1964;Camerer and Weber, 1992;Yu and Dayan, 2005). For example, canny decision-makers should more fully engage learning processes when confronted with unexpected uncertainty (Frisch and Baron, 1988;Yu and Dayan, 2005;Dayan and Yu, 2006). Thus, it should not be surprising that monkeys readily make this distinction.
Indeed, our findings imply that the cognitive processes motivating human preferences for certainty are shared with non-human primates. Furthermore, our data endorse the idea that ambiguity aversion does not reflect uniquely human faculties or motivations such as language, the need to justify one's decision, aversion to competition with a skilled opponent, the desire to avoid the embarrassment or regret of a decision that is later revealed to have been unwise, or feelings of competence within a domain of knowledge (Curley et al., 1986;Heath and Tversky, 1991;Kühberger and Perner, 2003).
Monkeys' simultaneous preference for risk and aversion to ambiguity seems surprising from an economic perspective (Ellsberg, 1961;Becker and Brownson, 1964;Frisch and Baron, 1988;Camerer and Weber, 1992;Fox and Tversky, 1995). Because unspecified probabilities are in some sense a form of compounded uncertainty, some economists have proposed that ambiguity aversion in humans may be an extension of risk aversion (Frisch and Baron, 1988). By contrast, our results suggest that monkeys' or small reward, respectively ( Figure 5B). We reasoned that if the monkeys simply associated the blue option with large reward, and selectively chose the option with the larger blue area, then behavior would not depend on the relative amounts of red and gray. By the same token, if the monkeys simply associated the color red with small reward, then behavior would not depend on the relative amounts of blue and gray. However, if the monkeys understood that the relative size of the red and blue sections corresponded to outcome probabilities, they would rapidly generalize to a threecolor bar.
We performed this task on three of the four monkeys. As in the standard task, the amount of blue and red indicated the probabilities of obtaining the large (333 μL) and small (67 μL) rewards, respectively. In this control task, the size of the gray bar indicated the probability of obtaining a medium sized reward (200 μL). As in the standard task, all probabilities were drawn from a uniform distribution. To reduce the possibility that the monkeys would learn this task and thus treat it differently from the standard task, we recorded behavior on only one brief (∼500 trials) session with no training. We found that choices varied lawfully as a function of both blue and red sections, indicating that monkeys attended to the lengths of both (Figure 3). The results of this control experiment demonstrate that monkeys readily use information about probabilities presented in an abstract and continuously varying form, and that they interpret occluders as obscuring that information.

discussion
Our findings demonstrate that monkeys prefer explicit information about reward probability distributions and avoid options in which this information is obscured -just as humans do. Moreover, monkeys' behavior in this task closely matched that of humans in a very similar task; human participants interpreted their task in a manner consistent with conventional definitions of risk and ambiguity (Ellsberg, 1961;Frisch and Baron, 1988;Camerer and Weber, 1992;Fox and Tversky, 1995). These findings indicate that