Learning about Expectation Violation from Prediction Error Paradigms – A Meta-Analysis on Brain Processes Following a Prediction Error

Modifying patients’ expectations by exposing them to expectation violation situations (thus maximizing the difference between the expected and the actual situational outcome) is proposed to be a crucial mechanism for therapeutic success for a variety of different mental disorders. However, clinical observations suggest that patients often maintain their expectations regardless of experiences contradicting their expectations. It remains unclear which information processing mechanisms lead to modification or persistence of patients’ expectations. Insight in the processing could be provided by Neuroimaging studies investigating prediction error (PE, i.e., neuronal reactions to non-expected stimuli). Two methods are often used to investigate the PE: (1) paradigms, in which participants passively observe PEs (”passive” paradigms) and (2) paradigms, which encourage a behavioral adaptation following a PE (“active” paradigms). These paradigms are similar to the methods used to induce expectation violations in clinical settings: (1) the confrontation with an expectation violation situation and (2) an enhanced confrontation in which the patient actively challenges his expectation. We used this similarity to gain insight in the different neuronal processing of the two PE paradigms. We performed a meta-analysis contrasting neuronal activity of PE paradigms encouraging a behavioral adaptation following a PE and paradigms enforcing passiveness following a PE. We found more neuronal activity in the striatum, the insula and the fusiform gyrus in studies encouraging behavioral adaptation following a PE. Due to the involvement of reward assessment and avoidance learning associated with the striatum and the insula we propose that the deliberate execution of action alternatives following a PE is associated with the integration of new information into previously existing expectations, therefore leading to an expectation change. While further research is needed to directly assess expectations of participants, this study provides new insights into the information processing mechanisms following an expectation violation.


INTRODUCTION
Patients' expectations have a great influence on their treatment and outcomes in psychotherapy (Greenberg et al., 2006), medical conditions as well as in patients undergoing surgery (Auer et al., 2016;Rief and Glombiewski, 2016). In addition, negative expectations about psychological interventions may lead to negative effects of psychotherapy (Ladwig et al., 2014). Rief et al. (2015) have proposed to consider dysfunctional expectations to be core features of mental disorders. It has been argued that dysfunctional behavior is guided by dysfunctional expectations of situational associations and outcomes. Hence, behavioral therapy would only be successful if there is a change of the dysfunctional expectations guiding the behavior. These dysfunctional expectations are pre-existing assumptions about contingencies with a high subjective associative strength, i.e., subjective certainty. Patients would have to experience an expectation violation, i.e., a state, in which the expected outcome and the actual outcome differ, to induce a change in their expectations about the contingencies. This corresponds to a relearning of the contingencies, i.e., a state, in which they perceive a difference between expected outcome and the actual outcome, which would induce a change in their expectations about the contingencies. It is hypothesized that depending on various information processing variables, expectations might either be changed or maintained after an expectation violation situation. Thus, the relearning is either successful and persists over time or the relearning might be only temporary or depending on contextual factors.
The particular mechanisms underlying the information processing and the persistence and change of expectations have remained unclear. Clinical observations suggests that patients with mental disorders are particularly resistant to expectation change and the perception on expectation violations (Rief et al., 2015;Rief and Glombiewski, 2016). There are promising new approaches examining immunization as one of the processing strategies following expectation violation (Kube et al., 2016). This could explain why even after a successful expectation violation, the expectation is not changed. The patients perceive the violation of their pre-existing expectation but attribute the situation to contextual factors, e.g., the setting. Thus, the confrontation with an aversive stimulus with aim of reducing an emotional response, as is commonly used in psychotherapeutic settings, might not always be enough to induce a persistent expectation change. Craske et al. (2014) proposed methods of maximizing such exposure techniques, which are supposed to increase the inhibitory learning of the old expectation about the contingencies. One of these methods is the active testing of the pre-existing expectation. This is suggested to facilitate the relearning of the contingencies and to stabilize the newly learned expectation, thus inducing an expectation change.
The change of dysfunctional expectations is theorized as a crucial mechanism for therapeutic success. The investigation of cognitive processes facilitating an expectation change vs. maintenance following an expectation violation might pose a promising approach for cognitive behavioral therapy. Thus, we propose to compare the cognitive processing of a more passive confrontation with the aim of reducing an emotional response and an active approach by testing the expectation.  The neuroimaging research on learning provides experimentally designed expectation violations. One of the concepts consistently associated with successful learning is the so-called prediction error, i.e., the neurological response to an unexpected stimulus. Learning research has mainly focused on reinforcement learning, whereby the expectations comprises predictions about reward and/or punishment (Karuza et al., 2014). Many studies use partial reinforcement or probabilistic learning paradigms. It can be argued that changes in behavioral strategies in these paradigms also reflect changes in underlying expectations regarding the contingencies of reward and punishment. Hence, in paradigms, in which no behavioral adaptation is necessary, i.e., a passive observation of contingencies, might diminish the attention on expectation violations. We argue that participants in both paradigms compute prediction errors and their relearning of the contingencies is successful. In alignment with the approach by Craske et al. (2014) to maximize inhibitory learning by actively testing the expectation, we hypothesize a different cognitive processing of "active" paradigms, which encourage a behavioral adaptation and FIGURE 2 | Overview of the meta-analysis results for all studies (p < 0.05, FDR). The significant clusters are comprised of activity in the superior frontal gyrus, middle frontal gyrus, anterior cingulate, cingulate gyrus, claustrum, insula, caudate head, precentral gyrus, putamen, lateral globus pallidus, caudate body, red nucleus, thalamus, parahippocampal gyrus, superior parietal lobule, declive, lingual gyrus and the fusiform gyrus (from left to right). MNI coordinates are presented below each coronal view.
"passive" paradigms, in which contingencies are observed. Since the concepts of prediction error and expectation violation are identical in matters of meaning for the preexisting expectation, it seems likely that clinical research can benefit from an insight of neuroimaging research on prediction error. Examining the functional magnetic resonance imaging (fMRI) results provided by research on prediction error might provide insights in the cognitive processes associated with the information processing during expectation violations.
Our aim is to review fMRI studies investigating two different prediction error paradigms. The first paradigm encourages strategic behavioral changes throughout the course of experiments while the second one requires a passive observation. A contrast analysis will be performed to identify differences in brain activity between these two paradigm categories. A meta-analysis summarized the current findings on brain areas associated with prediction error (Garrison et al., 2013). They found a consistent association of the pallidum, the striatum and medio-frontal structures with prediction error. These structures are also associated with the fronto-striatal circuits. The circuit is defined as circular connections between the caudate nucleus, putamen, thalamus and prefrontal regions (Leh et al., 2007). Dysfunctions in this circuit are associated with impaired behavioral adaptation such as poor set shifting performance, e.g., in a go/no-go tests or stimulus-bound behavior (Mega and Cummings, 1994). Several disorders are linked to fronto-striatal circuit dysfunctions, such as Huntington's disease (Beste et al., 2012), Parkinson's Disease (Owen, 2004) and obsessive-compulsive disorder (Maltby et al., 2005;Marsh et al., 2014). All clinical pictures are associated with behavioral and cognitive perseverations (Mega and Cummings, 1994). It therefore seems likely to assume the fronto-striatal circuit to be involved in the expectation violation processing and the resulting expectation and behavioral adaptation. We will perform a metaanalysis involving prediction error followed by a behavioral adaptation to an uncertain environment. We expect a consistent activation in the striatum and media-frontal areas.

Literature Selection
We conducted a systematic literature search to identify neuroimaging studies of prediction error using PubMed 1 , Web of Science 2 , and Neurosynth 3 databases. We searched for articles in the English language using the keywords "prediction error" AND "fMRI" and did not specify a time span for date of publication. The search revealed 8'610 results as of July 2016. To narrow the results, a second search was performed using the keywords "prediction error" AND "fMRI" AND "behavior change" as well as "prediction error" AND "fMRI" AND "observational learning". Again, no time span was specified. These searches revealed 111 results and four results, respectively. The abstracts of these articles were examined to select potential matches for our inclusion criteria. We also scanned the reference lists of the results to search for additional articles, which met our inclusion criteria. We retrieved the full text of 72 articles for further examination. We predefined study selection criteria to minimize ambiguousness in the study selection. The criteria can be requested of the corresponding author. Studies were included when they met the following criteria: (1) experimental prediction error paradigm and (2) report of voxel-wise-brain analysis for a prediction error main effect, which yielded a total of 59 articles. We excluded studies which did not report prediction error for healthy adults or used medication in their experiment (n = 6 studies excluded). We did this to include only prediction errors which arise from an unexpected change in contingencies in alignment with the clinical model. Of these studies, we precluded those articles failing to experimentally induce a prediction error by changing the contingencies between stimuli and outcome (n = 10 studies excluded). A flowchart of the selection process is shown in Figure 1. The studies included in the meta-analysis are listed in Table 1.

Contrast Selection
We included all analyses which contrasted prediction error brain activity with brain activity during expectation confirming trails or paradigm specific variations of these contrasts. Of the 43 studied that met all inclusion criteria, we included 60 contrasts in the analysis. If the coordinates were reported in Talairach space they were transformed to Montreal Neurologic Institute (MNI) space using the GingerAle software (Eickhoff et al., 2009, which utilizes the icbm2tal transform algorithm (Lancaster et al., 2007).
In total, we included 446 foci into the analysis.

Activation Likelihood Estimation (ALE)
We performed an activation likelihood estimation (ALE) analysis using the Software GingerAle (Eickhoff et al., 2009. The algorithm assesses above-chance clustering between experiments, using a probability distribution centered at each of the foci used in the analysis. Since the spatial relationship is assumed to be fixed in each experiment, the ALE analysis infers random effects (Eickhoff et al., 2009). We used the algorithm described in Turkeltaub et al. (2012), which organizes the foci by subject group (as opposed to study affiliation). This prevents an influence of multiple foci from one experiment on the Meta-Analysis results (Turkeltaub et al., 2012). We performed three Meta-Analyses: (1) studies which encourage a behavioral strategic adaptation following a prediction error, (2) studies, which employed a passive observational paradigm, and (3) an analysis of all studies, which was necessary to perform the contrast analysis. In line with previous studies (Garrison et al., 2013), we defined a false discovery rate (FDR) method with p < 0.05 and a minimal cluster volume of 50 mm 3 . We then performed a contrast analysis of the "active" behavioral subset and the "passive" observational study subset. This analysis allows the subtraction of two datasets to compare differences in brain activity between these two. To this end, a pooled dataset is created, which then serves as basis for two randomly created datasets with the same number of foci as the original datasets. A permutation of subtractions of simulated datasets are compared to the results of the original datasets. We used an uncorrected p-value p < 0.05 since the single analyses were already corrected with FDR (Eickhoff et al., 2011). We chose a minimal cluster volume of 50 mm 3 for the contrast analysis. Papaya 4 was used to superimpose the ALE cluster results on a T1 brain template (Colin27_T1_seg_MNI.nii 5 ).

Meta-Analysis across All Studies of Prediction Error
Twenty-one significant clusters were identified by the ALE metaanalysis of all 43 studies. The results show activation in the right basal ganglia and the right insula (see details in Figure 2 and Table 2). There was no clear indication of laterality in the main analysis.

Meta-Analyses for Behavioral and Observational Paradigms
When analyzing all prediction error studies, which employed a behavioral reaction following a prediction error, the ALE metaanalysis revealed 17 significant clusters. We found activation in the striatum, the insula and the claustrum (see details in Figure 3 and Table 3).
The ALE meta-analysis of all prediction error studies, which employed a passive paradigm revealed four significant clusters. We found activation in the putamen, the lateral globus pallidus, declive and the lingual gyrus (see details in Figure 4 and Table 3).
In both analyses, no clear indication of laterality was found.

Subtraction Analysis
The details of the ALE subtraction analysis are shown in Table 4 and Figure 5. In the contrast behavior -passive, we found five significant clusters, comprising parts of the striatum, the insula and the fusiform gyrus. There was a tendency of left sided structures to be more active in prediction error paradigms encouraging behavioral adaptation. We found no significant clusters in the contrast passive -behavior. It is often suggested to apply corrected thresholds to the contrast analyses, such as a FDR threshold. Therefore, we replicated the subtraction analyses with more conservative thresholds. We applied a corrected FDR threshold of p < 0.05 to the subtraction analysis. We found no significant clusters in the contrast passive -behavior. The significant clusters of the contrast behavior -passive do not survive the corrected threshold.

DISCUSSION
We performed a subtraction analysis of two different prediction error paradigms. One encourages a behavioral adaptation to changing contingencies while the second paradigm requires a passive observation of contingencies. Our aim was to gain a better understanding of why and how psychological interventions focusing on expectation violation lead to behavioral changes in some but not all cases. Therefore, we analyzed differences in prediction error involving on one hand the execution of an action alternative and on the other hand no behavioral change. We wanted to identify cognitive processes being involved in underlying expectations about contingencies guiding the behavior. As a major result when contrasting studies employing the two paradigms discussed earlier, we found significantly more activation in the left medial globus pallidus, the left caudate body, the right caudate head and putamen as well as the left fusiform gyrus and the left insula.

All Studies of Prediction Error
When performing a meta-analysis containing all prediction error studies our results are in line with previous research (Garrison et al., 2013). We found activation in the striatum, the insula, thalamus as well as fronto-medial structures. The Putamen and the Caudate body are part of the striatum whose association with memory processes is consistent with previous literature (Grahn et al., 2009;Provost et al., 2015). The insula has been associated primarily with fear conditioning (Kircher et al., 2013) but also with reinforcement learning for reward (Lawrence et al., 2014) as well for avoidance learning (Palminteri et al., 2012). Consistent with our hypothesis we also found activation in the areas associated with the fronto-striatal circuits (Leh et al., 2007). In addition to the striatum, we found activation in the globus pallidus, the thalamus and frontal structures, i.e., the left superior, media and middle frontal gyrus.

Prediction Error Followed by Behavioral Adaptation or Passive Observation
When contrasting the differences in neuronal activity of prediction errors computed in active behavioral adaptation and passive observational paradigms we found higher activation in the striatum, the insula and the fusiform gyrus.
The medial globus pallidus is part of the four corticostriatal loops, which are responsible for executive function, visual processing, motor function and motivational evaluation (Seger, 2006). It serves as an output nucleus of the basal ganglia and projects to the thalamus, the centromedian nucleus, and the pedunculopontine nucleus (Nauta and Mehler, 1966). These structures are associated with goal-directed motor actions as well as reward learning and evaluation (Hong and Hikosaka, 2008;Haber and Knutson, 2009;Sescousse et al., 2013).
The putamen is associated with novel motoric executions as well as in ambiguous action tendencies, i.e., if the best motoric strategy is unclear (Grahn et al., 2009). Moreover, due to findings of strong connectivity of the putamen with prefrontal regions, it is suggested that the putamen has a cognitive rather than solely motoric function (Provost et al., 2015).
The caudate body has been shown to be involved in cognitive tasks such as categorization and reward information assessment in monkeys (Yanike and Ferrera, 2014) as well as in humans (Packard and Knowlton, 2002). Further, it has been suggested, that the caudate nucleus is involved in evaluating outcomes post-decision (Badre, 2012;Kepecs and Mainen, 2012).
Most studies do not specifically differentiate between distinct parts of the striatum, but investigate the striatum in its entirety. The striatum has been associated with strategizing in avoidance learning (Palminteri et al., 2012), failure or success to learn associations in instrumental conditioning (Schönberg et al., 2007;Horga et al., 2015), decision making and motor initiation (Nagano-Saito et al., 2014).
The insula is associated with the perception and processing of interoception of emotional states (Zaki et al., 2012;Simmons et al., 2013).
The fusiform gyrus is associated with facial and body recognition (Peelen and Downing, 2005) as well as a sensitivity to visual words (McCandliss et al., 2003). The area of the fusiform gyrus showing peak activation is also associated with object recognition (Bar et al., 2001).
The functions of these areas can be incorporated into the processing of prediction errors computed in a behavioral paradigm. The higher activation of the putamen in prediction errors with behavioral changes might be due to the determination of a novel motoric behavior and its initiation. Due to its' evaluative properties, the caudate nucleus could function as a constant evaluation unit, comparing expected and actual outcomes. The involvement of the insula cannot be explained by the emotional valence of the stimuli used in the studies, since not all the studies comprising the insula cluster contained emotional content, such as negative feedback. They share, however, a high level of uncertainty in their paradigms, e.g., temporal uncertainty or ambiguous stimuli or categories. The processing of uncertainty has also been shown to be associated with insula activity (Simmons et al., 2008;Sarinopoulos et al., 2010), which could be interpreted as an aversive and thus emotional state. Rief et al.'s (2015) model proposes that following an expectation violation, various information processing mechanisms decide whether an expectation is changed and integrated or maintained and reinforced. In order to shed light on the cognitive processes involved in an expectation change following an expectation violation, we investigated the brain areas more active in FIGURE 4 | Overview of the meta-analysis results for the studies reporting passive observational paradigms (p < 0.05, FDR). The significant clusters are comprised of activity in the putamen, lateral globus pallidus, declive and the lingual gyrus (from left to right). MNI coordinates are presented below each coronal view.

Integration into a Clinical Model of Expectation Change and Persistence
FIGURE 5 | Overview of the meta-analysis results of the behavior -no behavior subtraction analysis (p < 0.05, uncorrected). The significant clusters are comprised of activity in the putamen, caudate head, caudate body, insula, medial globus pallidus, and the fusiform gyrus (from left to right). MNI coordinates are presented below each coronal view. paradigms encouraging a behavioral strategic change following a prediction error. The striatum might be involved in learning the specific contingencies between stimulus and outcome. This might eventually form an expectation about the action strategies resulting in a rewarding outcome. However, when facing an expectation violation, the caudate body might signal a nonrewarding outcome, even though the same behavioral strategy has been employed. On the other hand, if the environment encourages a passive behavior, i.e., no action has to be taken following an expectation violation, the individual is not required to determine a behavioral alternative. The difference in expectation and outcome could be solved by mechanisms such as immunization, leading to an expectation persistence (Kube et al., 2016). In contrast, if the environment encourages or even enforces the use of action alternatives, e.g., an active prediction error paradigm or a therapeutic setting, a behavioral reaction to the situation would be necessary. In such a case, the putamen could be involved in determining a novel behavior and initiate the action alternative by projecting to the medial globus pallidus. This structure could then initiate the motoric aspect of the action alternative. The thalamus, the centromedian nucleus, and the pedunculopontine nucleus could be involved in assessing the reward when employing the new behavior. If the action alternative leads to a satisfying result, i.e., a rewarding outcome, the behavior is integrated and leads to an expectation change.
The involvement of the insula especially in prediction error paradigms encouraging behavioral adaptation suggests an emotional component to be important. Due to its association with avoidance learning, the insula might be involved in assessing aversive outcomes following an action alternative. This contrasts with the reward assessment of thalamus, centromedian nucleus and pedunculopontine nucleus. It might be possible, that the avoidance of an unwanted outcome, e.g., a negative emotional state, is as important as the gain of a rewarding outcome. The aversion of a negative emotional state might be a rewarding outcome in itself which has to be considered when assessing the reward of an action alternative.

Limitations
A few methodological limitations have to be considered. First, the studies we included used various paradigms, showing a wide range of stimuli, tasks and underlying mathematical models. However, there is evidence of different brain activity involved in various types of learning and in particular in model-based (i.e., goal-directed actions) vs. model-free (i.e., habit-based actions) approaches (Maia, 2009;Wunderlich et al., 2012). Moreover, the ALE meta-analysis itself has a few limitations. Coordinate-based analyses accumulate power across studies (Costafreda, 2009) and cannot reproduce the same quality in results as image-based meta-analyses (Salimi-Khorshidi et al., 2009). A third limitation is that the results of the subtraction analysis do not survive a FDR corrected threshold. Considering this restriction, the results of the contrast analysis have to be interpreted with caution. Eickhoff et al. (2016) recommend a minimal sample size of 17 studies for the ALE meta-analysis. We could only include 19 studies using a passive observational paradigm. This suggests that the statistical power may be rather small for the subtraction analysis, explaining why our results did not survive the FDR corrected threshold. For future research, it is necessary to repeat the analyses with a larger study sample to increase the statistical power. This will allow a more decisive analysis of the differences in neurological activity between active behavioral and passive observational prediction error paradigms.

CONCLUSION
This meta-analysis sheds light into the cognitive processes involved in the execution of action alternatives following an expectation violation. The information processing involved is strongly associated with reward evaluation of newly found behavioral adaptations. However, further research is needed in order to explicitly investigate the expectations of participants of prediction error paradigms regarding their behavioral strategies.

AUTHOR CONTRIBUTIONS
LD: Did the major part of the work regarding conception and methodology of the article; performed the analyses and the evaluation of the results; approves of the manuscript being published; agrees on being accountable for all aspects of the work, ensuring that questions regarding quality and accuracy of the work are investigated appropriately and resolved. WR: Substantially contributed to the conception of the article, revised the manuscript critically and contributed to the content; approves of the manuscript being published; agrees on being accountable for all aspects of the work, ensuring that questions regarding quality and accuracy of the work are investigated appropriately and resolved.