Experimental and clinical usefulness of crossmodal paradigms in psychiatry: an illustration from emotional processing in alcohol-dependence

Introduction

Crossmodal processing can be globally defined as the ability to build a unitary representation of one's environment on the basis of stimulations coming from different sensorial modalities (Driver and Spence, 2000). In everyday life, human beings are confronted with a constant flow of multi-sensorial stimulations, and the capacity to integrate them is thus crucial for daily adaptive behaviors like social interactions, spatial attention or perceptuo-motor coordination (Lalanne and Lorenceau, 2004; Campanella and Belin, 2007). The importance of these crossmodal processes has led many researchers to investigate their behavioral and cerebral correlates, and the exploration of crossmodal mechanisms now constitutes a flourishing field in cognitive psychology and neuroscience (Calvert et al., 2001; De Gelder and Bertelson, 2003; Amedi et al., 2005). Indeed, hundreds of studies have been conducted during the last two decades on crossmodality among healthy participants, and huge advances have undeniably been made in understanding the developmental, psychological and cerebral correlates of crossmodality (particularly of face-voice integration). A wide variety of multimodal tasks have been used, requiring the integration of gender, identity (e.g., Joassin et al., 2011a,b; Love et al., 2011) or emotional features (e.g., Dolan et al., 2001; Pourtois et al., 2005; Ethofer et al., 2006a,b, 2013; Kreifelts et al., 2007, 2009, 2010; Robins et al., 2009; Müller et al., 2011), and these various results led to the identification of several brain areas specifically involved in the multisensory integration (mainly the superior parietal lobule, inferior occipital, middle frontal and superior temporal sulci).

Crossmodal Processes in Psychiatry: a Serious Lack of Data

The exploration of crossmodality in healthy populations has obviously gained a central position in the experimental psychology and neuroscience domains and has come to maturity, as notably illustrated by the development of integrative models structuring the numerous experimental data available (e.g., Campanella and Belin, 2007 for a review). However, there is a patent contrast between the large number of studies exploring the efficient crossmodal processing and the paucity of data currently available concerning the impairment of this processing in pathological populations. Indeed, while scientific knowledge is classically enriched by results obtained from impaired populations, very few clinical crossmodal research projects have been conducted up to now. Crossmodal integration impairments have been explored in several populations presenting perceptual alterations (e.g., visual or auditory loss, Zupan and Sussman, 2009; Barone, 2010; Massida et al., 2011), but very few studies have focused on crossmodal processing in neurological and psychiatric populations. Among these studies, main results concerned schizophrenia (De Gelder et al., 2005; Pearl et al., 2009; Szycik et al., 2009; Seubert et al., 2010a; Van den Stock et al., 2011), autism (van der Smagt et al., 2007; Mongillo et al., 2008; Foss-Feig et al., 2010; Kwakye et al., 2011) and Alzheimer's disease (Delbeuck et al., 2007), and suggested large-scale crossmodal deficits in these populations, while unimodal processes may be preserved (e.g., De Jong et al., 2009). As these psychopathological states have been described as disconnection syndromes (Friston and Frith, 1995; Melillo and Leisman, 2009), these crossmodal impairments might be related to an alteration of binding abilities needed to connect and integrate the different sensorial inputs. Moreover, as crossmodal integration is crucial in daily life, these alterations could be at least partly responsible for cognitive and social alterations observed in psychiatric states. Up to now however, as only scarce data are available in schizophrenia and autism, and as crossmodality has not been explored in other psychiatric states, many questions have remained unexplored concerning the crossmodal processing in psychiatry.

This striking paradox between the extensive knowledge concerning “normal” crossmodal processing, notably for the integration of emotional stimuli (e.g., De Gelder et al., 1999; Pourtois et al., 2000, 2002; Chen et al., 2010; Kreifelts et al., 2010; Regenbogen et al., 2012; Klasen et al., 2012; Ethofer et al., 2013) and the very few data on impaired crossmodal integration in clinical populations thus constitutes a strong limit of the current knowledge on crossmodality, and voices recently rose to promote the development of crossmodal research among clinical populations, with a double aim. On the one hand, there is a need to obtain a better description of the impairments associated with pathological states, particularly by offering a more ecological and complete evaluation of the deficits. On the other hand, a renewal of the understanding of crossmodal integration among healthy subjects is urgently needed. Indeed, if behavioral deficits in crossmodal processing are observed in a clinical population, comparing the cerebral activations between this population and healthy controls will offer strong insights concerning the brain regions associated with crossmodal processing. As summarized by Laurienti et al. (2005), “the use of clinical populations can add to the battery of study designs available to the imaging scientist investigating multisensory integration.” Despite the great promise of this perspective, very little research has up to now attempted to improve our understanding of both pathological states and the mechanisms of multisensory integration in general through crossmodal research in clinical populations.

Underlining the Usefulness of Crossmodal Research in Psychiatry

In view of the present limitations related to crossmodal research in psychiatric populations, the central objective of the present paper is to underline the usefulness of exploring crossmodal processing among clinical populations, and to help prepare the ground for the expansion of this innovative research topic. First, our recent studies exploring emotional crossmodal processing in alcohol-dependence will be described in order to illustrate the possibilities offered by this research field. Indeed, emotional decoding impairments have been found to be involved in the relapse after detoxification (e.g., Kornreich et al., 2001; Zywiak et al., 2003), and have been extensively investigated using visual or auditory stimulation (e.g., Monnot et al., 2001; Townshend and Duka, 2003, respectively). Capitalizing on these unimodal explorations, our research group has recently conducted several studies using audio-visual bimodal paradigms to increase the ecological validity of the experimental designs. The complementary use of behavioral, electrophysiological and neuroimaging techniques allowed obtaining the first insights concerning multimodal integration in alcohol-dependence. Second, these initial results will be discussed to show how they can (together with preliminary results obtained for multimodal integration in other psychiatric states) be extended to develop a coherent and ambitious research program using various psychiatric populations and sensory modalities. Finally, a conclusive section will underline the various potential clinical applications and the fundamental implications that this emerging project could bring.

Emotional Deficits in Alcohol-Dependence

Alcohol-Dependence: a Serious Mental Health Problem

Alcohol-dependence is the most frequent psychiatric diagnosis worldwide and is listed among the three more detrimental health problems (Harper and Matsumoto, 2005). Around 10% of the adult population in Western countries fulfils the diagnosis criteria for alcohol-dependence, and excessive alcohol consumption directly leads to 2.5 million deaths per year worldwide (World Health Organization, 2011). In view of these large-scale deleterious consequences of alcohol-dependence, extensive efforts have been conducted during the last decades to obtain a better understanding of alcohol-dependence at clinical and fundamental levels, particularly concerning the physiological, behavioral and cerebral impairments associated with chronic excessive alcohol consumption. Alcohol-dependence is known to have deleterious effects on many body systems (e.g., hepatic, cardio-vascular or gastrointestinal), but also on the central nervous system (see Oscar-Berman and Marinkovic, 2007 for a review,). Indeed, it has been extensively established that alcohol-dependence leads to major cerebral damage (McIntosh and Chick, 2004; Harper, 2007), particularly affecting white matter (Brooks, 2000; Oscar-Berman and Marinkovic, 2003), but also sub-cortical [e.g., amygdala (Cowen et al., 2004; Fein et al., 2006), insula, thalamus and cerebellum (Szabo et al., 2004; De Bellis et al., 2005)] and cortical [mainly temporal and frontal lobes (Kril et al., 1997; Harper and Matsumoto, 2005; Chanraud et al., 2007)] areas. These brain impairments correlate with the lifetime dose of ethanol consumed (Nicolás et al., 1997). At a functional level, many studies have explored the behavioral correlates of these cerebral effects, and have repeatedly shown impaired performance in a large range of cognitive abilities, ranging from perceptual (e.g., Blusewicz et al., 1977; Spitzer and Ventry, 1980; Spitzer, 1981) and attentional (e.g., Smith and Oscar-Berman, 1992; Sullivan et al., 1993; Noël et al., 2001) abilities to memory and executive functions (e.g., Bechara et al., 2001; Oscar-Berman et al., 2004; Flannery et al., 2007; Pitel et al., 2007). Nevertheless, in contrast with this extensive exploration of the consequences of alcohol-dependence on cognition, the evaluation of emotional abilities has long been neglected in this pathology.

The Emotional Deficit

Emotional states clearly have a major influence on most aspects of our lives, as emotions plays a role in every human's decisions, motivations or social interactions. Moreover it has been repeatedly shown that affective disturbances (e.g., impaired ability to identify or regulate one's own emotional states or to decode other persons' emotions) constitute a central characteristic of most mental diseases from a clinical point of view. Despite this obvious importance of emotions in psychiatry, the interest for experimental exploration of affective impairments in alcohol-dependence only rose during the last decade. While this field is still in its infancy, it already clearly appears that alcohol-dependence is associated with major impairments in a wide range of emotional abilities. Several recent research projects identified an impaired performance in various emotional functions among alcohol-dependent individuals, notably for alexithymia (Taieb et al., 2002; Uzun et al., 2003), emotional intelligence (Riley and Schutte, 2003; Szczepanska et al., 2004; Cordovil de Susa Uva et al., 2010) and empathy (Martinotti et al., 2009; Maurage et al., 2011a). More centrally for the present purpose, a deficit has also been consistently observed for the decoding of the emotions expressed by faces (Oscar-Berman et al., 1990; Frigerio et al., 2002; Clark et al., 2007; Marinkovic et al., 2009; Maurage et al., 2009a, 2011b) and voices (Monnot et al., 2001, 2002; Uekermann et al., 2005). Recently detoxified alcohol-dependent individuals globally over-estimate the intensity of the emotions conveyed by visual and auditory stimuli, have an erroneous interpretation of emotions and are not aware of their deficit (Philippot et al., 1999; Kornreich et al., 2001, 2002). While several contradictory results have emerged, describing a preserved decoding of visual (Uekermann and Daum, 2008) or auditory (Oscar-Berman et al., 1990) stimulations, this emotional decoding deficit is now strongly established as it has been replicated in a wide variety of paradigms and stimulus sets (e.g., morphed or ambiguous faces, Frigerio et al., 2002), and among individuals with various abstinence durations (Townshend and Duka, 2003; Montagne et al., 2006; Foisy et al., 2007a). The causal link between alcohol-dependence and emotional impairment is still unclear as no longitudinal study has up to now directly explored this question, but individuals presenting a high risk of developing alcohol-dependence (i.e., children of alcohol-dependent patients) have strong emotional disturbances, and notably (1) altered activations of the brain areas involved in the elicitation and decoding of emotional feelings, particularly the amygdala (Glahn et al., 2007), (2) reduced social skills abilities and inefficient emotional coping strategies, as explored by self-report questionnaires (Segrin and Menees, 1996), and (3) higher frequency of psychopathological states known to have a strong effect on mood and emotional states, like depression or anxiety (Sher et al., 1991). It can thus be postulated that these emotional deficits might at least partly precede the appearance of alcohol-dependence. Moreover, while initially described for negative emotions, this impairment has been shown to be generalized to positive affective states if more complex emotional stimuli are used: when more various emotional and mental states are involved in the decoding task, alcohol-dependence leads to a deficit for negative and positive emotions (but not for neutral mental states), as recently shown in a study (Maurage et al., 2011b) exploring the performance of alcohol-dependent individuals at the “Reading the Mind in the Eyes” test (Baron-Cohen et al., 2001). It should also be underlined that this emotional decoding deficit appears specific for emotional features and cannot be fully explained by the more general cerebral and cognitive deficits observed in alcohol-dependence. Indeed, it has been shown that, when emotional decoding is compared with other tasks of similar complexity involving the identification of facial features (e.g., age, gender, or race detection tasks), alcohol-dependent individuals present a global deficit for performance and reaction times in each task. Nevertheless, when the general cognitive impairment is controlled for (using a reaction time subtraction method), alcohol-dependence is no more associated with an alteration for non-emotional face processing tasks but the emotional decoding deficit persists, suggesting that this deficit is specific and not only due to general cognitive alterations (Foisy et al., 2007b; Maurage et al., 2008a). To sum up, it is now clearly recognized that alcohol-dependence is associated with a general impairment for the decoding of the emotional content of stimulations, which is present for faces and voices, but also for other affective stimuli like music (Kornreich et al., 2013) or body postures (Maurage et al., 2009a).

A More General Deficit Affecting Social Cognition

Importantly, it has been suggested that these emotional decoding impairments might influence social interactions and participate in the maintenance of the pathological state (Walitzer and Dearing, 2006). Indeed, as the development and preservation of adapted social communication is largely based on the ability to correctly express one's own emotional states and to accurately perceive (and react to) those expressed by others (Feldman et al., 1991), the emotional deficits might give rise to impaired interpersonal interactions and could increase the social problems frequently observed in alcohol-dependence. This assumption has not yet been directly tested in alcohol-dependence, but it has been shown that the intensity of emotional decoding alterations (specifically the over-estimation of negative emotions) is strongly correlated with the presence of interpersonal problems (as evaluated by the Inventory of Interpersonal Problems, Horowitz et al., 1988), and particularly with self-control difficulties in social contexts (Maurage et al., 2009a). While it has to be directly tested by prospective studies, this proposal of an involvement of emotional and interpersonal difficulties on the time course of alcohol-dependence recently received further empirical support by means of studies offering a specific exploration of the interpersonal abilities in alcohol-dependence. It has notably been shown that alcohol-dependent individuals have an impaired ability to understand humor (Uekermann et al., 2007), irony (Amenta et al., 2013) and mental states (Thoma et al., 2013), which are crucial abilities to correctly interact in interpersonal contexts. Moreover, alcohol-dependence is also associated with maladaptive self-beliefs in social context: alcohol-dependent individuals present an over-estimation of the social standards that have to be reached in order to be positively evaluated by others during social interactions, and their inability to actually reach these exaggerated social standards reduces their self-esteem and self-confidence when interacting with others (Maurage et al., 2013a). Finally, alcohol-dependence is also associated with a higher sensitivity to social rejection and ostracism (Maurage et al., 2012a), further underlining the role potentially played by emotional and interpersonal disturbances in this pathology.

Overall, better understanding the wide range of emotional disabilities related to alcohol-dependence (e.g., identifying, expressing and regulating one's own emotions, decoding and correctly reacting to others' emotions) is thus essential at fundamental level, but also for clinical practice, as they might have a role in the maintenance of this pathology, notably by hampering the development of satisfactory interpersonal links, thus potentially leading to social isolation and social stigma (Schomerus et al., 2011a,b). This social isolation could in turn reinforce the excessive alcohol consumption (used as a coping strategy to face isolation) and lead to a vicious circle (e.g., Carton et al., 1999). To sum up, the emotion decoding impairment in alcohol-dependence is now clearly identified and has a significant clinical importance, notably in view of its links with interpersonal problems. However, this deficit has up to now been exclusively explored using paradigms with low ecological validity, namely using only unimodal stimuli (faces or voices). It is thus unclear whether this deficit is maintained, reduced or increased in experimental designs that are closer to real life, specifically when crossmodal stimuli are used.

Crossmodal Emotional Alterations in Alcohol-Dependence

As it has been described in the previous section, the affective deficits related to alcohol-dependence are now clearly established, particularly concerning the alterations in the decoding of emotional faces and voices. However, all these studies were based on unimodal explorations, i.e., on the separate presentations of faces and voices. Nevertheless, sensory events are not experienced in isolation in everyday life, as we are constantly immersed in a flow of multiple sensory cues carrying information from different sensory modalities. Crossmodal processing is thus the rule rather than the exception and this is particularly true for emotions, as the perception and production of emotional states are routinely based on several sensory aspects (e.g., emotional facial expressions and emotional prosody in crossmodal face-voice stimuli). Therefore, while constituting a valuable first exploration, the unimodal investigations of affective processing among alcohol-dependent individuals conducted up to now are insufficient to comprehend the complexity of emotion decoding processing in this population and should be extended to more ecological crossmodal designs. Following this observation, and on the basis of earlier crossmodal studies which explored the integration of emotional stimuli among healthy controls by means of electrophysiological (e.g., De Gelder et al., 1999; Pourtois et al., 2000, 2002; Chen et al., 2010) and neuroimaging techniques (e.g., Dolan et al., 2001; Ethofer et al., 2006a,b, 2013; Kreifelts et al., 2007, 2009, 2010; Müller et al., 2011), three studies were conducted in our research group to explore, for the first time to the best of our knowledge, the crossmodal emotional decoding in alcohol-dependent participants. Importantly, these studies joining behavioral, electrophysiological and neuroimaging techniques will illustrate the usefulness of a complementary approach combining cognitive psychology and neurosciences approaches to determine the modification of audio-visual emotional decoding in alcohol-dependence. It should also be noted that these three studies are focused on the comparison between recently detoxified alcohol-dependent participants (i.e., individuals diagnosed with alcohol-dependence according to DSM-IV criteria and recruited during their third week of treatment in a detoxification center) and healthy controls paired for age, gender and education. Moreover, alcohol-dependent participants had abstained from alcohol for at least 2 weeks before the experiment took place, thus excluding any potential influence of acute alcohol intoxication on the results observed. Finally, in order to ensure that the emotional decoding deficits were indeed associated with alcohol consumption and not with biasing variables, several control measures were conducted: (1) participants had normal or corrected-to-normal visual and auditory abilities; (2) major medical problems, neurological disease (including epilepsy) and other psychiatric diagnoses (as assessed by an exhaustive psychiatric examination), including polysubstance abuse, constituted exclusion criteria in both groups; (3) subclinical psychiatric comorbidities (particularly depression and anxiety) were controlled for by means of questionnaires [i.e., Beck Depression Inventory (Beck and Steer, 1987) for depression, State-Trait Anxiety Inventory (Spielberger et al., 1983) for anxiety]. These measures were entered as covariables in our statistical analyses to control for the influence of these subclinical psychiatric comorbidities.

Do Alcohol-Dependent Individuals Present an Emotional Crossmodal Facilitation Effect?

The first exploration of crossmodal emotional processing in alcohol-dependence (Maurage et al., 2007a) was based on the elicitation of a “crossmodal facilitation effect.” Most classical crossmodal paradigms [e.g., McGurk (McGurk and McDonald, 1976) and ventriloquist (e.g., Alais and Burr, 2004) effects] are based on inhibition effects (i.e., to a deteriorated performance in crossmodal conditions as compared to unimodal). Nevertheless, several more recent studies have developed paradigms leading to a facilitation effect (Calvert et al., 2001; Teder-Sälejärvi et al., 2002), in which congruent bimodal (audio-visual) stimulation leads to better performance (i.e., higher correct response rates and/or shorter reaction times) than unimodal ones. This facilitation effect is considered as the behavioral marker of successful crossmodal integration of stimuli from different modalities (Calvert et al., 2001), and the absence of this facilitation effect would conversely index impaired crossmodal integration. This study was thus based on a design eliciting a facilitation effect to evaluate the presence of this effect among alcohol-dependent individuals. More precisely, an emotion-detection task was used in which participants were presented with emotional facial expressions and voices [i.e., audiotapes enunciating a semantically neutral name with an emotional prosody, taken from a validated battery (Maurage et al., 2007b)] depicting anger or happiness (see Figure 1 for an illustration of the experimental design). Auditory and visual stimuli were presented separately (unimodal conditions) or simultaneously (crossmodal condition, with emotionally congruent face-voice pairs). Morphed faces were used (40–60% level morphs depicting 40% of happiness and 60% of anger, or conversely) in order to increase the perceptual difficulty of faces and to obtain similar levels of difficulty for vision and audition [as faces are classically processed more rapidly than voices (Ellis et al., 1997; Schweinberger et al., 1997; Joassin et al., 2004)], which is needed to obtain a facilitation effect. Participants (20 alcohol-dependent inpatients and 20 paired controls) had to decide as quickly as possible which emotion was displayed in the stimulus (anger or happiness). While alcohol-dependent individuals were not significantly impaired for crossmodal processing in terms of performance, reaction times results first showed that alcohol-dependent participants were globally slower than controls whatever the experimental condition, which is a classical visuo-motor slowing effect associated with alcohol-dependence (e.g., Fein et al., 1990). But the central outcome of this study is that, as illustrated in Figure 2, while control participants showed a clear facilitation effect (i.e., significantly shorter reaction times in the audio-visual condition than in the unimodal auditory and visual), alcohol-dependent individuals did not present this effect, as no differences were observed in this group according to the experimental condition. Alcohol-dependence is thus associated with an absence of crossmodal facilitation effect. As the facilitation effect is the behavioral marker of efficient crossmodal processing, these results show that alcohol-dependence is associated with impaired auditory-visual integration of complex ecological stimuli. These results constitute the first direct evidence of a crossmodal impairment in alcohol-dependence. Nevertheless, as no control neutral condition was used, it cannot be asserted that this deficit is specific for crossmodal emotional processing as it might also be present for non-emotional crossmodal integration. Moreover, this initial study focused on the behavioral description of the specific deficit for crossmodal processing in alcohol-dependence and did not allow us to explore the cerebral correlates of the deficit. Two complementary studies were thus performed to explore the brain impairments related to this audio-visual integration deficit.

FIGURE 1

Figure 1. Illustration of the experimental design related to the crossmodal facilitation experiment, with the successive arrival of (1) a fixation cross, (2) the stimulus, and (3) an inter-stimuli black screen.The three categories of stimuli are illustrated, namely visual (angry face), auditory (angry voice) and crossmodal (happy face-voice).

FIGURE 2

Figure 2. Reaction times observed in the crossmodal facilitation experiment for the three modalities (A, Auditory; V, Visual; AV, Auditory-Visual) among alcohol-dependent participants (on the left) and controls (on the right). This panel shows that the facilitation effect (i.e., shorter reaction times for AV condition as compared to A and V ones) is present among controls but absent among alcohol-dependent participants (NS, Non-significant; *p < 0.05). Adapted from Maurage et al. (2007a).

What are the Brain Correlates Leading to the Alteration of Emotional Crossmodal Processes in Alcohol-Dependence?

At the neurophysiological level

The second study (Maurage et al., 2008b) aimed at describing the brain alterations leading to audio-visual integration impairment, by means of event-related potentials (ERP). ERP record the brain's electrical activity during cognitive tasks with a high temporal resolution and allow us to identify the electrophysiological component associated with the onset of a dysfunction, and then to infer the cognitive stage related to this impairment (Rugg and Coles, 1995). ERP have been widely explored in alcohol-dependent participants during the last decades. The initial explorations (see Hansenne, 2006 for a review) repeatedly described a deficit (reduced amplitude and delayed latency) of the P3b, a long-lasting positive parietal deflection functionally associated with the decisional stage of processing (Polich, 2004, 2007). However, other studies described a deficit in earlier visual ERP components, like P100 (Ogura and Miyazato, 1991), N170 or N200 (Kathmann et al., 1996). These deficits for P100 (linked to early visual processing) and more importantly for N170 (linked to specific processing of faces) suggest that the impairment in alcohol-dependence begins before the decisional level (P3b), namely at the visuo-spatial level of cognitive processing (Maurage et al., 2007c). Therefore, ERP clearly help to identify the precise stage (e.g., perceptual, attentional or decisional) at which a behavioral deficit originates, and were used here to determine the initial cognitive stage responsible for the crossmodal integration impairment in alcohol-dependence, with the following central question: does the crossmodal deficit start at an early, perceptive stage or only at later processing steps? This second study also explored the potential differential deficit observed for positive (i.e., happiness) vs. negative (i.e., anger) emotions. An emotion detection task was performed by 15 alcohol-dependent participants and 15 paired controls, with visual and auditory stimuli (similar to those presented in the first study) presented separately or simultaneously for 700 ms. Participants had to decide as fast as possible whether the face, voice or face-voice stimulus was an angry, happy or neutral emotional expression. ERP were recorded using a 32 electrode cap to obtain, for each participant, several electrophysiological components of interest (P100, N170–N2, P3b) for each experimental condition (visual, auditory or audio-visual) and each emotion (anger, happiness, neutral). First, alcohol-dependent individuals were less accurate than control participants to identify the emotion depicted in face or voice presented alone, these data being in line with the repeated observation of a deficit in the emotion decoding in alcohol-dependence (e.g., Philippot et al., 1999; Townshend and Duka, 2003; Maurage et al., 2009a). Moreover, alcohol-dependence was associated with slower reaction times, which confirms earlier results (Beatty et al., 1996; Verma et al., 2006) showing a global visuo-motor slowing down in alcohol-dependence, independently of the task or stimuli used. More centrally, the results clearly confirmed the ERP deficits classically observed in alcohol-dependence, as alcohol-dependence was associated with reduced amplitude and delayed latency of the N170/N2 and P3b components for visual and auditory stimulations, thus confirming the ERP alterations repeatedly described in this pathological state (Hansenne, 2006). However, the main result of this study concerned the group differences for the cerebral activations specifically associated with crossmodal processing. Indeed, we used a classical subtraction technique (Teder-Sälejärvi et al., 2002; Joassin et al., 2004) to isolate the electrophysiological activities directly related to visuo-auditory integration, as the auditory (A) and visual (V) unimodal conditions were subtracted from the auditory-visual bimodal condition (AV) using the following formula: AV − [A + V]. Group comparisons on these specific crossmodal activities showed that alcohol-dependence leads to highly reduced brain activity during integrative processes. Moreover, this deficit is particularly present for anger stimuli, with a strong impairment starting as early as 100 ms after stimulus appearance (while the deficits for happiness and neutral stimuli only appeared after 200–300 ms and were far less marked). Finally, a source location analysis was conducted by means of standardized weighted low-resolution electromagnetic tomography (swLORETA, Palmero-Soler et al., 2007), a technique allowing to accurately reconstruct nearby current sources on the basis of the electroencephalographic data. This analysis showed that the anger crossmodal processing impairment is indexed by a reduction in frontal activity. These data, shown in Figure 3, thus complement the results obtained in the first study by showing (1) that early crossmodal processing of emotional stimulation is impaired in alcohol-dependence, particularly for anger, and (2) that this deficit is associated with a reduction of the electrophysiological activations specifically linked with integrative processes, particularly in frontal areas. However, it should be noted that the electrophysiological deficit found in the present study was not related to reaction times or accuracy alterations, as no specific deficit for emotional stimulations (as compared to neutral ones) was observed among alcohol-dependent individuals at behavioral level. This absence of specific emotional deficit at the behavioral level might be partly explained by the very low sensitivity of the task, which was very easy (leading to a ceiling effect in performance, with more than 90% of correct answers). As this combination between relatively preserved emotional processing at the behavioral level and strongly impaired emotional processing at the cerebral level has been repeatedly observed in electrophysiological and neuroimaging studies in alcohol-dependence (e.g., Maurage et al., 2013b), a potential complementary explanation is that alcohol-dependent individuals might develop alternative strategies to compensate for their deficit. For example, in the present study, they might focus on one sensory modality to compensate for their crossmodal deficit, which might partly hide their emotional decoding deficit. In line with this, it can not be totally excluded that part of the electrophysiological alterations observed here in alcohol-dependent individuals as compared to controls might be due to the use of these alternative strategies. This proposal of the use of alternative strategies to compensate for a deficit in easy tasks should nevertheless be confirmed by specific studies varying the difficulty of the task and the possibility to use these alternative strategies. Moreover, due to their low spatial resolution, ERP are not able to precisely localize the brain areas involved in this integration deficit. Therefore, these results had to be confirmed and complemented by the use of neuroanatomical techniques, which was the central objective of the third study.

FIGURE 3

Figure 3. Source reconstruction analysis (using swLORETA software) of the cerebral generators among alcohol-dependent participants (left) and controls (right), for anger (above) and happiness (below) electrophysiological subtraction [AV−(A+V)] waves. Alcohol-dependent individuals exhibited a highly reduced frontal activation as compared to controls for anger stimulations. Adapted from Maurage et al. (2008b).

At the anatomical level

This third study (Maurage et al., 2013b) aimed at precisely locating the cerebral regions responsible for impaired crossmodal processes in alcohol-dependence, by means of functional magnetic resonance imaging (fMRI). One the one hand, alcohol-dependence is known to be associated with major cerebral consequences, particularly in white matter, limbic, temporal and frontal areas. On the other hand, the emotional impairments presented by alcohol-dependent individuals are also well documented, particularly for the decoding of visual or auditory stimulation. Nevertheless, these cerebral and emotional alterations have traditionally been explored separately, and very little is known about the cerebral correlates of emotional impairments in alcohol-dependence. To our knowledge, only a few studies have specifically focused on this topic, comparing the brain activations of recently detoxified alcohol-dependent participants with that of controls during the presentation of emotional scenes (Heinz et al., 2007) or emotional facial expressions (Salloum et al., 2007; Marinkovic et al., 2009). These results show that alcohol-dependence is associated with reduced activity in a wide range of brain areas during the processing of emotional stimuli, encompassing frontal, parietal and temporal regions which are not specifically involved in emotion processing. More precisely, the most statistically significant activity reductions were shown in regions playing a role in emotional processing, particularly the inferior frontal gyrus, anterior cingulate cortex and limbic structures (particularly the amygdala and hippocampus). A more recent study (Schulte et al., 2010) also suggested that alcohol-dependence is associated with white matter abnormalities, thus leading to disconnections between brain areas, and mainly between cortical and limbic structures. As the cortico-limbic connections are central for the processing and interpretation of emotional signals, this white matter deficit could play a major role in the affective disorders observed in alcohol-dependence. Nevertheless, these studies were exclusively based on the presentation of visual emotional stimuli, and the brain correlates of auditory or audio-visual emotional processing remains unexplored.

Alcohol-Dependence Leads to Serious Crossmodal Alterations

On the basis of the two studies presented above and of earlier results suggesting that brain areas dedicated to emotional processing are impaired or disconnected in alcohol-dependence, an fMRI study was conducted to explore the brain correlates of crossmodal emotional processing among alcohol-dependent participants. More precisely, an emotion detection task (illustrated in Figure 4) was administered to 12 alcohol-dependent participants and 12 paired controls while their brain activity was recorded using fMRI. The stimuli and task were identical to those presented in the first study, with a binary emotional decision (anger-happiness) on unimodal (morphed face or voice) or crossmodal (morphed face and voice presented simultaneously) stimulation. Brain activations during unimodal and crossmodal conditions were first computed (by subtracting the activations observed during a rest period without stimulation from those activations), and then the classical AV − [A + V] comparison was performed to isolate regions specifically involved in the integration of emotional faces and voices in both groups. It should first be underlined that alcohol-dependent individuals, while showing reduced unimodal activations (particularly in the middle frontal gyrus) presented a globally preserved pattern of cerebral activations during the separate processing of faces and voices. However, the central result of this study concerned crossmodal activations and the activations of the specific brain areas related to the integration of audio-visual stimulation. In the control group, the AV − [A + V] subtraction distinguished two categories of activations: on the one hand, several activations were found in unimodal regions (i.e., superior temporal gyrus for voices and fusiform gyrus for faces), showing that crossmodal stimulations provoke an enhanced activation in cerebral regions specialized in visual or auditory processing, which has been repeatedly observed among healthy participants (e.g., Calvert et al., 1999; Ghazanfar et al., 2005). On the other hand and more importantly, specific multimodal regions were revealed by the subtraction, namely middle frontal gyrus, superior parietal lobule and superior parietal gyrus. This is in line with earlier studies (e.g., Joassin et al., 2011a,b) showing that these brain regions are specifically activated in crossmodal conditions, as they receive multiple inputs from modality-specific regions and integrate them into a unitary and coherent representation of the environment (Rämä and Courtney, 2005; Bernstein et al., 2008). In the alcohol-dependent group however, the only significant activations for crossmodal stimulations were found in the unimodal regions cited above (mainly in the auditory regions), with a total absence of activations in the specific crossmodal areas. As further shown in the group comparison for crossmodal activities (Figure 5), it thus appears that alcohol-dependence is associated with a large and specific crossmodal deficit, indexed here by a lack of activation in the regions normally dedicated to the integration of inputs coming from different sensory modalities: alcohol-dependence is therefore associated with serious dysfunctions of the activation and connectivity between the cerebral regions involved in the multimodal perception of the social environment. Finally, psycho-physiological interactions (PPI) analyses allowed us to determine the functional connectivity between unimodal and crossmodal areas. Control participants presented a coherent connectivity pattern with on the one hand increased connectivity within unimodal regions (bilateral fusiform and superior temporal gyri), which confirms the enhanced unimodal connections in crossmodality (Kriegstein and Giraud, 2004), and on the other hand increased connectivity between unimodal and crossmodal areas (inferior occipital gyrus, middle frontal gyrus, superior parietal lobule), underlining the efficient functioning of the crossmodal cerebral network. Conversely, alcohol-dependent individuals did not present this pattern as unimodal areas were partially inter-connected but were not connected with crossmodal ones. As compared to controls, alcohol-dependent individuals showed highly reduced connectivity between unimodal and crossmodal areas, particularly with the middle frontal gyrus. These functional connectivity results thus suggest that the crossmodal deficit in alcohol-dependence could be partly due to disrupted connectivity within the crossmodal network, reducing the connections between unimodal and crossmodal areas. These data are preliminary and will have to be confirmed in future studies using larger groups and alternative experimental designs (Goebel and van Atteveldt, 2009; Love et al., 2011), notably by including neutral stimuli to explore the emotional specificity of these crossmodal alterations. However, they reinforce our earlier behavioral and electrophysiological results showing an emotional crossmodal processing deficit in alcohol-dependence, and offer the first description of the specific cerebral correlates of this impairment.

FIGURE 4

Figure 4. Illustration of the experimental design related to the neuroimaging study, showing the three experimental conditions, namely visual (left), auditory (center) and crossmodal (right) stimulations. Crossmodal stimulations were always congruent (i.e., depicting the same emotion in face and voice).

FIGURE 5

Figure 5. Group comparison showing the cerebral regions presenting significantly reduced activations among alcohol-dependent participants when compared to controls for the [AV − [A + V]] contrast in the neuroimaging study (p < 0.05 corrected for multiple comparisons at cluster size). Adapted from Maurage et al. (2013b).

Fundamental and Clinical Implications

The three studies presented above present a coherent pattern of results by describing similar specific audio-visual integration impairments in alcohol-dependence. Moreover, the use of different experimental methods and techniques provide complementary views of this impairment from behavioral, electrophysiological and neuroimaging data. Several implications can thus already be outlined on the basis of these results at experimental, fundamental and clinical levels, in order to lay the foundations for potential therapeutic interventions and future experimental investigation of these integrative processes.

First at the experimental level, the observation that the emotional decoding deficit in alcohol-dependence, widely described for unimodal stimulation, is increased in crossmodal stimulation, has shed new light on these earlier results and could influence future studies in the domain. Indeed, as crossmodal situations are omnipresent, our results suggest that earlier studies based on unimodal stimulation (and often using basic stimuli) underestimated the deficits in alcohol-dependence. This crossmodal impairment could also explain the hiatus between the relatively mild deficit frequently observed when presenting unimodal stimuli in experimental situations among alcohol-dependent subjects (e.g., Oscar-Berman et al., 1990; Beatty et al., 2000; Uekermann et al., 2005) and the obvious impairments observed in ecological situations, and notably in clinical observations. The present results could thus lead to a re-evaluation of earlier studies using unimodal stimuli, which probably underestimated the deficit present in real life situations. These results should also encourage future studies to use crossmodal stimulation in order to correctly evaluate various cognitive and emotional deficits in the processing of social stimuli. More generally, as emotional contexts in everyday life are most often characterized by the simultaneous perception of stimulations from different sensory modalities, our results argue for the development of more ecologically valid experimental designs, for example by means of video clips or virtual reality paradigms. Much progress has already been made in this direction for the evaluation of crossmodal processing among healthy participants (e.g., Vatakis and Spence, 2006; Barkhuysen et al., 2010; Petrini et al., 2010, 2011), and several studies have already used emotional crossmodal movies to explore face-voice integration (e.g., Kreifelts et al., 2010; Ethofer et al., 2013) but it has not been applied to clinical populations yet.

Second, from a more theoretical point of view, the development of experimental work on crossmodal processing in alcohol-dependence and other psychiatric states, could complement the results obtained among healthy participants and help to further renew the knowledge of crossmodal integration in general. Indeed, the exploration of impaired cognitive functions among clinical populations has traditionally been used, in neurology and neuropsychology, in order to add to those observations made among healthy individuals and to give a more exhaustive description of normal functioning. For instance, studies conducted among patients with cerebral lesions provided the description of double dissociations in memory or attentional systems, thus refining the theoretical models proposed for these systems (e.g., Listerud et al., 2009; Cohn et al., 2010; Barbeau et al., 2011). Specifically for the present topic, our fMRI results, showing that the crossmodal integration impairment in alcohol-dependence is associated with reduced activity in middle frontal gyrus, superior parietal lobule and superior parietal gyrus, confirms that these regions are necessary for a correct integration between faces and voices and thus reinforce the results obtained among healthy participants. This first step underlines the fact that exploring impaired crossmodal processing can offer promising perspectives, notably a better understanding of normal integration functioning.

Third, at the clinical level, the present results clearly confirm earlier data suggesting that emotional impairments (i.e., impaired ability to regulate one's own emotions or to correctly interpret others' emotional states) are a critical deficit in alcohol-dependence. The crossmodal paradigms used highlight that the impairments are more intense when experimental designs are closer to real life emotional situations. It is therefore obvious that emotional perturbations are involved in alcohol-dependence and should thus be considered in clinical settings, notably because they significantly contribute to relapse after detoxification,. Indeed, it has been shown by self-report questionnaires that more than 40% of alcohol-dependent patients identified the presence of emotional disturbances (e.g., persistent negative emotions, depression, anxiety) as being the most important causal factor explaining their relapse (Zywiak et al., 2003). Surprisingly, the experimental studies and theoretical models proposing therapeutic programs for alcohol-dependent individuals have up to now mainly focused on cognitive and behavioral aspects (e.g., development of coping strategies, motivation to change, DiClemente et al., 2003), and the emotional variables have been neglected. However, more recent models of addictions, and particularly the dual-process models (Wiers et al., 2007, 2013; Noël et al., 2010) offered a central position to emotional variables by considering that the addictive states are mostly due to an imbalance between “affective” (involved in automatic-impulsive behaviors) and “reflective” (involved in controlled-deliberate behaviors) systems. The present data, together with earlier results describing emotional alterations among alcohol-dependent individuals, further underline the importance of this affective system and encourage the development of therapeutic approaches focused on the rehabilitation of emotional abilities. Therapeutic programs have recently been developed to improve emotional decoding abilities among clinical populations by specifically training facial expression decoding (e.g., Face Tales Program, Philippot and Power, 2010). Applying them to alcohol-dependence, in complement with the classical rehabilitation programs including psychiatric and psychological therapy, might reduce the relapse rate after detoxification. Future development of these therapeutic proposals should include not only visual emotional stimuli, but also auditory and crossmodal stimulation, in order to develop more realistic emotion decoding rehabilitation programs. More globally, therapeutic interventions could also be improved through communication re-education programs in alcohol-dependence, focusing on crossmodal processing of the expression and identification of emotions in social settings.

Finally, in line with this clinical perspective, the specific crossmodal deficit for anger stimuli described in our electrophysiological study makes particular sense at the therapeutic level. Indeed, many clinical studies (e.g., Bartek et al., 1999; Karno and Longabaugh, 2005) have underlined that alcohol-dependent individuals have considerable difficulties to manage their anger, leading to aggressive behaviors and impulsivity in interpersonal relations. While the links between the regulation one's own anger and the ability to decode the anger expressed by others have not been directly explored yet, these two capacities might be simultaneously impaired in alcohol-dependence and sum up to increase interpersonal problems. However, although some studies have suggested that alcohol-dependence is associated with a relatively specific deficit in anger emotional facial expression decoding (e.g., Philippot et al., 1999), other studies have not replicated these results (Foisy, 2005; Uekermann et al., 2005) and this deficit has not been described for other stimuli (notably auditory prosody). This discrepancy between an obvious clinical deficit and contrasting experimental results could be explained by the fact that previous studies used only unimodal stimuli (mainly emotional facial expressions). These stimuli are artificial because in everyday social interactions, multimodal stimuli, and mainly auditory-visual stimuli, are much more common. Using more ecologic stimuli, our study established, at the electrophysiological level, the specific crossmodal deficit for anger in alcohol-dependence that has been repeatedly observed at clinical level. The development of future therapeutic programs should thus particularly emphasize the need to take into account this particular deficit for anger expression and decoding among alcohol-dependent individuals.

Perspectives for Future Research

As outlined above, our studies have to be considered as a very initial exploratory step in the examination of emotional crossmodal processing among clinical populations. Indeed, these studies focused on a specific clinical population and used a small subset of the possible emotional stimuli and sensory modalities. Nevertheless, when combined with the few previous data sets obtained among other psychiatric populations, these results constitute a reliable and promising basis for the development of an ambitious research program aiming at determining the behavioral and cerebral correlates of impaired crossmodal integration in psychiatry, and finally leading to strong fundamental propositions as well as clinical applications. Before ending this paper, four main directions that should be developed in future research will be proposed, each focusing on the extension of previous results and proposing a diversification of the emotional stimuli used, sensory modalities included and psychiatric populations explored.

Promoting the Use of More Emotional Expressions

A main limitation of the results presented above is that they only considered a very low number of emotional states (i.e., happy, angry, neutral). A central direction for future research will be to diversify the emotional stimulation used to determine the potential differential deficits associated with different emotional states in alcohol-dependence. It can indeed be hypothesized that alcohol-dependent individuals' emotional crossmodal deficit will vary according to stimulus valence. Our ERP results suggested a specific deficit for anger as compared to happy and neutral stimuli. This specific deficit makes sense at the clinical level and could lead to the development of innovative therapeutic programs. Nevertheless, it is not clear whether this impairment is really limited to anger or is more general, as it could for example be present for every negative emotion. It is thus necessary to develop crossmodal experimental paradigms that explore a broader set of emotions, and particularly of negative ones (e.g., disgust, fear, sadness) to confirm our results and separate the hypotheses of an anger-specific deficit vs. a general deficit for negative emotions. It has also been suggested (e.g., Philippot et al., 1999; Maurage et al., 2008c) that alcohol-dependence could be associated with a particular deficit for decoding and correctly reacting to emotional states which have a high interpersonal value, and particularly which are associated with a social evaluation aspect or moral judgment (e.g., anger, disgust, contempt, e.g., Hutcherson and Gross, 2011), as compared to emotions which express more self-focused feelings (e.g., fear, sadness). Crossmodal paradigms, due to their high ecological validity, could be very useful in exploring these hypothetical propositions on the differential deficit between emotions presenting high vs. limited social value in alcohol-dependence. A recent study (Lambrecht et al., 2012) has performed this differential exploration between others-oriented and self-oriented emotions among healthy controls, and this experimental approach might be applied in alcohol-dependence to explore the impact of interpersonal value on emotional decoding. More generally, future studies focusing on integration processes in alcohol-dependence should also go beyond the exploration of classical emotion decoding. Indeed, emotional abilities are not limited to this basic emotion decoding as daily life forces us to identify and correctly react to far more various and subtle emotional signals. More complex affective abilities (e.g., empathy, emotional intelligence) are thus also needed to develop and maintain satisfactory interpersonal relations, and several studies have recently shown that alcohol-dependence is associated with impairments for these abilities (e.g., Martinotti et al., 2009; Maurage et al., 2011a,b). These results, in line with recent studies exploring more complex emotional states among healthy people (e.g., Basile et al., 2011; Wagner et al., 2011), underline the need to go further than conventional emotion labels and to use more subtle and ecological paradigms in order to develop a better understanding of emotional impairments in alcohol-dependence. Nevertheless, these studies were conducted by means of unimodal visual stimuli, and thus have a low ecological validity. The use of crossmodal paradigms exploring complex emotional states and affective abilities will enrich these preliminary results by bringing experimental designs closer to daily situations and thus by offering a more valid description of alcohol related emotional and affective disorders. In this view, it would also be particularly important to use dynamic emotional faces synchronized with dynamic voices, as static materials do not capture the liveliness and true form of the facial expressions that typically occur in day-to-day interactions (Sato et al., 2004), and as the emotion content of these “static” non-canonical stimuli is processed by mental strategies and neural events distinct from their more ecologically relevant dynamic counterparts (Kilts et al., 2003).

Going Beyond Auditory and Visual Modalities

The crossmodal literature puts a strong emphasis on the integration between visual and auditory modalities. This focus is justified by the fact that vision and audition are by far the most dominant sensory modalities among human beings. Nevertheless, the near total absence of data concerning the other senses, and particularly the “chemical senses” (i.e., olfaction and taste) is surprising, as they play an underestimated but crucial role in the daily life of healthy as well as clinical populations (Schiffman, 1997). Indeed, olfactory and gustatory stimulation can also carry a strong emotional valence (e.g., Winston et al., 2005; Greimel et al., 2006; Shepherd, 2006), and exploring the integration between these emotional stimulations and visual or auditory ones could constitute a promising perspective to develop and renew crossmodal integration knowledge. More specifically, olfaction has been shown to play a crucial role in the development and maintenance of alcohol-dependence (e.g., Kareken et al., 2004; Little et al., 2005), but olfactory processing has up to now been studied very little in this pathology. Recent studies by our group exploring the olfactory abilities associated with excessive alcohol consumption (Maurage et al., 2011c,d) confirmed earlier results (Rupp et al., 2003, 2004, 2006) showing impaired processing of olfactory stimulations in alcohol-dependence, and gave the first insights concerning the cerebral correlates of this deficit (by means of ERP measures). Interestingly, our results showed that olfactory impairments are highly correlated with executive function deficits, and specifically with confabulation problems. These results suggest that these two abilities could rely on the same brain structures (and particularly on the orbitofrontal cortex), and that olfaction measures could be useful to shed new light on the exploration of executive and emotional impairments in alcohol-dependence. This is in line with recent proposals suggesting that olfactory measures could be a reliable cognitive marker in psychiatric disorders (Turetsky et al., 2009; Rupp, 2010). It thus appears that olfaction research is currently becoming a blooming research field among clinical populations. But once again, all these explorations have up to now been limited to unimodal stimulation while in real life situations, olfactory stimulations most often occur in combination with stimulation coming from other sensory modalities. This is particularly true for emotional contexts, and crossmodal explorations combining several senses (beyond audition and vision) are thus urgently needed to develop this new field of research. Very few studies have explored the crossmodal integration of emotional olfactory stimulation, by focusing on the influence of olfactory cues on facial expression decoding (Leppänen and Hietanen, 2003; Seubert et al., 2010a,b). These preliminary results replicated the classical facilitation effect, thus confirming the presence of genuine olfactory-visual integration among healthy participants. Neuroimaging data have also suggested that, while some brain areas (e.g., middle frontal gyrus) could be activated for every crossmodal interaction, independent of the sensory modalities engaged, other structures (mostly the anterior insula) could be specialized for olfactory-visual integration (e.g., Gottfried and Dolan, 2003; Small, 2004). Finally, they showed that schizophrenic patients present an impairment of this olfactory-visual integration, particularly for negative emotional stimuli, which suggests that crossmodal impairments among psychiatric populations could be independent of the sensory modalities involved. On the basis of these innovative explorations, future studies should thus explore, among healthy as well as clinical populations, the correlates of the crossmodal integration between the “chemical senses” and vision or audition. A more ambitious aim could be to go one step further toward ecological validity, by combining more than two sensorial modalities. Indeed, while our emotional experience is frequently based on the simultaneous perception of several sensory modalities, only bimodal stimulation paradigms have been proposed up to now. The recent technical advancements, and notably the growth of virtual reality, could lead to the development of experimental designs stimulating all the senses and thus open new perspectives for crossmodal processing explorations.

Applying Crossmodal Paradigms to Other Psychiatric Populations

The crossmodal studies presented in this chapter exclusively explored alcohol-dependence, and more specifically recently detoxified alcohol-dependent individuals. This population is of course only a specific part of the people presenting alcohol related problems, and more globally of the psychiatric patients. It thus appears important to underline the potential extension of these studies to other populations, in the field of alcohol abuse and dependence, but also in other psychiatric states, with the long term objective of developing a sound and integrative approach of crossmodal processing in clinical populations. Concerning alcohol related problems the literature on cerebral, cognitive and emotional impairments associated with alcohol consumption has classically been focused on long-term alcohol-dependence (namely on the exploration of the impairments due to long term chronic excessive alcohol consumption). Nevertheless, a new field of research has risen during the last decades, aiming at exploring the roots of alcohol addiction, namely the appearance and chronification of the deficits during the development of alcohol-dependence. On the one hand, many studies have been conducted among populations at high risk of becoming alcohol-dependent, mainly among children of alcohol-dependent individuals (Van der Stelt et al., 1994; Lieberman, 2000; Porjesz et al., 2005). These studies have suggested that several cerebral and cognitive impairments could be present before the development of alcohol-dependence, and thus be a causal factor rather than a consequence of excessive alcohol consumption. Our intention is not to go into the details of this important literature, but to underline that these explorations were once again exclusively based on unimodal stimulation. Crossmodal studies among children of alcohol-dependent individuals (notably for emotional abilities, which are still unexplored in this population) could thus give a more ecological and valid exploration of the deficits that are present before the development of alcohol-dependence. On the other hand and more recently, several projects have been conducted concerning the consequences of binge drinking (i.e., the excessive but episodic alcohol consumption, typically observed among adolescents and young adults and considered to be an “entrance door” toward alcohol-dependence, e.g., McCarty et al., 2004; Enoch, 2006). Recent studies have shown that binge drinking leads to cognitive effects (e.g., Giancola, 2002; Townshend and Duka, 2005; Zeigler et al., 2005), and we recently extended these results by suggesting that binge drinking habits rapidly lead to impaired processing of emotional auditory stimulation, and that this alcohol consumption pattern is particularly deleterious for brain functioning (Maurage et al., 2009b, 2012b). Nevertheless, it is still unknown whether these deficits are modified or not when several stimulations are presented together, and crossmodal studies would thus help to extend and clarify these preliminary results. Concerning the exploration of emotional crossmodal processing in psychiatry, it is surprising to notice that very few studies have been conducted among these clinical populations. Many projects have been proposed in recent years in order to explore the visual or auditory decoding of emotional stimulations among a wide variety of psychiatric states, like depression, autism, anxiety, anorexia nervosa and drug addiction (e.g., Mejias et al., 2005; Mendlewicz et al., 2005; Rossignol et al., 2005; Bhatara et al., 2010), but emotional crossmodal paradigms have only been used in a very limited number of these pathological states. Several studies (De Gelder et al., 2005; De Jong et al., 2009; Pearl et al., 2009; Szycik et al., 2009) have explored the integration of emotional stimulation among schizophrenic patients and consistently described emotional crossmodal deficits in this psychiatric state, notably indexed by reduced audio-visual integration ability and by a vision-audition imbalance (i.e., a dominance of the visual stimulation on the auditory one reducing crossmodal performance). These results, together with those obtained in alcohol-dependence, show that crossmodal processing impairments constitute a crucial aspect of psychiatric states, and should thus encourage the development of emotional crossmodal research among other psychiatric states. This is particularly true among populations which are known to present unimodal emotional decoding deficits, in order to answer the following central question: How does crossmodal integration occur when unimodal outputs are impaired? In other words, do some psychiatric populations manage to compensate their deficit in the processing of unimodal emotional stimuli by taking profit of the simultaneous presentation of two stimulations, or are all psychiatric states associated with increased processing impairments in crossmodal situations, as it has been observed in alcohol-dependence and schizophrenia?

Using Crossmodal Tasks in Real Clinical Settings

The final objective of this research area is also to offer new ways to manage clinical interventions in real clinical settings. Some preliminary data with clear potential clinical application have already been gathered (Campanella et al., 2010, 2012). Indeed, there is a growing literature base demonstrating that, throughout the information processing stream, a number of early and late neuroelectric features appear to be anomalous in various psychiatric populations. In all of these studies, the primary and most commonly reported finding has been P300 abnormalities (see Hansenne, 2006 for a review). P300 alterations have been highly important in the assessment of the pathophysiological mechanisms responsible for psychopathological states, as it is commonly acknowledged that a reduction of P300 amplitude is: (1) a state marker of depression, i.e., a biological marker that is altered during the disease but that stabilizes after clinical remission (Karaaslan et al., 2003); (2) a trait marker of schizophrenia, i.e., a biological parameter that is changed during and after the disease (Mathalon et al., 2000); and (3) a vulnerability marker of alcoholism, i.e., a biological variable that is altered before the emergence of the disease (high-risk children of alcoholic parents) (e.g., Hill et al., 1999). Such markers, if present, could be used to aid diagnosis, as prognostic elements, or to assist in choosing the most appropriate treatment for psychiatric disorders. They can also enhance our knowledge about the nature and the extent of cognitive damage, and offer deeper theoretical insights into both the aetiology and pathophysiology of the illness. Overall, such markers can improve early detection of illness, and, as such, facilitate more effective and targeted interventions (see Van der Stelt and Belger, 2007 for a review).

Nevertheless, the clinical sensitivity of the P300 has been hampered by the fact that its parameters (amplitude, latency) are diagnostically unspecific and not reliable enough to be useful for individual patients (Pogarell et al., 2007). Therefore, a current and important challenge for neurophysiologists is to discover novel and appropriate procedures to enhance the applicability and sensitivity of the P3b component in clinical settings. In this view, trying to increase P300 sensitivity, Campanella et al. (2010, 2012) have recently proposed a new variant of the classical oddball procedure using bimodal (visual-auditory) stimulations. The main idea was that, as bimodal face-voice associations require more “complex” interactions between unimodal (sensory) and multimodal (integrative) processes which work in parallel and influence each other, using a bimodal oddball design might enhance the sensitivity of the procedure by increasing the observable P300 differences evoked by “only” unimodal processes. To test this hypothesis, Campanella et al. (2010, 2012) compared two groups of participants: one group was healthy, and the other consisted of people displaying anxious and depressive tendencies. Both groups were submitted to unimodal (visual; auditory) and bimodal oddball tasks. Main results suggested that when the two groups of subjects differ in their subclinical level of comorbid anxiety and depression, unimodal visual and auditory oddball tasks may not allow us to detect this difference using P300 amplitude modulations, but a crossmodal task has greater power to detect even subclinical symptoms. Obviously, these results are preliminary and should be replicated on clinical populations: such experiments are currently underway in our laboratory, with the main aim to be able to index several steps of clinical severity in a pathological state. Moreover, it clearly appears that ERP data should not focus “only” on the P300 component: indeed, for instance, data have shown that (1) P300 deficits are correlated with previous “early” ERP alterations (e.g., Maurage et al., 2007c); and (2) the combination of different ERP components may be helpful to discriminate between different groups of patients. Price et al. (2006) compared and contrasted four electrophysiological endophenotypes -mismatch negativity, P50, P300, and antisaccades-, and analyzed their covariance on the basis of a single cohort of schizophrenic patients, family members and controls, tested with all paradigms. Data showed that the use of an electrophysiological battery has provided novel information on the characteristics of these features in schizophrenia and family member groups. In particular, it has highlighted the heterogeneity of electrophysiological features within these groups and how a combination of features could serve to minimize the impact of such heterogeneity. This outlines the urgent need in further studies to develop multisite guidelines to record a battery of electrophysiological measures that may be compared and used across studies.

Conclusion

The exploration of crossmodal processing among healthy controls has now become an extensive research field: behavioral as well as cerebral correlates of the integration processes between sensory modalities have been precisely explored among animal and human populations, leading to comprehensive models on this topic. Nevertheless, this maturity of the knowledge concerning “normal” crossmodal processes is in total contradiction with what can be observed in clinical states. Indeed, as outlined above, very little has been done up to now to attempt to understand how these crossmodal processes are impaired among neurological and psychiatric populations, and this astonishing lack of interest has had a detrimental effect on the advances that can be made in this topic.

The main aim of the present paper was thus to underline the urgent need to explore the crossmodal integration abilities among these populations, as progressing in this direction could lead to central implications at clinical and fundamental levels. First, for clinical aspects, using crossmodal designs among clinical populations would lead to a better understanding of the impairments presented by inpatients in real life situations (and notably in emotional contexts). This would provide a more ecological exploration of the cognitive, cerebral and affective deficits in these populations, thus complementing and clarifying earlier results. This could also lead to the development of new therapeutic interventions, using crossmodal clinical settings to rehabilitate impaired abilities (e.g., by means of virtual reality). Second, for fundamental research, while the data obtained among clinical populations have traditionally constituted a strong method to improve the understanding of normal abilities in neuropsychology and neuroscience (with the well-known proposition that exploring an impaired system helps to understand its healthy functioning), this approach has received very little attention in crossmodal processing research. Developing the exploration of integration abilities among clinical populations could shed a new light on the several questions that are still unresolved in this research field.

By describing our research focusing on emotional crossmodal integration in alcohol-dependence, we have only presented here what can be considered as a modest first step toward a real and ambitious research program allowing to precisely describing the crossmodal processing abilities among psychiatric populations. We indeed believe that our work, together with the few studies conducted in schizophrenia, constitute seminal results that should be developed in the future. More specifically, studies to come should extend this exploration of crossmodal processing in at least four main ways, by diversifying the stimuli (i.e., using a wider range of emotional but also non emotional stimuli), the sensory modalities (particularly by including the “chemical senses” in the crossmodal designs), the populations explored (i.e., studying the crossmodal processes among other populations with substance-abuse, but also among other psychiatric and neurologic states), and by adapting experimental paradigms to real needs of current clinical settings. These proposals for future studies are just illustrations of the many prospects offered by this largely unexplored field. In short, everything is still to be done concerning crossmodal processing in psychiatric populations, and our hope is thus that the preliminary data described in the present chapter will open the door to fresh, diverse and complementary studies.

Funding

The Authors are funded by the Belgian Fund for Scientific Research (F. N. R. S., Belgium), but this fund did not exert any editorial direction or censorship on any part of this article.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

Pierre Maurage and Salvatore Campanella are Research Associates at the Belgian Fund for Scientific Research (F. N. R. S., Belgium).

References

Alais, D., and Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal integration. Curr. Biol. 14, 257–262.

Pubmed Abstract | Pubmed Full Text

Amedi, A., von Kriegstein, K., van Atteveldt, N. M., Beauchamp, M. S., and Naumer, M. J. (2005). Functional imaging of human crossmodal identification and object recognition. Exp. Brain Res. 166, 559–571. doi: 10.1007/s00221-005-2396-5

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Amenta, S., Noël, X., Verbanck, P., and Campanella, S. (2013). Decoding of emotional components in complex communicative situations (irony) and its relation to empathic abilities in male chronic alcoholics: an issue for treatment. Alcohol. Clin. Exp. Res. 37, 339–347. doi: 10.1111/j.1530-0277.2012.01909.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Barbeau, E. J., Pariente, J., Felician, O., and Puel, M. (2011). Visual recognition memory: a double anatomo-functional dissociation. Hippocampus 21, 929–934.

Pubmed Abstract | Pubmed Full Text

Barkhuysen, P., Krahmer, E., and Swerts, M. (2010). Crossmodal and incremental perception of audiovisual cues to emotional speech. Lang. Speech 53, 3–30. doi: 10.1177/0023830909348993

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Baron-Cohen, S., Wheelwright, S., Hill, J., Raste, Y., and Plumb, I. (2001). The “Reading the mind in the eyes” test revised version: a study with normal adults, and adults with Asperger syndrome or high-functioning autism. J. Child Psychol. Psychiatry 42, 241–251. doi: 10.1111/1469-7610.00715

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Barone, P. (2010). Is the primary visual cortex multisensory? Comment on “Crossmodal influences on visual perception” by Prof. Ladan Shams. Phys. Life Rev. 7, 291–292. doi: 10.1016/j.plrev.2010.07.002

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Bartek, J. K., Lindeman, M., and Hawks, J. H. (1999). Clinical validation of characteristics of the alcoholic family. Nurs. Diagn. 10, 158–168. doi: 10.1111/j.1744-618X.1999.tb00047.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Basile, B., Mancini, F., Macaluso, E., Caltagirone, C., Frackowiak, R. S., and Bozzali, M. (2011). Deontological and altruistic guilt: evidence for distinct neurobiological substrates. Hum. Brain Mapp. 32, 229–239. doi: 10.1002/hbm.21009

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Beatty, W. W., Hames, K. A., Blanco, C. R., Nixon, S. J., and Tivis, L. J. (1996). Visuospatial perception construction and memory in alcoholism. J. Stud. Alcohol 57, 136–143.

Pubmed Abstract | Pubmed Full Text

Beatty, W. W., Tivis, R., Stott, D., Nixon, S. J., and Parsons, O. A. (2000). Neuropsychological deficits in sober alcoholics: influences of chronicity and recent alcohol consumption. Alcohol Clin. Exp. Res. 24, 149–154. doi: 10.1111/j.1530-0277.2000.tb04584.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Bechara, A., Dolan, S., Denburg, N., Hindes, A., Anderson, S. W., and Nathan, P. E. (2001). Decision making deficits, linked to a dysfunctional ventromedial prefrontal cortex, revealed in alcohol and stimulant abusers. Neuropsychologia 39, 376–389. doi: 10.1016/S0028-3932(00)00136-6

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Beck, A. T., and Steer, R. A. (1987). Beck Depression Inventory Manual. San Antonio, TX: Psychological Corporation.

Bernstein, L. E., Auer, Jr. E. T., Wagner, M., and Ponton, C. W. (2008). Spatiotemporal dynamics of audiovisual speech processing. Neuroimage 39, 423–435. doi: 10.1016/j.neuroimage.2007.08.035

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Bhatara, A., Quintin, E. M., Levy, B., Bellugi, U., Fombonne, E., and Levitin, D. J. (2010). Perception of emotion in musical performance in adolescents with autism spectrum disorders. Autism Res. 3, 214–225. doi: 10.1002/aur.147

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Blusewicz, M. J., Dustman, R. E., Schenkenberg, T., and Beck, E. C. (1977). Neuropsychological correlates of chronic alcoholism and aging. J. Nerv. Ment. Dis. 165, 348–355. doi: 10.1097/00005053-197711000-00006

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Brooks, P. J. (2000). Brain atrophy and neuronal loss in alcoholism: a role for DNA damage? Neurochem. Int. 37, 403–412. doi: 10.1016/S0197-0186(00)00051-6

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Calvert, G. A., Brammer, M. J., Bullmore, E. T., Campbell, R., Iversen, S. D., and David, A. S. (1999). Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport 10, 2619–2623. doi: 10.1097/00001756-199908200-00033

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Calvert, G. A., Hansen, P. C., Iversen, S. D., and Brammer, M. J. (2001). Detection of auditory visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 14, 427–438. doi: 10.1006/nimg.2001.0812

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Campanella, S., and Belin, P. (2007). Integrating face and voice in person perception. Trends Cogn. Sci. 11, 535–543. doi: 10.1016/j.tics.2007.10.001

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Campanella, S., Bruyer, R., Froidbise, S., Rossignol, M., Joassin, F., Kornreich, C., et al. (2010). Is two better than one? A cross-modal oddball paradigm reveals greater sensitivity of the P300 to emotional face-voice associations. Clin. Neurophysiol. 121, 1855–1862. doi: 10.1016/j.clinph.2010.04.004

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Campanella, S., Delle-Vigne, D., Kornreich, C., and Verbanck, P. (2012). Greater sensitivity of the P300 component to bimodal stimulation in an event-related potentials oddball task. Clin. Neurophysiol. 123, 937–946. doi: 10.1016/j.clinph.2011.10.041

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Carton, J. S., Kessler, E. A., and Pape, C. L. (1999). Nonverbal decoding skills and relationship wellbeing in adults. J. Nonverb. Behav. 23, 91–100. doi: 10.1023/A:1021339410262

CrossRef Full Text

Chanraud, S., Martelli, C., Delain, F., Kostogianni, N., Douaud, G., Aubin, H. J., et al. (2007). Brain morphometry and cognitive performance in detoxified alcoholdependents with preserved psychosocial functioning. Neuropsychopharmacology 32, 429–438. doi: 10.1038/sj.npp.1301219

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Chen, Y. H., Edgar, J. C., Holroyd, T., Dammers, J., Thonnessen, H., Roberts, T. P., et al. (2010). Neuromagnetic oscillations to emotional faces and prosody. Eur. J. Neurosci. 31, 1818–1827. doi: 10.1111/j.1460-9568.2010.07203.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Clark, U. S., Oscar-Berman, M., Shagrin, B., and Pencina, M. (2007). Alcoholism and judgments of affective stimuli. Neuropsychology 21, 346–362. doi: 10.1037/0894-4105.21.3.346

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Cohn, M., Moscovitch, M., and Davidson, P. S. (2010). Double dissociation between familiarity and recollection in Parkinson's disease as a function of encoding tasks. Neuropsychologia 48, 4142–4417. doi: 10.1016/j.neuropsychologia.2010.10.013

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Cordovil de Susa Uva, M., de Timary, P., Cortesi, M., Mikolajczak, M., du Roy de Blicquy, P., and Luminet, O. (2010). Moderating effect of emotional intelligence on the role of negative affect in the motivation to drink in alcohol-dependent subjects. Pers. Individ. Differ. 48, 16–21. doi: 10.1016/j.paid.2009.08.004

CrossRef Full Text

Cowen, M. S., Chen, F., and Lawrence, A. J. (2004). Neuropeptides: implications for alcoholism. J. Neurochem. 89, 273–285. doi: 10.1111/j.1471-4159.2004.02394.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

De Bellis, M. D., Narasimhan, A., Thatcher, D. L., Keshavan, M. S., Soloff, P., and Clark, D. B. (2005). Prefrontal cortex, thalamus, and cerebellar volumes in adolescents and young adults with adolescent-onset alcohol use disorders and comorbid mental disorders. Alcohol. Clin. Exp. Res. 29, 1590–1600. doi: 10.1097/01.alc.0000179368.87886.76

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

De Gelder, B., and Bertelson, P. (2003). Multisensory integration, perception and ecological validity. Trends Cogn. Sci. 7, 460–467. doi: 10.1016/j.tics.2003.08.014

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

De Gelder, B., Bocker, K. B., Tuomainen, J., Hensen, M., and Vroomen, J. (1999). The combined perception of emotion from voice and face: early interaction revealed by human electric brain responses. Neurosci. Lett. 260, 133–136. doi: 10.1016/S0304-3940(98)00963-X

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

De Gelder, B., Vroomen, J., de Jong, S. J., Masthoff, E. D., Trompenaars, F. J., and Hodiamont, P. (2005). Multisensory integration of emotional faces and voices in schizophrenics. Schizophr. Res. 72, 195–203. doi: 10.1016/j.schres.2004.02.013

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

De Jong, J. J., Hodiamont, P. P., Van den Stock, J., and de Gelder, B. (2009). Audiovisual emotion recognition in schizophrenia: reduced integration of facial and vocal affect. Schizophr. Res. 107, 286–293. doi: 10.1016/j.schres.2008.10.001

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Delbeuck, X., Collette, F., and Van der Linden, M. (2007). Is Alzheimer's disease a disconnection syndrome? Evidence from a crossmodal audio-visual illusory experiment. Neuropsychologia 45, 3315–3323. doi: 10.1016/j.neuropsychologia.2007.05.001

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

DiClemente, C. C., Jordan, L., Marinilli, A. S., and Nidecker, M. (2003). “Psychotherapy in alcoholism treatment,” in Handbook of Clinical Alcoholism Treatment, eds B. A. Johnson, P. Ruiz, and M. Galanter (Baltimore, MD: Lippincott, Williams, and Wilkins), 102–110.

Dolan, R. J., Morris, J. S., and de Gelder, B. (2001). Crossmodal binding of fear in voice and face. Proc. Natl. Acad. Sci. U.S.A. 98, 10006–10010. doi: 10.1073/pnas.171288598

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Driver, J., and Spence, C. (2000). Multisensory perception: beyond modularity and convergence. Curr. Biol. 10, 731–735. doi: 10.1016/S0960-9822(00)00740-5

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Ellis, H. D., Jones, D. M., and Mosdell, N. (1997). Intra- and inter-modal repetition priming of familiar faces and voices. Br. J. Psychol. 88, 143–156. doi: 10.1111/j.2044-8295.1997.tb02625.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Enoch, M. A. (2006). Genetic and environmental influences on the development of alcoholism: resilience vs. risk. Ann. N.Y. Acad. Sci. 1094, 193–201. doi: 10.1196/annals.1376.019

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Ethofer, T., Anders, S., Erb, M., Droll, C., Royen, L., Saur, R., et al. (2006a). Impact of voice on emotional judgment of faces: an event-related fMRI study. Hum. Brain Mapp. 27, 707–714. doi: 10.1002/hbm.20212

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Ethofer, T., Pourtois, G., and Wildgruber, D. (2006b). Investigating audiovisual integration of emotional signals in the human brain. Prog. Brain Res. 156, 345–361. doi: 10.1016/S0079-6123(06)56019-4

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Ethofer, T., Bretscher, J., Wiethoff, S., Bisch, J., Schlipf, S., Wildgruber, D., et al. (2013). Functional responses and structural connections of cortical areas for processing faces and voices in the superior temporal sulcus. Neuroimage 76, 45–56. doi: 10.1016/j.neuroimage.2013.02.064

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Fein, G., Bachman, L., Fischer, S., and Davenport, L. (1990). Cognitive impairments in abstinent alcoholics. West J. Med. 152, 531–537.

Pubmed Abstract | Pubmed Full Text

Fein, G., Landman, B., Tran, H., McGillivray, S., Finn, P., Barakos, J., et al. (2006). Brain atrophy in long-term abstinent alcoholics who demonstrate impairment on a simulated gambling task. Neuroimage 32, 1465–1471. doi: 10.1016/j.neuroimage.2006.06.013

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Feldman, R. S., Philippot, P., and Custrini, R. J. (1991). “Social competence and non-verbal behaviour”, in Fundamentals of Non-verbal Behavior, ed R. S. Feldman (New York, NY: Cambridge University Press), 329–350.

Flannery, B., Fishbein, D., Krupitsky, E., Langevin, D., Verbitskaya, E., Bland, C., et al. (2007). Gender differences in neurocognitive functioning among alcohol-dependent Russian patients. Alcohol. Clin. Exp. Res. 31, 745–754. doi: 10.1111/j.1530-0277.2007.00372.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Foisy, M. L. (2005). La Reconnaissance des Expressions Faciales émotionnelles dans l'alcoolisme: Spécification de la Nature et de l'origine des Biais évaluatifs. Unpublished Ph.D. Thesis, Catholic University of Louvain, Louvain-la-Neuve, Belgique.

Foisy, M. L., Kornreich, C., Fobe, A., D'Hondt, L., Pelc, I., Hanak, C., et al. (2007a). Impaired emotional facial expression recognition in alcohol dependence: do these deficits persist with midterm abstinence? Alcohol Clin. Exp. Res. 31, 404–410. doi: 10.1111/j.1530-0277.2006.00321.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Foisy, M. L., Kornreich, C., Petiau, C., Parez, A., Hanak, C., Verbanck, P., et al. (2007b). Impaired emotional facial expression recognition in alcoholics: are these deficits specific to emotional cues? Psychiatry Res.150, 33–41.

Pubmed Abstract | Pubmed Full Text

Foss-Feig, J. H., Kwakye, L. D., Cascio, C. J., Burnette, C. P., Kadivar, H., Stone, W. L., et al. (2010). An extended multisensory temporal binding window in autism spectrum disorders. Exp. Brain Res. 203, 381–389. doi: 10.1007/s00221-010-2240-4

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Frigerio, E., Burt, D. M., Montagne, B., Murray, L. K., and Perrett, D. I. (2002). Facial affect perception in alcoholics. Psych. Res. 113, 161–171. doi: 10.1016/S0165-1781(02)00244-5

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Friston, K. J., and Frith, C. D. (1995). Schizophrenia: a disconnection syndrome? Clin. Neurosci. 3, 89–97.

Pubmed Abstract | Pubmed Full Text

Ghazanfar, A. A., Maier, J. X., Hoffman, K. L., and Logothetis, N. K. (2005). Multi-sensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J. Neurosci. 25, 5004–5012. doi: 10.1523/JNEUROSCI.0799-05.2005

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Giancola, P. R. (2002). Alcohol-related aggression during the college years: theories, risk factors and policy implications. J. Stud. Alcohol 14, 129–139.

Pubmed Abstract | Pubmed Full Text

Glahn, D. C., Lovallo, W. R., and Fox, P. T. (2007). Reduced amygdala activation in young adults at high risk of alcoholism: studies from the Oklahoma family health patterns project. Biol. Psychiatry 61, 1306–1309. doi: 10.1016/j.biopsych.2006.09.041

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Goebel, R., and van Atteveldt, N. (2009). Multisensory functional magnetic resonance imaging: a future perspective. Exp. Brain Res. 198, 153–164. doi: 10.1007/s00221-009-1881-7

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Gottfried, J. A., and Dolan, R. J. (2003). The nose smells what the eye sees: crossmodal visual facilitation of human olfactory perception. Neuron 39, 375–386. doi: 10.1016/S0896-6273(03)00392-1

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Greimel, E., Macht, M., Krumhuber, E., and Ellgring, H. (2006). Facial and affective reactions to tastes and their modulation by sadness and joy. Physiol. Behav. 89, 261–269. doi: 10.1016/j.physbeh.2006.06.002

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Hansenne, M. (2006). Event-related brain potentials in psychopathology: clinical and cognitive perspectives. Psychol. Belg. 46, 5–36.

Harper, C. (2007). The neurotoxicity of alcohol. Hum. Exp. Toxicol. 26, 251–257. doi: 10.1177/0960327107070499

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Harper, C., and Matsumoto, I. (2005). Ethanol and brain damage. Curr. Opin. Pharmacol. 5, 73–78. doi: 10.1016/j.coph.2004.06.011

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Heinz, A., Wrase, J., Kahnt, T., Beck, A., Bromand, Z., Grüsser, S. M., et al. (2007). Brain activation elicited by affectively positive stimuli is associated with a lower risk of relapse in detoxified alcoholic subjects. Alcohol. Clin. Exp. Res. 31, 1138–1147. doi: 10.1111/j.1530-0277.2007.00406.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Hill, S. Y., Shen, S., Locke, J., Steinhauer, S. R., Konicky, C., Lowers, L., et al. (1999). Developmental delay in P300 production in children at high risk for developing alcohol-related disorders. Biol. Psychiatry 46, 970–981. doi: 10.1016/S0006-3223(99)00032-3

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Horowitz, L. M., Rosenberg, S. E., Baer, B. A., Ureno, G., and Villasenor, V. S. (1988). Inventory of interpersonal problems: psychometric properties and clinical applications. J. Consult. Clin. Psychol. 56, 885–892. doi: 10.1037/0022-006X.56.6.885

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Hutcherson, C. A., and Gross, J. J. (2011). The moral emotions: a social-functionalist account of anger, disgust, and contempt. J. Pers. Soc. Psychol. 100, 719–737. doi: 10.1037/a0022408

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Joassin, F., Pesenti, M., Maurage, P., Verreckt, E., Bruyer, R., and Campanella, S. (2011a). Cross-modal interactions between human faces and voices involved in person recognition. Cortex 47, 367–376. doi: 10.1016/j.cortex.2010.03.003

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Joassin, F., Maurage, P., and Campanella, S. (2011b). The neural network sustaining the crossmodal processing of human gender from faces and voices: an fMRI study. Neuroimage 54, 1654–1661. doi: 10.1016/j.neuroimage.2010.08.073

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Joassin, F., Maurage, P., Bruyer, R., Crommelinck, M., and Campanella, S. (2004). When audition alters vision: an eventrelated potential study of the cross-modal interactions between faces and voices. Neurosci. Lett. 369, 132–137. doi: 10.1016/j.neulet.2004.07.067

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Karaaslan, F., Gonul, A. S., Oguz, A., Erdinc, E., and Esel, E. (2003). P300 changes in major depressive disorders with and without psychotic features. J. Affect. Disord. 73, 283–287. doi: 10.1016/S0165-0327(01)00477-3

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kareken, D. A., Claus, E. D., Sabri, M., Dzemidzic, M., Kosobud, A. E. K., Radnovich, A. J., et al. (2004). Alcohol-related olfactory cues activate the nucleus accumbens and the ventrotemental area in high-risk drinkers: preliminary findings. Alcohol. Clin. Exp. Res. 28, 550–557. doi: 10.1097/01.ALC.0000122764.60626.AF

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Karno, M. P., and Longabaugh, R. (2005). An examination of how therapist directiveness interacts with patient anger and reactance to predict alcohol use. J. Stud. Alcohol 66, 825–832.

Pubmed Abstract | Pubmed Full Text

Kathmann, N., Soyka, M., Bickel, R., and Engel, R. R. (1996). ERP changes in alcoholics with and without alcohol psychosis. Biol. Psychiatry 39, 873–881. doi: 10.1016/0006-3223(95)00289-8

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kilts, C. D., Egan, G., Gideon, D. A., Ely, T. D., and Hoffman, J. M. (2003). Dissociable neural pathways are involved in the recognition of emotion in static and dynamic facial expressions. Neuroimage 18, 156–168. doi: 10.1006/nimg.2002.1323

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Klasen, M., Kenworthy, C. A., Mathiak, K. A., Kircher, T. T., and Mathiak, K. (2012). Supramodal representation of emotions. J. Neurosci. 31, 13635–13643. doi: 10.1523/JNEUROSCI.2833-11.2011

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kornreich, C., Blairy, S., Philippot, P., Dan, B., Foisy, M. L., Hess, U., et al. (2001). Impaired emotional facial expression recognition in alcoholism compared with obsessive-compulsive disorder and normal controls. Psychiatry Res. 102, 235–248. doi: 10.1016/S0165-1781(01)00261-X

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kornreich, C., Brevers, D., Canivet, D., Ermer, E., Naranjo, C., Constant, E., et al. (2013). Impaired processing of emotion in music, faces and voices supports a generalized emotional decoding deficit in alcoholism. Addiction 108, 80–88. doi: 10.1111/j.1360-0443.2012.03995.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kornreich, C., Philippot, P., Foisy, M. L., Blairy, S., Raynaud, E., Dan, B., et al. (2002). Impaired emotional facial expression recognition is associated with interpersonal problems in alcoholism. Alcohol Alcohol. 37, 394–400. doi: 10.1093/alcalc/37.4.394

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kreifelts, B., Ethofer, T., Grodd, W., Erb, M., and Wildgruber, D. (2007). Audiovisual integration of emotional signals in voice and face: an event-related fMRI study. Neuroimage 37, 1445–1456. doi: 10.1016/j.neuroimage.2007.06.020

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kreifelts, B., Ethofer, T., Huberle, E., Grodd, W., and Wildgruber, D. (2010). Association of trait emotional intelligence and individual fMRI-activation patterns during the perception of social signals from voice and face. Hum. Brain Mapp. 31, 979–991. doi: 10.1002/hbm.20913

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kreifelts, B., Ethofer, T., Shiozawa, T., Grodd, W., and Wildgruber, D. (2009). Cerebral representation of non-verbal emotional perception: fMRI reveals audiovisual integration area between voice- and face-sensitive regions in the superior temporal sulcus. Neuropsychologia 47, 3059–3066. doi: 10.1016/j.neuropsychologia.2009.07.001

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kriegstein, K. V., and Giraud, A. L. (2004). Distinct functional substrates along the right superior temporal sulcus for the processing of voices. Neuroimage 22, 948–955. doi: 10.1016/j.neuroimage.2004.02.020

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kril, J. J., Halliday, G. M., Svoboda, M. D., and Cartwright, H. (1997). The cerebral cortex is damaged in chronic alcoholics. Neuroscience 79, 983–998. doi: 10.1016/S0306-4522(97)00083-3

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kwakye, L. D., Foss-Feig, J. H., Cascio, C. J., Stone, W. L., and Wallace, M. T. (2011). Altered auditory and multisensory temporal processing in autism spectrum disorders. Front. Integr. Neurosci. 4, 1–11. doi: 10.3389/fnint.2010.00129

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Lalanne, C., and Lorenceau, J. (2004). Crossmodal integration for perception and action. J. Physiol. Paris 98, 265–279. doi: 10.1016/j.jphysparis.2004.06.001

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Lambrecht, L., Kreifelts, B., and Wildgruber, D. (2012). Age-related decrease in recognition of emotional facial and prosodic expressions. Emotion 12, 529–539. doi: 10.1037/a0026827

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Laurienti, P. J., Perrault, T. J., Stanford, T. R., Wallace, M. T., and Stein, B. E. (2005). On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Exp. Brain Res. 166, 289–297. doi: 10.1007/s00221-005-2370-2

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Leppänen, J. M., and Hietanen, J. K. (2003). Affect and face perception: odors modulate the recognition advantage of happy faces. Emotion 3, 315–326. doi: 10.1037/1528-3542.3.4.315

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Lieberman, D. Z. (2000). Children of alcoholics: an update. Curr. Opin. Pediatr. 12, 336–340. doi: 10.1097/00008480-200008000-00009

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Listerud, J., Powers, C., Moore, P., Libon, D. J., and Grossman, M. (2009). Neuropsychological patterns in magnetic resonance imaging-defined subgroups of patients with degenerative dementia. J. Int. Neuropsychol. Soc. 15, 459–470. doi: 10.1017/S1355617709090742

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Little, H. J., Stephens, D. N., Ripley, T. L., Borlikova, G., Duka, T., Schubert, M., et al. (2005). Alcohol withdrawal and conditioning. Alcohol. Clin. Exp. Res. 29, 453–464. doi: 10.1097/01.ALC.0000156737.56425.E3

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Love, S., Pollick, F. E., and Latinus, M. (2011). Cerebral correlates and statistical criteria of cross-modal face and voice integration. Seeing Perceiving 24, 351–367. doi: 10.1163/187847511X584452

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Marinkovic, K., Oscar-Berman, M., Urban, T., O'Reilly, C. E., Howard, J. A., Sawyer, K., et al. (2009). Alcoholism and dampened temporal limbic activation to emotional faces. Alcohol. Clin. Exp. Res. 33, 1880–1892. doi: 10.1111/j.1530-0277.2009.01026.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Martinotti, G., Di Nicola, M., Tedeschi, D., Cundari, S., and Janiri, L. (2009). Empathy ability is impaired in alcohol-dependent patients. Am. J. Addict. 18, 157–161. doi: 10.1080/10550490802544391

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Massida, Z., Belin, P., James, C., Rouger, J., Fraysse, B., Barone, P., et al. (2011). Voice discrimination in cochlear-implanted deaf subjects. Hear Res. 275, 120–129 doi: 10.1016/j.heares.2010.12.010

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Mathalon, D. H., Ford, J. M., and Pfefferbaum, A. (2000). Trait and state aspects of P300 amplitude reduction in schizophrenia: a retrospective longitudinal study. Biol. Psychiatry 47, 434–449. doi: 10.1016/S0006-3223(99)00277-2

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Maurage, P., Campanella, S., Philippot, P., Pham, T., and Joassin, F. (2007a). The crossmodal facilitation effect is disrupted in alcoholism: a study with emotional stimuli. Alcohol Alcohol. 42, 552–559. doi: 10.1093/alcalc/agm134

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Maurage, P., Joassin, F., Philippot, P., and Campanella, S. (2007b). A validated battery of vocal emotional expressions. Neuropsychol. Trends 2, 63–74.

Maurage, P., Philippot, P., Verbanck, P., Noel, X., Kornreich, C., Hanak, C., et al. (2007c). Is the P300 deficit in alcoholism associated with early visual impairments (P100, N170)? An oddball paradigm. Clin. Neurophysiol. 118, 633–644. doi: 10.1016/j.clinph.2006.11.007

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Maurage, P., Campanella, S., Philippot, P., Martin, S., and de Timary, P. (2008a). Face processing in chronic alcoholism: a specific deficit for emotional features. Alcohol Clin. Exp. Res. 32, 600–606. doi: 10.1111/j.1530-0277.2007.00611.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Maurage, P., Philippot, P., Joassin, F., Pauwels, L., Pham, T., Alonso Prieto, E. A., et al. (2008b). The auditory-visual integration of anger is impaired in alcoholism: an ERP study. J. Psychiatry Neurosci. 33, 111–122.

Pubmed Abstract | Pubmed Full Text

Maurage, P., Campanella, S., Philippot, P., Vermeulen, N., Constant, E., Luminet, O., et al. (2008c). Electrophysiological correlates of the disrupted processing of anger in alcoholism. Int. J. Psychophysiol. 70, 50–62. doi: 10.1016/j.ijpsycho.2008.05.572

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Maurage, P., Campanella, C., Philippot, P., Charest, I., Martin, S., and de Timary, P. (2009a). Impaired emotional facial expression decoding in alcoholism is also present for emotional prosody and body postures. Alcohol Alcohol. 44, 476–485. doi: 10.1093/alcalc/agp037

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Maurage, P., Pesenti, M., Philippot, P., Joassin, F., and Campanella, S. (2009b). Latent deleterious effects of binge drinking over a short period of time revealed by electrophysiological measures only. J. Psychiatry Neurosci. 34, 111–118.

Pubmed Abstract | Pubmed Full Text

Maurage, P., Grynberg, D., Noël, X., Joassin, F., Philippot, P., Hanak, C., et al. (2011a). Dissociation between affective and cognitive empathy in alcoholism: a specific deficit for the emotional dimension. Alcohol Clin. Exp. Res. 35, 1662–1668.

Pubmed Abstract | Pubmed Full Text

Maurage, P., Grynberg, D., Noël, X., Joassin, F., Hanak, C., Verbanck, P., et al. (2011b). The “Reading the Mind in the Eyes” test as a new way to explore complex emotions decoding in alcoholism. Psych. Res. 190, 375–378.

Pubmed Abstract | Pubmed Full Text

Maurage, P., Callot, C., Philippot, P., Rombaux, P., and de Timary, P. (2011c). Chemosensory event-related potentials in alcoholism: a specific impairment for olfactory function. Biol. Psychol. 88, 28–36. doi: 10.1016/j.biopsycho.2011.06.004

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Maurage, P., Callot, C., Chang, B., Philippot, P., Rombaux, P., and de Timary, P. (2011d). Olfactory impairment is correlated with confabulation in alcoholism: towards a multimodal testing of orbitofrontal cortex. PLoS ONE 6:e23190. doi: 10.1371/journal.pone.0023190

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Maurage, P., Joassin, F., Philippot, P., Heeren, A., Vermeulen, N., Mahau, P., et al. (2012a). Disrupted regulation of social exclusion in alcohol-dependence: an fMRI study. Neuropsychopharmacology 37, 2067–2075. doi: 10.1038/npp.2012.54

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Maurage, P., Joassin, F., Speth, A., Modave, J., Philippot, P., and Campanella, S. (2012b). Cerebral effects of binge drinking: respective influences of global alcohol intake and consumption pattern. Clin. Neurophysiol. 123, 892–901. doi: 10.1016/j.clinph.2011.09.018

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Maurage, P., de Timary, P., Moulds, M. M., Wong, Q., Collignon, M., Philippot, P., et al. (2013a). Maladaptive social self-beliefs in alcohol-dependence: a specific bias towards excessive high standards. PLoS ONE 8:e58928. doi: 10.1371/journal.pone.0058928

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Maurage, P., Joassin, F., Pesenti, M., Grandin, C., Heeren, A., Philippot, P., et al. (2013b). The neural network sustaining crossmodal integration is impaired in alcohol-dependence: an fMRI study. Cortex 49, 1610–1626. doi: 10.1016/j.cortex.2012.04.012

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

McCarty, C. A., Ebel, B. E., Garrison, M. M., DiGiuseppe, D. L., Christakis, D. A., and Rivara, F. P. (2004). Continuity of binge and harmful drinking from late adolescence to early adulthood. Pediatrics 114, 714–719. doi: 10.1542/peds.2003-0864-L

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

McGurk, H., and McDonald, J. (1976). Hearing lips and seeing voices. Nature 264, 746–748. doi: 10.1038/264746a0

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

McIntosh, C., and Chick, J. (2004). Alcohol and the nervous system. J. Neurol. Neurosurg. Psychiatry 75, 16–21. doi: 10.1136/jnnp.2004.045708

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Mejias, S., Rossignol, M., Debatisse, D., Streel, E., Servais, L., Guérit, J. M., et al. (2005). Event-related potentials (ERPs) in ecstasy (MDMA) users during a visual oddball task. Biol. Psychol. 69, 333–352. doi: 10.1016/j.biopsycho.2004.11.010

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Melillo, R., and Leisman, G. (2009). Autistic spectrum disorders as functional disconnection syndrome. Rev. Neurosci. 20, 111–131. doi: 10.1515/REVNEURO.2009.20.2.111

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Mendlewicz, L., Linkowski, P., Bazelmans, C., and Philippot, P. (2005). Decoding emotional facial expressions in depressed and anorexic patients. J. Affect. Disord. 89, 195–199. doi: 10.1016/j.jad.2005.07.010

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Mongillo, E. A., Irwin, J. R., Whalen, D. H., Klaiman, C., Carter, A. S., and Schultz, R. T. (2008). Audiovisual processing in children with and without autism spectrum disorders. J. Autism. Dev. Disord. 38, 1349–1358. doi: 10.1007/s10803-007-0521-y

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Monnot, M., Lovallo, W. R., Nixon, S. J., and Ross, E. (2002). Neurological basis of deficits in affective prosody comprehension among alcoholics and fetal alcohol-exposed adults. J. Neuropsychiatry Clin. Neurosci. 14, 321–328. doi: 10.1176/appi.neuropsych.14.3.321

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Monnot, M., Nixon, S., Lovallo, W., and Ross, E. (2001). Altered emotional perception in alcoholics: deficits in affective prosody comprehension. Alcohol. Clin. Exp. Res. 25, 362–369. doi: 10.1111/j.1530-0277.2001.tb02222.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Montagne, B., Kessels, R. P., Wester, A. J., and de Haan, E. H. (2006). Processing of emotional facial expressions in Korsakoff's syndrome. Cortex 42, 705–710. doi: 10.1016/S0010-9452(08)70408-8

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Müller, V. I., Habel, U., Derntl, B., Schneider, F., Zilles, K., Turetsky, B. I., et al. (2011). Incongruence effects in crossmodal emotional integration. Neuroimage 54, 2257–2266. doi: 10.1016/j.neuroimage.2010.10.047

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Nicolás, J. M., Estruch, R., Salamero, M., Orteu, N., Fernandez-Solà, J., Sacanella, E., et al. (1997). Brain impairment in well-nourished chronic alcoholics is related to ethanol intake. Ann. Neurol. 41, 590–598. doi: 10.1002/ana.410410507

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Noël, X., Bechara, A., Brevers, D., Verbanck, P., and Campanella, S. (2010). Alcoholism and the loss of willpower: a neurocognitive perspective. J. Psychophysiol. 24, 240–248. doi: 10.1027/0269-8803/a000037

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Noël, X., Van der Linden, M., Schmidt, N., Sferrazza, R., Hanak, C., Le Bon, O., et al. (2001). Supervisory attentional system in nonamnesic alcoholic men. Arch. Gen. Psychiatry 58, 1152–1158. doi: 10.1001/archpsyc.58.12.1152

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Ogura, C., and Miyazato, Y. (1991). Cognitive dysfunctions of alcohol dependence using event related potentials. Arukoru KenkyutoYakubutsu Ison. 26, 331–340.

Pubmed Abstract | Pubmed Full Text

Oscar -Berman, M., Hancock, M., Mildworf, B., Hutner, N., and Weber, D. A. (1990). Emotional perception and memory in alcoholism and aging. Alcohol. Clin. Exp. Res. 14, 383–393. doi: 10.1111/j.1530-0277.1990.tb00491.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Oscar-Berman, M., Kirkley, S. M., Gansler, D. A., and Couture, A. (2004). Comparisons of Korsakoff and non-Korsakoff alcoholics on neuropsychological tests of prefrontal brain functioning. Alcohol. Clin. Exp. Res. 28, 667–675. doi: 10.1097/01.ALC.0000122761.09179.B9

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Oscar-Berman, M., and Marinkovic, K. (2003). Alcoholism and the brain: an overview. Alcohol. Res. Health 27, 125–133.

Pubmed Abstract | Pubmed Full Text

Oscar-Berman, M., and Marinkovic, K. (2007). Alcohol: effects on neurobehavioral functions and the brain. Neuropsychol. Rev. 17, 239–257. doi: 10.1007/s11065-007-9038-6

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Palmero-Soler, E., Dolan, K., Hadamschek, V., and Tass, P. A. (2007). SwLORETA: a novel approach to robust source localization and synchronization tomography. Phys. Med. Biol. 52, 1783–1800. doi: 10.1088/0031-9155/52/7/002

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Pearl, D., Yodashkin-Porat, D., Katz, N., Valevski, A., Aizenberg, D., Sigler, M., et al. (2009). Differences in audiovisual integration, as measured by McGurk phenomenon, among adult and adolescent patients with schizophrenia and age-matched healthy control groups. Compr. Psychiatry. 50, 186–192. doi: 10.1016/j.comppsych.2008.06.004

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Petrini, K., Crabbe, F., Sheridan, C., and Pollick, F. E. (2011). The music of your emotions: neural substrates involved in detection of emotional correspondence between auditory and visual music actions. PLoS ONE 6:e19165. doi: 10.1371/journal.pone.0019165

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Petrini, K., McAleer, P., and Pollick, F. E. (2010). Audiovisual integration of emotional signals from music improvisation does not depend on temporal correspondence. Brain Res. 1323, 139–148. doi: 10.1016/j.brainres.2010.02.012

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Philippot, P., and Power, M. (2010). FaceTales. Available online at: http://www.ipsp.ucl.ac.be/recherche/projets/FaceTales/en/Home.htm doi: 10.1016/j.heares.2010.12.010

CrossRef Full Text

Philippot, P., Kornreich, C., Blairy, S., Baerts, I., Den Dulk, A., Le Bon, O., et al. (1999). Alcoholics' deficits in the decoding of emotional facial expression. Alcohol. Clin. Exp. Res. 23, 1031–1038.

Pubmed Abstract | Pubmed Full Text

Pitel, A. L., Beaunieux, H., Witkowski, T., Vabret, F., Guillery-Girard, B., Quinette, P., et al. (2007). Genuine episodic memory deficits and executive dysfunctions in alcoholic subjects early in abstinence. Alcohol. Clin. Exp. Res. 31, 1169–1178. doi: 10.1111/j.1530-0277.2007.00418.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Pogarell, O., Juckel, G., Norra, C., Leicht, G., Karch, S., Schaaff, N., et al. (2007). Prediction of clinical response to antidepressants in patients with depression: neurophysiology in clinical practice. Clin. EEG Neurosci. 38, 74–77. doi: 10.1177/155005940703800208

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Polich, J. (2004). Clinical application of the P300 event-related brain potential. Phys. Med. Rehabil. Clin. North Am. 15, 133–161. doi: 10.1016/S1047-9651(03)00109-8

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Polich, J. (2007). Updating P300: an integrative theory of P3a and P3b. Clin. Neurophysiol. 118, 2128–2148. doi: 10.1016/j.clinph.2007.04.019

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Porjesz, B., Rangaswamy, M., Kamarajan, C., Jones, K. A., Padmanabhapillai, A., and Begleiter, H. (2005). The utility of neurophysiological markers in the study of alcoholism. Clin. Neurophysiol. 116, 993–1018 doi: 10.1016/j.clinph.2004.12.016

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Pourtois, G., de Gelder, B., Bol, A., and Crommelinck, M. (2005). Perception of facial expressions and voices and of their combination in the human brain. Cortex 41, 49–59. doi: 10.1016/S0010-9452(08)70177-1

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Pourtois, G., de Gelder, B., Vroomen, J., Rossion, B., and Crommelinck, M. (2000). The time-course of intermodal binding between seeing and hearing affective information. Neuroreport 11, 1329–1333. doi: 10.1097/00001756-200004270-00036

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Pourtois, G., Debatisse, D., Despland, P. A., and de Gelder, B. (2002). Facial expressions modulate the time course of long latency auditory brain potentials. Brain Res. Cogn. Brain Res. 14, 99–105. doi: 10.1016/S0926-6410(02)00064-2

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Price, G. W., Michie, P. T., Johnston, J., Innes-Brown, H., Kent, A., Clissa, P., et al. (2006). A multivariate electrophysiological endophenotype, from a unitary cohort, shows greater research utility than any single feature in the Western Australian family study of schizophrenia. Biol. Psychiatry 60, 1–10. doi: 10.1016/j.biopsych.2005.09.010

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Rämä, P., and Courtney, S. M. (2005). Functional topography of working memory for face or voice identity. Neuroimage 24, 224–234. doi: 10.1016/j.neuroimage.2004.08.024

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Regenbogen, C., Schneider, D. A., Gur, R. E., Schneider, F., Habel, U., and Kellermann, T. (2012). Multimodal human communication–targeting facial expressions, speech content and prosody. Neuroimage 60, 2346–2356. doi: 10.1016/j.neuroimage.2012.02.043

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Riley, H., and Schutte, N. S. (2003). Low emotional intelligence as a predictor of substance-use problems. J. Drug Educ. 33, 391–398. doi: 10.2190/6DH9-YT0M-FT99-2X05

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Robins, D. L., Hunyadi, E., and Schultz, R. T. (2009). Superior temporal activation in response to dynamic audio-visual emotional cues. Brain Cogn. 69, 269–278. doi: 10.1016/j.bandc.2008.08.007

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Rossignol, M., Philippot, P., Douilliez, C., Crommelinck, M., and Campanella, S. (2005). The perception of fearful and happy facial expression is modulated by anxiety: an event-related potential study. Neurosci. Lett. 377, 115–120. doi: 10.1016/j.neulet.2004.11.091

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Rugg, M. D., and Coles, M. G. H. (1995). Electrophysiology of Mind. Oxford: Oxford University Press.

Rupp, C. I. (2010). Olfactory function and schizophrenia: an update. Curr. Opin. Psychiatry 23, 97–102. doi: 10.1097/YCO.0b013e328336643f

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Rupp, C. I., Fleischhacker, W., Drexler, A., Hausmann, A., Hinterhuber, H., and Kurz, M. (2006). Executive function and memory in relation to olfactory deficits in alcohol-dependent patients. Alcohol. Clin. Exp. Res. 30, 1355–1362. doi: 10.1111/j.1530-0277.2006.00162.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Rupp, C. I., Fleischhacker, W., Hausmann, H., Mair, D., Hinterhubern, H., and Kurz, M. (2004). Olfactory functioning in patients with alcohol dependence: impairments in odor judgments. Alcohol Alcohol. 39, 514–519. doi: 10.1093/alcalc/agh100

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Rupp, C. I., Kurz, M., Kemmler, G., Mair, D., Hausmann, A., Hinterhuber, H., et al. (2003). Reduced olfactory sensitivity, discrimination, and identification in patients with alcohol dependence. Alcohol. Clin. Exp. Res. 27, 432–439. doi: 10.1097/01.ALC.0000057945.57330.2C

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Salloum, J. B., Ramchandani, V. A., Bodurka, J., Rawlings, R., Momenan, R., George, D., et al. (2007). Blunted rostral anterior cingulate response during a simplified decoding task of negative emotional facial expressions in alcoholic patients. Alcohol. Clin. Exp. Res. 31, 1490–1504. doi: 10.1111/j.1530-0277.2007.00447.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Sato, W., Kochiyama, T., Yoshikawa, S., Naito, E., and Matsumara, M. (2004). Enhanced neural activity in response to dynamic facial expressions of emotion: an fMRI study. Cogn. Brain Res. 20, 81–91. doi: 10.1016/j.cogbrainres.2004.01.008

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Schiffman, M. (1997). Taste and smell losses in normal aging and disease. JAMA 278, 1357–1362. doi: 10.1001/jama.1997.03550160077042

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Schomerus, G., Corrigan, P. W., Klauer, T., Kuwert, P., Freyberger, H. J., and Lucht, M. (2011a). Self-stigma in alcohol dependence: consequences for drinking-refusal self-efficacy. Drug Alcohol Depend. 114, 12–17. doi: 10.1016/j.drugalcdep.2010.08.013

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Schomerus, G., Lucht, M., Holzinger, A., Matschinger, H., Carta, M. G., and Angermeyer, M. C. (2011b). The stigma of alcohol dependence compared with other mental disorders: a review of population studies. Alcohol Alcohol. 46, 105–112. doi: 10.1093/alcalc/agq089

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Schulte, T., Muller-Oehring, E. M., Pfefferbaum, A., and Sullivan, E. V. (2010). Neurocircuitry of emotion and cognition in alcoholism: contributions from white matter fiber tractography. Dialogues Clin. Neurosci. 12, 554–560.

Pubmed Abstract | Pubmed Full Text

Schweinberger, S. R., Herholz, A., and Sommer, W. (1997). Recognizing famous voices: influence of stimulus duration and different types of retrieval cues. J. Speech Lang. Hear. Res. 40, 453–463.

Pubmed Abstract | Pubmed Full Text

Segrin, C., and Menees, M. M. (1996). The impact of coping styles and family communication on the social skills of children of alcoholics. J. Stud. Alcohol. 57, 29–33.

Pubmed Abstract | Pubmed Full Text

Seubert, J., Kellermann, T., Loughead, J., Boers, F., Brensinger, C., Schneider, F., et al. (2010a). Processing of disgusted faces is facilitated by odor primes: a functional MRI study. Neuroimage 53, 746–756. doi: 10.1016/j.neuroimage.2010.07.012

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Seubert, J., Loughead, J., Kellermann, T., Boers, F., Brensinger, C. M., and Habel, U. (2010b). Multisensory integration of emotionally valenced olfactory-visual information in patients with schizophrenia and healthy controls. J. Psychiatry Neurosci. 35, 185–194. doi: 10.1503/jpn.090094

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Shepherd, G. M. (2006). Smell images and the flavour system in the human brain. Nature 444, 316–321. doi: 10.1038/nature05405

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Sher, K. J., Walitzer, K. S., Wood, P. K., and Brent, E. E. (1991). Characteristics of children of alcoholics: putative risk factors, substance use and abuse, and psychopathology. J. Abnorm. Psychol. 100, 427–448. doi: 10.1037/0021-843X.100.4.427

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Small, D. A. (2004). Crossmodal integration – insights from the chemical senses. Trends Neurosci. 27, 123–124. doi: 10.1016/j.tins.2004.01.002

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Smith, M. E., and Oscar-Berman, M. (1992). Resource-limited information processing in alcoholism. J. Stud. Alcohol. 53, 514–518.

Pubmed Abstract | Pubmed Full Text

Spielberger, D. C., Gorsuch, R. L., Lushene, R. L., Vagg, P. R., and Jacobs, G. A. (1983). Manual for the State-Trait Anxiety Inventory. Palo Alto, CA: Consulting Psychologists Press.

Spitzer, J. B. (1981). Auditory effects of chronic alcoholism. Drug Alcohol Depend. 8, 317–335. doi: 10.1016/0376-8716(81)90041-7

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Spitzer, J. B., and Ventry, I. M. (1980). Central auditory dysfunction among chronic alcoholics. Arch. Otolaryngol. 106, 224–229. doi: 10.1001/archotol.1980.00790280032006

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Sullivan, E. V., Mathalon, D. H., Zipursky, R. B., Kersteen-Tucker, Z., Knight, R. T., and Pfefferbaum, A. (1993). Factors of the Wisconsin Card sorting test as measures of frontal lobe function in schizophrenia and in chronic alcoholism. Psychiatry Res. 46, 175–199. doi: 10.1016/0165-1781(93)90019-D

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Szabo, Z., Owonikoko, T., Peyrot, M., Varga, J., Mathews, W. B., Ravert, H. T., et al. (2004). Positron emission tomography imaging of the serotonin transporter in subjects with a history of alcoholism. Biol. Psychiatry 55, 766–771. doi: 10.1016/j.biopsych.2003.11.023

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Szczepanska, L., Baran, J., and Mikolaszek-Boba, M. (2004). Connection between personality and emotional intelligence in groups of patients after suicidal attempts and ethanol dependent persons. Przeglev Lekvi. 61, 287–291.

Pubmed Abstract | Pubmed Full Text

Szycik, G. R., Münte, T. F., Dillo, W., Mohammadi, B., Samii, A., Emrich, H. M., et al. (2009). Audiovisual integration of speech is disturbed in schizophrenia: an fMRI study. Schizophr. Res. 110, 111–118. doi: 10.1016/j.schres.2009.03.003

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Taieb, O., Corcos, M., Loas, G., Speranza, M., Guilbaud, O., Perez-Diaz, F., et al. (2002). Alexithymia and alcohol dependence. Ann. Med. Interne (Paris). 153, 51–60.

Pubmed Abstract | Pubmed Full Text

Teder-Sälejärvi, W. A., McDonald, J. J., Di Russo, F., and Hillyard, S. A. (2002). An analysis of audiovisual crossmodal integration by means of event-related potentials (ERP) recordings. Cogn. Brain Res. 14, 106–114. doi: 10.1016/S0926-6410(02)00065-4

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Thoma, P., Winter, N., Juckel, G., and Roser, P. (2013). Mental state decoding and mental state reasoning in recently detoxified alcohol-dependent individuals. Psychiatry Res. 205, 232–240. doi: 10.1016/j.psychres.2012.08.042

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Townshend, J. M., and Duka, T. (2003). Mixed emotions: alcoholics' impairments in the recognition of specific emotional facial expressions. Neuropsychologia 41, 773–782. doi: 10.1016/S0028-3932(02)00284-1

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Townshend, J. M., and Duka, T. (2005). Binge drinking, cognitive performance and mood in a population of young social drinkers. Alcohol. Clin. Exp. Res. 29, 317–325. doi: 10.1097/01.ALC.0000156453.05028.F5

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Turetsky, B. I., Hahn, C. G., Borgmann-Winter, K., and Moberg, P. J. (2009). Scents and nonsense: olfactory dysfunction in schizophrenia. Schizophr. Bull. 35, 1117–1113. doi: 10.1093/schbul/sbp111

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Uekermann, J., and Daum, I. (2008). Social cognition in alcoholism: a link to prefrontal cortex dysfunction? Addiction 103, 726–735. doi: 10.1111/j.1360-0443.2008.02157.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Uekermann, J., Channon, S., Winkel, K., Schlebusch, P., and Daum, I. (2007). Theory of mind, humour processing and executive functioning in alcoholism. Addiction 102, 232–240. doi: 10.1111/j.1360-0443.2006.01656.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Uekermann, J., Daum, I., Schlebusch, P., and Trenckmann, U. (2005). Processing of affective stimuli in alcoholism. Cortex 41, 189–194. doi: 10.1016/S0010-9452(08)70893-1

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Uzun, O., Ats, A., Cansever, A., and Ozsahin, A. (2003). Alexithymia in male alcoholics: study in a Turkish sample. Compr. Psychiatry 44, 349–352. doi: 10.1016/S0010-440X(03)00009-9

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Van den Stock, J., de Jong, S. J., Hodiamont, P. P., and de Gelder, B. (2011). Perceiving emotions from bodily expressions and multisensory integration of emotion cues in schizophrenia. Soc. Neurosci. 6, 537–547. doi: 10.1080/17470919.2011.568790

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

van der Smagt, M. J., van Engeland, H., and Kemner, C. (2007). Brief report: can you see what is not there? Low-level auditory-visual integration in autism spectrum disorder. J. Autism Dev. Disord. 37, 2014–2019. doi: 10.1007/s10803-006-0346-0

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Van der Stelt, O., and Belger, A. (2007). Application of electroencephalography to the study of cognitive and brain functions in schizophrenia. Schizophr. Bull. 33, 955–970. doi: 10.1093/schbul/sbm016

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Van der Stelt, O., Gunning, W. B., Snel, J., Zeef, E., and Kok, A. (1994). Children of alcoholics: attention, information processing and event-related brain potentials. Acta Paediatr. 404, 4–6. doi: 10.1111/j.1651-2227.1994.tb13375.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Vatakis, A., and Spence, C. (2006). Audiovisual synchrony perception for speech and music assessed using a temporal order judgment task. Neurosci. Lett. 393, 40–44. doi: 10.1016/j.neulet.2005.09.032

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Verma, R. K., Panda, N. K., Basu, D., and Raghunathan, M. (2006). Audiovestibular dysfunction in alcohol dependence. Are we worried? Am. J. Otolaryngol. 27, 225–228. doi: 10.1016/j.amjoto.2005.09.005

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Wagner, U., N'Diaye, K., Ethofer, T., and Vuilleumier, P. (2011). Guilt-specific processing in the prefrontal cortex. Cereb Cortex. 21, 2461–2470. doi: 10.1093/cercor/bhr016

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Walitzer, K. S., and Dearing, R. L. (2006). Gender differences in alcohol and substance use relapse. Clin. Psychol. Rev. 26, 128–148. doi: 10.1016/j.cpr.2005.11.003

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Wiers, R. W., Bartholow, B. D., van den Wildenberg, E., Thush, C., Engels, R. C., Sher, K. J., et al. (2007). Automatic and controlled processes and the development of addictive behaviors in adolescents: a review and a model. Pharmacol. Biochem. Behav. 86, 263–283. doi: 10.1016/j.pbb.2006.09.021

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Wiers, R. W., Gladwin, T. E., Hofmann, W., Salemink, E., and Ridderinkhof, K. R. (2013). Cognitive bias modification and cognitive control training in addiction and related psychopathology: mechanisms, clinical perspectives, and ways forward. Clin. Psych. Sci. 1, 192–212. doi: 10.1177/2167702612466547

CrossRef Full Text

Winston, J. S., Gottfried, J. A., Kilner, J. M., and Dolan, R. J. (2005). Integrated neural representations of odor intensity and affective valence in human amygdala. J. Neurosci. 25, 8903–8907. doi: 10.1523/JNEUROSCI.1569-05.2005

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

World Health Organization. (2011). Global Status Report on Alcohol and Health. Geneva. Available online at: http://www.who.int/substance_abuse/publications/global_alcohol_report. Accessed on 05 June 2013.

Zeigler, D. W., Wang, C. C., Yoast, R. A., Dickinson, B. D., McCaffree, M. A., Robinowitz, C. B., et al. (2005). The neurocognitive effects of alcohol on adolescents and college students. Prev. Med. 40, 23–32. doi: 10.1016/j.ypmed.2004.04.044

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Zupan, B., and Sussman, J. E. (2009). Auditory preferences of young children with and without hearing loss for meaningful auditory-visual compound stimuli. J. Commun. Disord. 42, 381–396. doi: 10.1016/j.jcomdis.2009.04.002

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Zywiak, W. H., Westerberg, V. S., Connors, G. J., and Maisto, S. A. (2003). Exploratory findings from the reasons for drinking questionnaire. J. Subst. Abuse. Treat. 25, 287–292. doi: 10.1016/S0740-5472(03)00118-1

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text