# NEUROIMAGING OF AFFECTIVE EMPATHY AND EMOTIONAL COMMUNICATION

EDITED BY : Argye E. Hillis and Katherine P. Rankin PUBLISHED IN : Frontiers in Neurology

#### Frontiers Copyright Statement

© Copyright 2007-2019 Frontiers Media SA. All rights reserved. All content included on this site, such as text, graphics, logos, button icons, images, video/audio clips, downloads, data compilations and software, is the property of or is licensed to Frontiers Media SA ("Frontiers") or its licensees and/or subcontractors. The copyright in the text of individual articles is the property of their respective authors, subject to a license granted to Frontiers.

The compilation of articles constituting this e-book, wherever published, as well as the compilation of all other content on this site, is the exclusive property of Frontiers. For the conditions for downloading and copying of e-books from Frontiers' website, please see the Terms for Website Use. If purchasing Frontiers e-books from other websites or sources, the conditions of the website concerned apply.

Images and graphics not forming part of user-contributed materials may not be downloaded or copied without permission.

Individual articles may be downloaded and reproduced in accordance with the principles of the CC-BY licence subject to any copyright or other notices. They may not be re-sold as an e-book.

As author or other contributor you grant a CC-BY licence to others to reproduce your articles, including any graphics and third-party materials supplied by you, in accordance with the Conditions for Website Use and subject to any copyright notices which you include in connection with your articles and materials.

All copyright, and all rights therein, are protected by national and international copyright laws.

The above represents a summary only. For the full conditions see the Conditions for Authors and the Conditions for Website Use. ISSN 1664-8714 ISBN 978-2-88945-690-1 DOI 10.3389/978-2-88945-690-1

#### About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

#### Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

#### Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

#### What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office: researchtopics@frontiersin.org

# NEUROIMAGING OF AFFECTIVE EMPATHY AND EMOTIONAL COMMUNICATION

Topic Editors:

Argye E. Hillis, Johns Hopkins University School of Medicine, United States Katherine P. Rankin, University of California, San Francisco, United States

A recent explosion of research, both with neurotypical adults and individuals with brain lesions, has been devoted to delineating the auditory, cognitive, and motor processes underpinning affective empathy and emotional communication. This Research Topic highlights this line of investigation by bringing together a methodologically diverse range of neuroimaging studies that further advance our knowledge of the precise neural mechanisms by which these critical aspects of human interaction are accomplished, how they break down after brain damage, and how they recover, laying the groundwork for developing effective interventions for people with deficits in these functions.

Citation: Hillis, A. E., Rankin, K. P., eds. (2019). Neuroimaging of Affective Empathy and Emotional Communication. Lausanne: Frontiers Media. doi: 10.3389/978-2-88945-690-1

# Table of Contents

*04 Editorial: Neuroimaging of Affective Empathy and Emotional Communication*

Argye E. Hillis

*06 Emotion Regulation Using Virtual Environments and Real-Time fMRI Neurofeedback*

Valentina Lorenzetti, Bruno Melo, Rodrigo Basílio, Chao Suo, Murat Yücel, Carlos J. Tierra-Criollo and Jorge Moll

*21 Modulation of Cognitive and Emotional Control in Age-Related Mild-to-Moderate Hearing Loss*

Artyom Zinchenko, Philipp Kanske, Christian Obermeier, Erich Schröger, Arno Villringer and Sonja A. Kotz

*37 Differences in Cortical Structure and Functional MRI Connectivity in High Functioning Autism*

Alessandra M. Pereira, Brunno M. Campos, Ana C. Coan, Luiz F. Pegoraro, Thiago J. R. de Rezende, Ignacio Obeso, Paulo Dalgalarrondo, Jaderson C. da Costa, Jean-Claude Dreher and Fernando Cendes

*58 Cognitive and Affective Perspective-Taking: Evidence for Shared and Dissociable Anatomical Substrates*

Meghan L. Healey and Murray Grossman

*66 Neuroanatomy of Shared Conversational Laughter in Neurodegenerative Disease*

Peter S. Pressman, Suzanne Shdo, Michaela Simpson, Kuan-Hua Chen, Clinton Mielke, Bruce L. Miller, Katherine P. Rankin and Robert W. Levenson

*74 Affective Empathy in Behavioral Variant Frontotemporal Dementia: A Meta-Analysis*

Andrew R. Carr and Mario F. Mendez


Janelle N. Beadle, Sergio Paradiso and Daniel Tranel

*109 Right Hemisphere Regions Critical for Expression of Emotion Through Prosody*

Sona Patel, Kenichi Oishi, Amy Wright, Harry Sutherland-Foggio, Sadhvi Saxena, Shannon M. Sheppard and Argye E. Hillis

*116 The Affective Nature of Formulaic Language: A Right-Hemisphere Subcortical Process*

Diana Van Lancker Sidtis and John J. Sidtis

# Editorial: Neuroimaging of Affective Empathy and Emotional Communication

Argye E. Hillis\*

Johns Hopkins University School of Medicine, Baltimore, MD, United States

Keywords: emotion, functional neuroimaging, prosody, empathy, lesion-deficit brain mapping

**Editorial on the Research Topic**

#### **Neuroimaging of Affective Empathy and Emotional Communication**

This e-book brings together studies of the neural networks underlying affective empathy and emotional communication, from investigators from eight countries. The studies use a variety of methodologies, including EEG, task-related fMRI, resting state fMRI, PET, measures of cortical structure, parcel-based lesion symptom mapping, voxel based morphometry (VBM), facial electromyography, and real-time fMRI feedback. The investigations included a variety of populations, including healthy participants and people with hearing loss, autism, frontotemporal dementia (FTD), Alzheimer's disease (AD), primary progressive aphasia (PPA), progressive supranuclear palsy (PSP), corticobasal syndrome (CBS), and focal brain lesions. Although results do not converge on a single neuroanatomical or functional model of affective empathy or communication of emotions, they provide new insights into how we express, recognize, and share the emotions of other people.

Lorenzetti et al. studied emotion regulation in eight healthy controls using real-time fMRI neurofeedback (rtfMRI-NFB), as a proof of concept that this procedure might be useful in modulating complex emotions in people with anxiety, stress, or impaired empathy. The rtfMRI-NFB software provided a virtual environment to induce tenderness and anguish, using brain-computer interface and music. The procedure provided a robust method for both real-time measurement of the neural correlates of tenderness and anguish and voluntary modulation of these emotions. During tenderness, participants recruited the septo-hypothalamic area, medial frontal cortex, temporal pole, and precuneus. During anguish, participants recruited the amygdala, dorsolateral prefrontal, and additional regions associated with negative affect.

Zinchenko et al. reported data from two EEG experiments of 21 participants with hearing loss and 21 age-matched healthy controls, who observed multimodal video clips with facial expressions matched or mismatched with corresponding vocalizations. Participants categorized emotions (emotional conflict) of the clips. Negative stimuli modulated behavioral conflict processing in the normal hearing, but not in the hearing loss group. Yet, the amplitude difference in N100 responses between congruent and incongruent stimuli was larger in negative relative to neutral conditions in both groups across tasks. In the emotional conflict task, the hearing loss group performed at chance. They conclude that hearing loss affects processing of emotional acoustic cues and alters the behavioral benefits of emotional stimuli on cognitive and emotional control, despite preservation of early neural responses.

Pereira et al. report differences in cortical structure and functional connectivity in the default mode network (DMN) in rsfMRI in 22 participants with high-functioning autism (ASD) compared to 29 healthy controls. ASD patients had decreased gray matter volume and cortical thickness in

Edited and reviewed by: Jan Kassubek, Universität Ulm, Germany

> \*Correspondence: Argye E. Hillis argye@jhmi.edu

#### Specialty section:

This article was submitted to Applied Neuroimaging, a section of the journal Frontiers in Neurology

Received: 16 September 2018 Accepted: 27 September 2018 Published: 26 October 2018

#### Citation:

Hillis AE (2018) Editorial: Neuroimaging of Affective Empathy and Emotional Communication. Front. Neurol. 9:875. doi: 10.3389/fneur.2018.00875

**4**

cingulate, temporal lobes, and amygdala. Participants with ASD had reduced connectivity between the posterior cingulate cortex and areas of the executive control component of the DMN and higher connectivity between the anteromedial prefrontal cortex and areas of the sensorimotor component of the DMN. Decreased cortical thickness in right inferior frontal lobe correlated with poorer social functioning.

Healy and Grossman reviewed evidence from functional imaging in healthy participants and from behavioral and structural imaging studies in patients with bvFTD to identify shared and separate anatomical substrates of cognitive perspective-taking (ability to infer another's thoughts or beliefs) and affective perspective-taking (ability to infer another's emotions). They report that both types of perspective-taking engage temporoparietal junction, precuneus, and temporal poles, while only affective perspective-taking engages limbic system regions and basal ganglia. Furthermore, cognitive perspective-taking engages dorsomedial and dorsolateral prefrontal cortex, while affective perspective-taking engages ventromedial prefrontal cortex.

Pressman et al. examined the neural regions critical for shared conversational laughter in 75 participants with neurodegenerative disease, including AD, bvFTD, PPA, CBS, and PSP. In tapes of brief unrehearsed conversation with a partner, laughter was manually labeled and the timing of that laughter relative to the partner's laughter was identified. A voxel-based morphometry analysis of abnormal timing of laughter revealed the role of left precuneus and right fusiform gyrus.

Carr and Mendez carried out a meta-analysis of studies of affective empathy in bvFTD, who typically have atrophy of medial prefrontal, insula, and anterior temporal cortex. They found that people with bvFTD showed only a modest impairment in affective compared to controls across tasks. The most marked impairment was found in empathic concern.

Hua et al. reported evidence that empathy in bvFTD is undermined by enhanced positive emotional reactivity. They used facial electromyography of 26 participants with bvFTD and 25 healthy controls, as they identified emotions displayed in photographs of positive, negative, and neutral emotional faces. Participants with bvFTD showed impaired emotion recognition and greater reactivity of Zygomaticus major (ZM) (which is active during positive emotional reactions like smiling), compared to controls. Higher ZM reactivity was associated with worse negative emotion recognition. VBM revealed that smaller volume in the thalamus, midcingulate cortex, posterior insula, anterior temporal pole, amygdala, precentral gyrus, and inferior frontal gyrus was associated with greater ZM reactivity in bvFTD.

Beadle et al. identified a critical role of ventromedial prefrontal cortex in helping others who are suffering, by inducing empathy in eight participants with focal damage to the vmPFC and healthy controls, and measuring in real time their emotional responses and empathetic behavior. Those with damage to the vmPFC gave less money than healthy participants to a confederate who was suffering (a confederate).

Patel et al. found that listener ratings of prosody in 41 acute ischemic RH stroke patients positively correlated with four acoustic measures. Reductions in each of these four "prosody acoustic measures" was predicted by lesion load in pars opercularis, supramarginal gyrus, or associated white matter tracts (and not control regions).

Finally, Sidtis and Sidtis reviewed evidence from fMRI and PET studies of healthy participants and lesion-deficit association studies, of the neural bases of formulaic language, such as expletives and idioms that convey emotions. Evidence suggests that a right hemisphere-subcortical network modulates formulaic language.

Together, these studies provide evidence for a complex rightdominant network, including (but not limited to) ventromedial prefrontal cortex, amygdala, inferior frontal gyrus, insula, and temporal pole that mediates expression and recognition of emotion as well as emotional empathy. Each study reveals unique, novel insights about this complex network and stimulates future research in this field.

## AUTHOR CONTRIBUTIONS

The author confirms being the sole contributor of this work and has approved it for publication.

**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Hillis. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Emotion Regulation Using Virtual Environments and Real-Time fMRI Neurofeedback

Valentina Lorenzetti 1,2,3†, Bruno Melo4,5†, Rodrigo Basílio<sup>4</sup> , Chao Suo<sup>3</sup> , Murat Yücel <sup>3</sup> , Carlos J. Tierra-Criollo<sup>5</sup> and Jorge Moll <sup>4</sup> \*

<sup>1</sup> School of Psychology, Faculty of Health Sciences, Australian Catholic University, Melbourne, VIC, Australia, <sup>2</sup> Department of Psychological Sciences, Institute of Psychology Health and Society, University of Liverpool, Liverpool, United Kingdom, <sup>3</sup> Brain and Mental Health Laboratory, School of Psychological Sciences and Monash Institute of Cognitive and Clinical Neurosciences, Monash University, Melbourne, VIC, Australia, <sup>4</sup> D'Or Institute for Research and Education, IDOR, Rio de Janeiro, Brazil, <sup>5</sup> Biomedical Engineering Program, COPPE, Federal University of Rio de Janeiro, Rio de Janeiro, Brazil

#### Edited by:

Argye Hillis, Johns Hopkins Medicine, United States

#### Reviewed by:

Konstantinos Kalafatakis, University of Bristol, United Kingdom Yuzheng Hu, National Institute on Drug Abuse (NIDA), United States

> \*Correspondence: Jorge Moll jorge.moll@idor.org

†These authors have contributed equally to this work.

#### Specialty section:

This article was submitted to Applied Neuroimaging, a section of the journal Frontiers in Neurology

Received: 26 February 2018 Accepted: 14 May 2018 Published: 24 July 2018

#### Citation:

Lorenzetti V, Melo B, Basílio R, Suo C, Yücel M, Tierra-Criollo CJ and Moll J (2018) Emotion Regulation Using Virtual Environments and Real-Time fMRI Neurofeedback. Front. Neurol. 9:390. doi: 10.3389/fneur.2018.00390

Neurofeedback (NFB) enables the voluntary regulation of brain activity, with promising applications to enhance and recover emotion and cognitive processes, and their underlying neurobiology. It remains unclear whether NFB can be used to aid and sustain complex emotions, with ecological validity implications. We provide a technical proof of concept of a novel real-time functional magnetic resonance imaging (rtfMRI) NFB procedure. Using rtfMRI-NFB, we enabled participants to voluntarily enhance their own neural activity while they experienced complex emotions. The rtfMRI-NFB software (FRIEND Engine) was adapted to provide a virtual environment as brain computer interface (BCI) and musical excerpts to induce two emotions (tenderness and anguish), aided by participants' preferred personalized strategies to maximize the intensity of these emotions. Eight participants from two experimental sites performed rtfMRI-NFB on two consecutive days in a counterbalanced design. On one day, rtfMRI-NFB was delivered to participants using a region of interest (ROI) method, while on the other day using a support vector machine (SVM) classifier. Our multimodal VR/NFB approach was technically feasible and robust as a method for real-time measurement of the neural correlates of complex emotional states and their voluntary modulation. Guided by the color changes of the virtual environment BCI during rtfMRI-NFB, participants successfully increased in real time, the activity of the septohypothalamic area and the amygdala during the ROI based rtfMRI-NFB, and successfully evoked distributed patterns of brain activity classified as tenderness and anguish during SVM-based rtfMRI-NFB. Offline fMRI analyses confirmed that during tenderness rtfMRI-NFB conditions, participants recruited the septo-hypothalamic area and other regions ascribed to social affiliative emotions (medial frontal / temporal pole and precuneus). During anguish rtfMRI-NFB conditions, participants recruited the amygdala and other dorsolateral prefrontal and additional regions associated with negative affect. These findings were robust and were demonstrable at the individual subject level, and were reflected in self-reported emotion intensity during rtfMRI-NFB, being observed with both ROI and SVM methods and across the two sites. Our multimodal

**6**

VR/rtfMRI-NFB protocol provides an engaging tool for brain-based interventions to enhance emotional states in healthy subjects and may find applications in clinical conditions associated with anxiety, stress and impaired empathy among others.

Keywords: fMRI, emotion regulation, neurofeedback, BCI, region of interest, support vector machine, virtual reality, virtual environments

#### INTRODUCTION

Neurofeedback (NFB) is a novel application of brain-computer interfaces that aids real-time voluntarily regulation of brain activity. Mounting evidence shows that NFB has promising effects to enhance behavior, cognitive and emotional processes in normative samples (1–5). NFB has also been preliminary used to restore aberrant neurobiology and symptoms in neurological conditions (e.g., stroke, traumatic brain injury) and in psychopathology (e.g., ADHD, autism, depression, addiction) (1–7). Real-time functional magnetic resonance imaging (rtfMRI) based NFB has the potential to provide insight in understanding the mechanisms of psychological states (8–10). These include affiliative emotions (11) underpinned by deep brain nuclei (12, 13) the activity of which is unlikely to be robustly measured via surface electroencephalography.

rtfMRI NFB tools can be used to study the causal mechanisms of complex emotions and to inform evidence-based personalized interventions to enhance and recover aberrant emotional states (and their neural substrates) in normative and clinical samples. One key practical human challenge of fMRI studies includes participants being distracted and experiencing difficulties to feel valid psychological states in the scanner environment, particularly when trying to sustain complex emotions.

Recent studies have combined immersive virtual environments with multiple sensory modalities to deliver psychological/cognitive interventions, and to enhance their effectiveness via engaging and motivating individuals to practice (14–16).

Only two proof of concept studies have combined rt-NFB with virtual environments as brain computer interfaces (BCI). An electroencephalography-based NFB study computed brain activity from about 500 participants collectively, during an interactive game of relaxation and concentration over one night (16), where individual's level of brain activity could not be discerned. A separate rtfMRI-NFB paradigm used a virtual fire interface to up-regulate and down-regulate brain activity in eight healthy participants—but this was devoid of any emotional states and far from multimodal and immersive (17).

It remains untested whether rt-NFB platforms integrating multisensory virtual environments can successfully recruit complex emotions and sustain these emotions long and strong enough to probe their underlying neural correlates. Such a platform can advance NFB applications, via (i) increasing the ecological validity of rtfMRI-NFB experiments, and their relevance for the daily experiences of emotions outside of experimental settings, (ii) adapting NFB interfaces to the individual and target population so these are more relatable, engaging and effective in generating and sustaining complex emotions that maximize the success of rtfMRI-NFB interventions (18–20).

This study aims to demonstrate the feasibility of an engaging rtfMRI-NFB interface that can be individually tailored and, specifically, to provide a proof of concept for a rtfMRI-NFB integrating a virtual environment as a BCI and musical stimuli using both local (region of interest, ROI) and distributed (support vector machine, SVM) analyses. The FRIEND Engine Framework system (21) was enhanced and adapted for this aim. We recruited healthy young adults performing rtfMRI-NFB during complex emotion experiences, including tenderness—a positive affiliative emotion - and anguish—a self-reflective negative emotion (11, 13, 22–25).

We also aimed to validate the functional anatomy of these complex emotions during rtfMRI-NFB. After the real-time data was collected, we ran offline fMRI data analyses to verify the effects of the real-time neurofeedback task on brain activity using standard preprocessing and statistical analysis methods.

We hypothesized that participants would voluntary change the color of a virtual environment in the BCI during rtfMRI-NFB using the activity of the following regions: (i) for the tenderness condition, the septo-hypothalamic area (when using ROI-based rtfMRI-NFB method) and other brain areas ascribed to positive affiliative emotions i.e., medial orbitofrontal areas (when using SVM-based rtfMRI-NFB method) (11, 25–27); and (ii) for the anguish condition, the amygdala (during the ROI-based fMRI-NFB method) and also lateral prefrontal cortices implicated in negative affect (e.g., anguish, fear, anxiety, negative mood, stress, psychological pain), and in psychopathologies where negative affect is a feature [e.g., depression and generalized anxiety disorder (28–32)] (during SVM-based rtfMRI-NFB).

#### MATERIALS AND METHODS

#### Participants

We used a single subject, repeated measures design with two identical assessments on two consecutive days, counterbalanced by rtfMRI-NFB method (i.e., ROI and SVM). We recruited eight psychiatrically and neurologically healthy postgraduate research students, free of psychoactive medication and with normal or corrected-to-normal vision. Four participants were recruited from the D'Or Institute for Research and Education (IDOR) in Rio de Janeiro, Brazil (approved by the Ethics and Scientific committees of the Copa D'Or Hospital, Rio de Janeiro, Brazil - No 922.218). To validate the protocol in a different scanner and institution, we also recruited four participants from the Monash Biomedical Imaging (MBI) at Monash University in Melbourne, Australia (MUHREC CF15/1756 - 2015000893). All volunteers provided written informed consent prior to study participation.

#### Design of the Neurofeedback BCI

Supplementary video 1 and Figure 1 show the BCI used for the rt-fMRI NFB. The BCI comprised a virtual environment as a medium to convey sensory feedback to participants in real time, in association with ongoing tenderness, anguish and neutral emotional states. The virtual environment was created by editing the Unity 3D asset Autumnal Nature Pack (Unity 3D, https:// assetstore.unity.com/packages/3d/environments/autumnal-

nature-pack-3649) and displayed a first-person navigation at walking speed through hills and cornfields, with a total duration of 10′ 8 ′′ (Supplementary Video 1). The virtual environment was prepared to alternate between different trial types: neutral (30′′), tenderness (46′′) and anguish (46′′).

The trial types were displayed via changes in the base color hues of the virtual environment and via specific music excerpts. Music excerpts were fixed for each trial type, and not influenced by current neural/psychological states (no music for Neutral, mild, gentle music for Tenderness and eerie, distorted music for Anguish). Music excerpts were selected from 20 audio tracks, all normalized using the root mean square feature of Audacity software (Audacity, http://www.audacityteam.org). The audio tracks were previously rated to have comparable volume, pace, and rhythm. For the rtfMRI-NFB task runs, four excerpts for tenderness and four excerpts for anguish were played.

Neutral trials were characterized by a normal colored virtual landscape displayed in the BCI with no background music. Tenderness trials were characterized by a change in the color of the virtual landscape to orange and were accompanied by tenderness music excerpts. Anguish trials commenced when the color of the environment turned to purple hues and were accompanied by anguish music excerpts.

## Neurofeedback Task

#### Task Practice Outside the MRI

For training purposes, we recorded a video showing a sample of the virtual environment. The video lasted as long as one run of the rtfMRI-NFB task (10′ 8 ′′) and was used by participants to practice tenderness, anguish and neutral states before the MRI. With this practice, participants could learn which music tracks and VR color changes in the BCI corresponded to tenderness, anguish and neutral trials.

#### Neurofeedback Interface

As shown in **Figure 1**, instead of a classic thermometer, the color of the virtual environment was used as BCI changed in real time with increased engagement of the neural activity/pattern corresponding to distinct target emotional states—orange for tenderness trials, purple for anguish trials and natural light tones for neutral trials. Participants were instructed to experience tenderness or anguish as intensely as possible in the respective trials and to increase the intensity of their emotion to turn in real time, the color of the virtual environment BCI to as orange as possible during tenderness trials, and as purple as possible during anguish trials, which increased in turn the corresponding neural activity/pattern.

#### Training Run

During the training MRI run for rtfMRI-NFB, participants were instructed to feel the tenderness, anguish and neutral states as intensely as possible. This allowed mapping the brain regions that were most engaged by each individual while experiencing the emotions. We mapped and used the activity in these brain regions for each participant as a source for rtfMRI-NFB. The musical stimuli were delivered with MRI-compatible headphones (MR-Confon, http://www.mr-confon.de). The volume of the song excerpts was adjusted for each participant to a level where they could comfortably hear the music while performing rtfMRI-NFB.

#### Neurofeedback Task

For half of the sample, the rtfMRI-NFB task started with a tenderness trials block at baseline and follow up. The other half started the task with an anguish trials block at both assessments.

The fMRI protocol comprised four runs: a training run and three rtfMRI-NFB runs (10′ 8 ′′ each). The training run allowed mapping which brain regions the participant engaged while experiencing tenderness and anguish. The three subsequent rtfMRI-NFB runs provided participants with continuous feedback (every 2′′) on their brain activity in the form of updating the color of the virtual scenario in the BCI. The more participants engaged the target brain regions corresponding to tenderness and anguish states, the more the virtual environment would turn into orange and purple shades, respectively. During neutral rtfMRI-NFB trials, participants were not required to change the color of the virtual environment and this remained at baseline color.

#### Neurofeedback Methods: ROI and SVM

rtfMRI-NFB was delivered online and continuously via an updated platform of the FRIEND Engine Framework v 0.5 (21). We defined feedback signal as a sensory input to the participant (i.e., the color hue saturation of the dynamic virtual scenarios presented visually to participants in the BCI). This metric was determined by a number reflecting the hemodynamic state of a priori brain regions (or network of regions). Participants were instructed to enhance the target emotion as to intensify the color hue of the virtual environment BCI from neutral (baseline scenario hue) to orange (tenderness trials) or to purple (anguish trials).

We used two different rtfMRI-NFB methods to compute brain activity unknowingly to participants, test the capability of this software and explore whether the patterns of brain activity were more robustly recruited via either SVM based rtfMRI-NFB or ROI based rtfMRI-NFB. Half of the sample was randomly allocated to SVM method on day one and ROI method on day two, and the opposite order was used for the other half. We counterbalanced the presentation of the emotion trial types (**Figure 2**). The visual feedback

purple as participants increasingly engage target brain regions for tenderness and anguish trials.

on participants' brain activity was equivalent across ROI and SVM acquisitions although these relied on different metrics.

The rtfMRI-NFB ROI method computed percentage signal change (PSC) within the 10% most active voxels with an a priori defined ROI, measured across four blocks of the first training NFB run. ROIs included the septo-hypothalamic area when contrasting tenderness versus neutral conditions (11) and the right amygdala for when contrasting anguish versus neutral (33, 34). The feedback value was given by the equation

$$\frac{\overline{ROI}\_{\text{curr}\_{\text{-}}\text{vol}} - \sum\_{k=1}^{B} \frac{1}{\text{sig}(k)} \sum\_{k=1}^{B} \text{sig}\left(k\right) \overline{ROI}\_{k}}{\sum\_{k=1}^{B} \frac{1}{\text{sig}(k)} \sum\_{k=1}^{B} \text{sig}\left(k\right) \overline{ROI}\_{k}} \quad , \tag{1}$$

where ROIcurr\_vol is the mean of the ROI on the current volume, B is the number of volumes in the previous baseline condition and ROI<sup>k</sup> is the mean of the kth volume (21) weighted by a sigmoid function, sig k . The feedback value was used to modulate the color of the virtual environment, so that the higher the percent signal change, the more participants changed the color to orange and purple for the conditions of tenderness and anguish, respectively.

The SVM rtfMRI-NFB method provides the distance of a new observation relative to a separating hyperplane. It is a multivariate pattern analysis method that classifies the pattern of brain activity that best segregates between distinct conditions, which in our study comprised tenderness and anguish (i.e., all computed relative to the previous neutral block). We used a SVM classifier with a linear kernel and a cumulative training, meaning that all brain activity patterns observed during the rtfMRI-NFB task thus far are used at the end of the run to retrain and update the SVM classifier/model to use in the following runs. The projected value of a new observation was used to define the neurofeedback information, in our case, the color tonality of the virtual environment. For a new image volume, composed by real numbers x t , the projected value was given by x <sup>t</sup>w+b, where w is a vector containing the hyperplane coefficients and b is a constant (21). The more the pattern of brain activity segregated/classified the conditions, the more the color of the virtual landscape in the BCI turned to orange and purple, respectively.

The SVM rtfMRI-NFB method used a feature selection mask that included brain regions implicated in positive affiliative emotions (e.g., frontal, temporal, parietal and subcortical areas), and that excluded from SVM training and decoding those areas involved in sensorimotor or visuospatial processing (11).

#### MRI Data Acquisition

MRI and rtfMRI-NFB data were acquired in the two sites using a 3T Philips Achieva - at the D'Or Institute for Research and Education, in Rio de Janeiro, Brazil (Site 1) - and a 3T Siemens Magnetom Skyra - at the Monash Biomedical Imaging facility and the Brain and Mental Health Imaging laboratory, Monash Institute of Cognitive and Clinical Neurosciences, Monash University in Melbourne, Australia (Site 2).

Immediately before the rtfMRI-NFB task, we acquired highresolution anatomical images. In Site 1 we used an isotropic T1-weighted 3D turbo field echo sequence (TR/TE = 7.2/3.4 (s), flip angle = 8 ◦ , matrix size 240 × 240, FOV = 240 mm<sup>2</sup> , slice thickness = 1 mm, 170 slices, slice order ascending). Head motion was minimized via foam padding and straps over the forehead and under the chin. In Site 2 we used an isotropic T1 MP-RAGE scan (with TR/TE = 2.3/2.0 (s), flip angle = 9 ◦ , matrix size 256 × 240, FOV = 256 × 240 (mm), slice thickness = 1 mm, 170 slices, slice order descending).

fMRI data from the training run and the rtfMRI-NFB task comprised a total of 1,216 EPI volumes acquired over 40′ 32′′ in four runs (i.e., each run comprised 304 volumes and lasted 10′ 8 ′′). In both sites fMRI data were acquired with T2<sup>∗</sup> -weighted EPI (BOLD contrast), with TR/TE = 2,000/22 (ms), matrix = 64 × 64, FOV = 240 mm<sup>2</sup> , flip angle = 90◦ , isotropic voxel = 3.75 mm<sup>3</sup> , 24 slices. Before each fMRI run, we collected five dummy volumes for T1 equilibration. In Site 1 we used an optimized sequence with SENSE factor of 1.5 and dynamic stabilization to enhance temporal signal-to-noise (35) in brain areas prone to susceptibility effects (i.e., basal forebrain, ventromedial prefrontal cortex).

#### Behavioral Methods

The assessment protocol is overviewed in **Figure 3**. One week before baseline assessment, participants were contacted to identify the most effective personalized cognitive strategies to elicit tenderness, anguish and neutral states that would have been used by them during rtfMRI-NFB to up-regulate the underlying neural substrates. Tenderness was defined as a positive and affiliative (but not romantic) emotion experienced toward significant others, anguish as a negative and upsetting emotion not necessarily involving others, and neutral as emotionally neutral. Participants were also provided with a list of 20 sentences or mantras for each emotion, to use as a source to reflect on cognitive strategies to elicit tenderness and anguish states.

At baseline assessment, we collected participants sociodemographic data. At both baseline and 1 day follow up assessment, we administered questionnaires immediately before and after the MRI scan and rtfMRI-NFB to monitor changes in affect (Supplementary Table 1). Questionnaires included the Beck Depression Inventory [BDI (36)], the "state" subscale of the State and Trait Anxiety Inventory [STAI (37)] and Positive And Negative Affect Scale [PANAS (38)].

We administered visual analog scales (VAS) in between all the MRI runs (comprising a training run and three rtfMRI-NFB runs) to monitor participants' experience of (i) tenderness, anguish and neutral states (from 1 = very mild to 5 = very intense), (ii) how useful they found the emotion regulation strategies (from 1 = very little to 5 = very useful), (iii) how easy they found to use the virtual environment BCI (from 1 = extremely difficult and 5 = extremely easy), (iv) how easy they found to change the color of the virtual environment BCI during rtfMRI-NFB (from 1 = extremely difficult and 5 = extremely easy), (v) fatigue (from 1 = not at all and


FIGURE 3 | Outline of the assessment protocol. Assessments were identical across sites and days (baseline and day 2), with questionnaires administration before, during and after the MRI assessment and training/NFB runs. Unknowingly, each participant was delivered NFB using a distinct NFB method (either SVM = support vector machine or ROI = region of interest) on each of the two assessment days (gray box). Half of the participants delivered a ROI NFB method at baseline and a SVM NFB method at the one-day follow up. The other half underwent SVM NFB first and ROI NFB at follow up. BDI = Beck Depression Inventory (36), ROI = region of interest; STAI = Spielberger State and Trait Anxiety Inventory (37); PANAS = Positive and Negative Affect Scale (38). ERQ = Emotion Regulation Questionnaire (39), BVS = Body Vigilance Scale (40), SLS = Satisfaction with Life Scale (41).

5 = extremely) and (vi) focus (from 1 = not at all and 5 = extremely).

After MRI, we administered the Emotion Regulation Questionnaire (39), Satisfaction with Life Scale (41), and Body Vigilance Scale (40). After the follow-up assessment (end of day 2), participants were administered VAS scales to rate (from 1 = not at all, to 10 = extremely) how much the music excerpts evoked ten different positive and negative emotional states including anguish and tenderness, enchantment, transcendence, strength, serenity/peace, joy, nostalgic, sadness and tension.

## Off-Line Statistical Analyses

#### Behavioral Data

Participants' strategies to up-regulate tenderness, anguish and neutral states were qualitatively described. Chi-square and Ttests were run to compare participants' sex, age, questionnaire and VAS data between sites.

Repeated measures ANOVAs were run using site as a betweensubject factor (site 1 and site 2) and assessment time as a repeated measure (pre MRI and post MRI) to assess the effects of site and NFB task on BDI, STAI and PANAS scores.

Repeated measures ANOVAs were performed using site as between-subject factor and MRI run as repeated measure (one training run and three rtfMRI-NFB runs) to assess their effect on participants' experiences during rtfMRI-NFB (i.e., emotion intensity, how useful they found their emotion regulation strategies and to change the color of the virtual environment BCI during rtfMRI-NFB).

Two linear mixed models tested the effect of the three rtfMRI-NFB runs, the method to compute brain activity in real time (SVM and ROI) and assessment site, on the change in the color of the virtual environment BCI during rtfMRI-NFB (i.e., the degree of orange saturation for the tenderness condition, and purple saturation during the anguish condition).

Finally, t-tests compared emotion ratings of the music excerpts between the anguish tenderness and neutral conditions. We used IBM SPSS Statistics v22.0.0.0.

#### MRI Data Processing

MRI data was processed offline using Statistical Parametric Mapping 12 software v6470 (SPM12; www.fil.ion.ucl.ac.uk/spm).

#### **Offline MRI data pre-processing and first level analysis**

Offline MRI data Preprocessing included realignment, slice timing, normalization using T1-weighted images and smoothing. We corrected first level analysis results for artifacts, outliers and motion correction parameters. First level MRI data were quality checked to identify problematic volumes (e.g., distortions, movements, etc.) visually using the Medical Image Processing, Analysis, and Visualization tool (MIPAV, https://mipav.cit.nih. gov/), and automatically to identify artifacts of movements over 3 mm for translation and 0.02 radians for rotation via the Artifact Detection Tools (ART; http://www.nitrc.org/projects/ artifact\_detect). We used as a high-pass filter the double of the max length time between the same stimuli, 456 s for SVM and 152 s for ROI.

#### **Offline fMRI fixed effect group analysis.**

We first run t-contrasts to examine how brain activity was affected by emotion type (Tenderness vs. Anguish and Anguish vs. Tenderness), rtfMRI-NFB method (SVM and ROI) and assessment site. To gain power to detect these effects in a small group of participants, we analyzed the rtfMRI-NFB data using a fixed effects model (42, 43). First, to test our hypotheses on the engagement of specific ROIs during the conditions tenderness and anguish, SVC were applied using the ROIs from the ROI based rtfMRI-NFB method with a whole-brain voxel threshold of p < 0.005, uncorrected. Second, to test whether the hypothesized

regions were still implicated using a more conservative approach, we ran analyses of rtfMRI-NFB data using FWE correction at a whole brain level with p < 0.05.

## RESULTS

Results are summarized starting with sample demographic characteristics and questionnaire data (e.g., mood, anxiety, personalized strategies), followed by a description of the ratings of task variables (e.g., rtfMRI-NFB task, intensity of the emotions at the end of the rtfMRI-NFB runs, experience of the BCI, audio tracks) and brain activity patterns for the rtfMRI-NFB conditions of tenderness and anguish (i.e., small volume FWE corrected results), followed by whole brain FWE results for the whole group and separately by experimental site and rtfMRI-NFB method (ROI and SVM).

## Sample Characteristics

Sample demographic and questionnaire data are overviewed in **Table 1**. We recruited eight 23 to 28-year-old participants separately from two sites (Site 1, N = 4; Site 2, N = 4). The groups from the two sites were matched by age, sex and scores for depression, positive affect, satisfaction with life, emotion regulation and body vigilance during MRI.

## Personalized Strategies to Achieve the Target Emotions

Participants' emotion regulation strategies varied. Neutral emotional states were achieved by recalling non-salient personal memories and imagined trivial scenarios and by mentally repeating neutral mantras (e.g., the world is round/full of water/a planet, I am laying in the MRI scanner, I am laying down, the leaves move).

Tenderness states were achieved and maintained via strategies including thoughts of loved partners, friends, young relatives or pets, pleasant memories (e.g., of own childhood, playing with nieces/nephews, memorable moment with loved partner and friends), and via repeating mantras (e.g., the world is beautiful/love/safe/generous/has love everywhere; people love each other; people are nice; friends are special; love is all that matters/is everything, I am love, affection exists).

Anguish states were experienced via recalling memories and imagining negative scenarios (e.g., illness/death/arguments with close people, cruelty to pets, war, stuck in the MRI room/in a fire/in water drowning/in own mind) and via repeating unpleasant mantras (e.g., "the whole world is dying").

## Ratings of Emotions and of Neurofeedback Task Variables

Supplementary Table 1 and **Figure 4** overview the effect of NFB run (1-to-3) and site (Site 2 vs. Site 1) on emotion regulation and rtfMRI-NFB variables.

The intensity of the emotions during the rtfMRI-NFB task was rated as "moderately intense" for anguish and between "moderately intense" and "intense" for tenderness. The emotion intensity ratings were affected by site (Site 2 > Site 1) but participants experienced a similar intensity of emotions across the rtfMRI-NFB runs.

All participants found their strategies to be "moderately useful," across the rtfMRI-NFB runs (i.e., non significant effect of rtfMRI-NFB run) and this was affected by site (Site 2 > Site 1).

The virtual environment BCI was rated as "moderately" easy to use during tenderness, anguish and neutral conditions across participants from the three rtfMRI-NFB runs and the two sites, through the neutral condition was affected by site (Site 1 > Site 2).

Participants rated that it was "neither difficult nor easy" to detect the color change in the virtual environment BCI across all rtfMRI-NFB runs, but more markedly in one site (Site 1 > Site 2).

TABLE 1 | Summary of demographic and questionnaires data by Site 1 and site 2.


Site 1 = D'Or Institute for Research and Education, Rio de Janeiro; Site 2 = Monash Biomedical Imaging, Monash University, Melbourne; Mean (standard deviation values); ERQ, emotion regulation questionnaire (44); satisfaction with life scale (41); Body vigilance scale (40); BDI, Beck depression inventory (36); STAI, state and trait anxiety inventory (37); PANAS, positive affect and negative affect scale (45). \*These results did not survive Bonferroni correction for multiple comparisons. Bold fonts indicate p < 0.05.

Finally, there was an effect of site on the level of tiredness (Site 1 > Site 2) and focus (Site 2 > Site 1), and both were "moderate" across all rtfMRI-NFB runs.

## Rating of the Audio Tracks Used During the Conditions "Anguish" and "Tenderness"

Participants' ratings of the emotions induced by the audio tracks during rt-fMRI NFB (anguish and tenderness conditions) are overviewed in Supplementary Table 2. The music tracks used elicited significantly higher levels of tenderness and positive emotions (i.e., enchantment, transcendence, strength, serenity, joy) and trend-like higher level of nostalgia, potentially as participants' evoked past experiences. The music tracks used during anguish elicited significantly higher levels of anguish, sadness and tension.

## Level of Real Time Color Change of the Virtual Environment in the BCI During rtfMRI-NFB

**Figure 5** shows the change in the color of the virtual environment BCI during rtfMRI-NFB using two distinct NFB methods. The change in color of the BCI was significantly affected by rtfMRI-NFB runs (Tenderness: F = 7.53, df = 2, p = 0.001, Anguish: F = 6.78, df = 2, p = 0.001), assessment site (Site 2 > Site 1, Tenderness: F = 27.16, df = 1, p < 0.001, Anguish: F = 4.17, df = 1, p = 0.041) and rtfMRI-NFB method (ROI > SVM for Site 1: F = 34.132, df = 1, p < 0.001; SVM > ROI for Site 2: F = 21.03, df = 1, p < 0.001). Results separated by site reveal similar patterns and are shown in Supplementary Figure 5.

## Offline fMRI Analyses on Brain Activity During rtfMRI Neurofeedback

Mean values are illustrated in red dots, and median values in white dots.

Offline fMRI data analysis of rtfMRI-NFB runs confirm that participants successfully recruited the hypothesized areas and additional brain regions at a group level. Brain activity within the septo-hypothalamic ROI during tenderness rtfMRI-NFB trials was first examined using SVC FWE correction (p < 0.05, k = 5).

The tenderness rtfMRI-NFB condition significantly engaged the predicted septo-hypothalamic area (k = 48, T = 5.19, x = 3, y = 14, z = −7). The same results emerged when repeating the analyses with SVC FWE correction separately by site (Site 1, k = 31, T = 3.72, x = 0, y = 14, z = −10; and Site 2, k = 5, T = 3.99, x = −9, y = 8, z = −16) and separately by NFB method (SVM: k = 77, T = 4.62, x = 0, y = 11, z = −13, and ROI: k = 7, T = 4.13, x = 3, y = 14, z = −7).

We also examined brain activity during the rtfMRI-NFB task with a whole brain approach and FWE correction (p < 0.05, k = 5) (**Figure 6** and **Tables 2**, **3**). The results also show engagement of the septo-hypothalamic area and the frontal pole (including medial orbitofrontal regions), the temporal pole and the precuneus. Similar results emerged when examining the rtfMRI-NFB data separately by site (Supplementary Figure 2 and Supplementary Table 3) and by

NFB method (Supplementary Figure 3 and Supplementary Table 4). Notably, the same patterns emerged in individual participants' activation maps, shown in Supplementary Figure 4.

(N = 16, two scans per subject) via fixed-effect analysis and whole-brain FWE

correction with p < 0.05, T > 4.716).

Brain activity during anguish rtfMRI-NFB trials was first examined using a small volume FWE correction (p < 0.05, k = 5). The right amygdala area was robustly engaged in the whole sample (k = 214, T = 8.43, x = 24, y = −10, z = −13), and also when examining data separately by site (Site 1: k = 42, T = 5.24, x = 33, y = −7, z = −7, and Site 2: k = 236, T = 8.99, x = 30, y = −7, z = −22) and by NFB method (SVM: k = 176, T = 6.43, x = 33, y = −4, z = −7, and ROI: k = 174, T = 6.98, x = 24, y = −10, z = −13).

Anguish vs. Tenderness results (see **Figure 6** and **Table 3**) during rtfMRI-NFB using a whole brain approach with FWE correction (p < 0.05, k = 5) show the recruitment of the amygdala, frontal regions (i.e., polar, superior and middle areas), parietal regions (i.e., angular and supramarginal gyri, juxtapositional lobule), temporal (middle and inferior) and other cortical regions (lateral occipital, central operculum, cerebellum). Similar though weaker pattern of brain activity emerged when examining the results separately by site (Supplementary Table 2) and by rtfMRI-NFB method (Supplementary Table 3). Finally, the same patterns were apparent in individual


Table shows all local maxima separated by > 20 mm, surviving threshold of p < 0.05 (FWE-corrected), t > 4.7160, df = 18,548 minimum extent = 5. Regions were automatically labeled using the HarvardOxford-maxprob-thr0 atlas. x, y, and z =Montreal Neurological Institute (MNI) coordinates in the left-right, anterior-posterior, and inferior-superior dimensions, respectively. \*subcallosal region.

TABLE 3 | Overview of local maxima for brain activity during Anguish versus Tenderness neurofeedback conditions, across the whole brain.


Table shows all local maxima separated by >20 mm, surviving threshold of p < 0.05 (whole brain FWE-corrected), t > 4.7160, df = 18548, minimum extent = 5. x, y, and z = Montreal Neurological Institute coordinate in the left-right, anterior-posterior, and inferior-superior dimensions, respectively. Regions were automatically labeled using the HarvardOxford-maxprob-thr0 atlas and regions with an \* were the nearest location of activations using the same atlas.

participants' activation maps shown in Supplementary Figure 5.

## DISCUSSION

We provide for the first-time proof of concept and demonstrate feasibility of the implementation of rtfMRI-NFB using virtual environment BCI and music to elicit and measure the neural correlates of specific, complex emotional states. In line with our expectations, real-time up-regulation of tenderness engaged the septo-hypothalamic area and other regions previously implicated in positive affiliative emotions (i.e., medial frontal cortex and temporal pole, precuneus). Additionally, online up-regulation of anguish recruited a widespread network of regions ascribed to negative affect, including the amygdala, dorsolateral prefrontal and additional regions. These effects were corroborated by individual brain activation maps, and by group activation maps across the two experimental sites and the two NFB methods, as well as by self-reported emotions experienced during NFB. Our findings preliminarily validate the notion that individuals can experience powerful emotional states and recruit relevant brain networks in real time using a novel multisensory rtfMRI-NFB tool comprising a virtual environment BCI.

Up-regulation of tenderness states recruited three clusters of brain areas previously implicated in positive affiliative emotions. These include the septo-hypothalamic region, the frontal pole, the medial orbitofrontal cortex, the temporal pole and the precuneus. The validity of our findings on tenderness-related brain networks is corroborated by the involvement of these regions in previous fMRI work on affiliative emotions (11, 22, 25) and their specificity to the experience of tenderness is supported by participants' reports that their tenderness states increased/were sustained during the NFB tenderness condition.

We show that the septo-hypothalamic region was key for the experience of tenderness states. This is consistent with our previous rtfMRI-NFB study also targeting tenderness (11). Yet, this region may be ascribed to affiliative emotions generally including but not limited to tenderness [e.g., empathy, compassion, guilt and others (23)]. Indeed, previous fMRI experiments targeting positive affiliative emotions other than tenderness implicate the septo-hypothalamic region (11, 22, 25). Also, lesion evidence shows abnormal prosocial affect in patients affected by lesions of the septohypothalamic area (13) and by neurological disorders (i.e., frontotemporal dementia) compromising this area (46, 47).

rtfMRI-NFB during the tenderness condition recruited - in addition to the septo-hypothalamic area - the medial prefrontal (i.e., frontal medial, middle frontal gyrus), temporal and parietal regions (i.e., precuneous). This is consistent with neurobiological evidence and theories of affiliative emotions, suggesting that our rtfMRI-NFB study was successful. Yet, we failed to detect activity in the subgenual/ventral cingulate cortices (22, 48), which have been implicated in the neurobiology of additional affiliative emotions (e.g., compassion and guilt). This discrepancy may be explained by the different cognitive demands required in the current rtfMRI-NFB study and previous fMRI studies (11, 23– 27, 47), particularly as this was the only study to use personalized strategies to increase and maintain the intensity of the emotions and to use emotions to voluntary regulate brain activity in real time. Given the pilot nature of our study and the many elements included in the experiment (e.g., rtfMRI-NFB, virtual environment BCI, real time fMRI, mood induction, personalized strategies, audio tracks and others) further assessments are required to determine specific methodological factors in our study played a role in the partially discrepant findings with the literature to date.

rtfMRI-NFB during the anguish conditions, recruited a much more widespread network of regions comprising the amygdala and fronto-parietal, temporal and other cortical regions. The recruitment of the amygdala is consistent with our hypothesis and previous fMRI evidence on negative affect (49–52). Our results mirror those from previous fMRI studies on negative emotions that also implicate temporal (51), prefrontal (53–56), frontal polar (57, 58), and parietal regions (54). The overlapping brain networks between our study and previous work on negative affect suggest that our rtfMRI-NFB protocol successfully recruited the target brain network. Future work contrasting distinct complex negative emotions is required to clarify if this network is ascribed to anguish specifically rather to negative emotions that are intense, arousing and potentially threatening including but not limited to anguish—such as fear, emotional pain and anxiety (57, 59, 60).

The anguish condition engaged a widespread pattern of brain regions. Additional higher order cognitive control brain areas may have been recruited due to the complex cognitive demands associated with the task, including attention control, evaluation and voluntary regulation of negative emotions, cognitive efforts required for maintaining complex emotions (54, 57, 61–67). Indeed, participants reported to habituate quickly to anguish states, as the thoughts that originally elicited anguish, were no longer effective after a short period. Participants used additional cognitive strategies to maintain anguish states, including to think of new memories and thoughts and imagine other scenarios.

We did not directly compare SVM and ROI rtfMRI-NFB methods given the pilot nature of the study and the fundamentally distinct measures of brain activity. Yet, we explored whether the hypothesized networks were recruited more robustly using either method. Both ROI and SVM rtfMRI-NFB methods recruited similar networks and showed comparable accuracy rates. This is interesting as SVM has been recognized to be superior to ROI in handling low signal to noise in areas susceptible to artifacts, decoding complex brain states with high sensitivity and accounting for individual variability (68, 69).

This issue cannot be resolved in this pilot study as it relies on a small sample size. Yet, our goal was to deliver a proof of concept for a novel real-time fMRI neurofeedback approach and software tool that can be used in future studies aiming to test mechanistic or clinical hypotheses, and not to provide definitive evidence for the superiority of ROI over SVM approaches (or vice-versa) or to establish unequivocally the role of fMRI neurofeedback in helping volunteers achieve emotional states more efficiently. This pilot methodological study demonstrates the feasibility of this novel neurofeedback method and software tool and its usability across research centers and teams to provide real-time emotional neurofeedback using virtual scenarios, employing either ROI or SVM-based metrics.

#### Limitations

Our study presents some important limitations. First, while selfreported emotions and previous work corroborated the patterns of brain activity, the lack of an active control condition (e.g., sham feedback from a separate region, artificially created or from another dataset) prevents the understanding of whether confounding variables have driven our results (e.g., rtfMRI-NFB, task practice, arousal, general intentional/motivation factors, others). Nonetheless, we would like to emphasize that this is a proof of concept study not aimed at showing differences between real and sham conditions, but at providing key insights on the technological implementation of multimodal, fMRI-NFB using a virtual environment as BCI and its feasibility for conducting single-subject studies.

Second, we did not use a rtfMRI-NFB transfer run to examine if participants had learned or could transfer the skills outside the MRI environment. We prioritized to acquire brain data from rtfMRI-NFB to test our new platform (4).

Third, we did not measure emotion subjectively in a continuous fashion, but at the end of each neurofeedback run. Our pilot real-time fMRI neurofeedback study did not aim to test statistically significant effects in emotional learning/enhancement across runs. Yet, our study provides evidence for feasibility along with guidelines, a protocol, and a free software tool that enables other researchers to conduct (emotional) fMRI neurofeedback integrated with a VR/game platform.

Fourth, we used a set of matched audio tracks for the conditions of tenderness and anguish to minimize systematic differences due to using different music tones and rhythms. However, the valence of the different audio tracks may have engaged distinct neural networks possibly confounding our results (70). We used the same audio tracks for all participants and these may have not helped all equally to achieve the target emotions, due to inter-individual differences in taste in music, personalities and other psychological variables. Personalized audio tracks may have been more effective in eliciting powerful and individually salient emotional states. However, participants' ratings of the audio tracks show that these induced the target tenderness, anguish and other positive and negative emotions.

Additionally, participants used different strategies to experience different emotions or the same emotion over time, which were qualitatively described and not controlled for in the brain activity analyses. The use of discrepant strategies may have biased brain activity (i) during NFB tenderness and anguish blocks, which were derived relative to the previous neutral blocks (ii) measured post-acquisition when contrasting tenderness and anguish. On the other end, personalized strategies ensured that each individual found the best way to feel valid emotional states. Our findings from participants' rating of their emotion intensity and the consistent patterns of brain activity in individual brain activation maps suggest that the target neurobehavioral states were achieved despite—or because of—personalized emotion regulation strategies.

Patterns of brain activity may differ from subject to subject or from session to session. This differential responsiveness means that the fixed-effect statistical analyses may not be appropriate when trying to generalize inferences (42). In our case, this analysis fits well since we are working with a restricted group that has been trained to perform the emotional task, and making inferences to an additional group of subjects was not our goal (43). Instead, providing robust results at the individual subject level is an important step toward clinical applications.

#### Future Directions and Conclusions

This novel rtfMRI-NFB platform is a promising tool for future experiments and interventions, particularly as the virtual environment BCI and musical excerpts can be individually customized to maximize participant's engagement. This platform can be changed or replaced by other multisensory approaches (tactile, auditory, sensory, etc.) according to specific experimental/clinical intervention needs, and is compatible with other platforms routinely used in experimental psychology and neuroscience research (e.g., MATLAB, EPrime, Presentation, Python, R and others). Participants successfully and voluntarily shifted from a brain pattern of intense negative emotions to a pattern of positive affiliative emotions. Our findings may contribute to the understanding of the neurobiological mechanisms of psychological interventions that boost positive affiliative emotions—such as compassion focused therapies (71) and loving-kindness meditation—and neuroplasticity (72, 73). Our study may inform the development of non-invasive, brainbased therapies that boost positive affiliative emotions—possibly even via hyper scanning—that have beneficial effects for a range of psychopathologies—e.g., depression, borderline personality disorder, psychopathy, and others.

In sum, we validated a novel rtfMRI-NFB protocol and instrument using a multimodal stimulation for future experimental and clinical intervention. We warrant replication studies using active control conditions [e.g., sham rtfMRI-NFB, biofeedback, psychotherapy, pharmacotherapy, physiotherapy, or other physical interventions (4)].

Future developments for rtfMRI-NFB platforms incorporating virtual environments as BCI may include providing feedback on different properties of brain functions including but not limited to connectivity and multiple ROIs concurrently (both possible with the Friend Engine platform), and tailoring rtfMRI-NFB tasks with multi-sensory BCIs to the needs of the individual and target population in large samples (e.g., videogame like interface for children, feared stimuli in participants with phobias, mannequins that can move with brain activity in patients with stroke with impaired motor function), to identify the characteristics of those who respond best and least and inform evidence based interventions. Our results warrant further rtfMRI-NFB studies using personalized interfaces in large cohorts to examine the therapeutic potential of rtfMRI-NFB in clinical samples, and its ability to enhance cognitive and emotional wellbeing in normative populations.

#### DATA AVAILABILITY

The datasets used in the current study are available from the corresponding author on reasonable request.

## AUTHOR CONTRIBUTIONS

VL and BM led the study execution, protocol setup, data analysis, and the writing of all the aspects of the manuscripts from start to completion. RB led the stimulus and neurofeedback setup in the D'Or laboratory in Rio and provided assistance in setting up the neurofeedback platform at Monash University in Melbourne. CS contributed to data collection and led the neurofeedback software setup in the Melbourne site. MY and CT-C advised on the running of the project and revised the manuscript. JM advised on the running of all the aspects of the study, overviewed the experimental protocol setup and the running of the fMRI analyses; and contributed to the writing of the first and other drafts of the manuscript.

#### FUNDING

The experiment conducted at IDOR was funded by different grants from the National Council for Scientific and Technological Development (CNPq, Ref:311623/2014-0), Research Support Foundation of the State of Rio de Janeiro (FAPERJ, Ref: E-26/202.962/2015), and intramural grants from the D'Or Institute for Research and Education (IDOR, PNeuro). The experiment conducted at Monash University was supported by the Monash Biomedical Imaging – Psychology Grant 2015.

#### REFERENCES


#### ACKNOWLEDGMENTS

Ms. Patricia Bado, Ms. Julie Wein, and Mr. Sebastian Hoefle at D'Or Institute for Research and Education, IDOR, Rio de Janeiro, Brazil, have contributed to study design, participants' recruitment, and data collection. Dr. Juan Domininguez (Australian Catholic University), Dr Pascal Molenberghs (The University of Melbourne), Dr. Bryan Paton (The University of Newcastle), Dr. Parnesh Raniga (University of Sydney) have contributed to the setup of the neurofeedback platform at the Monash Biomedical Imaging facility, Monash University, Melbourne, Australia.

#### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fneur. 2018.00390/full#supplementary-material


regulation of negative emotion. J Cogn Neurosci. (2002) 14:913–21. doi: 10.1162/089892902760191135


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Lorenzetti, Melo, Basílio, Suo, Yücel, Tierra-Criollo and Moll. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Modulation of Cognitive and Emotional Control in Age-Related Mild-to-Moderate Hearing Loss

Artyom Zinchenko1,2,3, Philipp Kanske4,5, Christian Obermeier <sup>2</sup> , Erich Schröger <sup>6</sup> , Arno Villringer <sup>2</sup> and Sonja A. Kotz 2,7 \*

1 International Max Planck Research School on Neuroscience of Communication (IMPRS NeuroCom), Leipzig, Germany, <sup>2</sup> Department of Neuropsychology, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany, <sup>3</sup> Department Psychologie, Ludwig-Maximilians-Universität München, Munich, Germany, <sup>4</sup> Chair of Clinical Psychology and Behavioral Neuroscience, Faculty of Psychology, Technische Universität Dresden, Dresden, Germany, <sup>5</sup> Department of Social Neuroscience, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany, <sup>6</sup> Institute of Psychology, University of Leipzig, Leipzig, Germany, <sup>7</sup> Department of Neuropsychology and Psychopharmacology, Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, Netherlands

#### Edited by:

Argye Hillis, Johns Hopkins Medicine, United States

#### Reviewed by:

Konstantinos Kalafatakis, University of Bristol, United Kingdom Claude Alain, Rotman Research Institute (RRI), Canada \*Correspondence:

Sonja A. Kotz kotz@cbs.mpg.de; sonja.kotz@maastrichtuniversity.nl

#### Specialty section:

This article was submitted to Applied Neuroimaging, a section of the journal Frontiers in Neurology

Received: 01 March 2018 Accepted: 30 August 2018 Published: 19 September 2018

#### Citation:

Zinchenko A, Kanske P, Obermeier C, Schröger E, Villringer A and Kotz SA (2018) Modulation of Cognitive and Emotional Control in Age-Related Mild-to-Moderate Hearing Loss. Front. Neurol. 9:783. doi: 10.3389/fneur.2018.00783 Progressive hearing loss is a common phenomenon in healthy aging and may affect the perception of emotions expressed in speech. Elderly with mild to moderate hearing loss often rate emotional expressions as less emotional and display reduced activity in emotion-sensitive brain areas (e.g., amygdala). However, it is not clear how hearing loss affects cognitive and emotional control mechanisms engaged in multimodal speech processing. In previous work we showed that negative, task-relevant and -irrelevant emotion modulates the two types of control in younger and older adults without hearing loss. To further explore how reduced hearing capacity affects emotional and cognitive control, we tested whether moderate hearing loss (>30 dB) at frequencies relevant for speech impacts cognitive and emotional control. We tested two groups of older adults with hearing loss (HL; N = 21; mean age = 70.5) and without hearing loss (NH; N = 21; mean age = 68.4). In two EEG experiments participants observed multimodal video clips and either categorized pronounced vowels (cognitive conflict) or their emotions (emotional conflict). Importantly, the facial expressions were either matched or mismatched with the corresponding vocalizations. In both conflict tasks, we found that negative stimuli modulated behavioral conflict processing in the NH but not the HL group, while the HL group performed at chance level in the emotional conflict task. Further, we found that the amplitude difference between congruent and incongruent stimuli was larger in negative relative to neutral N100 responses across tasks and groups. Lastly, in the emotional conflict task, neutral stimuli elicited a smaller N200 response than emotional stimuli primarily in the HL group. Consequently, age-related hearing loss not only affects the processing of emotional acoustic cues but also alters the behavioral benefits of emotional stimuli on cognitive and emotional control, despite preserved early neural responses. The resulting difficulties in the multimodal integration of incongruent emotional stimuli may lead to problems in processing complex social information (irony, sarcasm) and impact emotion processing in the limbic network. This could be related to social isolation and depression observed in the elderly with age-related hearing loss.

Keywords: ERPs, aging, hearing loss, cognitive conflict, emotional conflict, affective modulation, executive control

## INTRODUCTION

Healthy aging is often accompanied by a progressive decline in hearing capacity or even hearing loss [HL; (1)]. The prevalence of HL is close to 40% in those of 65 years or older (2) and mild-to-moderately severe sensorineural HL affects up to 33% of the world's adult population (3, 4). Hearing loss modulates the processing of acoustic information in the auditory cortex as well as along the ascending auditory pathways. For instance, Alain et al. (5) used magnetoencephalography to measure auditory evoked fields (AEFs) in a task where participants listened to complex sounds that were either in tune (congruent condition) or had a mistuned component (incongruent condition). The authors found that the incongruent condition elicited an enlarged object-related negativity (ORN) in participants with hearing impairments. The ORN is an event-related potential (ERP) component that reflects the perception of a mistuned low tonal element of a complex tone [e.g., (6–8)]. The authors suggested that HL increases neural excitability in auditory cortex which could be related to deficits in inhibitory control. Finally, in addition to inhibitory control, HL can also considerably influence general and emotional well-being in the elderly (9, 10).

Recent neuroimaging work showed that HL is also associated with a specific neuronal reorganization, most notably in networks responding to emotional stimuli (3). The authors reported that HL reduces the engagement of the limbic regions during processing of affective stimuli (e.g., including the left amygdala, left parahippocampus etc.), likely due to affected processing of acoustic features or valence. Furthermore, it was shown that negative sounds improve the functioning of "backward connections from the amygdala to the auditory cortex," while the "forward connections from the auditory cortex to the amygdala" are modulated by the acoustic features of a sound (11). Therefore, it is likely that continuous loss of hearing acuity may affect the reported connectivity patterns during processing of emotional sounds and result in hindered perception or misclassification thereof (3).

The correct identification of non-verbal acoustic and facial affective cues is a vital component of adequate interpersonal communication (12). However, this process becomes particularly challenging when the emotional valence of different communication channels (auditory, visual) is incompatible, resulting in emotional conflict (13, 14). Processing of such conflict is costly as shown in slower responses, increased error rates and conflict- and emotion-specific ERP components (15–18).

For instance, Zinchenko et al. (19) ran EEG experiments where they presented participants (groups of older and younger adults) with multisensory dynamic stimuli: short video clips of actors facially expressing and vocalizing negative or neutral emotions. The incongruence was created between non-emotional vowel category (cognitive task of Experiment 1) and emotional valence of visual and audio dimensions (emotional task of Experiment 2). More specifically, in Experiment 1 participants were asked to identify the vowel (i.e., "A" or "O") and ignore the emotional valence of stimuli, while in Experiment 2 the task was to report emotion of the voice (negative or neutral) regardless of matching or mismatching emotional and neutral facial expressions. Therefore, the authors varied the emotion of the target dimension (neutral, negative) and the nature of conflict was either emotional or cognitive.

As a result, negative emotions improved emotional conflict processing in younger (18) and older adults (19). In more detail, the conflict effect (i.e., RT difference between incongruent and congruent conditions) was smaller in the negative emotion condition relative to the neutral condition. Similarly, negative emotion was also shown to modulate conflicts that arise between opposing non-emotional stimulus dimensions [i.e., cognitive conflicts; (19); see also (20, 21), for similar findings]. Besides behavioral modulation of cognitive and emotional conflicts, negative emotions also resulted in conflict-specific ERP responses (18, 19).

Specifically, younger adults showed a conflict specific dissociation of the N100 during processing of cognitive and emotional conflicts (18). The N100 is a negative-going wave that peaks 80–120 ms after sound onset and was most often found over the fronto-central region of the scalp [see (22) for auditory N100]. This component is modulated by attention (23), emotion (24), and congruence (25). In the cognitive conflict task, the conflict effect was observed to be larger for negative relative to neutral trials, while in the emotional conflict task the conflict effect was more pronounced in neutral as compared to negative trials (18). Another component sensitive to conflict processing is the P200 (positive wave that peaks around 200 ms post-stimulus). The P200 increases for emotional compared to neutral stimuli (26, 27), and its amplitude decreases for incongruent stimuli in both cognitive and emotional conflict tasks (18, 28). Lastly, a well-described neural marker of conflict processing is the N200 (i.e., negative-going deflection that peaks 200–350 ms poststimulus) that elicits larger amplitude in response to incongruent than congruent stimuli (29–31). The N200 conflict effect was observed at fronto-central (20, 30, 32) and posterior electrodesites (18, 33).

Although the role of HL in various cognitive functions has been studies extensively in the last years (34–36), relatively little is known how decreasing hearing capacity affects the role of emotions in cognitive and emotional conflict processing. The detection of conflict in processing of emotional information is vital in successful interpersonal communication and social adaptation. Therefore, it is possible that social isolation observed in HL older adults (37) may at least be partially related to problems in the processing of complex social information that may contain emotional conflict. In order to test this hypothesis, we used multisensory dynamic stimuli and manipulated them in a way to make emotion either taskirrelevant (the conflict was created between non-emotional stimulus dimensions; cognitive conflict) or task-relevant, where emotional stimulus dimensions were made incongruent [emotional conflict; see (18)]. Specifically, we explored whether the cognitive and emotional conflicts influence early perceptual processes [e.g., N100, P200; (18)] and whether negative emotion is able to modulate the two types of conflict [N200; (20, 38)] in participants with varying degrees of hearing capacity.

Based on previous findings that HL results in a reduction of gray matter volume in frontal cortex and particularly controlspecific anterior cingulate cortex (ACC), we expected that the HL group would result in particularly hindered performance in both cognitive conflict task (39, 40) and in the emotional conflict task (19). Additionally, it was expected that negative targets would improve performance in both cognitive and emotional tasks in the NH group (19). On the other hand, as several previous studies indicated reduced capabilities to process emotional information related to moderate HL [e.g., (3)], we hypothesized that emotional targets would have no influence on both types of conflict in HL older adults. Due to its precise temporal resolution and millisecond precision, measuring EEG further allowed testing at what stage does HL influence processing of affective information. Finally, multisensory emotional and cognitive information optimizes behavioral responses in older adults (41–45). Therefore, we used dynamic multisensory emotional and neutral video stimuli in order to elicit the most optimal behavioral and neural responses [e.g., (46, 47)].

In line with our recent findings, we expected that incongruent cognitive and emotional tasks would result in a bigger N100 amplitude increase for negative than for neutral targets in older adults with NH (19). Additionally, we predicted that incongruent relative to congruent trials will result in a smaller P200 response in the two types of conflict (18, 28). These conflict-specific N100 and P200 responses were previously found at either anterior (28, 48, 49) or posterior electrode-sites (48, 50) in younger adults and at anterior electrode-sites only in older adults (19).

We also expected that incongruent stimuli would elicit a larger N200 responses in both groups (18, 51). We hypothesized that emotional targets would not modulate the N200 in the two conflict types, since the modulation of this component seems to be observed for static unimodal pictures (20) but not for dynamic multisensory videos (18, 19).

## METHODS

#### Participants

Twenty-one NH older adults (see **Table 1** for demographic information) and 21 older adults with HL with normal or corrected-to-normal vision participated in Experiment 1 and Experiment 2. The order of the two Experiments was counterbalanced and we kept at least 7 days in between the two testing days. All participants were right-handed (Edinburgh Handedness Inventory score ME = 89.6, SD = 11.7). Sample size was determined on the basis of previous studies that used identical paradigm and stimuli [e.g., (18, 19)]. On the basis of effect size measures provided in these studies, we determined that our sample size would be appropriate to detect an f(U) effect size of 0.33 with 85% power (partial eta2 = 0.1, groups = 2, number of measurements = 4), given an alpha level of 0.05 and a nonsphericity correction of 1. Participants had no history of alcoholism, drug abuse, neurological or psychiatric disorders as assessed via Structured Clinical Interview in DSM-IV [SCID-I; (52)] at the Day Clinic for Cognitive Neurology, University of Leipzig. Additionally, we used an instant dipstick drug test (Drogentest Multi-8/2-DT, Diagnostik Nord) to examine TABLE 1 | Subject demographics and clinical characteristics.


The table contains hearing thresholds (in dB) for frequencies between 250 and 8,000 Hz for the right and left ears. \*p< 0.05; \*\*p < 0.01; \*\*\*p < 0.001.

a possible use of eight drugs (amphetamine, buprenorphine, benzodiazepines, cocaine, methamphetamine, morphine/opiates, methadone, and cannabis) in both hearing groups. Older adults in the two groups did not differ in mean age [t(40) = −1.76, p > 0.05] or mean years of education: HL group (all 11.6 years, SD = 2), NH group [mean = 11.2 years, SD = 1.46, t(40) = −1.18, p > 0.2]. The two hearing groups came from the "Leipzig Cohort for Mind-Body-Emotion Interactions" (LEMON) database.

Participants were screened with a pure-tone audiometric testing. As a result, older adults in the NH group showed thresholds equal to or lower than 30 dB in both ears at [all] frequencies crucial for speech perception [500–4,000 Hz, (53)]. Participants in the HL group had thresholds between 30 and 70 dB in [at least one of] the corresponding frequencies, which corresponded to mild to moderately-severe HL. **Table 1** reports average hearing information at frequencies of interest for both groups. The HL participants did not rely on hearing aids.

Additionally and in line with previous literature (54), participants completed the Adult Temperament Questionnaire [effortful control subscale, ATQ; (55)] and Depression Anxiety Stress Scale [DASS; (56)]. Both groups had comparable results for effortful control, stress, anxiety, and depression (see **Table 1** for details).

Participants were asked to rate expressiveness, arousal, and emotion identification of the complete videos, video streams alone, and audio streams alone [see **Table 2** and stimulus material below for details; (57)]. The groups did not differ with regard to perceived expressiveness and arousal of the stimuli. Furthermore, the NH group rated the emotional material as more emotional compared to the neutral material. On the other hand, the HL group rated emotional voices as neutral and emotional faces as even more negative relative to neutral stimuli and relative to the NH group (see **Supplementary Material** for details). A written informed consent form was obtained from all participants and they were paid ∼30 e for participation. The experiment was conducted in accordance with the principles of the Declaration of Helsinki and was also approved by the Ethics Committee of the University of Leipzig.

#### Stimulus Material

We validated experimental design, procedure and stimuli of the current study in our previous work (18, 19). Short video clips depicted either a male or a female actor articulating the vowel "A" and "O" in a neutral and negative (i.e., angry) tone of voice (see **Figure 1A**). The sounds in all videos were normalized to 70 dB by means of root mean square using Final Cut Pro 7 (Apple Inc.). In Experiment 1, we used these original videos to create 8 congruent and 8 incongruent stimuli by matching or mismatching vocalizations of the face and voice (e.g., voice pronouncing "A" with facial lip movement corresponding to "A" vs. "O," respectively). Participants were asked to report the vowel pronounced by the voice ("A," "O"). The onset of the original video sound was used for the overlay with the mismatching sound. In Experiment 2, we modified videos used in Experiment 1 and created 12 congruent and 12 incongruent emotional conflict videos. For this purpose, we mismatched the emotional valence of the face and voice (e.g., face [lip movements] pronouncing a neutral "A" and the corresponding audio "A" that is pronounced emotionally, **Figure 1**). Again, the onset of the original video sound was used for the precise overlay of the incongruent voice with the facial expression and lip movement in both incongruent conditions (negative [neutral] face—neutral [emotional] voice). Note that in Experiment 2 we always matched the vocalization of the face and voice. The task in this experiment was to report the emotional valence of the voice (negative, neutral). Thus, Experiments 1 and 2 were very similar, but differed in the task instruction and the combination of audio and visual stimuli. Additionally, the video duration in all conditions varied from 1 to 2 s (see **Table 4**). All conditions in Experiments 1 and 2 were comparable in time before the audio onset and total video durations (see **Table 3** and **Supplementary Material** for details).

We observed no differences between conditions with regard to emotion identification, expressiveness and arousal (see **Supplementary Material** for details). We also tested whether videos differed with regards to movement. For this purpose, we quantified per-pixel changes in light intensity (luminance) between video frames (58). Subsequently, we used a Kruskal– Wallis test to compare the two emotion and two vowel conditions. As a result, there were no differences except for negative relative to neutral stimuli showing a higher number of movements (X² = 5.33, p < 0.05). Since angry expressions are naturally more dynamic and intense [e.g., (59)] the observed difference is expected in naturalistic stimuli. Nevertheless, these motion differences should have no effect on final results, since we focused on the interaction of congruence and emotion. Finally, we found no motion differences between different vowels (X² = 1.25, p > 0.2).

Both Experiment 1 and Experiment 2 consisted of four blocks with 52 videos in each block (negative = 26 videos, incongruence = 50%) that were pseudo-randomized and administered in a 2 (emotional, neutral) by 2 (congruent, incongruent) factorial design.

#### Procedure

Both Experiments were performed in a sound-attenuated booth. Participants were seated about 1 m from a computer screen and audio stimuli were delivered via headphones. After 200 ms fixation cross participants watched videos stimuli in full duration (i.e., response did not terminate video presentation; see **Figure 1B**). In Experiment 1, the task was to identify vocalization of voices (either "A" or "O"), while emotional valence of the face and voice were (i) task-irrelevant and (ii) always matched. In Experiment 2, the task was to report the emotional valence of the voice (negative, neutral). We also introduced probe trials (10% of all trials presented randomly throughout experiment) when participants were additionally asked to report the vocalization of the face (i.e., lip movement, i.e., "A" or "O" in Experiment 1; emotion of the face in Experiment 2). This was done to ensure that faces were not ignored. These questions were not limited in time, and were not included into further analyses (all participants answered >90% questions correctly in both Experiments). Main questions had a response time-window of 1000 ms and started from voice onset. Participants saw a "try to respond faster" sign for 200 ms in case if they did not respond within the given timewindow. In case of an incorrect response the word "incorrect" appeared on the screen. We counterbalanced button presses across participants and introduced a random intertrial duration between 1,000 and 2,000 ms. Lastly, in order to make sure that participants understood the task requirement we asked them to write the instructions down on a sheet of paper. All participants were able to correctly describe the task.

## EEG Recording and Pre-processing

We used Brain Vision Recorder (Brain Products GmbH, Munich, Germany) to record data from 59 Ag/AgCl electrodes (10-10 system) at a sampling rate of 500 Hz. The reference was at left mastoid, and ground was at the sternum. We measured vertical and horizontal electro-oculogram to reject artifacts and kept impedance level below 5 k.

For the EEG data analyses we used the FieldTrip (v0.20120501) toolbox (60) running on Matlab 8.1 R2013a (The Mathworks, Natrick, USA). After re-referencing electrodes offline to linked mastoids we split the data into longer epochs (±2,000 ms time-locked to the voice onset) and rejected those epochs that contained excessive muscle activity or jump artifacts. We then band-pass filtered the data using a two-pass Butterworth IIR filter with a frequency pass-band of 0.1–100 Hz (order of

#### TABLE 2 | Results of the video rating.


four).We also applied principal components analysis after preprocessing, thus reducing dimensionality of the data and preserving α = 0.99 of the variance (61). A fastica algorithm was used for the independent component analysis (ICA). In the following step we have rejected components that showed ocular, muscle, heart, and electrode artifacts (number of components removed in Experiment 1: mean = 12, SD = 3, ∼16% of trials; in Experiment 2: mean = 14, SD = 4.1, ∼15% of trials). Finally, we have visually inspected individual epochs and discarded those epochs that contained artifacts.

#### Data Analysis

Smaller epochs time-locked to the voice onset (−200 to 1,000 ms) were selected for the statistical analysis. First, we band-pass filtered continuous EEG data using a two-pass Butterworth IIR filter with a frequency pass-band of 0.5–30 Hz, and then calculated averaged activity for each participant and for each session and condition after applying a 200 ms baseline correction before the voice onset (18, 19). Furthermore, in line with previous literature (18, 20, 48), four regions of interest (ROIs) were defined: left anterior (FP1, AF3, AF7, F3, F5, F7, FC3, FC5, FT7), right anterior (FP2, AF4, AF8, F4, F6, F8, FC4, FC6, FT8), left posterior (CP3, CP5, TP7, P3, P5, P7, PO3, PO7, O1), and right posterior (CP4, CP6, TP8, P4, P6, P8, PO4, PO8, O2). The following time-windows were used to identify peak latencies separately for each participant and each condition: 70– 110 ms (N100), 140–225 ms (P200), and 240–380 ms (N200) as suggested by Luck and Kappenman (62). For a mean amplitude analysis we used averaged activity that fell within 40 ms (i.e., 20 ms before and after) of individual peaks from the group mean ERPs. Subsequently a repeated-measures ANOVA was calculated for each time-window, using emotion (emotional, neutral), congruence (congruent, incongruent), region (anterior, posterior), and side (left, right) as within-subject factors and group (NH, HL) as a between-subject factor. In the results section, we report statistically significant effects that involved the critical factors emotion, congruence, and group.

## RESULTS

## Experiment 1

## Behavioral Data

**RT data**

We report an interaction of emotion, congruence and group [F(1, 40) = 6.89, p < 0.02, η<sup>p</sup> <sup>2</sup> = 0.147; see **Figure 2**]. Posthoc analyses by group revealed an interaction of emotion and congruence in the NH group [F(1, 20) = 10.36, p = 0.004, ηp <sup>2</sup> = 0.341] but not in the HL group [F(1, 20) = 0.323, p > 0.5, ηp <sup>2</sup> = 0.016]. In the NH group, the conflict effect was smaller for negative emotion targets [97 ms; F(1, 20) = 32.18, p = 0.01, ηp <sup>2</sup> = 0.617] than neutral targets p124 ms; F(1, 40) = 64.119, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.762].

#### **Errors**

Incongruent stimuli elicited more errors than congruent stimuli [F(1, 40) = 6.54, p < 0.02, η<sup>p</sup> <sup>2</sup> = 0.150]. No other main effects or interactions reached significance (all p's > 0.05).

#### ERP Data

#### **N100 range**

We found main effects of emotion [F(1, 40) = 5.38, p < 0.03, η<sup>p</sup> <sup>2</sup> = 0.119; see **Figure 3**] and congruence [F(1, 40) = 13.04, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.246], as well as an interaction of emotion and congruence [F(1,40) = 9.54, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.193]. Post-hoc analyses revealed larger mean congruence effect for negative [F(1, 40) = 19.36, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.326] but not for neutral stimuli [F(1, 40) = 0.116, p > 0.7, η<sup>p</sup> <sup>2</sup> = 0.003]. The interaction between emotion, congruency, and group was not significant (see **Figure 4**).

#### **P200 range**

We report an interaction of region and congruence [F(1, 40) = 7.00, p < 0.02, η<sup>p</sup> <sup>2</sup> = 0.149]. Incongruent stimuli elicited an increased amplitude over the anterior electrode-sites

FIGURE 1 | (A) Multimodal video stimuli of Experiments 1 and 2: Example of the female actor (publication of these images was approved by the actor) vocalized the interjections "A" and "O" in either a negative or neutral tone of voice. Incongruence was created by a mismatch of the vocalizations and the video components in Experiment 1 and mismatches in emotion of audio and video components in Experiment 2. (B) Example of a trial sequence. The trial started with a 200 ms fixation point that was followed by a video clip played in full length. The length of videos was varied (see Tables 3, 4 for details). The response time window was activated with the voice onset and lasted until response or a maximum of 1,000 ms. We also introduced a random inter-trial interval between 1,000 and 2,000 ms. Please note that speech bubbles "neutral" and "negative" refer to the [audio] target dimension (not visual dimension).

[F(1, 40) = 5.56, p < 0.03, η<sup>p</sup> <sup>2</sup> = 0.122], but not over posterior sites [F(1, 40) = 0.127, p > 0.7, η<sup>p</sup> <sup>2</sup> = 0.03].

#### **N200 range**

We found no significant main effects or interactions in the N200 time range.

To summarize, in Experiment 1 we tested whether taskirrelevant emotion influences cognitive conflict processing in two elderly groups of participants with different hearing capacities. As a result, emotion facilitated behavioral conflict processing by reducing the conflict effect in the NH but not in the HL group. However, negative emotion modulates cognitive conflict in the N100 of both hearing groups, putatively indicating that emotion modulates early conflict-specific processing in spite of HL. Interestingly, both groups showed a control-specific P200 conflict effect only at anterior electrode-sites. Finally, we did not find a N200 conflict effect in either one of the groups. In Experiment 2 we further tested whether HL modulates the role emotional valence of the target in the emotional conflict task.

#### Experiment 2 Behavioral Data<sup>1</sup>

<sup>1</sup>Please find the results of an Omnibus ANOVA that compares data across both Experiments 1 and 2, adding the within-group factor of conflict type (cognitive, emotional) at the end of the results section.

#### TABLE 3 | Timing of video stimuli of Experiment 1.


#### TABLE 4 | Timing of video stimuli of Experiment 2.


#### **RT data**

We observed an interaction of emotion, congruence, and group [F(1, 40) = 10.78, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.212; see **Figure 2**]. Post-hoc by group revealed an interaction of emotion and congruence in the NH group [F(1, 20) = 15.03, p < 0.01, ηp <sup>2</sup> = 0.429] but not in the HL group [F(1, 20) = 1.13, p > 0.3, ηp <sup>2</sup> = 0.054]. In the NH group, the conflict effect was smaller for negative [93 ms; F(1, 20) = 17.53, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.467] than neutral trials [175 ms; F(1, 20) = 128.22, p < 0.001, ηp <sup>2</sup> = 0.865].

#### **Errors**

We found main effect of congruence [F(1, 40) = 116.03, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.730] and an interaction of congruence and group [F(1, 40) = 67.15, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.610]. Post-hoc analyses revealed that the conflict effect was larger in the HL group [46.6%,

F(1,20) = 86.08, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.811] than the NH group [6.7%, F(1, 20) = 21.42, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.482].

#### **N100**

We report an interaction of emotion and congruence [F(1,40) = 13.68, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.255; see **Figure 5]**. Incongruent stimuli elicited larger N100 amplitudes than congruent stimuli in the negative emotion condition [F(1,41) = 17.69, p < 0.001, ηp <sup>2</sup> = 0.301] but not in the neutral condition [F(1,41) = 1.21, p > 0.25, η<sup>p</sup> <sup>2</sup> = 0.029]. We observed no main effect or interactions with the factor group (all p's > 0.05; see **Figure 6**).

#### **P200**

The main effect of emotion was significant [F(1, 40) = 11.25, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.220]. The interaction of emotion and region was also significant [F(1, 40) = 9.2, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.187]. We observed that negative stimuli elicited smaller P200 responses than neutral stimuli and this effect was larger over the anterior brain region [F(1, 41) = 13.22, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.244] relative to posterior sites [F(1, 41) = 4.39, p = 0.042, η<sup>p</sup> <sup>2</sup> = 0.097].

#### **N200**

We observed an interaction of emotion and group [F(1, 40) = 4.23, p < 0.05, η<sup>p</sup> <sup>2</sup> = 0.096]. Negative stimuli elicited marginally larger N200 responses in the HL group [F(1, 20) = 4.28, p = 0.052, ηp <sup>2</sup> = 0.176], but not in the NL group [F(1, 20) = 0.340, p > 0.5, η<sup>p</sup> <sup>2</sup> = 0.017]. We also found an interaction of region and congruence [F(1, 40) = 14.28, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.263]. Posthoc analyses revealed that incongruent relative to congruent stimuli led to smaller N200 responses at anterior electrode-sites [F(1, 41) = 4.2, p < 0.5, η<sup>p</sup> <sup>2</sup> = 0.097], but not at posterior ones [F(1, 41) = 2.064, p > 0.15, η<sup>p</sup> <sup>2</sup> = 0.048].

In summary, Experiment 2 tested how HL modulates the role of emotion of the target in emotional conflict processing. As expected, we found that negative stimuli improved processing of emotional conflict by reducing the RT conflict effect in the NH group but not in the HL group. The N100 response showed an age-independent interaction of emotion and control: incongruent negative as compared to neutral stimuli resulted in larger N100 responses than congruent stimuli. We also found that the effect of emotion differed across the regions in the P200 of both groups, and it also varied between the two hearing groups in the N200.

#### **Omnibus ANOVA**

In the omnibus ANOVA we directly compared the results of Experiment 1 and Experiment 2. For each time window, a repeated-measures ANOVA was calculated using conflict type (cognitive, emotional), emotion (emotional, neutral), congruence (congruent, incongruent), region (anterior, posterior), and side

(left, right) as within-subject factors and hearing group (normal hearing, hearing loss) as a between-group factor.

#### Results Behavioral Data

#### **RT data**

We found a 4-way interaction of experiment x congruence x emotion x group [F(1, 40) = 4.84, p = 0.034, η<sup>p</sup> <sup>2</sup> = 0.108]. **Figure 2** shows that the benefit of negative emotion was more pronounced in the emotional than the cognitive conflict task. Nevertheless, the pattern of results was identical for both experiments: the conflict effect was reduced for emotional compared to neutral conflicts in the NH group but not in the HL group.

#### **Error**

We found an interaction of conflict type, congruence, and group [F(1, 40) = 46.09, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.535]. Incongruent relative to congruent trials resulted in increased errors in Experiment 1 [F(1, 40) = 6.54, p < 0.02, η<sup>p</sup> <sup>2</sup> = 0.150]. In Experiment 2, the conflict effect was larger in the HL group [F(1, 20) = 86.08, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.811] than the NH group [F(1, 20) = 21.42, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.482].

#### **N100**

We observed an interaction of emotion x congruence [F(1, 40) = 10.51, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.208] as well as an interaction of region x congruence x emotion [F(1, 40) = 6.41, p < 0.02, ηp <sup>2</sup> = 0.138]. Follow-up analysis over the two brain regions revealed an interaction of emotion and congruence in the anterior brain region [F(1, 41) = 12.01, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.226], but not in the posterior brain region [F(1, 41) = 2.84, p > 0.05, ηp <sup>2</sup> = 0.065].

#### **P200**

We found a significant main effect of emotion [F(1, 40) = 10.61, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.210] as well as an interaction of region and emotion [F(1, 40) = 15.98, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.286]. Follow-up analyses showed that neutral stimuli generated larger amplitude

than negative stimuli in the anterior brain region [F(1, 41) = 14.78, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.265], but not in the posterior brain region [F(1, 41) = 2.19, p > 0.1, η<sup>p</sup> <sup>2</sup> = 0.051]. We also observed an interaction of region and congruence. Incongruent stimuli produced enhanced amplitude in the anterior brain region, [F(1, 41) = 9.14, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.182], but not in the posterior region [F(1, 41) = 1.49, p > 0.2, η<sup>p</sup> <sup>2</sup> = 0.035].

Finally, we also found an interaction of congruence and hearing group [F(1, 40) = 4.16, p = 0.048, η<sup>p</sup> <sup>2</sup> = 0.094]. Incongruent relative to congruent stimuli elicited enhanced amplitude in the HL group [F(1, 40) = 10.51, p = 0.002, ηp <sup>2</sup> = 0.208], but not in the NH group [F(1, 20) = 5.215, p = 0.033, ηp <sup>2</sup> = 0.207].

#### **N200**

In the N200 we observed no main effect or interactions involving factors experiment, group, congruence, and emotion.

#### Discussion

The present set of experiments investigated the role of age-related HL on the influence of emotion on cognitive and emotional control with behavioral and ERP measures. In what follows, we examine in detail the results of the two conflict tasks and finally conclude with a general discussion.

#### Cognitive Conflict

In this Experiment, participants were instructed to report the vowel expressed in videos (i.e., "A" or "O") regardless of its emotional quality. As a result, both groups showed prolonged responses to incongruent compared to congruent stimuli as well as delayed responses to negative than to neutral stimuli. Most importantly, we observed that negative targets reduced the conflict effect in the normal hearing (NH) group, but not in the hearing loss (HL) group. Finally, emotion modulated the N100 conflict effect in both hearing groups, and incongruent stimuli elicited an increased P200 amplitude specifically over anterior electrode-sites in both groups.

First, we found that emotion does not benefit conflict processing in the HL group. Additionally, the participants' ratings of the stimuli showed that HL individuals rated negative targets as less emotional than participants of the NH group (**Table 2**). This is in accordance with what was reported by Husain et al. (3), who showed that moderate HL results in reduced brain activity in response to emotional targets and to structural changes in brain regions that are known to be involved

in the processing of emotions. However, HL individuals rated negative visual stimuli as more negative than NH participants. This may imply that participants with HL relied more heavily on visual information when judging the emotional valence of stimuli and, as a form of overcompensation, rated negative visual stimuli as more negative.

In the EEG, we observed that emotional compared to neutral stimuli resulted in larger N100 conflict effect (18, 19) in both NH and HL adults. Negative emotion requires only some 100 ms to modulate early neural responses to incongruent stimuli in both hearing samples, potentially by increased or preferential allocation of attention to the target (18, 63). Due to high motivational relevance and saliency, visual and acoustic emotional stimuli attract attention (64, 65) and facilitate control processes (21, 66). Possibly, as emotional information in the visual domain was still available to the HL group and as these individuals seem to rely more heavily on visual information, the HL group could show an intact early neural response to the conflict, albeit without a corresponding behavioral facilitation. Alternatively, it is also possible that HL adults do actually process emotional characteristics of acoustic information to some degree during the early neural processing stages but not later on. In other words, this result implies that HL may reduce confidence for the perception of emotional tones with intact early neural responses to emotional auditory stimuli.

The P200 also resulted in a conflict effect over anterior electrode-sites in both hearing groups: incongruent P200 amplitude was smaller than congruent P200 amplitude. Increased attentional demands correlate with decreased P200 responses (28, 67). Consequently, the observed reduction in P200 to incongruent stimuli may be explained by distractor-related increase in attentional demands (15, 68).

Finally, we observed no conflict- or emotion-related main effects and interactions in the N200. This ERP component is evoked when prepotent responses have to be inhibited (31, 69). Therefore, the observed reduced effortful control and executive functions in older individuals may explain the absence of

the N200 conflict effect in both hearing groups [see (70) for comparable findings; (19, 71)].

To summarize, the HL group show intact initial processing of negative auditory stimuli, but reduced confidence at later processing stages. Further, the current results indicate that despite problems with processing of emotional auditory information, HL participants were able to process other acoustic features of sounds (interjections "Ah" and "Oh") as indicated by comparable overall conflict effects and error rates.

#### Emotional Conflict

In this experiment, the task was to report the emotion of the auditory stimulus dimension regardless of the emotion of the visual facial expression, while vocalizations of the face (lip movement) and voice were always matched and task-irrelevant. As a result, emotion facilitated behavioral performance in NH participants, but not in the HL group. Moreover, HL individuals performed at chance level, with error rates ∼50% in incongruent trials. In the EEG, we found a valence-specific N100 conflict response in the two groups: (i) the N100 amplitude was larger for incongruent relative to congruent stimuli, and (ii) this effect was greater for negative than neutral stimuli. Finally, in the P200 and N200 responses we also observed conflict- and valence-specific effects.

Behavioral RT conflict processing was improved for emotional stimuli in NH adults, while HL participants showed a chance performance in response to incongruent stimuli in the emotional conflict task. As HL was shown to diminish processing of acoustic emotional information in the current and previous studies (3, 11), processing of emotional conflict was especially problematic for HL adults. In other words, the HL group could have purely relied on the visual input due to the inability to make use of acoustic stimuli and, therefore, performed at chance level.

In the EEG, we found that emotional rather than neutral stimuli led to an increased N100 conflict effect: we found a larger N100 response to incongruent stimuli in the negative, but not in neutral trials. This effect was comparable in both hearing groups. Therefore, these results indicate that moderate HL does not diminish the processing of emotional cues completely as participants must have detected some emotional information in the acoustic signal that conflicted with the concurrent visual input.

Kumar et al. (11) showed that the backward connections from the amygdala to the auditory cortex were modulated by negative sounds. On the other hand, the acoustic features of a sound modulated the forward connections from the auditory cortex to the amygdala (11). These forward and backward projections are thought to function jointly to process acoustic stimuli (11). Husain et al. (3) hypothesized that hearing-loss related sound deprivation may lessen the available acoustic and/or valence information for the auditory cortex-amygdala interface. The authors propose that, people with HL may exhibit a dulled response to emotional stimuli as they may lack necessary acoustic or valence information required for an adequate emotional response. The current results demonstrate that processing of emotional stimuli is not delayed in HL participants, but these individuals tend to misclassify acoustic emotional information. Additionally, HL could have specifically impacted backward connections from the amygdala to the auditory cortex, thus letting some emotional information still reach the amygdala via the forward connections and to evoke emotion-specific early neural responses in the N100.

We also observed that negative stimuli elicited smaller P200 amplitude than neutral stimuli. Emotion-specific reduction in the P200 response may be driven by attentional capture by negative vocalizations (67), even in the HL group. As discussed above, this finding also implies that HL may result in reduced confidence in the perception of emotional sounds, while early neural responses to such stimuli remain intact.

In the N200 we observed that incongruent stimuli elicited larger responses than congruent stimuli over anterior, but not posterior electrodes. Previous findings suggest that the N200 is an index of conflict monitoring, with its amplitude varying as a function of attentional control required for conflict processing (31, 69, 72). Therefore, an increased N200 response may reflect increased executive demands to process the incongruent stimuli. Finally, negative stimuli elicited increased amplitudes in comparison to neutral stimuli in the HL group, but not in the NH group. We conclude that this may reflect additional demands, uncertainty and difficulty to process emotional stimuli in the HL participants as suggested by previous fMRI research (3).

## GENERAL DISCUSSION AND LIMITATIONS

The current results replicate previous findings that negative emotion facilitates both cognitive and emotional conflict processing by reducing the RT conflict effect (18–20, 73). Emotional stimuli attract attention due to their motivational relevance for survival (64, 65) and trigger cognitive control processes (66). Processing of emotional stimuli is also known to enhance the readiness to act (74) and speed up executive control in both conflict types (16, 18).

Interestingly, no emotion-related behavioral facilitation was observed in the HL group. It was suggested that the age-related gradual increase in HL may promote social isolation (2, 37). As a consequence, the emotion processing limbic network may also be impacted as has been shown in aging and tinnitus research (75– 77). Our results further indicate that social isolation in moderate HL may in part be caused by problems in processing emotional information. Although this topic has not been investigated in much detail in older individuals, it has been shown that children (of up to 9 years old) with mild-to-moderate HL are less able to understand complex social signals such as sarcasm, due to an inability to extract the sarcastic intonation from acoustic information (78). Additionally, Segal and Kishon-Rabin (79) showed that younger adults with mild HL may have problems with the comprehension of the stressed words in a sentence. Processing of emotional cues is even more challenging when they are complemented by emotional cues from different communication channels [audio, visual; (13, 14)]. As shown here, processing of such conflicts may become particularly burdensome for people with age-related HL.

Husain et al. (3) hypothesized that HL may diminish acoustic or valence cues required for the adequate processing of emotional information. Our results suggest that HL may not result in a general susceptibility to acoustic features that are available for processing as HL participants generally performed well in the cognitive conflict task. On the other hand, participants showed the strongest deprivation when the task required to identify the emotional valence of auditory targets. These findings are especially obvious since the multisensory stimuli in the two different conflict tasks were very similar.

Despite a lack of behavioral facilitation (Experiment 1) and chance level performance (Experiment 2), the HL group showed no difference in the emotion-modulated early (100 ms poststimulus) conflict-specific responses. It is possible that the preserved processing of emotional information from the visual domain could facilitate early conflict specific neural processing in HL group. However, this explanation does not apply in the emotional conflict task, where visual emotion information was not available in the incongruent emotional condition (i.e., incongruent combination of a neutral face and a negative voice). These results may imply that HL in the selected frequencies does not completely restrict the processing of emotional cues from the acoustic signal. It appears that HL older adults were still able to process emotional cues to a certain degree; however, this did not result in any behavioral benefits, probably due to reduced confidence in the processing of emotional cue. This hypothesis is in line with our rating results, as well as with previous findings of Picou (80), who showed that HL participants exhibited a reduced range of emotional ratings.

The current study tested whether age-related moderate HL impacts how negative emotions impact cognitive and emotional control. However, it remains open whether we would observe the same result for positive emotions (81–83). Specifically, there is increasing evidence of a positivity effect where elderly individuals preferentially allocate their attention to and have a better memory of positive than negative/neutral stimuli (84–86); however, see (87, 88); for no positivity bias in aging]. In other words, processing of positive emotion information may be specifically important in aging and future studies should examine whether moderate HL may impact positive emotional conflicts as well.

Finally, the age of the actors in the videos could potentially be a limiting factor. Specifically, it was shown that people of different ages seem to preferentially attend to and have higher exposure to faces of their own than another age groups (89), and this may also be true for same-age voices. Considering that we used videos of younger individuals in the current study, this could be a limiting factor as older adults could process faces of younger adults differently than faces of their own age (90). Future studies should aim at controlling this factor.

## CONCLUSION

Age-related moderate HL changes the processing of acoustic and, potentially through compensation, visual emotional cues. As a result, people with HL may show reduced behavioral benefits for emotional stimuli in cognitive and emotional control in a multisensory environment. Importantly, such changes in multisensory integration of incongruent emotional cues may impact the emotion processing limbic network and could

## REFERENCES


contribute to social isolation and depression that is sometimes observed in related to age-related HL.

## AUTHOR CONTRIBUTIONS

AZ, PK, CO, AV, ES, and SK: study design; AZ: data collection; AZ, CO, and SK: data analysis; AZ, PK, ES, AV, and SK: editing of final manuscript.

## SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fneur. 2018.00783/full#supplementary-material


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Zinchenko, Kanske, Obermeier, Schröger, Villringer and Kotz. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Differences in Cortical Structure and Functional MRI Connectivity in High Functioning Autism

Alessandra M. Pereira1,2, Brunno M. Campos <sup>1</sup> , Ana C. Coan<sup>1</sup> , Luiz F. Pegoraro<sup>3</sup> , Thiago J. R. de Rezende<sup>1</sup> , Ignacio Obeso4,5, Paulo Dalgalarrondo<sup>3</sup> , Jaderson C. da Costa2,6, Jean-Claude Dreher <sup>4</sup> and Fernando Cendes <sup>1</sup> \*

<sup>1</sup> Neuroimaging Laboratory, School of Medical Sciences, The Brazilian Institute of Neuroscience and Neurotechnology, University of Campinas, Campinas, Brazil, <sup>2</sup> Department of Pediatrics, Pontifícia Universidade Católica do Rio Grande do Sul, Porto Alegre, Brazil, <sup>3</sup> Department of Psychiatry, State University of Campinas, Campinas, Brazil, <sup>4</sup> Center for Cognitive Neuroscience, Reward and Decision Making Group, Centre National de la Recherche Scientifique, UMR 5229, Lyon, France, <sup>5</sup> Centro Integral en Neurociencias A.C., Hospital HM Puerta del Sur en Madrid, Madrid, Spain, <sup>6</sup> Brain Institute (InsCer), Pontifícia Universidade Católica do Rio Grande do Sul, Porto Alegre, Brazil

#### Edited by:

Argye Hillis, Johns Hopkins Medicine, United States

#### Reviewed by:

Elysa Jill Marco, University of California, San Francisco, United States Roma Siugzdaite, Ghent University, Belgium

> \*Correspondence: Fernando Cendes fcendes@unicamp.br

#### Specialty section:

This article was submitted to Applied Neuroimaging, a section of the journal Frontiers in Neurology

Received: 14 October 2017 Accepted: 18 June 2018 Published: 10 July 2018

#### Citation:

Pereira AM, Campos BM, Coan AC, Pegoraro LF, de Rezende TJR, Obeso I, Dalgalarrondo P, da Costa JC, Dreher J-C and Cendes F (2018) Differences in Cortical Structure and Functional MRI Connectivity in High Functioning Autism. Front. Neurol. 9:539. doi: 10.3389/fneur.2018.00539 Autism spectrum disorders (ASD) represent a complex group of neurodevelopmental conditions characterized by deficits in communication and social behaviors. We examined the functional connectivity (FC) of the default mode network (DMN) and its relation to multimodal morphometry to investigate superregional, system-level alterations in a group of 22 adolescents and young adults with high-functioning autism compared to age-, and intelligence quotient-matched 29 healthy controls. The main findings were that ASD patients had gray matter (GM) reduction, decreased cortical thickness and larger cortical surface areas in several brain regions, including the cingulate, temporal lobes, and amygdala, as well as increased gyrification in regions associated with encoding visual memories and areas of the sensorimotor component of the DMN, more pronounced in the left hemisphere. Moreover, patients with ASD had decreased connectivity between the posterior cingulate cortex, and areas of the executive control component of the DMN and increased FC between the anteromedial prefrontal cortex and areas of the sensorimotor component of the DMN. Reduced cortical thickness in the right inferior frontal lobe correlated with higher social impairment according to the scores of the Autism Diagnostic Interview-Revised (ADI-R). Reduced cortical thickness in left frontal regions, as well as an increased cortical thickness in the right temporal pole and posterior cingulate, were associated with worse scores on the communication domain of the ADI-R. We found no association between scores on the restrictive and repetitive behaviors domain of ADI-R with structural measures or FC. The combination of these structural and connectivity abnormalities may help to explain some of the core behaviors in high-functioning ASD and need to be investigated further.

Keywords: autism spectrum disorders, functional connectivity, MRI, cortical thickness, default mode network (DMN), social communication, stereotyped behavior

## INTRODUCTION

Autism spectrum disorders (ASD) represent a complex group of neurodevelopmental conditions characterized by deficits in social behaviors, including both interpersonal social processes and selfreferential thought (1). This condition is reported to affect 1 in 59 individuals according to the last CDC update of autism's estimated prevalence (2). The pathology of ASD is currently considered a disruption of brain development time-course with a wide range of heterogeneity among patients (3). The specific neurobiological substrates of this lifelong developmental disability remain unclear. Several studies reported a combination of structural abnormalities along with atypical brain connectivity in ASD (4–15). These abnormalities could help explain some of the symptoms of ASD and their severity.

Early investigations in ASD showed an increase in total brain volume at 2–4 years of age persisting into childhood but not adolescence (16). Some areas increase more than others, including frontal and temporal regions and the amygdala, while other structures present reduction in volume, such as the corpus callosum (17–26), probably indicating dysfunction of intra- and inter-hemispheric connectivity (15, 27–36). The first generation of studies using brain imaging failed to report consistent localized neocortical brain dysfunction (37, 38). However, structural neuroimaging has indicated various sites of anatomical abnormalities, providing some clues for a better understanding of this condition (17, 39–44).

Despite some inconsistencies, there is a trend from more recent studies which have observed regional increases of gray matter (GM) accompanied by local reductions of white matter (WM) (6, 38, 45). These findings support an increased local but reduced long-distance cortico-cortical reciprocal activity and functional coupling (46–48). Converging lines of evidence suggest that ASD is a complex disorder of brain connectivity (49, 50), involving aberrant functional connectivity (FC) within the default mode network (DMN), as well as between the DMN and several cortical and subcortical areas (13, 15, 27, 30, 31, 34– 36, 44, 51–70, 107, 135).

The DMN is a set of structures known to be particularly engaged when participants are at rest **(Figure 1)**. Anatomically, this network consists of the posterior cingulate cortex (PCC), retrosplenial cortex, lateral parietal/angular gyrus, medial prefrontal cortex, superior frontal gyrus, regions of the temporal lobe, and the parahippocampal gyrus (54, 71–73, 79). Many have speculated that the DMN function may extend beyond cognitive processes and encompass the role of maintaining homeostasis between excitatory and inhibitory neuronal responses (74, 75). Others have argued that it is active when contemplating scenarios and events, when the mind is wandering, or when conducting lower-level observations of the individual's external surroundings (76–79). More recently, the "developmental disconnection model," proposed by many authors, links the core symptoms of ASD to weak FC between remote cortical regions and an excess of FC within local regions (80–82). For recent reviews in the topic see references (6, 15, 37, 43, 44, 50, 57, 67, 83–86).

It is currently unclear the extent of regions overlap between abnormal structural and functional connectivity in ASD patients

FIGURE 1 | The DMN constituent components. The blue square placed in the posterior cingulate cortex illustrates the seed position described in the methods.

and its relationship with different clinical presentations in the spectrum of this condition (26, 87, 88). The understanding of the relationship between structural and functional alterations is also compromised by the high heterogeneity of individuals and the age-related differences reported among different ASD groups (26). The comparison between brain structure and function in a single group of ASD individuals with similar phenotypic pattern can shed light on these complex interactions and establish a link with clinical symptomatology in these patients.

We aimed to characterize the relationships between structural and functional abnormalities in a cohort of patients with highfunctioning autism. We performed a high resolution multimodal structural (cortical thickness, gyrification index, surface area and GM volume) and functional (resting-state FC) analysis to detect superregional, system-level alterations attempting to establish a neurobiological foundation to pathology and clinical symptoms in this part of the spectrum of autism adolescents and young adults with high-functioning autism without associated depression, psychosis, seizures, or other major psychiatric disorders.

## METHODS

#### Participants

We recruited 22 adolescents and young adults with ASD and 29 normal controls from the local community and the University of Campinas. This study was approved by the Ethics Committee of the University of Campinas (plataformabrasil.saude.gov.br; reference number: CAAE 02388012.5.0000.5404; number of the approved ethical statement: 190409). All participants provided written informed consent approved by the Ethics Committee. For the participants younger than 18 years of age, we obtained informed consent from parents or guardians, as well as from the participants themselves.

A trained and qualified clinician made the diagnosis of ASD using the DSM-5 criteria after interviewing the family and examining each patient. A second investigator confirmed the diagnosis using the "Current" Scores of the Autism Diagnostic Interview-Revised (ADI-R) (89). The ADI-R is a clinical diagnostic instrument for assessing autism in children and adults (89). The ADI-R provides a diagnostic algorithm for autism as described in both the ICD-10 and DSM-IV and is one of the most important validated ASD measures available in Brazil. The clinician's observation provides the opportunity to put the patient's behavior into the context of knowledge about other patients, but information from caregivers provides a broader context needed in understanding the patient's day to day behavior in a wide range of situations, his or her history, as well as family expectations, resources, and experiences and other important contextual factors. Thus, patient's testing and parent interviews should be viewed as complementary and necessary components of the diagnostic evaluation after the clinical evaluation and DSM-5 criteria. All patients were required to have a full-scale IQ greater than 85, as measured by the Wechsler Abbreviated Scale of Intelligence.

Exclusion criteria comprised a history of major psychiatric disorders (e.g., depression, psychosis), seizure, head injury, toxic exposure, facial dysmorphic features, and the evidence of genetic, metabolic, or infectious disorders. We also excluded individuals with secondary autism related to a specific etiology such as tuberous sclerosis or Fragile X syndrome (all included patients had a negative investigation of tuberous sclerosis and Fragile X syndrome).

Thirteen individuals in the ASD group were using a variety of psychoactive medications. Nine subjects were not under psychoactive drug treatment. Five subjects were taking psychostimulants, seven were taking antipsychotics, and six were taking selective serotonin reuptake inhibitors (SSRIs) for anxiety and compulsive behaviors. Six of these subjects were using more than one of the medications listed above. Participants were instructed not take any medication 1 day before their visit.

#### Neuroimaging Data Acquisition

We acquired functional and structural MRIs on a 3T scanner (Phillips, Achieva; Best, The Netherlands) with the following protocol:


MRI sequences were corrected for gradient non-linearity during the reconstruction step in the Phillips scanner. We performed a visual inspection of all structural and functional images to assess image quality, movement artifacts, and the existence of clinically relevant abnormalities.

## Image Processing and Analysis

Our MRI phenotyping combined group- and individuallevel analysis of GM volume, cortical thickness and folding complexity, which are three established in vivo markers of brain morphology and development. There was no difference between the groups on movement in the scanner for the structural imaging.

#### Voxel-Based Morphometry Analysis

We performed VBM with the VBM8/SPM8 toolbox (Wellcome Department of Cognitive Neurology, http://www.fil.ion.ucl.ac. uk) for detection of GM volume abnormalities. VBM allows the automated identification of the whole brain GM differences between groups (90). Post-processing of the T1-weighted images included normalization to the same stereotaxic space (MNI-152 template), modulation and segmentation of the images into GM, WM and cerebrospinal fluid (CSF). The DARTEL algorithm was included to increase the accuracy of the alignment between subjects (91). The resultant GM images were smoothed with a 10 mm FWHM isotropic Gaussian kernel. We excluded eight outliers (four ASD patients and four controls) detected in a quality test for image homogeneity and co-registration. Therefore, the final VBM analysis included 19 ASD patients and 25 controls (all other analyses from here on included the 22 patients and 29 controls).

We used two-sample t-tests (to adjust for multiple comparisons we considered a p < 0.001, minimum of 30 contiguous voxels) to search for areas of volume reduction or increase in ASD patients. First, we looked for areas of GM volume reduction or increase in the ASD group with age as covariable. As a second approach, we looked at the differences between groups in the correlation between age and GM volumes with total IQ as a covariate.

## Cortical Thickness, Surface Area, and Gyrification Analysis With FreeSurfer

We performed cortical reconstruction and volumetric segmentation with the FreeSurfer image analysis suite (http:// surfer.nmr.mgh.harvard.edu/), which is a well-validated method already described in previous publications (92–94). A single filled WM volume was generated for each hemisphere after intensity normalization, skull stripping, and image segmentation using a connected components algorithm. A surface tessellation was generated for each WM volume by fitting a deformable template. This resulted in a triangular cortical mesh for GM and WM surfaces in each hemisphere. Cortical thickness, then, was calculated as the shortest distance between GM and WM surfaces. Vertex-wise measurements of surface area were determined as the area of a vertex on the GM surface (5). We used the FreeSurfer default Gaussian filter of 10 mm FWHM to smooth the surfaces (92, 94).

Another volumetric measure obtained from FreeSurfer is the local gyrification index (LGI) which was developed by Schaer et al. (95). The LGI was defined as the ratio between the GM surface border and an outer border in successive coronal sections (96). To calculate this LGI, FreeSurfer uses both tessellated outer and inner contours of the pial surface, which were covered by a triangle mesh. For each vertex on the outer surface, a spherical region of interest is created with a standard size of 25 mm radius. Therefore, the LGI is given as the ratio between the outer area on the surface and the area comprehended in the real pial surface (95). Thus, the LGI for each vertex on the pial surface reflects the amount of cortex buried in its locality. The LGI values obtained were mapped onto a normalized cortical surface.

We then compared regional cortical thickness, surface area and gyrification index between autism and control groups using a general linear model (GLM) with age and total IQ as covariates. To correct for multiple comparisons, we performed a clusterbased correction (level of significance at α = 0.01) (97).

## ROI Analysis With Data Extracted From FreeSurfer

ROI measures of cortical thickness, cortical area and LGIs for 33 gyral regions generated by FreeSurfer (98, 99) (https://surfer. nmr.mgh.harvard.edu/fswiki/FsTutorial/AnatomicalROI# Groupstatsfiles) were corrected for total intracranial volume generated by FreeSurfer and exported to SPSS Statistics version 20 (IBM Corp. Released 2011. IBM SPSS Statistics for Windows, Version 20.0. Armonk, NY: IBM Corp.).

Group differences in gyral-level cortical thickness, cortical area, and LGIs were analyzed using mixed GLMs with diagnosis (autism vs. controls) as the between-subjects factor, the 33 gyral regions from both hemispheres (98, 99) as the within-subjects factors, also with age and total IQ as covariates. We also ran the same mixed GLM for subcortical volumes generated by FreeSurfer. All comparisons between controls and patients were Bonferroni corrected for multiple comparisons.

#### Resting-State Functional MRI Processing and Analysis

To perform the resting-state processing and analysis, we used the UF2C (User-Friendly Functional Connectivity; https://www. lniunicamp.com/uf2c) toolbox (100) on a PC running MATLAB 2013a (The MathWorks, Inc., Natick, MA, USA) with SPM8 (Wellcome Trust Centre for Neuroimaging). The UF²C toolbox (100) pipeline started with a standard image preprocessing protocol which includes: (i) functional realignment to the mean image (movement parameters are saved); (ii) structuralfunctional co-registration; (iii) structural segmentation into GM, WM and CSF tissues; (iv) functional and structural normalization (MNI 152); (v) functional image smoothing (kernel with double voxel sizes = 6 × 6 × 6 mm<sup>3</sup> ).

We used the functional and structural T1-weighted images of all subjects as data input. The GM, WM, and CSF maps were spatially adjusted (sinc interpolation [or Whittaker–Shannon interpolation] of third degree) to the functional image, aiming to obtain functional segmented maps (GM, WM, and CSF). A multilinear regression was performed including WM and CSF global signal fluctuations and six movement parameters (three translational and three rotational) to reduce their confounding influence on the GM signal (101). Subsequently, a band-pass filter (0.008–0.1 Hz) was applied to remove low-frequency drifts and artifacts arising from cardiac or respiratory rate (102).

To reduce the chance of false positives/negatives, we controlled the amount of motion during scanning sessions using a cumulative value of movement equal or higher than 3 mm (size of one voxel) using the first volume as a reference as the cut-off to exclude subjects from the analysis. One patient was excluded from the resting-state analyses due to excessive movements during the fMRI acquisition. There was no difference between groups in the amount of movement during the scans: multivariate general linear model, Tukey's corrected with maximum displacement on axes X (controls average 0.77 mm ±0.47; patients average 0.78 ± 0.49), Y (controls average 0.30 mm ± 0.1; patients average 0.41 ± 0.22), and Z (controls average 1.15 mm ± 0.47; patients average 1.35 ± 0.51), average framewise displacement (controls average 0.18 mm ± 0.04; patients average 0.24 ± 0.06), and derivative variance (DVAR) (control average 3.18% SD ± 0.40; patients average 3.22% SD ± 0.36) were added as variables.

We estimated the cross-correlations using a cubic seed (9 × 9 × 9 mm<sup>3</sup> ) to extract the reference time-series (64). The reference time-series was correlated with each gray matter voxel creating the correlation maps. We varied the seed position according to the analysis described below.

#### DMN Analysis

The motivation to investigate the connectivity of the whole brain to and from the DMN came from the fact that: (a) it is a very stable and reproducible network (103, 104), (b) several studies have shown alterations in the DMN in ASD, including high functioning autism (88), and (c) it connects to most regions of the brain, and in particular, to regions processing salience, attention, and negative affect (105). To study the DMN, we positioned the seed on the PCC (centered on the MNI coordinate −41 13 −29) because this is one of the most active areas within the DMN, and it is possible to place a seed region involving both hemispheres at once (the blue square in **Figure 1** illustrates the position of this seed). We used the standard seed-based FC methodology, in which the whole averaged time series of the seed region is used as a reference to calculate the correlation with the GM voxels. We performed these steps individually generating a 3D r-score map for each volunteer. We converted all individual rscore maps resultant from the connectivity analysis to z-scores (Fisher's transformation) and performed a spatial smoothing (6 × 6 × 6 mm<sup>3</sup> FWHM), aiming to reduce high discrepancies in neighbor voxels.

#### Other Seed Positions

Additional to the seed positioned in the PCC (from the DMN), we tested other four seeds that we judged relevant for ASD verbal communication and social skills, according to findings from previous publications (6, 37, 106): (i) bilateral medial frontal region (MNI 0 49 −3); (ii) left + right amygdala (MNI −23 −4 −20); (iii) left anterior hippocampus (MNI −24 −13 −20); (iv) left temporal pole (−41 13 −29). We used the same steps as described for the generation of the 3D r-score DMN maps to obtain individual 3D statistical maps for the functional connectivity maps derived from seeds in these four positions. We did not include seeds in other areas also considered important for ASD, such as the caudate, to avoid too many comparisons and to focus mainly on regions more directly related to emotional communication and interpersonal interactions.

The functional connectivity preprocessing was developed aiming to avoid possible confounding effects raised from structural variations. The functional images were segmented using the tissues probabilistic maps obtained from the T1WI, with consistent thresholds. This means that the resultant postprocessed functional images included only voxels with the upper threshold probability to be GM or GM/WM. Additionally, the seeds time series extraction applied an algorithm that excludes by the average time series, voxels with a temporal behavior that is considered a minor outlier regarding the others. These last steps exclude from the seed, voxels which are functionally discrepant (see **Supplementary Image 1**).

As in the previous section, all individual r-score maps resultant from the connectivity analysis to z-scores (Fisher's transformation) and performed a spatial smoothing (6 × 6 × 6 mm<sup>3</sup> FWHM), aiming to reduce high discrepancies in neighbor voxels. We applied a two-sample t-test (to adjust for multiple comparisons we considered a p < 0.001, with a minimum of 10 contiguous voxels) with age added as covariate to compare controls and patient's groups resulting in two t-maps: a map showing areas that were more functionally connected in controls than in patients and a map showing the opposite.

#### Correlations With the Clinical Phenotype

We explored how the neuroanatomical and functional differences observed in the ASD group may be related to the clinical outcome. For that purpose, we conducted multiple correlation analyses between the ROI measures of cortical thickness, cortical area, and LGIs for the 33 gyral regions of each hemisphere generated by FreeSurfer (98, 99) and values from the PCC seedbased functional connectivity analysis (Resting-state analysis) vs. the "Current" Scores obtained at the ADI-R (scores in each of the three content areas: communication and language, social interaction, and restricted, repetitive behaviors), with age and total IQ as covariates and with Bonferroni correction for multiple comparisons using SPSS Statistics version 20 (IBM Corp. Released 2011. IBM SPSS Statistics for Windows, Version 20.0. Armonk, NY: IBM Corp.).

#### Analyses of Overlapping of Abnormalities Across Modalities

We analyzed the number of voxels that coincided with the resting-state fMRI and structural analyses using co-registration of statistical maps. This procedure was automated and based on the maps matrix intersection, providing relative percentages of overlapping among maps. Maps with distinct resolution were interpolated using 4th degree B-Spline interpolation.

In addition, we also investigated if the areas of abnormalities were near or within the same anatomical sub-region by subregion by atlas labeling coincidence.

#### RESULTS

#### Subject Demographics and Global Brain Measures

There were no significant differences in age between ASD (n = 22; mean ± SD: 17.45 ± 3.29) and controls (n = 29; 18.48 ± 2.82, two-sample t-test, p = 0.24). There was no significant difference in sex ratios between groups (Fisher's exact test; p = 0.22). We found no significant differences in full scale and performance IQ (p = 0.1) but, as expected, the ASD group displayed significantly lower verbal scale IQ (p = 0.03; see **Table 1**). There were also no significant betweengroup differences in total brain volume or total surface area (p > 0.05).

All imaging analyses were covaried for age and total IQ and corrected for multiple comparisons as described in the methods.

#### Voxel-Based Morphometry (VBM) Analysis

VBM showed that individuals with ASD had reduced GM concentration in the cerebellum bilaterally (right anterior and posterior lobe and left posterior lobe), bilateral anterior cingulate, right middle, medial, and superior frontal gyrus, left fusiform gyrus, parahippocampus, amygdala, paracentral, and postcentral gyrus and claustrum. Increased GM concentration was detected in the right cerebellum and brainstem (**Figure 2**; **Table 2**).

In a correlation between age and GM volumes (i.e., areas with decreased GM volume in patients with increasing age as compared to controls), we observed that ASD participants had more age-related GM atrophy than controls exclusively in the left temporal lobe (temporal pole, middle temporal gyrus, parahippocampal gyrus, uncus) (p < 0.001, **Supplementary Image 2**; **Table 3**).

TABLE 1 | Summary of clinical data.


ADI-R, Autism Diagnostic Interview-Revised ("Current" Scores); ASD, autism spectrum disorder. There were no significant differences between the ASD and control groups in age, full IQ and performance IQ at p < 0.05 (two-tailed). There was no significant difference in sex ratios between groups (Fisher's exact test; p = 0.22). \*There was a significant difference in verbal IQ (p = 0.03). The following cutoff scores were used: ADI-R social, greater than 10; communication, greater than 6; and repetitive behavior, greater than 3. Rt to Lt, right to left ratio.

FIGURE 2 | Areas with decreased (cool colormap) and increased (hot colormap) cortical voxel-based morphometry in patients when compared to controls. In shades of blue (cool colormap), the most significant regions with decreased gray matter (voxel-based morphometry, two sample t-test, p < 0.001, cluster with at least 30 voxels) in patients compared to controls. In the hot colormap (black to yellow), regions of increased gray matter (voxel-based morphometry, two sample t-test p < 0.001 clusters with at least 30 voxels).

#### Cortical Thickness and Gyrification Index Using Freesurfer Vertex-by-Vertex Analysis

Individuals with ASD presented decreased cortical thickness in the right hemisphere over the cingulate, precentral, superior frontal, superior, and inferior parietal regions. In the left hemisphere, decreased cortical thickness was observed in the supramarginal, superior parietal, paracentral, precuneus, superior, and middle frontal and lingual gyrus (the areas of decreased cortical thickness are shown in red in **Figure 3A**), and increased thickness was observed in the postcentral area (**Table 4**).

The ASD group had increased cortical surface in the following areas in the right hemisphere: cingulate, precentral, and superior frontal regions (which coincided with regions with decreased cortical thickness), as well as middle frontal, pars triangularis, supramarginal, precuneus, paracentral, superior, and middle temporal, and lateral occipital regions. In the left hemisphere, the ASD group had increased surface areas in the superior and middle frontal and precuneus (coinciding with the regions with


TABLE 3 | Areas with significant gray matter VBM reduction influenced by the age in patients with ASD.


p < 0.001; cluster with at least 30 voxels. All these four areas had significantly reduced functional connectivity on the seed analysis (see Table 6).

reduced cortical thickness), as well as in the pre- and post-central, orbitofrontal, posterior cingulate, inferior parietal, temporal lobe (superior, middle, and inferior temporal regions) and insular regions (**Figure 3B**).

Gyrification was increased in the lingual, precuneus, superior temporal sulcus and superior parietal areas in the right hemisphere, and in the precentral and paracentral areas of the left hemisphere (**Figure 3C**).

#### Region of Interest (ROI) Analysis With FreeSurfer Data

When examining gyral-based differences in cortical thickness (ROI analysis with data extracted from FreeSurfer), which includes a larger number of voxels in each region measured by the vertex-by-vertex analysis, we observed increased thickness in the right posterior cingulate cortex, including the isthmus cingulate

FIGURE 3 | Regions with differences in cortical thickness, surface areas, and gyrification. The most significant clusters for group analysis using a GLM vertex-wise approach, between control and autism groups for left and right hemispheres. In red are areas of decreased and in blue are areas of increased values in patients with autism. All results were corrected for multiple comparisons (Cluster-based correction). (A) ASD presented decreased (in red) cortical thickness in the right cingulate, precentral, superior frontal, superior, and inferior parietal regions. In the left hemisphere, decreased cortical thickness was observed in the supramarginal, superior parietal, paracentral, precuneus, superior, and middle frontal and lingual gyrus, and increased thickness in the postcentral area. (B) Increased surface areas in the superior and middle frontal and precuneus (coinciding with the regions with reduced cortical thickness), as well as in the pre- and post-central, orbitofrontal, posterior cingulate, inferior parietal, temporal lobe (superior, middle, inferior temporal), and insular regions in ASD. (C) Increased gyrification in the lingual, precuneus, superior temporal sulcus, and superior parietal areas in the right hemisphere and the precentral and paracentral areas of the left hemisphere.

(which is a narrow cortical area that connects the posterior end of the cingulate gyrus with the parahippocampal gyrus), and in the right and left lateral orbitofrontal cortex as well as decreased cortical thickness in the left paracentral and posterior cingulate TABLE 4 | Areas of decreased cortical thickness by FreeSurfer vertex-wise analysis in patients with ASD.


Level of significance equal to 0.01. All results were corrected for multiple comparisons (cluster-based correction). S, sulcus; G, gyrus.

and in the right temporal pole in the ASD group compared to controls (**Supplementary Image 3**; **Table 5**).

The ROI analysis showed significantly increased cortical surface area only in the right anterior cingulate (p = 0.019, multivariate analysis with Bonferroni correction).

We found also increased gyrification index in the postcentral, precentral, superior parietal, and supramarginal regions of both hemispheres, in the right frontopolar and middle frontal regions, and in the left paracentral region (**Table 6**).

#### Subcortical Volumes

We found no differences between groups in the volumes of the amygdala, hippocampus, thalamus, or caudate.

#### Functional Connectivity

We first examined the FC patterns of the PCC, which is part of the DMN. Relative to the control group, ASD patients showed reduced FC with the PCC (i.e., between the posterior part of the DMN and other areas of the brain) which was more pronounced in the left hemisphere, including the middle temporal gyrus, inferior, and superior frontal gyrus, and anterior and posterior cingulate. Decreased connectivity was also observed in other regions outside the DMN: the right cerebellum, cuneus, and caudate (**Figure 4**). Increased FC in areas of the DMN occurred only in the right middle frontal gyrus. Outside the DMN regions, increased connectivity was present in the left caudate (**Figure 4**).

The analysis of the additional seed positions as described in Methods, showed decreased FC in ASD patients between the left amygdala and right claustrum, inferior parietal lobule, postcentral gyrus, cingulate gyrus, precentral gyrus, inferior frontal gyrus, middle frontal gyrus, and left postcentral gyrus; between the left anterior frontal region and the right superior frontal gyrus; between the left anterior hippocampus and bilateral temporal, right insula, and left precentral regions; between the left temporal pole and the left temporal and parietal, right temporal, frontal, parietal, and occipital regions (**Figure 4**; **Table 7**). Increased FC was observed between left amygdala and right superior frontal gyrus, and between the middle frontal regions and bilateral pre- and postcentral gyrus (**Figure 4**; **Table 7**).

## Imaging and Clinical Scores

#### Cortical Thickness and Symptomatology

Significant correlation (corrected for age and total IQ) was found in the right pars triangularis (part of the lateralized frontoparietal components of the DMN) (73), where reduced cortical thickness was associated with more impaired scores in the social domain of the Autism Diagnostic Interview-Revised (ADI-R) (r = −0.63; p < 0.001) (**Figure 5**). A significant negative correlation (r = −0.52; p = 0.02) was also found between cortical thickness in the left precentral and superior frontal regions (areas of the executive control and sensorimotor component of the DMN) (73) with communication scores on the ADI-R (**Figure 6**).



L, left; R, right. p < 0.05 (multivariate analysis with Bonferroni correction) for all the areas presented in the table.

TABLE 6 | Spatially distributed patterns of differences in the gyrification index in individuals with Autism Spectrum Disorder compared with controls—ROI analysis.


L, left; R, right. p<0.05 (multivariate analysis with Bonferroni correction) for all the areas presented in the table.

Reduced cortical thickness in these areas was associated with more severe scores on the ADI-R communication domain. Thicker cortices in the right temporal pole (r = 0.56; p = 0.01) and posterior cingulate (r = 0.50; p = 0.03) were associated with greater communication impairment as measured by the ADI-R communication domain (**Figure 6**). We found no correlations between the scores on the restrictive and repetitive behaviors (RRIB) domain of ADI-R and structural images.

#### Functional Connectivity and Symptomatology

There was a trend for significant association (that did not survive Bonferroni correction) between stronger connectivity indexes from PCC to the right temporal pole (p = 0.09) and left anterior hippocampus (p = 0.10) with worse symptom severity in the social domain on the ADI-R, controlling for age and total IQ. We found no correlations between the scores on the RRIB domain of ADI-R and FC.

#### Overlap of Abnormalities Across Modalities

The percentage of coincident maximum voxels abnormalities between resting-state FC and abnormal gray matter on VBM was <3%. However, we found a close localization of the FC abnormalities and GM reduction on VBM and changes in cortical thickness in FreeSurfer ROI analysis (**Figure 7**) and vertexwise analysis (**Figure 8**) in cingulate gyri of both hemispheres, left parahippocampal gyrus, postcentral gyrus, amygdala, and claustrum; right middle and superior frontal gyri, temporal pole, and cerebellum (**Table 8**).

Note that the lack of correspondence between the maps presented in **Figures 2**, **3** and the results in **Figures 7**, **8**, is because in **Figures 2**, **3** the maps show the most statistically significant clusters of abnormalities while in **Figures 7**, **8** the areas indicated do not correspond to the maximum voxel statistical location, but rather the sub-anatomical regions with significant differences (therefore, much larger than in **Figures 2**, **3**).

GM atrophy determined by VBM showed a closer anatomical relationship with reduced FC than surface measures by FreeSurfer. Interestingly, areas with decreased GM volume (middle and superior temporal gyri, parahippocampus, and amygdala/uncus, all in the left hemisphere) that correlated with increasing age in patients had reduced FC (see **Table 3**).

#### DISCUSSION

The diversity of neuroimaging results are likely explained by the heterogeneous nature of ASD, both among the subgroups within the spectrum, the variable comorbidities and on an individual level across the lifespan (15, 29, 35, 43, 44, 50, 57–60, 63, 67, 84–86, 107, 135). The individual differences in functional and structural organization, the idiosyncratic ASD connectivity and cortical atrophy maps, which change over the maturation of central nervous system, are themselves the core features of ASD, although its pathophysiological basis remains undetermined (13, 15). These findings underscore the need to address both age and severity when investigating functional and structural neuroimaging in ASD (15). Every imaging technique, both regarding acquisition and post-processing have their limitations and advantages and are in constant improvement of the quality of acquisition (better hardware) and algorithms of post-processing. These facts make it difficult to compare studies over the years. The use of multimodal imaging in a single study, in a similar age range and severity of symptoms, may provide a better description of the altered brain connectivity and structural changes, and its relationship with behavioral changes, than one imaging method alone. However, several multimodal studies have been performed with some contradictory findings, which by itself justify further studies (6, 15, 29, 35, 37, 43, 44, 50, 57–60, 63, 67, 84–86, 107, 135).

Different from most studies that focused on a single technique (27, 28, 108–112) or low functioning autism (113), or using a heterogeneous group of patients (114), our multimodal imaging

shades of blue (cool colormap), regions with maximum decreased functional connectivity (union of all seeds results, two sample t-test p < 0.001 clusters with at least 10 voxels) in patients compared to controls. In the hot colormap (black to yellow), regions of increased functional connectivity (two sample t-test, p < 0.001, cluster with at least 10 voxels).

investigation showed abnormalities across brain measures in young adults and adolescents with high-functioning autism. We showed reduced cortical thickness, increased cortical surface and increased gyrification, as well as abnormal functional connectivity, mostly co-localized in areas that are important hubs of the default mode network and other regions frequently linked to socio-emotional processing, such as cingulum, amygdala, insula, and temporal pole. Overall, our findings suggest aberrant functional connectivity involving a network of altered cortical structure.

We combined structural and functional connectivity analyses to detect complex brain abnormalities and to investigate how these alterations are related to each other and symptom severity in a group of individuals with high functioning autism. We observed that patients with ASD had decreased FC compared to controls between the PCC and anterior medial prefrontal cortex and left superior temporal cortex (temporal pole), both regions part of the DMN. Patients also exhibited greater diffuse subtle GM atrophy related to increasing age (in the VBM analysis), more pronounced in left temporal TABLE 7 | Areas of significantly decreased and increased connectivity in patients with ASD in comparison with a group of healthy individuals.


\*PCC, Posterior Cingulate Cortex bilaterally (posterior aspect of the DMN). Ant, anterior; Bil, Bilateral; All regions in the table had p < 0.001 (two-sample t-test), cluster with at least 10 voxels.

Interview-Revised) social domain.

regions (temporal pole, middle temporal gyrus, parahippocampal gyrus, and uncus). In addition, we showed areas of abnormal cortical structure, combining thinning, and thickening, increased surface area and gyrification index in different areas of the brain, involving frontal, parietal, and temporal areas that had abnormal FC. Overall these structural and functional abnormalities involved areas linked to: (a) visual processing and analysis of logical order of events (lingual gyrus), (b) encoding visual memories (temporal and posterior cingulate areas), (c) areas related to language, memory and emotion processing (temporal pole, middle temporal, parahippocampus, and uncus), (d) areas of the executive control component of the DMN, which has been associated with performance of executive functional tasks (anterior and posterior cingulate cortex, left middle temporal, inferior, and superior frontal gyrus), (e) areas of the sensorimotor component of the DMN (anteromedial prefrontal cortex and bilateral pre- and postcentral gyrus), (f) areas of the lateralized fronto-parietal components of the DMN related to executive and language functions (reduced cortical thickness in left frontal regions), and (g) areas of the auditory component of the DMN (temporal and parietal areas) (73, 115). In addition, more severe scores on the communication domain of the ADI-R were associated with increased cortical thickness in the right temporal and posterior cingulate gyrus, and there was a trend for worse symptoms in the social domain of the ADI-R to be associated with stronger connectivity between posterior cingulate cortices (DMN) and temporal regions (areas of the Auditory component of the DMN) (71–73).

Our findings taken together indicate that young adults and adolescents with high functioning autism present complex, subtle morphological cortical changes that may reflect different stages of neurogenesis, combined with aberrant connectivity within and outside the DMN.

FIGURE 6 | Correlations between cortical thickness (from FreeSurfer ROI analysis) and communication scores on the ADI-R. In the ASD group thicker cortices in the right temporal pole and right posterior cingulated were positively associated with greater communication impairment as measured by the ADI-R Autism Diagnostic Interview-Revised) social domain (Top), while thinner cortices in the left precentral and superior frontal regions correlated with greater communication impairment as in the ADI-R social domain (Bottom).

#### Structural Abnormalities

To date, neuroimaging studies in ASD have mainly investigated either cortical volume or cortical thickness in isolation, and combined measures of surface area and gyrification with functional data remain scarce (4). Studies in adults with ASD typically show cortical thickening of the frontal cortex (6, 116, 117), whereas the cortical thickness of the temporal lobe has been reported as increased or decreased in patients with ASD (118).

Abnormal brain structure has been reported with great variability in individuals with ASD, both enlargement, and

to the maximum voxel statistical location, but rather the sub anatomical regions with significant differences in patients with high functioning autism compared to controls. See Tables 2–7 for the centroid MNI coordinates of maximal abnormalities and Table 8 for a summary of the location of increased and decreased changes as compared to controls.

reduction of the GM (40, 46, 119). However, this variability is probably due to the highly heterogeneous age of the patients (from children to adults) and various phenotypes (5, 120). It is believed that in ASD there is a disruption of the time course of brain development and this could be the explanation for the detection of specific increased areas in children during an early phase of development and reduced areas (atrophy) in adults (40). Our findings, which included only ASD individuals with total IQ > 85, confirm this theory and add further evidence about specific types of abnormal cortical shape and volume in association to functional abnormalities. Another key aspect of our results is that we used multimodal imaging measures in the same patients to certify that the abnormalities are present across brain measures, different from most studies so far that focused on a single technique.

Volumetric studies of ASD in earlier MRI studies showed increased volumes in left frontal and temporal lobes across the 2- to the 11-year-age range (121) and in the dorsolateral prefrontal and medial frontal cortex in patients aged 2–5 years (122). A meta-analysis showed that brain size in autism was slightly reduced at birth, increased within the first year of life, and within normal range by adulthood (123). However, it is difficult to compare these studies since the methodologies for cortical volume measurements varied significantly (manual volumetry, VBM with different versions of SPM software, cortical thickness). Also, earlier studies used images with lower MRI field strength (1.5 T) as compared to the higher fields (3T MRI)

and higher resolution images used in more recent studies. More recent versions of SPM software (http://www.fil.ion.ucl.ac.uk/ spm/software/) have substantial algorithmic enhancements with more sophisticated registration models compared to previous versions and thus, making it difficult to compare earlier studies with more recent ones (45, 124). These aspects and the fact that our patient's ages ranged from 14 to 25 years (mean: 17.4 years) may explain why our VBM analyses (excluding the cerebellum and brainstem) did not show areas of increase GM and showed GM atrophy mainly in temporal and frontal areas.

VBM and FreeSurfer cortical measures use quite different methods and are expected to yield different results as we showed here. Our intention was not to compare these two methods, but rather to expand the search for structural changes in these patients in a multimodal way. We believe that these two techniques added information and were not redundant. VBM performs voxel-wise statistical analysis on smoothed (modulated) normalized segments (90, 124). VBM is a statistical parametric mapping of segmented tissue density and compares the local concentration of gray matter between two groups of subjects (90, 124). The interpretation of gray matter concentration or density depends on the preprocessing steps used (90, 124). However, VBM is a whole-brain unbiased, objective technique, with very reproducible results in similar circumstances (of image quality and software version), providing great sensitivity for localizing small-scale, regional differences in gray matter concentration (90, 124, 125). In addition, more rigorous methods for correction for multiple comparisons will reduce the false positives but also reduce the pickup rate of true positives.

FreeSurfer uses the cortical geometry to do inter-subject registration, which appears to have a much better matching of homologous cortical regions than other volumetric techniques. FreeSurfer allows measuring the two components of volume separately (thickness and surface area). These two measures are not similar and do not necessarily change in parallel as will be discussed below (37). FreeSurfer uses the white matter surface geometry for registration, which is completely independent to GM atrophy; therefore, GM alterations will not result in different registrations (92–94, 99). Therefore, one should not expect a total overlap between findings with VBM and FreeSurfer in the same group of subjects, as it was in this study.


Using FreeSurfer, we found significant differences in cortical thickness of ASD patients over frontal regions (superior, middle frontal regions, pars orbitalis) and temporal lobes (right temporal pole). This finding is consistent with previous reports suggesting that people with ASD have differences in frontal lobe neuronal integrity, function, anatomy, and connectivity. Furthermore, it has been suggested that individuals with ASD have a delay in frontal lobe maturation and that abnormalities in frontal lobe development may underlie some of the social impairments reported in people with ASD (39, 122, 126), which was corroborated by our results.

Cortical surface areas are usually, but not necessarily, increased (as illustrated in **Table 7**) in regions with reduced cortical thickness, which is biologically explained by the consequent increase in sulcation of the cortical mantle (i.e., with atrophy the sulci became deeper, thus increasing the area) (37). Therefore, explaining our finding of increased cortical surface areas coinciding with the regions with reduced cortical thickness described above, as well as in the preand post-central, orbitofrontal, posterior cingulate, inferior parietal, temporal lobes, and insular regions. However, cortical thickness and surface area measurements represent distinct aspects of the cortical architecture and may represent different early neurodevelopmental pathologies (37, 127, 128). Cortical thickness measurements appear to reflect the number of neurons within cortical minicolumns (mainly related to intermediate progenitor cells), while cortical surface area measurements may be related to the number of cortical minicolumns (mainly related to radial unit progenitor cells), according to the radial unit hypothesis (5, 37, 117, 127–129). Our findings suggest that, in addition to the well-documented early brain overgrowth in ASD, there is probably an arrested growth during late childhood, followed by accelerated regionally specific thinning during adolescence and young adulthood. More specifically, the present results complement earlier findings of thinner cortices in adults with ASD (5, 130– 132).

We found increased gyrification in temporal, parietal, and frontal areas in ASD, supporting previous studies that indicate that these are the core areas in ASD and are probably related to abnormalities in visual-spatial attention, selective attention, and visual-motor learning as well as in the mirror neuron system (133, 134). Gyrification represents the amount of cortex within sulcal folds in the surrounding area of measurement and is computed as the ratio between the surface of the outer surface of the brain and the surface of the corresponding area on the GM (pial) surface (37, 95, 129), which reflects an early developing process. It is believed that the brain in ASD goes through a stage of accelerated expansion during early childhood, and consequently, ASD patients are expected to have an increase in cortical folding to accommodate an increasing brain surface into the skull (37, 127). A closer inspection of **Figure 3**, reveals that the areas (representing the points of maximal statistical scores) of reduced cortical thickness, increased cortical surface areas and increased gyrification areas have a similar distribution in our group of young adults and adolescents with ASD.

Abnormalities found in our analysis could be implicated in the core behaviors often impaired in ASD: social and communication (medial frontal region, anterior cingulate) and repetitive and stereotyped behavior (medial and lateral orbitofrontal region).

## Resting-State Functional Connectivity

Findings from most studies have continued to support the broad notion that, overall, individuals with ASD have poorer connectivity in regions spanning long distances in the brain compared to controls, whereas connectivity seems to be increased in local circuits (6, 47). However, findings amongst studies on FC in ASD do not overlap [some with increased (106) and others with decreased (51, 86, 135) connectivity in similar areas], in part due to different techniques used (i.e., seed analyses of predetermined areas, region of interest analyses, etc.) and heterogeneity of patient groups and age range, as occurs with the structural data discussed above (53, 68, 107). Others have reported decreased connectivity of the DMN in adolescents and adult patients with ASD (14, 51, 52, 54, 87), associated with more severe symptoms (135, 136). We found increased connectivity in the ASD group in the right middle frontal gyrus, and a trend for an association between the right temporal pole and left anterior hippocampus FC strength and ADI-R social score, indicating that worse symptom severity was associated with more connectivity in this region. Overall, our results are similar to those observed by Supekar et al. (106) about brain hyperconnectivity predicting symptom severity in ASD. Individuals with greater FC showed more severe social deficits, and they argue that this brainbehavior relationship suggests that aberrant FC may underlie social deficits, which are some of the hallmarks of ASD (28, 106). Our results add to the growing evidence that regional DMN under-connectivity may underlie the pathogenesis of patients' clinical deficits and go further by showing that seed-based analysis reveals the reduction in connectivity also in areas outside the DMN (amygdala, insula, and temporal pole), supporting that ASD is not only a condition of under- or hyper-connectivity but also of aberrant FC (13, 14, 27, 29, 35, 37, 54–56, 69, 137–140).

#### The Role of the Temporal Pole

We found significant VBM cortical atrophy in ASD individuals when considering age, only in the left temporal lobe, including the left temporal pole. We also observed decreased FC in individuals with ASD between the left temporal pole and the remainder of left temporal and parietal regions. This region lies between the orbital frontal cortex and the amygdala, two of the region's most frequently linked to socio-emotional processing. The temporal pole is highly connected with the amygdala, hippocampus, parahippocampal gyrus, cingulate gyrus, orbitofrontal cortex, and the insula (141, 142). In addition, the temporal pole cortex extends topographically to the insula (ventrally) and the entorhinal cortex (medial-inferiorly) (142). The role of the temporal pole is key for various social and emotional functions, including mentalizing (theory of mind) (56, 66, 141, 143, 144). The impairment of theory of mind abilities is one of the most popular hypotheses about ASD (56, 66, 86, 144–146). Some studies using theory of mind tasks showed temporal pole activation (147–150), which give support to our interpretation of the temporal pole as a key node in ASD and social dysfunctions.

## Overlap Between Functional and Structural Findings

Our findings give further evidence that ASD is a network disorder, as revealed by the structural and functional abnormalities (112, 151). In a similar vein, Honey et al. observed that, although resting-state FC is variable and is often present between regions without direct structural linkage, its strength, persistence, and spatial statistics are nevertheless constrained by the large-scale anatomical structure of the human cerebral cortex (152).

We found no complete voxel overlap of areas of maximal GM reduction and areas of decreased connectivity in our patients, which is expected due to the different anatomical resolution between structural and functional images (original voxel sizes of 1 vs. 3 mm<sup>3</sup> , which became even more discrepant after spatial smoothing) and differences in post processing and analyses. However, close localization of the abnormalities was observed in cingulate gyri of both hemispheres, left parahippocampal gyrus, postcentral gyrus, amygdala, and claustrum, right middle, and superior frontal gyri, temporal pole, and cerebellum. Interestingly, GM atrophy determined by VBM showed a closer anatomical relationship with reduced FC than surface measures by FreeSurfer; particularly in areas with GM reduction in the left hemisphere that correlated with increasing age in ASD patients (middle and superior temporal gyri, parahippocampus, and amygdala/uncus). These differences may be explained by the distinct methods for quantification used by VBM and FreeSurfer, which may also reflect different biological substrates between GM volume vs. cortical thinning and cortical areas as discussed above. Nevertheless, our results support the notion that brain alterations in high functioning autism, although subtle and diffuse, converge into areas of structural and functional changes of higher order multisensory association cortex (58). Also, the lack of close correlation between cortical thickness and FC patterns [as also found in other diseases (100)] indicate that changes in cortical thickness or GM atrophy that are not severe enough to be seen on routine MRIs, do not impact directly on FC patterns. This observation is in line with studies of brain networks showing that structural and functional network communities rarely overlap; i.e., functional modules are not always directly connected anatomically [for review see (153)].

## LIMITATIONS

Limitations of our study include the potential effects of medication, a relatively small sample size that may have reduced statistical power and lack of information about puberty stages. However, the statistical significance of the results after correcting for multiple comparisons was remarkable. Our results cannot be generalized to younger and lower-functioning individuals with ASD since we studied a group that included only high functioning autism.

## CONCLUSION

We found cortical thinning and diffuse GM reduction, more pronounced in the left hemisphere, as well as decreased FC between the left hemisphere and PCC (posterior aspect of the DMN) in patients with high-functioning autism. Reduced cortical thickness in the right inferior frontal lobe correlated with higher social impairment, while thinner cortices in the left precentral and superior frontal regions and thicker cortices in the right temporal pole and posterior cingulated correlated with greater communication impairment.

The combination of these abnormalities might represent a neurobiological pattern of this end of the spectrum of autism disorders, indicating a network disorder and could help explain some of the core behaviors in ASD. We also believe that new techniques, such as cortical thickness measurements and surface morphometry could help to elucidate in more detail the patterns of abnormalities related to age and the neurodevelopmental process.

## AUTHOR CONTRIBUTIONS

AP and BC conducted data collection, data analyses and wrote the manuscript. AC, LP, TdR, IO, PD, and JdC contributed with study design and manuscript preparation and revision. J-CD contributed with study design, supervision, analyses of data, manuscript preparation and revision. FC contributed to study design and organization, supervision, funding, data analyses, manuscript preparation, and revision.

## FUNDING

This study was supported by CNPq (Conselho Nacional de Desenvolvimento Científico e Tecnológico) and FAPESP (Fundação de Amparo à Pesquisa do Estado de São Paulo), grant # 2013/07559-3. Part of this work was performed within the framework of the Laboratory of Excellence LABEX ANR-11-LABEX-0042 of Université de Lyon, within the program Investissement d'Avenir (ANR-11-IDEX-0007) operated by the French National Research Agency (ANR).

## ACKNOWLEDGMENTS

We are grateful to all individuals who underwent MRIs in this study for their helpful cooperation.

## SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fneur. 2018.00539/full#supplementary-material

Supplementary Image 1 | Z-scored average connectivity maps of all seeds from both groups. With (a) we indicate controls' average maps, with (b), patients' average maps. In (1), DMN maps, with seed on the posterior cingulated cortex; in (2) the seeds in the left temporal pole; in (3) with the seed on the left anterior hippocampus; in (4), with the seed on the left amygdala; in (5), the seed on the

interhemispheric medial frontal gyrus. The slices in (1), (2), and (5) were MNI axial: −32, −12, 18, 48, 78, and in (3) and (4) were MNI axial: −26, −12, 18, 48, 78.

Supplementary Image 2 | Areas of gray matter atrophy in voxel-based morphometry influenced by the age in patients with ASD. Gray matter atrophy determined by voxel-based morphometry, p < 0.001 clusters with at least 30 voxels.

## REFERENCES


Supplementary Image 3 | Inflated surface maps showing areas with increased and decrease cortical thickness in ASD compared to controls using ROI analysis. There was an increased thickness in right posterior cingulate (red), and in the right (green) and left lateral orbitofrontal cortex (blue) as well as decreased cortical thickness in the left paracentral (pink), posterior cingulate (yellow), and in the right temporal pole (orange) in the ASD group compared to controls.


cortex in children with autism. Hum Brain Mapp. (2014) 35:567–80. doi: 10.1002/hbm.22188


autism: key regions and temporal slope. Neuroimage (2011) 58:391–400. doi: 10.1016/j.neuroimage.2011.06.040


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Pereira, Campos, Coan, Pegoraro, de Rezende, Obeso, Dalgalarrondo, da Costa, Dreher and Cendes. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Cognitive and Affective Perspective-Taking: Evidence for Shared and Dissociable Anatomical Substrates

#### Meghan L. Healey 1,2 \* and Murray Grossman1,2 \*

*<sup>1</sup> Penn Department of Neurology and Frontotemporal Degeneration Center, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA, United States, <sup>2</sup> Neuroscience Graduate Group, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA, United States*

#### Edited by:

*Argye Hillis, Johns Hopkins Medicine, United States*

#### Reviewed by:

*Sadhvi Saxena, Beth Israel Deaconess Medical Center, Harvard Medical School, United States Yuzheng Hu, National Institute on Drug Abuse (NIDA), United States*

#### \*Correspondence:

*Meghan L. Healey mheal@pennmedicine.upenn.edu Murray Grossman mgrossma@pennmedicine.upenn.edu*

#### Specialty section:

*This article was submitted to Applied Neuroimaging, a section of the journal Frontiers in Neurology*

Received: *08 March 2018* Accepted: *06 June 2018* Published: *25 June 2018*

#### Citation:

*Healey ML and Grossman M (2018) Cognitive and Affective Perspective-Taking: Evidence for Shared and Dissociable Anatomical Substrates. Front. Neurol. 9:491. doi: 10.3389/fneur.2018.00491* Perspective-taking refers to the ability to recognize another person's point of view. Crucial to the development of interpersonal relationships and prosocial behavior, perspective-taking is closely linked to human empathy, and like empathy, perspective-taking is commonly subdivided into cognitive and affective components. While the two components of empathy have been frequently compared, the differences between cognitive and affective perspective-taking have been under-investigated in the cognitive neuroscience literature to date. Here, we define cognitive perspective-taking as the ability to infer an agent's thoughts or beliefs, and affective perspective-taking as the ability to infer an agent's feelings or emotions. In this paper, we review data from functional imaging studies in healthy adults as well as behavioral and structural imaging studies in patients with behavioral variant frontotemporal dementia in order to determine if there are distinct neural correlates for cognitive and affective perspective-taking. Data suggest that there are both shared and non-shared cognitive and anatomic substrates. For example, while both types of perspective-taking engage regions such as the temporoparietal junction, precuneus, and temporal poles, only affective perspective-taking engages regions within the limbic system and basal ganglia. Differences are also observed in prefrontal cortex: while affective perspectivetaking engages ventromedial prefrontal cortex, cognitive perspective-taking engages dorsomedial prefrontal cortex and dorsolateral prefrontal cortex (DLPFC). To corroborate these findings, we also examine if cognitive and affective perspective-taking share the same relationship with executive functions. While it is clear that affective perspectivetaking requires emotional substrates that are less prominent in cognitive perspectivetaking, it remains unknown to what extent executive functions (including working memory, mental set switching, and inhibitory control) may contribute to each process. Overall results indicate that cognitive perspective-taking is dependent on executive functioning (particularly mental set switching), while affective perspective-taking is less so. We conclude with a critique of the current literature, with a focus on the different outcome measures used across studies and misconceptions due to imprecise terminology, as well as recommendations for future research.

Keywords: perspective-taking, empathy, cognitive, affective, emotion, frontotemporal dementia, neuroimaging

## INTRODUCTION

Perspective-taking is a complex and multifaceted sociocognitive process that enables us to recognize and appreciate another person's point of view, whether it be the same or different from our own. Previous work has shown that perspective-taking is closely related to and a key aspect of human empathy, which refers to the ability to internally simulate and adopt the mental states of others. Perhaps unsurprisingly then, both perspectivetaking and empathy are critical in guiding successful social interactions, effective communication, and prosocial behavior. For example, an individual's perspective-taking capacity is known to predict the size of one's social network (1, 2), and empathy is known to predict altruistic giving, prosocial behavior, and overall life satisfaction (3–5). Despite such a fundamental role in today's society, however, there is still much to be learned about the cognitive and neural underpinnings of perspective-taking.

Perspective-taking is sometimes characterized along two dimensions: cognitive and affective. Cognitive perspective-taking may be defined as the ability to infer the thoughts or beliefs of another agent, while affective perspective-taking may be defined as the ability to infer the emotions or feelings of another agent. This distinction between cognitive and affective components raises an important question: are there dissociable anatomic substrates for cognitive and affective perspective-taking? Or, is there an independent perspective-taking module that can be applied to either emotional content or cognitive content?

This line of inquiry has been considered more commonly in the context of empathy, which is frequently divided into cognitive and affective components. Note that while there is agreement that these two different types of empathy exist, the terms themselves are imprecise and a variety of alternatives are offered throughout the cognitive neuroscience literature (6). Here, in this review, we define cognitive empathy as the ability to model the emotional states of others (e.g., "I understand what you feel"). As shown in **Figure 1**, this definition of cognitive empathy makes it tantamount to affective perspective-taking. Other commonly used terms include affective theory of mind and mentalizing, although we stress that these terms are poorly operationalized. Next, we define affective empathy as the ability to share the emotional experience of others (i.e., "I feel what you feel"). Affective empathy may also be referred to as experiencesharing and affect-sharing, among others. Thus, while both cognitive and affective empathy depend upon perspective-taking, the difference between the two processes is based on whether or not an individual not only recognizes but also adopts the other agent's emotion. This concept of affective empathy is related to emotional contagion, which refers to the automatic and primitive process by which observation of emotions in one agent triggers isomorphic emotions in a second agent. When an agent both experiences another's emotions (i.e., emotional contagion) and models them effectively (i.e., cognitive empathy/affective perspective-taking), affective empathy results. See **Figure 1** for a visual depiction of the relationship between emotional contagion and empathy.

These two types of empathy, cognitive and affective, may map onto two components of empathic processing, although there is

FIGURE 1 | A model of the relationship between empathy and perspective-taking. In this model, both perspective-taking and empathy are subdivided into cognitive and affective components. <sup>1</sup>Cognitive perspective-taking refers to the ability to make inferences about others' thoughts and beliefs. <sup>2</sup>Affective perspective-taking is the ability to make inferences about others' emotions and feelings. Affective-perspective taking is thus very closely related to cognitive empathy (illustrated by the dashed box). <sup>3</sup>Cognitive empathy is ability to model another agent's emotions. It may be a prerequisite to affective empathy. <sup>4</sup>Affective empathy results from a combination of cognitive empathy and emotional contagion. Here, the perceiver not only models the other agent's emotion, but also adopts it (i.e., affect sharing). Affect sharing thus distinguishes affective empathy from affective perspective-taking. <sup>5</sup>Emotional contagion refers to the process by which emotions in one agent trigger isomorphic emotions in another agent. Emotional contagion may occur without conscious awareness.

some debate on how these components may interact. One model specifies that cognitive and affective empathy are dissociable: they operate independently and depend on unique neural substrates (6–8). An alternative model suggests that the two are part of a single system and may even operate in sequence, such that one must first recognize the other agent's emotion and identify with it, and then successfully attribute the source of the emotion to the agent and inhibit one's own perspective (9, 10). A more recent model (10) synthesizes these two positions and proposes both shared and unique neural substrates.

To date, it appears that the majority of data, from both functional imaging studies in healthy adults and lesion studies in patients, support the view that cognitive and affective empathy are largely distinct processes (8, 11). For example, Shamay-Tsoory et al. (8) found a behavioral and anatomic double dissociation between cognitive and affective empathy: patients with lesions in the ventromedial prefrontal cortex showed a selective deficit in cognitive empathy and theory of mind while patients with lesions in the inferior frontal gyrus showed selective deficits in affective empathy and emotion recognition.

While there is mostly a consensus that the two types of empathy are in part dissociable processes (6), less discussed is whether or not there are unique anatomic substrates underlying the two different kinds of perspective-taking. This question is of crucial import as perspective-taking is itself the key process upon which empathy depends. Importantly, the distinction between cognitive and affective-perspective taking here is a fine-grained one: unlike empathy, there is no element of experience or affectsharing in perspective-taking. The primary distinction between cognitive and affective perspective-taking is rather the type of content that the perceiver is modeling. Accordingly, in this review, cognitive perspective-taking is defined as the ability to infer the thoughts or beliefs of another agent, while affective perspective-taking is defined as the ability to infer the emotions or feelings of another agent. See **Figure 1** for the relationship between the types of perspective-taking and empathy.

In addition to the different types of content being modeled in cognitive vs. affective perspective-taking, there are potential differences in how cognitive and affective perspective-taking may relate to or depend upon executive functions. Previous work has demonstrated that affective perspective-taking, as one might imagine, is tightly linked to emotion perception (12, 13). Unknown, however, is what construct(s) cognitive-perspectivetaking is related to. According to Miyake et al. (14), there are three postulated subdomains of executive function: mental set shifting, information updating and monitoring (i.e., working memory), and inhibitory control. Each of these executive functions, which are generally probed by different neuropsychological measures and supported by different brain regions, may play a unique role in cognitive or affective perspective-taking.

Therefore, we ask: what are the neural correlates of cognitive and affective perspective-taking? Are these processes supported by a single neural system or discrete neural systems? To answer these questions, we review neuroimaging studies from healthy adults and from individuals with focal neurodegenerative disease, namely, behavioral variant frontotemporal degeneration (bvFTD). We seek converging evidence for these anatomical findings by investigating if cognitive and affective perspectivetaking demonstrate the same or different relationships with executive functions. Data showing that cognitive and affective perspective-taking have the same relationship with executive function would constitute evidence for a single-system model and data showing that cognitive and affective perspective-taking have different relationships with executive function would constitute evidence for a two-system model. Considered together, our findings will help (1) further our theoretical understanding of perspective-taking, (2) explain individual differences in healthy adults and patterns of impairment in clinical populations, and (3) offer potential targets for interventions designed to enhance perspective-taking behavior.

## PERSPECTIVE-TAKING IN HEALTHY ADULTS

Recently, functional imaging studies have begun to compare and contrast the neural correlates of cognitive and affective perspective-taking, exploring the question of whether or not there is a core module for perspective-taking or if the two processes are largely dissociable. Here, we review only papers that investigate these two processes within a single task. Overall, results seem to indicate that affective and cognitive perspective-taking are related to brain activity in overlapping but separable neuroanatomic networks. For example, Völlm et al. (15) scanned subjects while presenting them with cartoon stories. Following each story, the subject had to indicate which of two pictures showed the main character's next action (cognitive perspective-taking) or which of two pictures showed an action that would make the main character feel better (affective perspective-taking). Despite the difference in outcome measures across conditions, the authors demonstrated common areas of activation in medial prefrontal cortex and temporoparietal junction. However, affective perspective-taking (referred to as "empathic perspective-taking" by the authors and defined as the ability to infer other's emotional experiences) elicited additional activations in paracingulate, anterior and posterior cingulate cortices, and amygdala, while cognitive perspective-taking (referred to as "theory of mind" stimuli by the authors and defined as the ability to attribute mental states to others) elicited additional activations in lateral orbitofrontal cortex, middle frontal gyrus, and superior temporal gyrus. Complementary to these results are the results of Hynes et al. (16). Using short written scenarios, Hynes et al. (16) revealed a differential role of the orbitofrontal cortex in affective vs. cognitive-perspective taking, with the medial orbitofrontal cortex (i.e., Brodmann's areas 11 and 25) preferentially involved in affective perspective-taking. Corradi-Dell'Acqua et al. (17) and Sebastian et al. (18) also demonstrated different patterns of activity in prefrontal cortex when contrasting cognitive and affective perspective taking. For instance, Sebastian et al. (18) collected fMRI while adult subjects were presented with cartoon vignettes. Both cognitive and affective conditions elicited activity in temporoparietal junction, precuneus, and temporal poles, while only affective perspective-taking recruited medial/ventromedial prefrontal cortex (vmPFC). The authors interpret the vmPFC finding as evidence that this region, with its connections to the insula, temporal pole, and amygdala, is well-suited to integrate affective and non-affective information during theory of mind processing. This conclusion mirrors the previous lesion study findings of Shamay-Tsoory et al. (19, 20).

More recently, Bodden et al. (21) collected fMRI while 30 healthy adults completed the Yoni task, adapted from Shamay-Tsoory et al. (22). In the Yoni task, statements are written on the top of the screen about what object character "Yoni" prefers (affective) or is thinking of (cognitive) and the participant's task is to select the correct option. Results indicated that there are both shared and distinct anatomic correlates of cognitive and affective perspective taking. For example, classic theory of mind regions including the superior temporal sulcus/temporoparietal junction and parietal regions in the right hemisphere were associated with both conditions. However, the orbitofrontal cortex, inferior frontal gyrus, and basal ganglia were only associated with affective perspective taking. Schlaffke et al. (23) showed similar results using cartoon picture stories. A direct contrast of affective vs. cognitive perspective-taking associated regions within the prefrontal cortex, posterior cingulate cortex, and basal ganglia with affective perspective-taking. Cognitive relative to affective perspective-taking, on the other hand, revealed precuneus and bilateral temporal lobes. While this may seem to at odds with the results of Sebastian et al. (18), who found the precuneus and temporal pole were engaged in both conditions, the results are actually not inconsistent. Even though there was higher activation in the cognitive condition in the precuneus and temporal lobe in Schlaffke et al. (23), they also demonstrated an overlap in activation with the affective condition.

Finally, Kalbe et al. (24) took a different approach and used repetitive transcranial magnetic stimulation (TMS) to examine cognitive and affective perspective-taking and in particular, the role of the right dorsolateral prefrontal cortex (DLPFC). Healthy male subjects performed a computerized version of the Yoni task while a single train of 900 1 Hz TMS was applied to the right DLPFC to reduce cortical excitability. TMS stimulation produced a selective impairment of cognitive but not affective perspectivetaking, suggesting that the neural networks supporting these processes are functionally independent.

In summary, the functional imaging literature seems to suggest affective perspective-taking may uniquely engage amygdala, basal ganglia, ventromedial prefrontal cortex, and inferior frontal gyrus. Cognitive perspective-taking, on the other hand, may uniquely engage the dorsomedial prefrontal cortex and DLPFC. Both processes may engage the temporoparietal junction and precuneus (25).

## PERSPECTIVE-TAKING IN FRONTOTEMPORAL DEGENERATION

While fMRI can associate patterns of brain activity with ongoing behavior, it is a correlative technique that cannot identify which brain regions are truly necessary for a given task. Therefore, it is important to complement fMRI studies with converging evidence from patient studies. Here, we test the relationship between cognitive and affective perspective-taking by studying bvFTD. bvFTD is a young-onset neurodegenerative disease characterized by executive and social limitations due to progressive atrophy in frontal and temporal regions (26). Loss of empathy and perspective-taking are hallmark features of bvFTD (26) and have been demonstrated through a variety of tasks, including the Interpersonal Reactivity Index (IRI), the Multifaceted Empathy Test (MET), and the Story-based Empathy Task (SET) (15, 27–30). Generally speaking, these tasks, although varying in method and modality, show that patients with bvFTD struggle to accurately infer others' mental states (e.g., thoughts, feelings, intentions) accurately and consequently fail to share their emotions as well. Finally, bvFTD is an appropriate lesion model to study empathy because the patterns of atrophy that are characteristic of the disease include regions that are hypothesized to play an important role in empathy as well.

Since bvFTD is a rare clinical population, there are only a few reports that have contrasted cognitive and affective perspective-taking within a single study. Search terms here included "frontotemporal dementia" and "perspective-taking" or "frontotemporal dementia" and "theory of mind." Studies were then narrowed down to those that contrasted cognitive and affective domains within the same patients. Most of this research focuses on the additional regions that must be engaged particularly when affective mental states are modeled. For example, Cerami et al. (31) administered the nonverbal SET, which was based on the earlier work of Vollm et al. (15). The task requires subjects to identify the correct ending of short comic strips that include intention attribution (i.e., cognitive perspective taking, as defined here), emotion attribution (i.e., affective perspective taking), or causal inference (a control condition). Despite its name, the SET does not actually assess empathy: at no point are subjects queried as to whether or not they shared the emotion of the main character. Thus, it is better described as a perspective-taking task. Results showed that patients with mild bvFTD were impaired in both intention attribution and emotion attribution, but were significantly worse at emotion attribution. Since the tasks were otherwise matched in difficulty, this within-group effect suggests there may be differences between the two processes. Structural imaging data also revealed differences between intention attribution and emotion attribution: although no unique regions were reported for intention attribution, results indicated that emotion attribution was uniquely related to gray matter density in the right amygdala, left posterior insula, and left posterior superior temporal sulcus extending into the temporoparietal junction. Interestingly, the finding that the temporoparietal junction is related to emotion but not intention attribution is inconsistent with the results from the healthy adult fMRI studies, which concluded that the temporoparietal junction is involved in both cognitive and affective perspective-taking. The precuneus, however, was observed for both types of attribution as expected. Importantly, no direct contrast between emotion and intention attribution was performed by Cerami et al. (31), so it remains possible that the difference in activation across conditions was not statistically significant. In confirmation of the healthy adult studies, the authors ultimately conclude that the aforementioned limbic and frontoinsular structures can be used to differentiate the two types of attribution or perspective-taking. Caminiti et al. (32) also found that mild bvFTD patients show an impaired ability to attribute cognitive and affective states to other agents using a similar version of the SET. They also examined how abnormal patterns of brain activity at rest may relate to performance, finding that patients with worse affective mentalizing performance showed weaker functional connectivity between medial prefrontal cortex and the attentional network, as well as reduced coherent activity within executive, sensorimotor, and fronto-limbic networks. These results are consistent with the earlier work of Cerami et al. (31).

There are also two behavioral studies contrasting cognitive and affective perspective-taking in bvFTD, the conclusions of which are well-aligned with the above imaging studies. For example, Torralva et al. (33) report differential cognitive and affective perspective-taking abilities at different stages of disease in bvFTD. Patients were classified as mild or moderate based on clinical disease rating (CDR) scores, with both groups showing impaired cognitive and affective perspective-taking on the faux pas recognition task (34). In the faux pas recognition task, patients read brief stories in which someone unintentionally commits a social faux pas (or not). When a faux pas is identified, patients are asked a question about the first character's intentionality (cognitive perspective-taking) and the second character's feelings (affective perspective-taking). The authors found that patients with mild bvFTD outperformed the moderate group in the cognitive condition, but not in the affective condition. Since affective perspective-taking deficits are present even at early stages of the disease, while cognitive perspectivetaking is preserved, this suggests unique perspective-taking modules for each type of content. Furthermore, while cognitive perspective-taking was correlated with executive function (i.e., mental flexibility as assessed by the Wisconsin Card Sorting Task), affective-perspective taking was not. This suggests (1) there may be a core deficit in affective-perspective taking in bvFTD that is less likely due to executive deficits and (2) since cognitive and affective perspective-taking have different relationships with executive function, they may consist of two systems that are at least partially dissociable. Dodich et al. (35) obtained similar results, but using the SET described previously. The authors showed that mild bvFTD patients were impaired in both conditions relative to healthy controls. A vectorial analysis then showed that the patients were disproportionately impaired on the affective, but not cognitive condition, compared to the basic abilities (i.e., causal inference) condition, again suggesting a relative deficit in affective (but not cognitive) perspective-taking in bvFTD. Taken as a whole, the imaging and behavioral studies in frontotemporal degeneration support the argument that cognitive and affective perspective-taking are at least partially dissociable. This conclusion is based on several lines of evidence: (1) affective perspective-taking in bvFTD can be selectively impaired, (2) cognitive, but not affective, perspective-taking in bvFTD is associated with executive function performance, and (3) limbic and frontoinsular structures are uniquely related to affective perspective-taking.

Finally, while the purpose of this review is to highlight the similarities and differences between cognitive and affective perspective-taking in frontotemporal degeneration, it is important to note that there is a more extensive body of literature on empathy itself in this patient population. For example, the IRI is commonly used to examine human empathy (36) and is often administered in bvFTD, either to the patient him/herself or to a relative or caregiver. The IRI is a 28-item survey that probes 4 domains: perspective taking, fantasy, empathic concern, and personal distress. Rankin et al. (30) examined the IRI in a mixed sample of neurodegenerative disease patients. When patients with bvFTD were analyzed independently, they showed behavioral impairments in both cognitive and emotional aspects of empathy. Results also indicated that global empathy (total score on the IRI) was related to atrophy in the right subcallosal gyrus in the inferior frontal cortex. Eslinger et al. (27) expanded upon these results by investigating the individual subscales of the IRI. The authors reported that the perspective-taking subscale of the IRI was related to right dorsolateral prefrontal cortex, which is consistent with the TMS results of Kalbe et al. (24) mentioned earlier, as well as the temporal pole and subcortical structures including the right amygdala and left caudate nucleus. The perspective-taking score from the IRI was also correlated with executive measures of mental flexibility, consistent with the report of Torralva et al. (33). Many authors argue that the perspective-taking subscale of the IRI is a proxy for cognitive empathy, but a careful item analysis suggests that may be a combination of both cognitive (e.g., "I believe there are two sides to every question and try to look at them both") and affective (e.g., "Before criticizing somebody, I try to imagine how I would feel if I were in their place") items. Indeed, Davis (36) describes perspective taking as the "tendency to spontaneously adopt the psychological point of view others," a definition which would encompass both cognitive and emotional mental states. Finally, regions related to empathic concern (e.g., "I often have tender, concerned feelings for people less fortunate than me") included the right medial frontal cortex. Empathic concern, as the name suggests, is closely related to affective empathy. Therefore, a single instrument is able to yield both a measure of perspective-taking and a measure of empathy. For this reason, it is suboptimal to combine subscales of the IRI to create a global empathy score: each subscale appears to be relatively independent and unique. More recently, Dermody et al. (37) also used the IRI to examine the neural bases of "cognitive" and affective empathy deficits in Alzheimer's Disease (AD) and bvFTD. While there was a cognitive empathy (i.e., IRI perspective-taking) deficit in both AD and bvFTD, there was an affective empathy deficit only in bvFTD. Deficits in bvFTD, but not AD, remained even after controlling for overall cognitive dysfunction. Perspective-taking deficits in bvFTD were related to bilateral frontoinsular, temporal, parietal, and occipital atrophy, while reduced empathic concern was related to left orbitofrontal, inferior frontal, and insular cortices.

## FUTURE RECOMMENDATIONS

Although promising, there are still important caveats to mention about the existing literature on both empathy and perspective-taking. For instance, much of the existing research on these topics uses questionnaire-based measures, such as the IRI (36). While the IRI is extensively used, it is not without methodological concerns. When patients are allowed to self-report, answers are likely biased: Sollberger et al. (38) demonstrated that bvFTD patients overestimate their own empathy on the IRI. Similar results were found by Massimo et al. (39), who showed that patients with bvFTD do poorly when asked to evaluate their own performance on cognitive tests, and by Williamson et al. (40), who asked subjects to predict their performance on tasks of everyday function. Indeed, Eslinger et al. (41) showed that bvFTD patients (termed "social-dysexecutive") overestimate their performance in 10 of 17 social and emotional domains relative to the judgments of their caregivers. Finally, Rankin et al. (42) demonstrated that, when asked to complete self-report questionnaires about their personalities, patients tend to overestimate their positive qualities and minimize their negative qualities. Patients may also misrepresent their own skills or behaviors for other reasons as well, including apathy. Apathy is frequently documented in patients with bvFTD (43) and could interfere with accurate test-taking abilities. The IRI is also often taken on behalf of patients by their family members or caregivers. Such caregiver and informant-based measures are also problematic. Caregivers are significantly burdened by patient disease, particularly when empathy and/or theory of mind are impaired (44). High levels of caregiver stress may prevent objective ratings and lead to negatively skewed results. Caregiver stress may also vary over the time course of the disease, which would affect the reliability of the data. As mentioned above, the IRI could also be improved by designing more precise subscales. For example, the perspective-taking subscale could be divided into scales for cognitive items and affective items only. Furthermore, in its current form, the empathic concern subscale always probes an individual's tendency to commiserate with another's suffering. Empathy, however, is the more general process of sharing another's feelings, whether they are positive or negative. Additional items could be included in the IRI that measure an individual's likelihood to share in another's joy, happiness, or excitement.

In addition to questionnaires, another popular method for assessing empathy and/or perspective-taking includes narrative-based measures. While these methods may be more ecologically valid and do not suffer from the same confounds as questionnaires, narratives and stories are inherently long, which makes them demanding in terms of executive resources. Patients with bvFTD have executive deficits (45, 46), which can potentially confound comprehension. Indeed, some studies have suggested that the impairment of traditional story-based theory of mind tasks may actually be reflective of deficits in working memory, rather than deficits in perspective -taking itself (47, 48).

To address these concerns, future work needs to develop new ecologically valid paradigms that require patients to actively use their perspective-taking abilities. For example, Healey et al. (49) developed a language-based (cognitive) perspective-taking task assessing a patient's sensitivity to the amount of information available to a conversational partner. Unlike narrative-based tasks, resource demands were minimal as patients only had to generate a brief speech sample describing the movement of a target object. Conditions varied depending on perspective-taking demand and how much information was shared with the conversational partner. Results indicated that patients with bvFTD were impaired at this task and that decreased performance was related to gray matter atrophy in medial prefrontal and lateral orbitofrontal cortices. Similarly, instead of using a questionnaire-based metric, Fernandez-Duque et al. (50) used naturalistic stimuli to explore empathy in frontotemporal dementia and Alzheimer's disease. Patients watched videotaped interviews of everyday people discussing emotionally charged events in their lives and answered questions about the interview. This study also demonstrated impaired performance in frontotemporal dementia patients relative to healthy elderly participants. Finally, Baez et al. (51) also highlight the need to use naturalistic stimuli when studying empathy and/or perspective-taking in bvFTD. Baez et al. (51) administered the empathy for pain task (EPT), which uses natural picture stimuli illustrating two individuals in order to assess empathy for another's pain when it is intentional vs. accidental. Following presentation of the picture stimuli, participants were asked to respond to questions in the cognitive domain (e.g., was the action done on purpose?) or affective domain (e.g., how sad do you feel for the victim?) bvFTD patients demonstrated deficits in both the cognitive and affective domains of empathy. The deficit in the cognitive, but not affective, domain could be explained by co-varying for co-existing deficits in executive function, consistent with other reports.

Finally, in examining the differences between cognitive and affective perspective taking, future patients studies need to also examine potential differences in white matter fractional anisotropy. To our knowledge, no study has yet to do this. Similarly, fMRI studies should conduct functional connectivity analyses to see if patterns of network connectivity can differentiate the two types of perspective-taking.

## CONCLUSIONS

To date, research seems to suggest that cognitive and affective perspective-taking are in part dissociable. Functional imaging studies have found both shared (e.g., temporoparietal junction, precuneus) and non-shared neural correlates of cognitive and affective perspective-taking, with limbic and basal ganglia structures uniquely involved in affective perspective-taking. There are also regional differences within the frontal lobe between cognitive and affective perspective taking (e.g., cognitive perspective taking elicits activation in dorsal regions, while affective perspective taking elicits activation in more ventral

the two types of perspective-taking share some cognitive and anatomic substrates. This core perspective-taking module is associated with the temporoparietal junction (TPJ) and precuneus (PCun). Cognitive and affective perspective-taking then diverge into separate components that are functionally dissociable, represented by the two separate boxes. Cognitive perspective-taking, in purple, uniquely engages dorsomedial prefrontal cortex (dmPFC) and dorsolateral prefrontal cortex (dlPFC). Affective perspective-taking, in orange, uniquely engages the amygdala (amyg), basal ganglia (BG), ventromedial prefrontal cortex (vmPFC), and inferior frontal gyrus (IFG).

regions). See **Figure 2** for a visual depiction of our findings. Perhaps more convincing, however, are the data from patients with behavioral variant frontotemporal degeneration. Behavioral and imaging data in this clinical group show unequal impairment in the two domains, differential relationships with executive function, and unique associations with gray matter atrophy, all of which suggest partially dissociable neural systems. However, there are only a handful of these studies to date, so future research needs to continue to explore empathy and perspective-taking in bvFTD. In doing so, studies must be careful to minimize executive demands, which could confound performance, and try to design stimuli that are as ecologically valid as possible. Finally, across the entire perspective-taking and empathy literature, there is a problem with imprecise terminology (e.g., theory of mind, mentalizing, perspective-taking are all used

## REFERENCES


interchangeably; affective perspective-taking is closely related to cognitive empathy), which makes it difficult to compare and contrast across studies (6). Clear operational definitions must be given whenever possible so we can begin to amass a stronger body of evidence regarding these two constructs.

## AUTHOR CONTRIBUTIONS

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

## FUNDING

Funding provided by NS101863, AG038490, AG017586, AG053488, and the Wyncote Foundation.


Neuropsychologia (2006) 44:950–8. doi: 10.1016/j.neuropsychologia.2005.0 8.009


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Healey and Grossman. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Neuroanatomy of Shared Conversational Laughter in Neurodegenerative Disease

Peter S. Pressman1,2,3 \*, Suzanne Shdo<sup>2</sup> , Michaela Simpson<sup>3</sup> , Kuan-Hua Chen<sup>3</sup> , Clinton Mielke<sup>2</sup> , Bruce L. Miller <sup>2</sup> , Katherine P. Rankin<sup>2</sup> and Robert W. Levenson<sup>3</sup>

<sup>1</sup> Division of Behavioral Neurology and Neuropsychiatry, Department of Neurology, University of Colorado Denver, Aurora, CO, United States, <sup>2</sup> Memory and Aging Center, University of California, San Francisco, San Francisco, CA, United States, <sup>3</sup> Berkeley Psychophysiology Center, University of California, Berkeley, Berkeley, CA, United States

#### Edited by:

Christian Gaser, Friedrich-Schiller-Universität-Jena, Germany

#### Reviewed by:

Donna Clark Tippett, Johns Hopkins Medicine, United States Jee Bang, Johns Hopkins University, United States

\*Correspondence: Peter S. Pressman peter.pressman@UCDenver.edu

#### Specialty section:

This article was submitted to Applied Neuroimaging, a section of the journal Frontiers in Neurology

Received: 01 February 2018 Accepted: 30 May 2018 Published: 15 June 2018

#### Citation:

Pressman PS, Shdo S, Simpson M, Chen K-H, Mielke C, Miller BL, Rankin KP and Levenson RW (2018) Neuroanatomy of Shared Conversational Laughter in Neurodegenerative Disease. Front. Neurol. 9:464. doi: 10.3389/fneur.2018.00464 Perceiving another person's emotional expression often sparks a corresponding signal in the observer. Shared conversational laughter is a familiar example. Prior studies of shared laughter have made use of task-based functional neuroimaging. While these methods offer insight in a controlled setting, the ecological validity of such controlled tasks has limitations. Here, we investigate the neural correlates of shared laughter in patients with one of a variety of neurodegenerative disease syndromes (N = 75), including Alzheimer's disease (AD), behavioral variant frontotemporal dementia (bvFTD), right and left temporal variants of semantic dementia (rtvFTD, svPPA), nonfluent/agrammatic primary progressive aphasia (nfvPPA), corticobasal syndrome (CBS), and progressive supranuclear palsy (PSP). Patients were recorded in a brief unrehearsed conversation with a partner (e.g., a friend or family member). Laughter was manually labeled, and an automated system was used to assess the timing of that laughter relative to the partner's laughter. The probability of each participant with neurodegenerative disease laughing during or shortly after his or her partners' laughter was compared to differences in brain morphology using voxel-based morphometry, thresholded based on cluster size and a permutation method and including age, sex, magnet strength, disease-specific atrophy and total intracranial volumes as covariates. While no significant correlations were found at the critical T value, at a corrected voxelwise threshold of p < 0.005, a cluster in the left posterior cingulate gyrus demonstrated a trend at p = 0.08 (T = 4.54). Exploratory analysis with a voxelwise threshold of p = 0.001 also suggests involvement of the left precuneus (T = 3.91) and right fusiform gyrus (T = 3.86). The precuneus has been previously implicated in the detection of socially complex laughter, and the fusiform gyrus has a well-described role in the recognition and processing of others' emotional cues. This study is limited by a relatively small sample size given the number of covariates. While further investigation is needed, these results support our understanding of the neural underpinnings of shared conversational laughter.

Keywords: laughter, communication, neuroanatomy, empathy, voxel-based morphometry

## INTRODUCTION

Laughter is an ancient and universal emotional expression that often supports social connection (1–3). Laughter primarily occurs in social situations (4), and may reflect recognition of a benign transgression against social expectations or norms rather than anything obviously humorous (5). Laughter occurs an average of five times per 10 min of conversation, usually after fairly mundane, rather than obviously humorous, statements (6). Sometimes, however, social laughter may simply be a nearly automatic response to another's laughter (7). Sharing in another's laughter correlates closely with measures of relationship quality, closeness, and social support (8).

While shared laughter can occur in various situations, e.g., while watching a television show, the emphasis of this study is on conversational laughter. Shared conversational laughter involves both laughter production and perception. Neuroanatomically, laughter production involves brainstem structures including the periaqueductal gray and the upper reticular formation (9), which are under the influence of basal ganglia, hypothalamus, premotor cortices, and basal temporal lobes (3), including the fusiform gyrus (10). Recent studies suggest that networks involved with laughter perception may vary depending on the laughter's nature (11–13). Researchers have characterized laughter in various ways, sometimes demonstrating that different types of laughter have distinctive acoustic properties (9, 11, 14–18). One of the more common distinctions is between "voluntary" and "involuntary" laughter (9, 14, 18), with the former being more internally driven, and the latter being more stimulus driven and externally provoked. The neural substrate supporting perception of voluntary, controlled laughter (as is commonly associated with social interaction) may differ from that supporting perception of involuntary laughter (commonly elicited by tickling). Functional MRI (fMRI) studies have suggested that "social" laughter (e.g., taunting or joyful) activated more medial prefrontal cortex and precuneus compared to tickling laughter, whereas tickling laughter predominantly activated the superior temporal gyrus (11). This may be due to the ambiguity of the social signal, which requires stronger engagement of internal mentalizing by the perceiver (12, 13).

Disorders that impact social interactions can also result in altered laughter patterns. For example, in a recent fMRI study, boys with disruptive behaviors had less mutual laughter and demonstrated less neural reactivity within the supplementary motor and bilateral superior temporal cortices when exposed to social laughter (19). Similarly, altered social dynamics are common among those with neurodegenerative diseases (20). Based on these altered patterns of social interaction, we previously identified different patterns of conversational laughter among some patients with neurodegeneration (21). The purpose of our current study was to investigate the neural correlates of shared conversational laughter during naturally occurring conversation among patients with one of a variety of neurodegenerative illnesses (N = 75). Patients with neurodegenerative diseases can serve as a valuable, naturally occurring brain lesion model, in which overlapping regions of volume loss can serve as an indicator of the neural underpinnings of particular behaviors. Each patient with a neurodegenerative disease was seen with a conversational partner in order to assess the frequency of shared laughter.

The neuroanatomy of shared laughter has been studied primarily using task based fMRI studies. While valuable, there are some limitations to the ecological validity of such studies. For example, many studies investigate laughter perception using actor-produced stimuli, which are to some extent all voluntary. Furthermore, classification schema themselves may not account for more ambiguous real-world scenarios. Here, we investigate the neural substrate of shared conversational laughter under conditions of high ecological validity, i.e. seminaturalistic conversations. While shared or "contagious" laughter has sometimes been described as being less volitional than other laughter types, the complexity of most social interactions may require interpretation and understanding of the other's laughter in order to assess the utility of sharing in that laughter (12, 13), in which case shared conversational laughter may be more controlled and associate more with prefrontal cortex and precuneus. Alternatively, if shared conversational laughter is more involuntary, shared conversational laughter may associate more with the superior temporal lobes (19).

## MATERIALS AND METHODS

#### Subjects

Subjects were selected from a standing data repository at University of California, Berkeley's Psychophysiology Laboratory derived from an assessment of emotional functioning that involved multiple emotion-eliciting tasks. The Institutional Review Boards of the University of California, San Francisco, and the University of California, Berkeley, approved the study. All subjects provided informed consent prior to participation.

In order to be included, subjects diagnosed with a neurodegenerative disease had to participate in a conversation (the task of interest) during which their healthy conversational partner laughed at least once. In addition, all subjects had to be part of a group of no fewer than five with a diagnosis of a similar neurological disorder. Only subjects with an MRI scan of the brain within 3 months of the task of interest were included. Seventy-five subjects met these criteria and were included in this research, including Alzheimer's disease (N = 11), behavioral variant frontotemporal dementia (bvFTD, N = 23), right and left temporal variants of semantic dementia [rtvFTD (N = 6), svPPA(N = 10)], nonfluent/agrammatic primary progressive aphasia (nfvPPA, N = 11), corticobasal syndrome (CBS, N = 7), and progressive supranuclear palsy (PSP, N = 7). Combined, these diseases relate to bilateral frontal, parietal, and temporal lobar degeneration, offering a wide range of regions that could support shared conversational laughter. Prior to being assessed at Berkeley, all subjects with a neurodegenerative illness underwent a detailed clinical evaluation, including a physical examination and neuropsychological testing. Following this evaluation, their diagnosis was determined by a panel of experts, including neurologists, neuropsychologists, speech pathologists, and nurses.

Alzheimer's disease was established by National Institute on Aging–Alzheimer's Association criteria, and included amnestic, dysexecutive, and behavioral subtypes (22, 23). Consensus criteria were also used to define corticobasal degeneration (24) and progressive supranuclear palsy (25). Primary progressive aphasias (semantic dementia [svPPA] and nfvPPA) were diagnosed using consensus criteria outlined in 2011 (26). In addition, some subjects were diagnosed with the right temporal variant of frontotemporal dementia (rtFTD) by the expert panel by means of available examination and structural magnetic resonance imaging (MRI) data. Those with bvFTD were required to meet 2011 international criteria for inclusion in the study (27, 28). Clinical characteristics of all research groups are detailed in **Table 1**. Overall, the mean age was 64.3 years, with a mean of 16.8 years of education, mean CDR of 0.8 and mean CDR box of 4.2. Participants were 37.3% female, 92% right-handed, and 94.7% Caucasian. There were no significant demographic differences between groups. Groups did differ in disease severity as assessed by Clinical Dementia Rating (CDR), CDR Box, and Mini Mental State Exam (MMSE) scores, as well as neuropsychological test scores, in the pattern expected for respective diagnoses. While not included in our analysis, basic demographic information was available on 67 out of 75 conversational partners. Spouses or romantic partners comprised 86.6% of conversational partners, with a mean age of 63.6 years (SD 12.3). Conversational partners were 58.2% female. No significant differences in conversational partner demographics were found between groups.

## Task of Interest

All assessments were conducted between 2002 and 2016 as part of a broader study of emotional function in neurological disease. Laboratory procedures for obtaining samples of conversations between patients and caregivers were derived from wellestablished methods (29). Each patient and their conversational partner, usually a family member or close friend, was instructed to discuss a mutually selected topic of continuing disagreement in their relationship in order to evoke emotional reactivity. Each conversation lasted between 10 and 15 min. Audio recordings of the conversations were obtained using unidirectional lavaliere microphones attached to each conversationalist.

## Acoustic Labeling

The audio from conversations was saved in WAV format. All speech and nonspeech sounds (such as laughter) were manually labeled in Praat, an acoustic analysis program (30, 31) by trained research assistants based on their own judgment of what constituted laughter (**Supplementary Figure 1**).

## Measure of Interest

Based on a previously described classification of conversational laughter (32), when a laugh was detected, it was categorized as being related to the partner's laughter if the laugh occurred during or within 3 s following the partner's laughter. Laughter was automatically categorized via a script written for Stata 13.0. The automated categorization of the first 100 laughs collected was compared with a human rater, with 100 percent agreement. All subsequent laughs were categorized using the automated procedure. For each participant, the probability of laughing relative to his or her partner's laughter was calculated as being the number of laughs relating to partner laughter divided by the total number of times the partner laughed.

## MRI Acquisition

All participants with a neurodegenerative disease underwent a structural MRI scan on a 1.5, 3, or 4T Magnetom VISION system (Siemens Inc. Iselin, N.J.) within 3 months of the conversation. T1-weighted whole brain images were obtained using a volumetric magnetization prepared rapid gradient echo MRI sequence (MPRAGE, TR/TE/TI=10/4/300 ms) with 15◦ flip angle, coronal orientation perpendicular to the double spin echo sequence, 1.0 × 1.0 mm<sup>2</sup> in-plane resolution and 1.5 mm slab. Scans were visually inspected for excessive movement prior to inclusion.

#### Image Pre-processing

Voxel-based morphometry (VBM) preprocessing and analysis were performed using the VBM8 toolbox (http://dbm.neuro. uni-jena.de/vbm/) and Statistical Parametric Mapping 8 (SPM8) software (http://www.fil.ion.ucl.ac.uk/spm/software/spm8/). Following bias-correction and tissue-classifications, segmented images were normalized to MNI space with a 1.0 mm cubic resolution using affine and nonlinear transformations via the diffeomorphic anatomical registration through exponentiated lie algebra (DARTEL) method (33, 34). DARTEL was also used to create a customized template based on 300 older healthy controls. Default parameters of the VBM8 toolbox were used in all preprocessing steps except for adding a previously described light clean-up procedure in the morphological filtering step (35). Spatially normalized, segmented, and modulated gray matter images were then smoothed with an 8-mm FWHM isotropic Gaussian kernel.

## Voxel-Based Morphometry Analysis

We used multiple regression design analyses to correlate the probability of a patient with neurodegeneration laughing relative to their partner's conversational laughter with gray matter atrophy across the sample of 75 participants. Age, gender, magnet strength, MMSE (as a proxy for disease severity), and TIV were used as standard covariates. The Geriatric Depression Scale (GDS) was considered as a potential covariate, but ultimately disregarded as GDS did not correlate with the probability of laughing in relation to the partner's laughter, i.e., the primary variable of interest (p = 0.35). Because VBM of neurodegenerative disease can lead to co-atrophy artifact, wherein regions unrelated to a task appear as statistically significantly related due to disease specific co-atrophy patterns, each diagnosis was also parameterized and entered as a confounding covariate in the analysis. This helps to ensure that regions of atrophy are associated with laughter probability only if those associations are present in more than one diagnostic group (36, 37), improving generalizability of results.

A voxel-wise threshold of p < 0.005 was used to threshold the resulting statistical parametric map, which was then corrected for multiple comparisons at p < 0.05 based on cluster extent and a TABLE 1 | Demographic and neuropsychological test scores of all included diagnostic groups.


CDR, Clinical Dementia Rating Score; CDR Box, Clinical Dementia Rating Sum of Box Scores; MMSE, Mini-Mental State Exam Score; CVLT, California Verbal Learning Test -II Score; BNT, Boston Naming Test; and GDS, Geriatric Depression Score. \*signifies between group differences at p < 0.01, \*\*signifies between group differences at p < 0.001.

custom-fit error distribution based on 1,000 data permutations (35). This permutation analysis also helps correct for deviations from parametric data distributions, that was required given a zero-inflated distribution of our data (38). Because we recognized that our sample size was relatively small given the high number of covariates involved, we permitted exploratory analysis at an unadjusted threshold of p < 0.001 should results indicate a region of marginal non-significance at a p < 0.10. SPM T-maps were superimposed on the Montreal Neurological Institute (MNI) single subject brain template using automated anatomical labeling included in the MRIcron software package.

## RESULTS

Across all groups in the 10-min conversation, the median number of laughs was 2, with a range of 0–38. Further information on laughter by group is listed in **Table 2**. The counts and probabilities are generally zero-inflated, with wide variation between individuals. Neither negative binomial regression nor zero-inflated Poisson (correcting for zero-inflation) found differences between groups in any laugh-related measure.

No voxels met the critical T value of 5.84. At a corrected voxelwise threshold of p < 0.005, a cluster in the left posterior cingulate gyrus demonstrated marginal non-significance at p = 0.08 (T = 4.54). We found decreased gray matter at a threshold of p < 0.001 in nine areas: left posterior cingulate gyrus (T = 3.97), left precuneus (T = 3.90), right fusiform gyrus (T = 3.87), right cerebellum (T = 3.63), left middle cingulate (T = 3.47), left supplementary motor cortex (T = 3.37), right posterior cingulate cortex (T = 3.36), left anterior cingulate cortex (T = 3.36), and right inferior temporal gyrus (T = 3.28) (**Table 3**, **Figure 1**).

## DISCUSSION

While they should be considered exploratory, our findings suggest that the probability of laughing due to someone else's laughter in everyday conversation may positively correlate with brain volumes in the posterior cingulate cortex. Additional exploratory analysis implicates the precuneus, right fusiform gyrus, left supplementary motor cortex, and left anterior cingulate, all of which have been previously been implicated with laughter in the past.

Our findings lend support to existing theories of the neural substrate of shared laughter using a methodology with greater ecological validity. Although our findings should be interpreted with caution, a potential role for the implicated regions has previously been supported by functional neuroimaging. For example, among healthy research participants Wildgruber and colleagues found the posterior cingulate and precuneus to be highly involved in the perception of socially complex laughter, such as that which conveys joy or taunting (12). These brain regions have been shown to be involved with networks that support mentalizing processes and theory of mind (39). This suggests that in most conversation, laughing even in response

#### TABLE 2 | Laughter characteristics by group.


Laugh counts and probabilities between populations. Mean and standard deviation are on top, followed by median and range on bottom in bold. Note zero-inflation frequently dropping the median below the mean. Due to high variance, no significant differences were found between any measures after assessing with models incorporating zero-inflation.

TABLE 3 | Neuroimaging correlates between volumes and probability of sharing in laughter.


T-value list for laughter correlates. List of all regions with a positive correlation between the brain volume and probability of laughing during or shortly after the healthy conversational partner's laugh at significance level p < 0.001.

to another's laughter is not necessarily automatic, but rather depends on the evaluation of the laughter's context and social implications prior to a response.

Our results also suggest involvement of the right fusiform gyrus in mutual laughter. The right fusiform gyrus has been widely implicated in the processes of facial and emotional recognition, but also in the more automatic generation of laughter when electrically stimulated (10). Others who have described correlations between laughter perception and the fusiform gyrus have suggested that visual imagery of laughing faces may be elicited by the laughter (40–42). While most of these studies investigated the acoustics of laughter alone, our participants were able to see their partner's face, which may account for some of this correlation.

Our exploratory results also implicate the anterior and middle cingulate gyrus and supplementary motor area (SMA). The cingulate and supplementary motor cortices have been previously associated with production of involuntary vocalizations, such as those associated with emotions (43). The SMA and pre-SMA have previously been correlated with listening to emotional signals, such as laughter, particularly when emotionally complex (13), and while producing related facial movement (44).

Contrary to what one would predict for an anatomy of predominantly involuntary laughter, we did not find involvement of the superior temporal lobes. Similarly, while some studies have correlated shared laughter to the anterior insula, we did not find this in our research. This may be due in part to a methodological limitation. As we co-varied for each disease type, regions that predominantly atrophied with only one disorder were essentially removed from the analysis. For example, anterior insular degeneration is common in bvFTD (20). Due to the relative specificity of this region to that disorder, including bvFTD as a covariate could essentially remove it from inclusion in our findings. Nevertheless, superior temporal lobar degeneration can be involved in a wider array of neurodegenerative diseases. We believe that the relative lack of their involvement here suggests a relatively small role compared to brain regions that mediate a more volitional sharing of conversational laughter.

#### Strengths and Weaknesses

This study is limited by a relatively small sample size given the number of covariates. Thus, results are exploratory, and should be interpreted accordingly. The covariate structure of this VBM was designed to avoid co-atrophy artifact—as discussed, however, this approach may fail to identify regions truly related to expression scores that are atrophied in only one diagnostic group. While this approach does increase plausible generalizability of results by ensuring correlations are present in more than one patient group, we only studied scans from patients with neurodegeneration, not healthy individuals. Other weaknesses include a paucity of information on patients' conversational partners. Furthermore, our experimental design and labeling system only captures audible laughter and does not easily permit exploration of causes of an individual's laughter. For this reason, we could not discern possible contributions of laughter due to phenomena like pseudobulbar affect, which would have a different contributing anatomy. The instruction to discuss a topic of mutual disagreement may have reduced our chances of eliciting laughter—other instructions may have increased the number of laughs in this study and lent even greater ecological validity to our results. This low level of laughter overall has been previously described using the same task. As in that publication, nfvPPA stands out from other groups in conversational laughter production, which may represent a paralinguistic

method of social connection to compensate for an oftenfrustrating apraxia of speech (21). Further studies in other samples would be necessary to confirm and help explain this behavior.

Despite those weaknesses, our results are consistent with previous research on brain regions involved with shared laughter. The use of a task with high ecological validity sheds further light on structures that are most likely to be relevant to sharing in everyday conversational laughter. Our findings support theories that envision this interaction as being less automatic than the response commonly elicited in more narrowly defined task-based designs. In everyday interaction, shared laughter likely depends on a degree of internalization and processing of other socially relevant information.

posterior cingulate gyrus (A–C), precuneus [images (A,C–F,I)], right fusiform gyrus [images (G–L)], and left supplementary cortex [images (B,I)].

#### Future Directions

As with previous studies of laughter in neurodegenerative disease, there is substantial variance in laugh behavior. Future studies may reduce this variance by focusing on longitudinal changes in laughter within individuals. Because shared conversational laughter is associated with higher measures of relationship quality, future studies should also consider exploring how changes in laugh behavior impact the relationship between patients and caregivers coping with neurodegenerative disease, as well as influence caregiver perceptions of behavioral manifestation of neurodegenerative disease as rated on standardized questionnaires and directly reported to physicians.

## CONCLUSIONS

A network including the cingulate cortex, precuneus, fusiform gyrus, and supplementary motor area likely mediates the probability of someone sharing in another person's conversational laughter. These findings are in accordance with and offer further ecological validity to prior models describing the perception and sharing in socially complex laughter.

## AUTHOR CONTRIBUTIONS

PP, KR, and RL Study conception and design. PP, MS, K-HC, SS, BM, KR, and RL Acquisition of data. PP and CM Analysis and interpretation of data. PP Drafting of manuscript. PP, MS, CM, K-HC, SS, BM, KR, and RL Critical revision.

## REFERENCES


## FUNDING

Funding was provided by the American Brain Foundation as well as grant numbers P50AG023501, P01AG019724, and 5R01NS050915-11 from the NIH National Institute on Aging.

## ACKNOWLEDGMENTS

We would like to thank the patients and staff at the UCSF Memory and Aging Center, as well as research assistants at both the Berkeley Psychophysiology Laboratory and the University of Colorado, Denver Anschutz Medical Campus for coding: Collin Barlow, Abbey Dykhouse, Juliet Small Ernst, William Harkin, Michelle Hough, Eric Lemieux, Pooja Mahtre, Lila Murphy, Christine Silver, Tu Tran, Catherine Waymel, and Lily Vittayarukskul. We also thank Dr. Huntington Potter of the Rocky Mountain Alzheimer's Disease Research Center for his support of this publication, and Heidi Chial for edits and comments on this manuscript.

## SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fneur. 2018.00464/full#supplementary-material

Supplementary Figure 1 | A sample praat grid labeling laughter. A depiction of how laughs were labeled for each speaker using Praat. Each speaker was represented by one tier for speech (<sp>), and another for nonspeech sounds (e.g., <laugh>). The sample represents an instance where Speaker 2 laughed following speech by Speaker 1. Speaker 1 then joined in that laughter before resuming speech.

perception of different laughter types. Neuroimage (2010) 53:1264–71. doi: 10.1016/j.neuroimage.2010.06.028


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The reviewer JB declared a past co-authorship with one of the authors BM to the handling Editor.

Copyright © 2018 Pressman, Shdo, Simpson, Chen, Mielke, Miller, Rankin and Levenson. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Affective Empathy in Behavioral Variant Frontotemporal Dementia: A Meta-Analysis

Andrew R. Carr 1,2 \* and Mario F. Mendez 1,2,3,4

index was particularly effected (d = 1.12).

<sup>1</sup> V.A. Greater Los Angeles Healthcare System, Los Angeles, CA, United States, <sup>2</sup> Department of Neurology, University of California, Los Angeles, Los Angeles, CA, United States, <sup>3</sup> Departments of Psychiatry and Biobehavioral Sciences, University of California, Los Angeles, Los Angeles, CA, United States, <sup>4</sup> David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, CA, United States

Background: Empathy deficits are a widely recognized symptom in the behavioral variant frontotemporal dementia (bvFTD), and although several reviews have examined cognitive empathy deficits, there are no meta-analytic studies on affective empathy deficits.

Objective: Identify salience of affective empathy in bvFTD.

Method: A thorough review of affective empathy found 139 possible studies, but only 10 studies included measures of affective empathy and met standardized criteria.

#### Edited by:

Argye Hillis, Johns Hopkins Medicine, United States

#### Reviewed by:

Maria Salsone, Consiglio Nazionale Delle Ricerche (CNR), Italy Guido Gainotti, Università Cattolica del Sacro Cuore, Italy

> \*Correspondence: Andrew R. Carr

connect@drewrcarrphd.com

#### Specialty section:

This article was submitted to Applied Neuroimaging, a section of the journal Frontiers in Neurology

Received: 25 February 2018 Accepted: 22 May 2018 Published: 12 June 2018

#### Citation:

Carr AR and Mendez MF (2018) Affective Empathy in Behavioral Variant Frontotemporal Dementia: A Meta-Analysis. Front. Neurol. 9:417. doi: 10.3389/fneur.2018.00417 Results: BvFTD patients demonstrated a modest impairment compared to controls across all tasks (d = 0.98). Empathic concern as measured by the interpersonal reactivity

Conclusions: This study provides evidence for an increased commitment to observing affective empathy in bvFTD and capturing its role in the disorder.

Keywords: affective empathy, behavioral variant frontotemporal dementia, empathic concern, reactivity index, empathy

## INTRODUCTION

The behavioral variant of frontotemporal dementia (bvFTD) is neurodegenerative disorder that preys upon the social centers of the frontal and temporal lobes. Early in the progression, individuals with bvFTD demonstrate marked socioemotional behavioral disturbances including lack of insight, emotional blunting, and social disinhibition (1). One of the most problematic social changes is their loss of empathy as it deeply impacts their relationships (2). The loss of empathy is one of the five behavioral criteria in the International Consensus Criteria for the diagnosis of bvFTD (1). However, the concept of empathy itself is complicated and continues to be poorly defined in studies of bvFTD.

Empathy can be broadly defined as identifying with other's feeling states (3). More precisely it is an awareness of inhabiting an affect state corresponding to an affect state of another through observing or imagining that other's state (4, 5). This involves multiple affective experiences and includes emotional contagion or "affect sharing" in addition to affective perspective-taking, an extension of mentalizing (6). Thus, empathy is frequently broken down into affective and cognitive components, primarily: affect sharing and mentalizing (7). Given this characterization, we might have better insight into the type of empathy deficits that bvFTD patients demonstrate.

Several recent reviews and meta-analytic studies highlight the role of mentalizing, the basis of cognitive empathy, in bvFTD. Mentalizing or "Theory of Mind" (ToM) involves apprehending the thoughts (cognitive ToM) or feelings (affective ToM) of others. Lesion studies have indicated that ventromedial frontal lesions result in deficits in ToM and cognitive empathy (8). These deficits have been used to contrast bvFTD from Alzheimer's disease (AD). Bora et al. (9) reviewed 30 studies finding ToM deficits in bvFTD patients compared AD patients particularly in recognizing social faux pas. Clearly there is utility in examining cognitive empathy or mentalizing in this population. In a recent review by Henry et al. (10) of studies totalling 312 patients with bvFTD, they found significant difficulty with mentalizing tasks among these patients. More germane to this study, they found that emotion recognition played a salient role in studies despite not capturing affect sharing itself.

Whereas several robust reviews of mentalizing help to illuminate the cognitive impact on empathy, there are relatively few studies examining emotional empathy or affect sharing. Studies can evaluate emotional empathy by gauging aspects of affective empathy or the presence of visceral reactions to others affective states (7). For example, several studies of bvFTD patients indicate a greater level of emotional blunting and callous interactions with loved-ones (11–13).

Deficits in emotional or affective empathy most prominently arises from disturbances in the medial frontal cortex and the anterior insula (8); however, deficits and may also arise from disease affecting bilateral amygdala (14), precentral gyrus (15) orbitofrontal cortex (16), inferior parietal lobule, brainstem, and thalamus (17). Given that bvFTD has early and prominent medial frontal (including anterior cingulate) and anterior insula degeneration, these patients may have a pronounced impairment inn affective empathy. Additionally, affective empathy involves functional connectivity among the ventral anterior insula, orbitofrontal cortex, amygdala, and perigenual anterior cingulate (18). The white matter tracts such as the right uncinate fasciculus lesions may also be problematic in bvFTD empathy (19). Measures of affective empathy frequently come in the form of self or caregiver inventories. A common measure used for empathy is the Interpersonal Reactivity Index (20, 21). The subscale

#### TABLE 1 | Characteristics of the publication.


Comprehensive affect testing system (CATS), interpersonal reactivity index empathic concern scale (IRI-E) and empathy for pain concern rating (EPT-Concern rating).

empathic concern assesses "other-oriented" feelings e.g., one's affective reaction to another's emotions. Previous literature have identified lower levels of empathic concern in bvFTD patients when rated by their caregivers (22–24); however, this is typically denied on self-reports (22). In one study (25) indicated that a reduced capacity for empathic concern in bvFTD is associated with relate decreases in left orbitofrontal cortex, left inferior frontal gyrus, left insular cortex, and the bilateral mid-cingulate gyrus.

Other measures of affective empathy involve direct behavioral tasks or observations. For example, the Picture Viewing Paradigms (26) attempts to capture affect sharing by having participants view an object or scene then report their level of distress or emotionality in response to the task. In another example, Oliver et al. (27), observed that bvFTD patients demonstrated lower levels of shared emotional experience, diminished arousal and more positive valence when viewing negative social scenarios. Finally, tasks with psychophysiological measures have been limited. One notable example demonstrates that bvFTD patients tend to have lower blood pressure than controls when viewing a video of a man completing a disgusting act (28). Across these studies, patients with bvFTD exhibit marked deficits sharing affective states of various stimuli.

We sought to summarize and evaluate the existing studies on affective empathy in bvFTD. The literature on affect sharing and empathic concern in bvFTD is reviewed. This quantitative review provides important point estimates that may clarify the magnitude of affective empathy deficits in bvFTD. Additionally, it can lead to recommendations for further investigation.

## METHODS

#### Literature Search

We conducted a systematic review of the literature by searching the following databases: PubMed, Psych INFO, Web of Science, and Google-Scholar. The search consisted of the following terms: "bvFTD," "bvFTD," "FTD," "empathy," "experiencing sharing," "affective empathy," "prosocial concern," "empathic concern," "empathic motivation," "IRI." The literature search began March 3, 2017 and concluded June 11, 2017.

#### Inclusion Criteria

We chose studies based upon the following criteria: (1) the use of an accepted international consensus criteria for bvFTD (1, 29); (2) the presence of a non-bvFTD comparison group; (3) the

#### TABLE 2 | Demographics of studies.


\*\*\*p < 0.01, \*p < 0.05. <sup>t</sup>Studies measured severity using the Clinical Dementia Rating.

#### TABLE 3 | Effect sizes of studies.


LCI95 indicates the lower limit of confidence interval and HCI95 indicates the higher limit of the confidence interval. NSNot Significant.

presence of statistics necessary for the calculation of effect size, and (4) the use of a measure of affective empathy with a primary focus on emotion sharing. Multifactor empathy measures that incorporated cognitive theory of mind or perspective taking were excluded.

As see in **Figure 1**, the initial search yielded 139 studies across the databases. Only 25 studies met the first three criteria: (1) the use of an accepted international consensus criteria for bvFTD; (2) the presence of a non-bvFTD comparison group; and (3) the presence of statistics necessary for the calculation of effect size. Of those 25 only 10 included a measure of affective empathy with appropriate statistics present. Of those studies not selected empathy was characterized by emotional recognition or cognitive empathy task such as Reading the Mind in the Eyes Task (31– 41). The characteristics of the included studies are included on **Table 1** and the demographic information (**Table 2**).

#### Outcomes

The primary objective of this review is to assess the impact of affective empathy in bvFTD. We include studies reporting the following outcomes. We examined affect matching evidenced by comprehensive affect testing system [CATS; (48)] and mirroring tasks (27). We also examined empathic concern ratings as evidenced by the interpersonal reactivity index empathic concern scale [IRI-E; (20)] and empathy for pain concern rating [EPT-Concern rating (49)]. We also explored outcomes for self and caregiver ratings as well as behavioral tasks.

#### Statistical Analysis

The authors combined the findings from the identified studies using the MetaEasy MS 1.04 Statistical package. Effect sizes were taken from pre-treatment measures in studies involving a repeated measure design.

Given the high likelihood of heterogeneity among the studies, the summary effect and 95% confidence intervals emerged from a random effects approach assuming both random and systematic error vary within the study's effect sizes using the DerSimonian-Laird (DL) approach (50). As such the Cochrane Q statistic tested heterogeneity and the I 2 assessed the variation around the mean effect (51). Publication bias was assessed using Orwin's fail safe N, Egger's regression intercept.

#### RESULTS

**Table 3** presents a study-by-study chart of the effect sizes on emotional empathy task in bvFTD relative to others. First an overall weighted mean effect size was calculated. A negative effect indicated the bvFTD group performed at a reduced ability compared to the reference group whereas a positive one indicated the reverse. The overall random effect DL model indicated a moderate effect size, d = 0.98 95%CI (−1.25, −0.71). Thus collapsed across all studies bvFTD patients are impaired in measures of

emotional empathy compared to other non FTLD groups. **Figure 2** depicts the ranges of the individual studies effect sizes.

However, the analysis yielded significant heterogeneity amongst studies, Q = 30.72, p<sup>q</sup> = 0.001. The I 2 estimate indicated at 64 percent difference in random and systemic error between the studies. This could be attributable to the types of measures used and the variability within bvFTD behavioral presentations. Few clear subdivisions could emerge. When examining solely the effect size of the IRI-Empathic Concern scale a relatively similar pattern immerged. The five studies identified had an overall effect of dDL = 1.12, 95%CI (−1.46, −0.08). However it too had a significant level of heterogeneity, Q = 12.33, p<sup>q</sup> = 0.031. In each of these studies bvFTD patients had a more difficult time with emotional concern than their peers. On the other ratings listed, there was a strong effect for caregiver rated measures of empathy, Z = −1.54, 95%CI (−2.17 to −0.91) but no effect for self-rated empathy, Z = −0.38, 95%CI (−1.01, −0.91).

In terms of their performance on task-based measures of emotional empathy, the results were relatively mixed. On Oliver et al.'s (27) study in which participants view a video and provided a response to indicate their emotional concern and emotional there was a trend toward significance, 95%CI (−1.06, 0.11). However, the Shany-Ur affect matching test there was a strong deficit in the bvFTD group (z = −1.53, 95%CI (−1.92, −1.14).

## DISCUSSION

The present meta-analytic study illuminates the magnitude of affective empathy deficits in bvFTD in the current published literature. This review examined the effects of 10 studies that depict impairments in affective empathy among patients with bvFTD. Given the magnitude of the effect size generated, it is likely that affective empathy plays a large role in the socioemotional alterations that characterize this disease. However, in addition to the paucity of studies, the overall heterogeneity issues across the samples indicate problems with measurement. Despite this, this meta-analysis indicates that the source of the impaired empathy in bvFTD extends beyond deficits in mentalizing to include significant primary deficits in affective empathy.

A lack of affective empathy is a central feature of bvFTD. This disorder is associated with neuropathology in areas of the brain that mediate affective empathy (52). These areas include medial frontal regions such as the ventromedial prefrontal cortex and anterior cingulate gyrus, the anterior insula, and associated areas such as the amygdale and the right anterior temporal lobe as well as corresponding neural networks according to Seeley et al. (53). It is not surprising that one of the main criteria and presentations of bvFTD is with impairments in expressions of empathy and sympathy toward other (54). These behaviors include a spectrum from simple lack of responsiveness to the concerns of loved ones to frank antisocial behaviors leading to trouble with society and the law (55). This metaanalysis supports this pathological and clinical profile of bvFTD.

Both rating scales and behavioral tasks find deficits in affective empathy. The tasks-based studies demonstrate clear problems with bvFTD patients' capacity to connect emotively with the world around them. These studies should be further replicated as they may produce insight to the individual behaviors within affective empathy, such as the lack of reciprocity in communication and the disconcerting prolongation of eye gaze (56). These compliment the robust inventories that speak to the day to day loss of affect connection which is a significant problem for family members of bvFTD (57).

The most frequent task used in the analysis was the IRI Empathic Concern scale. These studies indicated that caregivers generally feel a lack of warmth and connectedness to bvFTD patients. Although this caregiver assessment of empathic concern is only a proxy measure of affect sharing, it does indicate that bvFTD patients fail to convey affective empathy to those who know them. Given that an important evolutionary function of empathy is for prosocial connection (58), bvFTD patients fail to connect with the emotional experience of those they care about. Only the Rankin et al. (24) study failed to reach a significant effect size, which may be attributable to the use of older clinical criteria for bvFTD which has less specific socioemotional elements (29).

The task-based assessments yielded variable results. In the Shany-Ur study (42), the bvFTD patients had difficulty targeting the nonverbal aspects of affect sharing, and in the Oliver et al. study (27), the bvFTD patients did not show a significant effect. These studies differed in the required attention to nonverbal language processes and self-insight. Previous studies have shown that bvFTD patients have difficulty expressing their feeling states and lack the insight to know how well they are connecting to various social prompts (52, 59). They may instead report overlearned or social normed responses to various social situations (60). In other words, in scenarios with an easily detectable emotional prompt, like the Oliver et al study, they may respond typically, whereas when more subtly

REFERENCES


is involved in detecting emotionality from nonverbal aspects, as in the Shany-Ur study, they may disclose deficits in affect sharing.

This meta-analysis discloses several other findings from this research. One major takeaway is the paucity of studies using affective empathy as a core variable. Another finding is that most studies use a proxy measure such as a caregiver report. However, it is clear that direct task-based measures can be very informative in examining affective empathy in bvFTD. In particular, psychophysiological investigations of affective empathy can yield a more direct assessment of affective empathy among patients with bvFTD (12). Further connections to the basic sciences may help those studying early-onset dementias develop new paradigms for assessing socioemotional issues within this population.

As any study, this meta-analytic review is not without its limitations. Although many studies look at various aspects of empathy in bvFTD patients, there were surprisingly few that met all criteria. This is in large part due to the heterogeneity in research studies in this field. Often, exploring empathy within bvFTD patients is a secondary function of larger studies and uses crude measures to undertake such a task. More robust studies should be done to explore this as behavioral features are prominent in the diagnosis of this syndrome. Additionally, the use of a healthy control group excludes the typical comparisons of other dementia syndromes. Again, this was due to the low number of quality studies that involved multiple types of dementia patients. Future studies would do well to explore this more detail.

In conclusion, this review supports the presence of primary deficits in affective empathy among patients with bvFTD. Empathic concern, in particular, is a widely studied and broadly declined function in these patients. Future studies using task-based measures coupled with psychophysiological assessments and neuroimaging analysis would help further clarify this relationship and the brain-behavior mechanisms involved.

## AUTHOR CONTRIBUTIONS

AC participated in manuscript preparation, literature review, analysis, and formatting. MM participated in manuscript preparation, literature review, and analysis.


frontal gyrus versus ventromedial prefrontal lesions. Brain (2009) 132:617–27. doi: 10.1093/brain/awn279


positive stimuli in frontotemporal dementia. Neuropsychologia (2015) 67:14– 26. doi: 10.1016/j.neuropsychologia.2014.11.022


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Carr and Mendez. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Enhanced Positive Emotional Reactivity Undermines Empathy in Behavioral Variant Frontotemporal Dementia

Alice Y. Hua<sup>1</sup> , Isabel J. Sible<sup>2</sup> , David C. Perry <sup>2</sup> , Katherine P. Rankin<sup>2</sup> , Joel H. Kramer <sup>2</sup> , Bruce L. Miller <sup>2</sup> , Howard J. Rosen<sup>2</sup> and Virginia E. Sturm<sup>2</sup> \*

<sup>1</sup> Department of Psychology, University of California, Berkeley, Berkeley, CA, United States, <sup>2</sup> Department of Neurology, Memory and Aging Center, University of California, San Francisco, San Francisco, CA, United States

#### Edited by:

Kathrin Reetz, RWTH Aachen Universität, Germany

#### Reviewed by:

Mario F. Mendez, UCLA David Geffen School of Medicine, United States Janelle Beadle, University of Nebraska Omaha, United States

> \*Correspondence: Virginia E. Sturm virginia.sturm@ucsf.edu

#### Specialty section:

This article was submitted to Applied Neuroimaging, a section of the journal Frontiers in Neurology

Received: 01 March 2018 Accepted: 15 May 2018 Published: 04 June 2018

#### Citation:

Hua AY, Sible IJ, Perry DC, Rankin KP, Kramer JH, Miller BL, Rosen HJ and Sturm VE (2018) Enhanced Positive Emotional Reactivity Undermines Empathy in Behavioral Variant Frontotemporal Dementia. Front. Neurol. 9:402. doi: 10.3389/fneur.2018.00402 Behavioral variant frontotemporal dementia (bvFTD) is a neurodegenerative disease characterized by profound changes in emotions and empathy. Although most patients with bvFTD become less sensitive to negative emotional cues, some patients become more sensitive to positive emotional stimuli. We investigated whether dysregulated positive emotions in bvFTD undermine empathy by making it difficult for patients to share (emotional empathy), recognize (cognitive empathy), and respond (real-world empathy) to emotions in others. Fifty-one participants (26 patients with bvFTD and 25 healthy controls) viewed photographs of neutral, positive, negative, and self-conscious emotional faces and then identified the emotions displayed in the photographs. We used facial electromyography to measure automatic, sub-visible activity in two facial muscles during the task: Zygomaticus major (ZM), which is active during positive emotional reactions (i.e., smiling), and Corrugator supercilii (CS), which is active during negative emotional reactions (i.e., frowning). Participants rated their baseline positive and negative emotional experience before the task, and informants rated participants' real-world empathic behavior on the Interpersonal Reactivity Index. The majority of participants also underwent structural magnetic resonance imaging. A mixed effects model found a significant diagnosis X trial interaction: patients with bvFTD showed greater ZM reactivity to neutral, negative (disgust and surprise), self-conscious (proud), and positive (happy) faces than healthy controls. There was no main effect of diagnosis or diagnosis X trial interaction on CS reactivity. Compared to healthy controls, patients with bvFTD had impaired emotion recognition. Multiple regression analyses revealed that greater ZM reactivity predicted worse negative emotion recognition and worse real-world empathy. At baseline, positive emotional experience was higher in bvFTD than healthy controls and also predicted worse negative emotion recognition. Voxel-based morphometry analyses found that smaller volume in the thalamus, midcingulate cortex, posterior insula, anterior temporal pole, amygdala, precentral gyrus, and inferior frontal gyrus—structures that support emotion generation, interoception, and emotion regulation—was associated with greater ZM reactivity in bvFTD. These findings suggest that dysregulated positive emotional reactivity may relate to reduced empathy in bvFTD by making patients less likely to tune their reactions to the social context and to share, recognize, and respond to others' feelings and needs.

Keywords: facial electromyography, positive emotion, empathy, dysregulation, emotion recognition, frontotemporal dementia

## INTRODUCTION

The behavioral variant of frontotemporal dementia (bvFTD) is a neurodegenerative disease characterized by socioemotional decline (1). Patients with bvFTD exhibit dramatic changes in personality and behavior that lead to functional impairment (2). The behavioral symptoms in bvFTD emerge as neurodegeneration selectively targets the frontoinsula, anterior cingulate cortex, thalamus, hypothalamus, amygdala, ventral striatum, and brainstem—brain structures that together form the salience network, a system that supports emotion generation, interoception, and empathy (1–3). Early atrophy in the frontoinsula and anterior cingulate cortex, key salience network hubs, renders certain emotions more vulnerable than others in bvFTD (4, 5). Although specific negative (e.g., disgust) and self-conscious (e.g., embarrassment) emotions are diminished in bvFTD (6–8), certain positive emotions (e.g., happiness) appear to be relatively intact, if not enhanced (9). Some patients with bvFTD exhibit behaviors such as elevated mood, jocularity, and reward-seeking [e.g., pursuit of alcohol and sweets; (10–13)], symptoms that may reflect positive emotion dysregulation (9). Despite this heightened positivity, patients with bvFTD do not exhibit interpersonal warmth (14) or positive emotional responses to social cues that typically promote empathy, compassion, and prosociality (15–17).

Decline in empathy is a core diagnostic feature of bvFTD and is a symptom that has a profound impact on families and caregivers (1, 18, 19). Empathy refers to the ability to feel, understand, and respond to others' emotions (20, 21). As empathy degrades in bvFTD, patients become less sensitive to others' feelings and needs, impairments that erode even longstanding relationships. Numerous studies have shown that emotion recognition and perspective-taking, forms of "cognitive empathy," are impaired in bvFTD and reflect atrophy in the temporal pole, lateral temporoparietal cortex, and medial prefrontal cortex (22–29). Poor emotion recognition in bvFTD may be due, in part, to impairments in "emotional empathy" (24, 26, 30, 31), an automatic, primitive form of affect-sharing that facilitates emotion recognition (20, 32, 33). During emotional empathy, emotions travel rapidly across individuals via highly conserved visceromotor mirroring systems (20) that include salience network structures such as the frontoinsula, anterior cingulate cortex, midcingulate cortex, and thalamus (34–37). Emotional empathy fosters vicarious affective experience and emotional understanding by allowing individuals to simulate others' internal states. While sharing others' negative emotions can motivate other-oriented behaviors that alleviate suffering, sharing others' positive emotions can create mutual feelings of reward and enjoyment, pleasant feelings that solidify social bonds (38).

In the present study, we examined whether dysregulated positive emotions were associated with empathy impairments in bvFTD. We hypothesized that elevated positive emotional states may make patients with bvFTD less able to feel, recognize, and respond appropriately to others' emotions. Using facial electromyography (EMG), we measured automatic, subvisible facial muscle reactivity in patients with bvFTD and healthy controls as they viewed photographs of negative, positive, and self-conscious emotional faces. Emotional empathy, which is often assessed by measuring participants' reactions to others' physical or emotional pain (39, 40), can also be measured via facial mimicry (41)—the unconscious, rapid imitation of another's facial expressions. Facial mimicry activates emotion generation systems, enhances emotional experience, and facilitates emotion recognition (42–44). We expected that emotional empathy would be impaired in bvFTD due to atrophy in brain structures that support emotion generation, interoception, and emotion regulation.

Although one approach to quantifying emotional empathy impairments in bvFTD is to measure the extent to which patients' facial reactions are blunted yet context-appropriate (i.e., reduced negative facial reactivity to negative faces and reduced positive facial reactivity to positive faces), another approach is to examine the extent to which patients exhibit facial expressions that are intense but not tuned to the socioemotional context. Whereas previous studies of facial mimicry have focused only on the degree to which an observer's expression matches that of another person (31, 45), here we considered whether emotional empathy impairments in bvFTD may relate to dysregulated positive emotional reactivity (i.e., heightened positive emotional reactions to a wide range of emotional stimuli). We expected that patients with bvFTD who exhibited unmodulated positive emotional reactions to a variety of emotional faces would be worse at recognizing others' emotions and be less responsive to the feelings and needs of people they encounter in their everyday lives.

## MATERIALS AND METHODS

#### Participants

Participants included 26 patients with bvFTD recruited through the Memory and Aging Center at the University of California, San Francisco (UCSF) and 25 healthy controls recruited from the community. All participants underwent a detailed clinical interview, neurological examination, functional assessment, and neuropsychological evaluation. Participants completed the neuropsychological testing and diagnostic evaluation in close proximity to the laboratory assessment of emotion (within 4 months for patients and 12 months for healthy controls).

A clinician assessed disease severity using the Clinical Dementia Rating Scale [CDR; (46)]. CDR Total (scores range from 0 to 3) and Sum of the Boxes (CDR-Box; scores range from 0 to 18) scores were calculated for each participant. Higher scores on both CDR measures indicate greater functional impairment. A neuropsychologist assessed cognitive functioning through the Mini Mental State Examination [MMSE; scores range from 0 to 30, with higher scores indicating greater cognitive functioning; (47)] and a comprehensive cognitive battery that included tests of episodic memory (i.e., verbal and visual), executive functioning (e.g., set-shifting, working memory, and fluency), language functioning (e.g., semantic knowledge and confrontational naming), and visuospatial processing.

All patients met consensus research criteria for probable or possible bvFTD (1). The healthy control group underwent an identical neurological and cognitive work-up as the patients and had no history of neurological, psychiatric, or cognitive disorders. The healthy controls had CDR Total and CDR-Box scores of 0 as well as MMSE scores of 26 or above. See **Table 1** for demographic information and cognitive test scores for each group.

## Procedures

Participants came to the UCSF Center for Psychophysiology and Behavior for a laboratory-based assessment of emotion. All participants or their caregivers, when appropriate, provided informed consent to participate in the study. Participants were seated in a comfortable chair in a well-lit experiment room 1.75 m away from a 21-inch computer monitor. A remotely controlled camera recorded the testing session. The experimental procedures were approved by the UCSF Committee on Human Research.

At the beginning of the testing session, the experimenter used alcohol swabs and mildly abrasive pads to prepare each participant's skin for the application of surface electrodes. Two pairs of 4 mm wide Ag/AgCl electrodes were placed over the left Zygomaticus major (ZM; cheek) and left Corrugator supercilii (CS; brow) muscle regions following established facial EMG procedures (48). Whereas ZM contraction (which occurs during smiling) is an index of positive emotional reactivity, CS contraction (which occurs during frowning) is an index of negative emotional reactivity (30, 32). The left side of the face was chosen due to previous work that has shown that the right hemisphere of the brain plays a predominant role in the production of spontaneous emotional reactions (49). During electrode placement, the experimenter asked participants to smile and frown to verify that the electrodes indeed captured observable changes in ZM and CS activity. The electrodes were removed and reapplied if they did not capture the expected facial activity or if the inter-electrode impedance levels, which measure the resistance to direct current and reflect noise from the skin surface, were >10 kOhms. The EMG signals were acquired utilizing BIOPAC hardware (one EMG100C amplifier per muscle type) and software (AcqKnowledge version 4.2).

After EMG sensor placement, participants rated their subjective emotional experience of various positive (i.e., amused, compassionate, love or tenderness, and awe) and negative (i.e., afraid, sad, disgusted, and surprised) emotions on a Likert-type scale (0 = not at all to 4 = extremely). These ratings provided us with measures of participants' baseline subjective emotional experience.

## Emotion Recognition Task

Participants viewed ten photographs of a man displaying various discrete emotional facial expressions. The photographs were selected from the UC Davis Set of Emotion Expressions (50) and included a neutral expression as well as negative (e.g., disgusted, afraid, angry, sad, and surprised), positive (e.g., happy), and self-conscious (e.g., proud, embarrassed, and ashamed) expressions. At the beginning of the task, participants were only instructed to look at the photographs. Each trial was preceded by a 30 s resting baseline period in which participants viewed an "X" on the computer monitor. Participants viewed each photograph for 10 s. All participants viewed the photographs in the same order. After viewing the series of photographs, participants were then shown each photograph again and were asked to identify the emotion of the person in the photograph. They selected their answer from a list of options (i.e., afraid, angry, ashamed, disgusted, embarrassed, happy, neutral, proud, sad, or surprised). All task instructions, questions, and response options were presented visually on the computer monitor and verbally via audio recordings. E-Prime 2.0 software (Psychology Software Tools, Pittsburgh, PA) was used to present the stimuli. One patient with bvFTD did not speak and, thus, did not respond to the multiple choice questions.

#### Measures

#### Baseline Positive Emotional Experience

We computed total positive and total negative emotional experience composite scores by summing participants' baseline positive and negative emotion ratings, respectively.

#### Facial EMG Reactivity

#### **Impedance levels**

First, we calculated the mean impedance levels for ZM (healthy controls: M = 1.89, SE = 0.41 and patients: M = 1.33, SE = 0.21) and CS (healthy controls: M = 2.10, SE = 0.28 and patients: M = 1.75, SE = 0.17), which were well below the targeted 10 kOhms threshold. The groups did not differ in their mean ZM, F(1, 33) = 1.43, p = 0.24, or mean CS, F(1, 33) = 1.12, p = 0.30, inter-electrode impedance levels, which suggested that the electrode placement was adequate and similar for both groups.

#### **Data processing and quality checks**

Second, the EMG raw signals were filtered offline with a Bandpass Blackman 61 filter (28–500 Hz), integrated, and rectified. 100 ms bins were extracted from the integrated max channels during

#### TABLE 1 | Characteristics of participants by group.


<sup>a</sup>The healthy controls received the California Verbal Learning Test- II (16-word list) instead of the Short-Form. Their performance on the 20-min delay was in the average range expected for individuals their age (M = 12.67, SD = 2.14). bvFTD, behavioral variant frontotemporal dementia; MMSE, Mini-Mental State Examination; CDR Total, Clinical Dementia Rating Total score, and CDR-Box, Clinical Dementia Rating Sum of Boxes; IRI, Interpersonal Reactivity Index. Means (M) and standard deviations (SD) are listed for each group, unless otherwise noted.

the last 2 s of the resting baseline and the first 5 s during each photograph. Upon visual inspection, trials in which the raw EMG signals were excessively noisy (e.g., due to poor placement or wire interference) were deleted. The majority of trials for each group were maintained after this first stage of quality checking: 86% of participants (20 patients with bvFTD, 24 healthy controls) had 100% complete trials, and 96% of participants (24 patients with bvFTD, 25 healthy controls) had at least 70% complete trials.

#### **Extraneous movement**

Third, given that patients with bvFTD occasionally fidget during the testing session, we conducted an additional quality check of the EMG data. A research assistant reviewed the video recordings of each trial of the task and took detailed notes of any extraneous movements that might have impacted the EMG signals (e.g., face-touching, talking, facial twitches, sneezes, and coughs). These notes were then used during another visual inspection of the raw EMG signals. At this stage of the data quality review, 100 ms bins that corresponded to moments of extraneous movements (as determined by the videos) were flagged in the dataset. An index of extraneous movement during each trial was calculated by summing the number of 100 ms bins flagged per trial. Within patients, six bins on average were flagged across all the trials, and within healthy controls, one bin on average was flagged across the trials. The sum of flagged bins across trials was calculated for a total extraneous movement score that was used as a covariate in our analyses.

#### **Facial EMG reactivity**

We computed two types of scores for each muscle: (1) peak reactivity scores, which were calculated for each trial (and used in analyses that examined each trial separately) and (2) total reactivity scores, which were calculated across the trials (and used in the correlational behavioral and neuroimaging analyses that examined all trials together). To compute the peak reactivity scores, we followed the following procedure. First, baseline ZM and CS activity were calculated by averaging muscle activity during the last 2 s of the pre-trial baseline period. Second, we calculated the mean activity of ZM and CS during each 100 ms bin of the first 5 s of the trial. We focused on the first 5 s because previous studies of facial mimicry in patients with socioemotional impairments (e.g., autism and schizophrenia) have used trial windows that range from 1 to 7 s (43, 45, 51, 52), and we wanted to ensure we captured the peak muscle response in a narrow time window while still allowing for variable response times across individuals. Third, we calculated reactivity scores for each trial by subtracting each muscle's mean baseline activity level from its activity during each of the fifty 100 ms bins. Fourth, we identified the peak emotional reactivity score for each muscle to capture the maximum response during each trial. See **Figure 1** for an example of one trial. To calculate the total reactivity scores, we

summed the peak reactivity scores for ZM and CS across all of the trials, which captured each muscle's total maximal contraction across the task.

To determine whether extraneous movement influenced facial EMG reactivity, we conducted zero order Pearson correlations between total extraneous movement across all trials and total reactivity scores for ZM and CS. Neither total ZM reactivity, r(42) = 0.18, 95% CI [−0.12, 0.45], p = 0.24, nor total CS reactivity, r(43) = 0.04, 95% CI [−0.25, 0.33], p = 0.78, was associated with total extraneous movement. Thus, extraneous movement did not appear to affect the EMG reactivity scores.

#### Emotion Recognition

We calculated a total emotion recognition score, which was the sum of the correctly identified emotions across all ten trials. We also calculated negative (i.e., disgusted, afraid, angry, sad, and surprised), positive (i.e., happy), and self-conscious (i.e., proud, embarrassed, and ashamed) emotion recognition scores by summing the correct emotion recognition responses across each set of relevant trials. Recognition of the neutral face was also examined.

#### Real-World Empathy

A subset of participants (7 patients with bvFTD and 9 healthy controls) had informants who completed the Interpersonal Reactivity Index (IRI), a multidimensional measure of real-world empathic behavior, in close proximity to the laboratory-based emotions assessment (within 3 months for patients with bvFTD and 13 months for healthy controls). Given that patients with bvFTD typically lack insight into their behavioral and emotional symptoms, informant reports are a valid way to quantify patients' empathic deficits (23, 24). The IRI is composed of four 7-item subscales (24, 53). Each item was coded on a scale from 1 to 5 (scores for each subscale ranged from 7 to 35). We focused on the empathic concern (a subscale that measures emotional responsiveness to others) and perspective-taking (a subscale that measures the tendency to imagine another person's perspective) subscales because they are established measures of emotional empathy and cognitive empathy, respectively (23).

#### Neuroimaging

Forty-three participants (19 patients with bvFTD and 24 healthy controls) underwent research-quality structural magnetic resonance imaging (MRI) within close proximity to the emotional assessment (within 3 months for patients and 12 months for healthy controls). Structural MRIs were acquired on a 3.0 Tesla Siemens (Siemens, Iselin, NJ) TIM Trio scanner equipped with a 12-channel head coil located at the University of California, San Francisco, Neuroscience Imaging Center using volumetric MPRAGE (160 sagittal slices; slice thickness, 1.0 mm; FOV, 256 × 230 mm<sup>2</sup> ; matrix, 256 × 230; voxel size, 1.0 × 1.0 × 1.0 mm<sup>3</sup> ; TR, 2,300 ms; TE, 2.98 ms; flip angle, 9◦ ).

After visual inspection, five scans were excluded due to excessive motion or poor scan quality. Thus, 38 scans (23 healthy controls and 15 patients with bvFTD) were included in the neuroimaging analyses. For preprocessing, Statistical Parametric Mapping version 12 default parameters were employed with the light clean-up procedure in the morphological filtering step (http://www.fil.ion.ucl.ac.uk/spm/ software/spm12/). Structural T1 images were corrected for bias field, segmented into gray matter, white matter, and cerebrospinal fluid, and spatially normalized into Montreal Neurological Institute (MNI) space (54). Default tissue probability priors (voxel size, 2.0 × 2.0 × 2.0 mm<sup>3</sup> ) of the International Consortium for Brain Mapping were used. Segmented images were visually inspected for adequate gray matter segmentation. Segmented images were smoothed with an 8 mm full-width at half-maximum Gaussian kernel.

We used voxel-based morphometry (VBM) to examine the neural correlates of ZM and CS reactivity in the patients. Statistical maps for VBM were examined at p < 0.005, uncorrected. To derive a study-specific error distribution, we ran one thousand permutation analyses to calculate the onetailed T-threshold for correction with multiple comparisons (pFWE < 0.05) using vlsm2 (55). This type of permutation analysis uses a resampling approach for significance testing; a test statistic is compared with the null distribution calculated from the present dataset and is an accurate representation of Type 1 error at p < 0.05 across the entire brain (56). Images were overlaid with MRIcron (http://people.cas.sc.edu/rorden/ mricron/index.html) on a MNI average brain based on the gray and white matter templates used for preprocessing.

#### RESULTS

#### Participant Characteristics

We used analyses of variance (ANOVAs) to compare the groups on age, education, and total extraneous movement. We used a chi-square test to determine whether there were similar proportions of men and women in the patient and control groups. We then used those variables that were significantly different or approached significance (p < 0.10) as covariates in our analyses. We also used ANOVAs to compare the functional (CDR-Total and CDR-Box) and cognitive (MMSE and other neuropsychological measures) status between the groups. Means (M) and standard errors (SE) are presented for each analysis.

The patients with bvFTD and the healthy controls were similar in age, F(1, 49) = 3.59, p = 0.06. The bvFTD group had a greater proportion of men, χ 2 (1, <sup>N</sup>=51) = 5.67, p = 0.02, and fewer years of education, F(1, 49) = 8.49, p = 0.005, than the healthy controls. Thus, we included age, sex, and education in all of our analyses. We also included total extraneous movement, which was higher in the patients with bvFTD than the healthy controls, F(1, 49) = 8.40, p = 0.006, as an additional covariate in relevant analyses.

Patients were in the mild to moderate range of functioning (as indicated by the CDR) and had impaired cognitive functioning on numerous neuropsychological tests in a battery that included tests of verbal memory (California Verbal Learning Test Short Form 10-min recall) and visual episodic memory [Benson 10 min recall; (57)], confrontational naming [abbreviated Boston Naming Test; (58)]; set-shifting (Modified Trails correct lines per minute); working memory (digits backward); semantic fluency; phonemic fluency; and figural fluency. Demographic information and statistical comparisons for neuropsychological measures are presented in **Table 1**.

#### Baseline Positive Emotional Experience

Analyses of covariance (controlling for age, sex, and education) revealed that patients with bvFTD endorsed significantly greater TABLE 2 | Facial muscle reactivity by participant group for each facial expression.


EMG units are mV × 10<sup>6</sup> .

baseline positive emotional experience than healthy controls, F(1, 36) = 11.43, p = 0.002 (bvFTD: M = 6.09, SE = 1.07; healthy controls: M = 2.30, SE = 0.53). They did not differ significantly from the healthy controls in their baseline negative emotional experience, F(1, 36) = 3.52, p = 0.07 (bvFTD: M = 2.73, SE = 1.16; healthy controls: M = 0.3, SE = 0.13).

#### Facial EMG Reactivity

We first examined whether the healthy controls exhibited the expected pattern of facial reactions during the task. Consistent with previous facial EMG studies, in the healthy controls peak ZM reactivity was greater than peak CS reactivity during the positive emotion trial, and peak CS reactivity was greater than peak than ZM reactivity during the negative emotion trials. During the self-conscious trials, peak ZM reactivity was greater than CS reactivity, which is consistent with the fact that there is smiling behavior in the target's facial expression for two of the self-conscious trials (i.e., proud and embarrassed). The patients with bvFTD exhibited atypical reactions to numerous trials, as delineated in the analyses that follow. See **Table 2** for the means and standard errors of each group's peak muscle reactivity during each trial.

We next conducted mixed effects models (with participant as the random effect) to determine whether there were main effects of diagnosis or diagnosis X trial interactions on peak ZM and CS reactivity (controlling for age, sex, education, and total extraneous movement). These analyses revealed a significant diagnosis X trial interaction on peak ZM reactivity, F(1, 487) = 4.54, p = 0.03, but not on peak CS reactivity, F(1, 497) = 1.49, p = 0.22. Next, we used analyses of covariance (same covariates as above) to decompose the significant diagnosis X trial interaction on ZM reactivity. These analyses indicated that patients with bvFTD had significantly greater peak ZM reactivity during the neutral, F(5, 43) = 5.53, p = 0.02, disgusted, F(5, 41) = 7.45, p = 0.009, surprised, F(5, 41) = 4.59, p = 0.04, proud, F(5, 42) = 4.88, p = 0.03, and happy, F(5, 43) = 4.76, p = 0.04, trials than the healthy controls. To ensure that any demographic or behavioral differences between the groups were not influencing our results, we examined the associations between each muscle's peak reactivity and age, sex, education, and total extraneous movement in our mixed effects models. No significant associations emerged; thus, we concluded that these variables played a minimal role in our results.

To further explore whether sex was impacting our results, we excluded four female healthy controls and conducted a followup analysis in a subset of the sample that was sex-matched, χ 2 (1, <sup>N</sup>=45) = 3.39, p = 0.07. We conducted analyses of covariance (same covariates as above) to examine whether we found a similar pattern of heightened ZM reactivity in bvFTD during the neutral, disgusted, surprised, proud, and happy trials. These analyses found that patients with bvFTD continued to have significantly greater peak ZM reactivity during the disgusted trial than the healthy controls, F(1, 38) = 4.50, p = 0.04. Patients with bvFTD also continued to have greater peak ZM reactivity during the surprised, F(1, 38) = 3.12, p = 0.09, proud, F(1, 39) = 2.72, p = 0.11, and happy, F(1, 39) = 2.99, p = 0.09, trials compared to healthy controls though these results fell to trend levels due to loss of power in the smaller sample. Given that these analyses found a similar pattern of enhanced ZM reactivity in bvFTD during numerous trials, it is unlikely that sex differences between the patients and controls accounted for our results.

Finally, because patients with bvFTD reported elevated positive emotional experience before the task began, we also examined whether higher baseline positive emotional experience was associated with greater ZM reactivity. A linear regression (controlling for age, sex, education, and total extraneous movement) across the sample found no association between baseline positive emotional experience and total peak ZM reactivity, r(38) = 0.23, p = 0.17, t = 1.77, β = 0.31, p = 0.09.

#### Emotion Recognition

Analyses of covariance (controlling for age, sex, and education) revealed that patients with bvFTD had lower total emotion recognition scores than the healthy controls, F(1, 43) = 31.59, p < 0.001. Patients with bvFTD were worse than healthy controls at recognizing negative emotional faces, F(1, 44) = 38.75, p < 0.001 (percent correctly recognized in bvFTD vs. controls: 60 vs. 88% for angry, 36 vs. 92% for disgusted, 32 vs. 68% for afraid, 40 vs. 68% for sad, and 52 vs. 100% for surprised) and the positive emotional face, F(1, 44) = 4.33, p = 0.04 (84 vs. 100% for happy in bvFTD vs. controls), but their recognition of self-conscious emotional faces was not significantly impaired, F(1, 43) = 3.63, p = 0.06 (20 vs. 24% for embarrassed, 28 vs. 24% for ashamed, and 58 vs. 100% for proud in bvFTD vs. controls). The patients were also worse at recognizing the neutral face, F(1, 44) = 13.66, p < 0.001 (48 vs. 92% in bvFTD vs. controls).

#### Relationship Between Positive Emotion Dysregulation and Emotion Recognition Impairments

We conducted separate linear regressions to examine whether heightened ZM reactivity predicted worse emotion recognition across the sample. Linear regressions (controlling for age, sex, education, and total extraneous movement) revealed that greater total ZM reactivity across the trials predicted worse recognition of negative emotions, t = −2.09, β = −0.39, p = 0.04, but not worse recognition of positive, t = −0.29, β = −0.05, p = 0.77; self-conscious, t = −1.46, β = −0.23, p = 0.14; or neutral, t = 0.62, β = 0.10, p = 0.54, faces. Total CS reactivity, in contrast, was not associated with recognition of negative, t = 0.36, β = 0.07, p = 0.72; positive, t = 0.56, β = 0.09, p = 0.58; self-conscious, t = 0.18, β = 0.03, p = 0.86; or neutral, t = −1.41, β = −0.21, p = 0.17, faces.

To further investigate the association between positive emotion dysregulation and impaired emotion recognition, we ran a linear regression (controlling for age, sex, and education) to examine whether greater baseline positive emotional experience predicted worse negative emotion recognition. This analysis revealed that greater positive emotional experience at baseline also predicted worse negative emotion recognition, t = −3.54, β = −0.58, p = 0.001.

#### Relationship Between Positive Emotion Dysregulation and Real-World Empathy Impairments

We conducted separate linear regressions across the sample to examine whether heightened ZM reactivity predicted worse real-world empathic behavior as measured by the IRI. Linear regressions (controlling for age, sex, education, and total extraneous movement) revealed that elevated ZM reactivity was associated with worse real-world empathy. Greater total ZM reactivity across the trials predicted lower scores on the empathic concern, t = −3.05, β = −0.88, p = 0.01, and perspective-taking, t = −2.45, β = −0.75, p = 0.03, IRI subscales. See **Figure 2** for scatterplots. Total CS reactivity, however, was not associated with either empathic concern, t = −0.40, β = −0.17, p = 0.70, or perspective-taking, t = −0.25, β = −0.10, p = 0.81.

Given the relatively small sample size for the IRI analyses (n = 16), we also removed the covariates and conducted zeroorder Pearson correlation analyses to confirm that greater total ZM reactivity, but not CS reactivity, was associated with lower IRI scores. The associations that we detected above remained significant without the covariates: greater total ZM reactivity was associated with lower empathic concern, r(16) = −0.94, 95% CI [−0.99, −0.54], p = 0.004, and perspective-taking, r(16) = −0.85, 95% CI [−0.98, −0.12], p = 0.03. Similarly, total CS reactivity was not associated with either empathic concern, r(16) = −0.42, 95% CI [−0.76, 0.10], p = 0.11, or perspective-taking, r(16) = −0.31, 95% CI [−0.70, 0.22], p = 0.24, on the IRI.

To explore whether elevated baseline positive emotional experience also predicted worse real-world empathic behavior, we conducted separate linear regressions (controlling for age, sex, education) in which we tested whether greater subjective positive experience predicted IRI subscale scores. These analyses indicated that baseline positive emotional experience did not predict either empathic concern, t = 0.34, β = 0.18, p = 0.75, or perspective-taking, t = −0.45, β = −0.40, p = 0.67, subscale scores.

## Neural Correlates of Enhanced Zygomaticus major Reactivity

We first conducted a whole-brain analysis in which we compared the patients with bvFTD to the healthy controls (controlling for age, sex, and total intracranial volume) in order to identify regions with significant atrophy. As expected, patients with bvFTD had smaller volume in the insula, anterior cingulate cortex, striatum, and amygdala, among other regions, at the most stringent statistical threshold (pFWE < 0.05). See **Figure 3**.

In our behavioral analyses, the mixed effects models found a significant diagnosis X trial interaction on peak ZM reactivity, which indicated that patients with bvFTD had higher ZM reactivity than the healthy controls during multiple trials. To capture patients' generalized positive responsivity, we calculated a diagnosis (control = 0, patient = 1) X total ZM reactivity (peak reactivity across all trials) interaction term and entered this interaction term as the independent variable in a VBM analysis across the sample. Nuisance covariates included total ZM reactivity across all trials, diagnosis, disease severity (CDR-Box), and total intracranial volume (the total volume of gray matter, white matter, and cerebrospinal fluid volume to take into account differences in head size). Because age, sex, education, and total extraneous movement were not significantly associated with peak ZM reactivity in our behavioral analyses, we did not include these variables as covariates. In order to offset loss of power incurred by correction for multiple comparisons, we masked our analyses to brain regions that have been implicated in emotion generation, empathy, and facial expression: inferior frontal gyurs (pars triangularis), inferior frontal operculum, insula, cingulate, caudate, putamen, pallidum, thalamus, precentral gyrus, amygdala, and temporal pole. As shown in **Figure 3**, many of these regions were significantly affected in bvFTD. We also examined whether there were any brain regions in which larger gray matter volume was associated with greater ZM reactivity in bvFTD. Furthermore, we conducted parallel analyses for CS reactivity and baseline positive emotional experience.

The VBM analysis revealed that smaller volume in the bilateral thalamus and right midcingulate cortex was associated with greater ZM reactivity in bvFTD at the most stringent statistical threshold (pFWE < 0.05). Smaller volume in the right posterior insula, left anterior temporal pole, bilateral inferior frontal operculum, bilateral precentral gyrus, left midcingulate cortex, and right amygdala was also associated with greater total peak ZM reactivity in bvFTD (p < 0.005, uncorrected). See **Table 3** for T-scores and significance levels for all associated regions; **Figure 4** displays statistical maps. No brain regions emerged in which larger gray matter volume was associated with greater ZM reactivity in bvFTD (p < 0.005, uncorrected). There were also no regions associated with CS reactivity or baseline positive emotional experience in bvFTD at this threshold.

#### DISCUSSION

Our results suggest that atrophy in emotion-relevant brain structures underlies emotional empathy impairment in bvFTD and that a propensity for positive emotional states relates to patients' reduced sensitivity to the feelings and needs of others. Using facial EMG, we found that patients with bvFTD had heightened ZM reactivity in response to various types of emotional faces. Patients with bvFTD not only had greater ZM reactivity than healthy controls while viewing a positive (i.e., happy) face but also while viewing negative (i.e.,

The mask for our VBM analysis in red. (C) An overlay of both (A,B).

disgust and surprise), self-conscious (i.e., proud), and neutral faces. In contrast, patients with bvFTD did not differ from healthy controls in their CS reactivity during any trial. In addition, greater total ZM reactivity, but not CS reactivity, was associated with worse negative emotion recognition and real-world empathy. Furthermore, baseline positive emotional experience was heightened in bvFTD compared to healthy controls, an elevation in positive affect that also predicted worse negative emotion recognition. Taken together, these results suggest that both phasic (i.e., ZM reactivity) and tonic (i.e., baseline positive emotional experience) positive emotional responding may be dysregulated—and socially maladaptive—in bvFTD.

## Enhanced Positive Emotions May Undermine Negative Emotion Recognition and Real-World Empathy

Positive emotions confer numerous benefits such as facilitating approach behavior and fostering social connections (38). Dysregulated positive emotions—positive emotions that are too intense or are context-inappropriate—can be problematic, however, and lead to behavioral symptoms (9, 11, 12). In healthy adults, individuals with lower levels of self-reported emotional empathy exhibit greater ZM reactivity to negative (e.g., angry) faces than those with higher emotional empathy (59). In bipolar disorder, a disorder characterized by intermittent periods of mania [a phase defined by positive emotion dysregulation, inappropriate interpersonal boundaries, and risk-taking; (60)] and euthymia (a phase marked by the absence of manic symptoms), empathy may vary across the clinical course. Whereas, during euthymia, individuals with bipolar disorder exhibit typical, or even enhanced, emotion recognition, during mania they have difficulty identifying negative emotions in others (61). In bvFTD, heightened positive emotional reactivity may increase patients' pursuit of rewards and interest in certain types of humor (62, 63) but decrease their sensitivity to feel, know, and respond to others' feelings (15, 24, 64–66).

## Atrophy in Emotion-Relevant Brain Structures Alters Emotions and Empathy

The neuroimaging analyses found that smaller gray matter volume in the right midcingulate cortex (anterior and posterior divisions) and bilateral thalamus was associated with greater total ZM reactivity in bvFTD. The cluster in the thalamus included the medial pulvinar nucleus and extended into the vicinity of the parvocellular part of the mediodorsal nucleus and ventral posterior lateral nucleus, among others (67). At less stringent statistical thresholds, the left midcingulate cortex, bilateral inferior frontal gyri, bilateral precentral gyri, right amygdala, left temporal pole, and right posterior insula also emerged as regions in which smaller volume was associated with greater total ZM reactivity in bvFTD.

The thalamus is a key hub in afferent pathways that receive viscerosensory information and efferent pathways that support skeletomotor control (68–71). Disruption of thalamocortical loops in bvFTD, therefore, may have a significant effect on emotions and empathy (36, 40, 72–75). The temporal poles, which are critical for appraising the meaning of socioemotional stimuli (76), communicate with the amygdala, a region that is tightly connected with the medial pulvinar nucleus of the thalamus. This system promotes rapid processing of salient visual stimuli (including emotional faces) as well as other incoming sensory information (77, 78). Interoceptive signals from the visceral organs are also relayed to the medial pulvinar, the



Smaller volume in bilateral thalami, bilateral midcingulate cortex, bilateral precentral gyri, bilateral inferior frontal operculum, left anterior temporal pole, right amygdala, and right insula was associated with a greater interaction effect between peak zyomaticus major reactivity across all trials and the bvFTD diagnosis when controlling for peak zyomaticus major reactivity across all trials, diagnosis, functional status (CDR-Box), and total intracranial volume. Montreal Neurological Institute coordinates (x, y, z) given for maximum T-score for the cluster (cluster size > 150 mm3). Results are significant at p < 0.005 uncorrected.

Results considered significant at p < 0.005 uncorrected

\*denotes the cluster significant at pFWE < 0.05 †

signifies that these regions were included in the cluster above

mediodorsal, and ventral posterior lateral nuclei from brainstem centers (67, 72, 79). This afferent pathway, which has connections to the midcingulate cortex, posterior insula, and the salience network more broadly (69, 70, 72, 80), relays internal signals from the body to the brain and is critical for processing negative, noxious, and painful stimuli ((39, 73, 81). Dysfunction in this system in bvFTD may reduce patients' access to physiological changes that typically accompany shared emotional experiences that motivate empathic and prosocial actions (15).

Although dysfunction in interoceptive pathways may dampen empathy by diminishing negative emotional experience, empathy may also falter as patients are no longer able to suppress positive feeling states, especially in inappropriate contexts. The neuroimaging analyses also indicated that atrophy in regions that produce and regulate facial movements (52) that occur during voluntary facial imitation and spontaneous facial mimicry [e.g., inferior frontal gyrus and primary motor cortex; (52, 82–84)] was associated with greater ZM reactivity in bvFTD. Atrophy in the inferior frontal gyrus, a region with a critical role in behavioral and cognitive control (85, 86), may lead to dysregulated emotions in certain contexts. We speculate a combination of diminished reactivity to certain negative emotional cues (6, 7, 87) and enhanced sensitivity to certain positive emotional cues (9, 87) may make patients with bvFTD less able to tune their reactions to the social context and less likely to display empathic responses to others in need.

#### LIMITATIONS

The present study has several limitations that should be considered. First, we used a variety of emotional faces as stimuli negative, positive, self-conscious, and neutral—but our ability to assess empathy for positive emotions was limited because we only included a single positive (happy) emotional face. Given that we detected high ZM reactivity in bvFTD, an alternate explanation for our results is that patients emotional empathy for positive affective states (e.g., happiness and pride) is enhanced, but we believe this is unlikely. Sharing others' positive emotions fosters close relationships (88), but interpersonal functioning declines significantly in bvFTD (38). Happiness is often the only positive emotion that is assessed in empathy research (45, 49, 51, 52, 59), and it will be important for future studies to investigate how patients with bvFTD respond to other positive emotions, especially those that arise in social contexts [e.g., compassion and affection; (89, 90)]. One previous study found that when the lens on emotion recognition was widened to include patients' identification of numerous positive, negative, and self-conscious emotions, patients with bvFTD demonstrated widespread impairment on all tested emotions, regardless of valence (29). We hypothesize that a similar pattern would emerge for emotional empathy and that, despite enhanced ZM reactivity to the happy face in this study, emotional empathy in social situations that typically evoke shared positive feelings would be impaired in bvFTD.

Second, although previous research has found diminished negative emotional reactivity in bvFTD using facial EMG (31), we did not find impairments in CS reactivity in the present study. While it is possible that static negative emotional photographs were not intense enough to elicit a measurable CS response, previous studies have successfully used photographs to activate the CS muscle (49, 52, 91). In addition, the controls in our study displayed the expected CS response to the negative faces. We speculate that a proclivity for positivity in bvFTD interferes with context-appropriate empathic responding and, thus, may have obscured CS deficits in this sample. Tonic elevations in positive mood or affect may also predispose patients with bvFTD toward positive reactions to emotional—as well as non-emotional stimuli. If patients were presented with more ecologically valid emotional stimuli that unfolded over time—as emotional events occur in the real world—we hypothesize that patients with bvFTD would have impaired emotional empathy for others' negative as well as positive affective states.

Third, well-established models of empathy have proposed that emotional empathy occurs automatically and promotes cognitive empathy and prosocial actions (33, 92). Consistent with this framework, we examined whether facial EMG reactivity predicted emotion recognition. It is also possible, however, that patients' decline in emotion recognition underlies their

facial EMG reactivity alterations and enhanced positive reactions to others' negative emotional states. Patients who have a poor understanding of the social world and others' emotions may be less likely to mirror the emotions of those around them. Future longitudinal studies of bvFTD that investigate the earliest manifestations of empathy disruption would be important for further elucidating how cognitive empathy and emotional empathy interact and decline. Studies of individuals with mutations in C9ORF72, a genetic form of bvFTD that targets the medial pulvinar nucleus of the thalamus (93, 94), may help to shed light on this question given the critical role of the thalamus in emotional empathy.

## CONCLUSION

Emotional empathy is a tuning process during which an individual mirrors and shares the emotions of another person (20, 21, 33). In bvFTD, a disease in which there are profound alterations of emotions and empathy, dysregulation of positive emotions may make patients less able to share, recognize, and respond to the varied affective states of other people.

## ETHICS STATEMENT

This study was carried out in accordance with adherence to generally accepted practices for experimental research with human subjects. The protocol was approved by the UCSF Committee on Human Research. All subjects or legal guardians gave written informed consent in accordance with the Declaration of Helsinki.

## AUTHOR CONTRIBUTIONS

AH and VS: developed and designed the study; AH and IS: completed data collection; AH and VS: conducted analyses and interpretation of the data; AH, VS, IS, DP, KR, JK, HR, and BM: drafted or revised the manuscript; VS, BM, HR, JK, and KR: acquired financial support for the project leading to this manuscript.

## ACKNOWLEDGMENTS

This project was supported by grants from the NIH National Institute on Aging (P50AG023501, P01AG019724, R01AG052496, R01AG032306, R01AG057204, 1K23AG040127, and 1K23AG045289), and The Larry L. Hillblom Foundation (2013-A-029-SUP and 2005/2T). Time for this research was also supported by the National Institute of Mental Health (MH020006-16A1). We are grateful for the patients, healthy controls, and families that have participated in our research studies.

## REFERENCES


emotional state on the detection of change in emotional expression. Cogn Emot. (1988) **15**:853–864. doi: 10.1080/02699930143000194


neurodegenerative diseases. Neuropsychologia (2009) **47**:2812–27. doi: 10.1016/J.NEUROPSYCHOLOGIA.2009.06.006


C9orf72 hexanucleotide repeat expansion. Brain (2014) **137**:3047–60. doi: 10.1093/brain/awu248

94. Lee SE, Sias AC, Mandelli ML, Brown JA, Brown AB, Khazenzon AM, et al. Network degeneration and dysfunction in presymptomatic C9ORF72 expansion carriers. NeuroImage Clin (2017) **14**:286–297. doi: 10.1016/j.nicl.2016.12.006

**Conflict of Interest Statement:** KR is a co-author on this manuscript and an editor for this invited issue; thus, KR will not handle any editorial duties for this paper.

The other authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Hua, Sible, Perry, Rankin, Kramer, Miller, Rosen and Sturm. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Ventromedial Prefrontal Cortex Is Critical for Helping Others Who Are Suffering

#### *Janelle N. Beadle1 \*, Sergio Paradiso2 and Daniel Tranel3,4*

*1Department of Gerontology, University of Nebraska at Omaha, Omaha, NE, United States, 2Private Practice in Psychiatry and Psychotherapy, Catania, Italy, 3Department of Neurology, Roy J. & Lucille A. Carver College of Medicine, University of Iowa, Iowa City, IA, United States, 4Department of Psychological and Brain Sciences, University of Iowa, Iowa City, IA, United States*

Neurological patients with damage to the ventromedial prefrontal cortex (vmPFC) are reported to display reduced empathy toward others in their daily lives in clinical case studies. However, the empathic behavior of patients with damage to the vmPFC has not been measured experimentally in response to an empathy-eliciting event. This is important because characterizing the degree to which patients with damage to the vmPFC have lower empathic behavior will allow for the development of targeted interventions to improve patients' social skills and in turn will help family members to better understand their impairments so they can provide appropriate supports. For the first time, we induced empathy using an ecologically-valid empathy induction in neurological patients with damage to the vmPFC and measured their empathic emotional responses and behavior in real time. Eight neurological patients with focal damage to the vmPFC were compared to demographically-matched brain-damaged and healthy comparison participants. Patients with damage to the vmPFC gave less money in the empathy condition to a person who was suffering (a confederate) than comparison participants. This provides the first direct experimental evidence that the vmPFC is critical for empathic behavior toward individuals who are suffering.

Keywords: empathy, ventromedial prefrontal cortex, financial decision making, prosocial behavior, lesion study

## INTRODUCTION

Daily we encounter people who are suffering—strangers living on the street; friends suffering from cancer who can't pay their hospital bills; family members who have lost their homes to a fire. Traditional economic theories purport that we are rational actors who behave in ways that maximize our monetary gain, and therefore would be unlikely to donate to others in need (1). Yet, when people are asked to make financial decisions in daily life, researchers find that emotion (e.g., anger), not just rational thought, impacts our financial decisions toward others (2). A striking example of this can be seen in laboratory settings when people play economic decision making games, such as the Ultimatum Game (UG). When participants receive an offer that is perceived to be unfair, it is thought to elicit anger which in turn leads them to reject that offer, despite the negative financial impact of this choice (3).

From a neuroscience perspective, our financial decisions are thought to be guided by interacting brain systems involving cognition, emotion, and decision making (2, 4, 5). In fact, patients who have brain damage to a region implicated in decision making, the ventromedial prefrontal

#### *Edited by:*

*Argye Hillis, Johns Hopkins Medicine, United States*

#### *Reviewed by:*

*Clifford I. Workman, University of Chicago, United States Tino Prell, Friedrich Schiller Universität Jena, Germany*

> *\*Correspondence: Janelle N. Beadle jbeadle@unomaha.edu*

#### *Specialty section:*

*This article was submitted to Applied Neuroimaging, a section of the journal Frontiers in Neurology*

*Received: 03 December 2017 Accepted: 16 April 2018 Published: 25 May 2018*

#### *Citation:*

*Beadle JN, Paradiso S and Tranel D (2018) Ventromedial Prefrontal Cortex Is Critical for Helping Others Who Are Suffering. Front. Neurol. 9:288. doi: 10.3389/fneur.2018.00288*

**96**

cortex (vmPFC), have difficulty making advantageous financial decisions (6, 7). In other words, their decisions result in financial outcomes that are poorer than that of healthy adults.

Clinical case studies demonstrate that patients with damage to the vmPFC have a reduced capacity to make decisions, ranging from minor decisions about choosing a restaurant, to major decisions about monetary investments (8–10). Furthermore, laboratory-based research studies show that patients with damage to the vmPFC have difficulty on multiple tasks measuring financial decision making (6, 11–14, 64). For instance, they have difficulty learning which decks are financially advantageous in the Iowa Gambling Task, and consequently achieve less overall financial gain than healthy comparison participants (11–13). In the UG, patients with damage to the vmPFC reject unfair offers at a higher rate than healthy comparison participants (6). This results in the patients obtaining less money overall than healthy adults. Based on these studies, researchers have hypothesized that patients' decision making difficulties may derive from a reduced ability to utilize emotional information to guide decision making in an advantageous manner, as described by the somatic marker hypothesis (15–18).

Despite extensive research on financial decision making behavior in patients with damage to the vmPFC, we do not know how they behave in financial contexts where they witness another person who is suffering. This is an important question because many of our financial decisions occur in a social context. For example, a family member may need extra financial support if they develop a chronic illness, such as dementia. A long tradition of psychology and neuroscience research has characterized the behavior of healthy adults when they witness another person's suffering (19–21). Research has shown that an antecedent to motivate someone to help another person is a perception or awareness that the person is in need of help (22). For instance, while there are situations that may evoke empathic joy toward others, such as when a best friend gets offered their dream job, this type of situation is not likely to elicit help because the person is not in need. Furthermore, extensive research has shown that feelings of empathy also motivate people to help others when they perceive them to be in need (20, 23–26).

Empathy is thought to be made up of two components: (1) cognitive—one's ability to understand others' thoughts and emotions, and (2) emotional—one's ability to feel compassion and sympathy for the person in need or feel similarly to them (27). Individuals who experience high levels of empathy tend to show greater helping behaviors toward others in need than those experiencing low levels of empathy (20). Based on this body of research, the empathy-altruism hypothesis was developed which purports that empathic emotion is one mechanism for helping behavior toward others in need (23, 26).

Functional neuroimaging studies point to a broad network of brain regions involved in empathy, such as the vmPFC, amygdala, anterior cingulate, and anterior insula (28–31). Although functional neuroimaging studies provide important information about brain networks involved in empathy, lesion studies are able to determine which regions are critical for empathy to occur. There is a growing body of patient studies examining the degree to which lesions to regions including the anterior cingulate, insula, and amygdala affect empathy [(32–34); for review see Ref. (35)]. However, because only a small number of studies have investigated these regions using varying methodologies, currently there is no conclusive evidence that these regions are critical for empathy. In comparison, there is a long history of clinical and experimental research implicating the importance of the vmPFC for empathy (36–38). Therefore, due to the current state of the literature, we chose to focus on the vmPFC, because there is more substantial and consistent evidence that it is important for empathy.

Clinical case studies have shown that patients with damage to the vmPFC behave in ways that suggest they have reduced empathy toward others (39–41). However, these findings have not yet been demonstrated in a controlled, experimental context where participants with damage to the vmPFC show lower empathic behavior than healthy adults in response to an empathy-eliciting context. Furthermore, it has not yet been experimentally tested whether participants with damage to the vmPFC have reduced awareness of empathic information, reduced empathic emotion, or reductions in both domains in comparison to healthy adults. For empathic behavior towards others to occur, it is often motivated by both an awareness that the other person is in need and the experience of empathic emotion (22). If one or both these aspects are missing, the individual may exhibit lower empathic behavior. Therefore, assessing both the patients' awareness of empathic information and their empathic emotion may aid in understanding potential motivations for their empathic behavior.

The information generated in the present study is crucial in designing effective interventions to improve social functioning in patients with damage to the vmPFC, because it will help clinicians to target the cognitive or emotional domains that are reduced in patients with damage to the vmPFC. If only their empathic behavior is lower than healthy adults, this can be targeted with behaviorally focused social skills training. If they are lower on their awareness of perceiving empathic information from empathy-eliciting contexts, they could receive training on how to determine when a situation is likely to evoke empathy in others. If they are lower on feelings of empathy, they could receive training on techniques to increase one's empathy, such as imagining what the other person may be feeling. Furthermore, this is also important information for the patients' family and caregivers because it will help them to better understand what social skills might be most difficult for the patients, so they can provide appropriate support. Therefore, the present study addresses a gap in the knowledge by experimentally investigating empathic behavior, empathic feelings, and awareness of empathic information in response to an empathy-eliciting context.

For the first time, the current study directly examines how patients with damage to the vmPFC behave in a financial context when exposed to someone who is suffering. The study uses a novel, ecologically valid empathy induction designed to represent a real-world scenario that would be likely to induce empathy. Furthermore, converging methods were used to assess empathy and financial decision making towards a man who is suffering. Specifically, these methods included (1) behavior—measure of financial decision making toward a suffering individual, (2) emotional response—real time patient self-reports of empathic emotion toward the suffering individual, (3) trait empathy—selfreport assessing general tendency toward empathy in daily life completed by the patient (and patients' family member), and (4) theory of mind—ability to accurately understand and assess others' feelings and intentions.

The target group included eight neurological patients with focal damage to the vmPFC who were compared to a brain-damaged comparison (BDC) group and a healthy, normal comparison (NC) group. To reduce demand characteristics, participants were told that they would be playing an economic decision making game. During the course of the study, there was a neutral condition where the participant would overhear their opponent through the intercom talking about unemotional events from their day (e.g., playing a card game and reading the newspaper). The key target empathy induction condition involved the participant overhearing through the intercom system a second opponent discussing the anniversary of their son's death and their grieving process. Empathic behavior was measured implicitly by how much money they gave to each opponent on the economic game (i.e., empathy versus neutral condition). To measure in the moment self-report ratings of empathy in response to the empathy induction, participants completed a mood questionnaire before and after each induction condition. This questionnaire measured empathy, in addition to other relevant emotions (e.g., sadness, hostility, joviality, and personal distress). At the end of the study, participants also completed a theory of mind task where they were asked to assess the intentions and feelings of others through written scenarios. Finally, participants completed a questionnaire measuring empathy as a general tendency across the lifespan which was also completed by their family members, as a means of corroboration.

It was hypothesized that patients with damage to the vmPFC will show significantly lower empathic behavior in response to an empathic induction in which they witness another person's suffering than comparison groups. Furthermore, it was hypothesized that patients with damage to the vmPFC will show less empathic emotion than comparison participants in response to an empathy induction.

#### MATERIALS AND METHODS

#### Participants

Target participants included eight patients with focal damage to the vmPFC (see **Figure 1**). These patients were compared to NC (*N* = 8) and BDC (*N* = 8) groups. Comparison participants were matched to the target patients on age, education, gender, and full scale intelligence. All groups included five females and three males.

Kruskal–Wallis tests were used to compare age and education across the three groups. To compare chronicity between the BDC and vmPFC groups, a Mann–Whitney *U* test was used, and a Chi-square test was used to compare the two groups on type of etiology. In the present study, there were 19 statistical tests performed that were not testing *a priori* hypotheses. Therefore, we applied a false discovery rate correction for these tests (false discovery rate level: 0.05). There were no significant differences between groups on any of the demographic variables after the false discovery rate correction was applied [age: *X*(2) = 2.79, *p* = 0.25, Benjamini–Hochberg *p*-value = 0.59; education: *X*(2) = 4.94, *p* = 0.08; Benjamini–Hochberg *p*-value = 0.51; chronicity: *z*(14) = 2.53, *p* = 0.01, Benjamini– Hochberg *p*-value = 0.19; etiology: *X*(1) = 0, *p* = 1.00; Benjamini–Hochberg *p*-value = 1.00]. The BDC group included individuals with lesions outside of regions that have been previously implicated as being involved in empathy (**Tables 1** and **2**).

FIGURE 1 | Lesion overlap map of patients with damage to the ventromedial prefrontal cortex (vmPFC). The lesion overlap map of eight patients with damage to the vmPFC is shown. Images are presented using radiological convention. Warmer colors indicate greater numbers of patients whose lesions overlap in a particular region, whereas cooler colors indicate fewer lesion overlaps. Overlap was greatest bilaterally in the ventromedial prefrontal region, with the cortex and white matter in the right vmPFC being involved in all eight patients.

In the BDC group, the lesions also excluded regions that have been associated with numeracy and valuation. The patients in the vmPFC group did not have major impairments in intelligence or memory, and they did not have premorbid personality disorders (6).1 Mann–Whitney *U* tests were used

1Some of the target participants previously participated in another study from our laboratory examining the 1-shot version of the UG in the role of the Responder, the person who decides whether to accept or reject the offer. These include patients 1983, 0770, 0318, 2577, 2352, and 2391. This previous study did not include an empathy induction. In the present study, we investigate patients' responses to an empathy induction on the UG in the role of the Proposer, the person who makes the offer.

to compare the vmPFC group to the BDC group on relevant neuropsychological variables (WAIS-III Full Scale Intelligence Quotient—FSIQ, WAIS-III Working Memory Index—WMI, Trail Making Test Part A and B—TMT). There were no significant differences between the groups on any of the neuropsychological variables after the false discovery rate correction was applied [FSIQ: *z*(14) = 0.95, *p* = 0.34; Benjamini–Hochberg *p*-value = 0.68; WMI: *z*(14) = 0.74, *p* = 0.46, Benjamini– Hochberg *p*-value = 0.79; TMT-A: *z*(14) = 0.21, *p* = 0.83, Benjamini–Hochberg *p*-value = 0.97; TMT-B: *z*(14) = 0.00, *p* = 1.00; Benjamini–Hochberg *p*-value = 1.00]. This study was carried out in accordance with the recommendations of the


*Patients with damage to the ventromedial prefrontal cortex were case-matched on age, gender, education, and WAIS-III Full Scale Intelligence Quotient (FSIQ) to individuals from the two comparison groups. Chronicity, years between lesion onset and experimental testing session.*

*SAH, subarachnoid hemorrhage; vmPFC, patient with damage to the ventromedial prefrontal cortex; BDC, brain damaged comparison participant; NC, normal comparison participant; M, mean; SD, standard deviation; NA, not applicable.*

*Kruskal–Wallis tests were used to compare age and education across the three groups. Mann–Whitney U test was used to compare chronicity between the BDC and vmPFC groups, and a Chi-square test was used to compare the type of etiology. There were no significant differences between groups on any of the demographic variables. Nineteen statistical tests were performed that were not testing a priori hypotheses. Consequently, we applied a false discovery rate correction for these 19 tests (false discovery rate level: 0.05). (To preserve the confidentiality of the patients who participated in the study, age is presented as a range.)*

TABLE 2 | Neuropsychological characteristics of patients.


*FSIQ, WAIS-III Full Scale Intelligence Quotient; WMI, WAIS-III Working Memory Index; TMT A, Trail Making Test Part A; TMT B, Trail Making Test; vmPFC, patient with damage to the ventromedial prefrontal cortex; BDC, brain damaged comparison participant; M, mean; SD, standard deviation.*

*There were no significant differences between the groups on any of the neuropsychological variables when Mann–Whitney U tests were conducted. Nineteen statistical tests were performed that were not testing a priori hypotheses. Consequently, we applied a false discovery rate correction for these 19 tests (false discovery rate level: 0.05). NC group did not complete the neuropsychological testing portion of the study.*

Declaration of Helsinki and the University of Iowa Institutional Review Board with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the University of Iowa Institutional Review Board.

#### Experimental Design

The present experiment involved a quasi-experimental, crosssectional design. The independent variables included experimental condition (neutral, empathy) and participant group (vmPFC patients, brain damage comparison patients, and normal, healthy adult comparison participants). The study used a within-subjects design, and thus all participants received both the neutral and empathy experimental conditions.

A novel empathy induction was used to elicit empathy in an implicit fashion similar to how empathy is frequently evoked in daily life—specifically, hearing another person talk about their struggles, frustration, and profound sadness. Participants were led to believe that the purpose of the study was to play an economic game (the UG) against a series of two opponents through an intercom system, with the opponents located in a different testing room. In one condition, *empathy induction*, the participant overheard (through the intercom) their opponent discussing the recent death of their son with the Research Assistant. In another condition (*neutral*, no empathy induction), the participant overheard their opponent discussing neutral, mundane events with the Research Assistant (e.g., such as playing cards or eating breakfast). Each participant underwent both the neutral and empathy induction in the same testing session. Due to the small number of available patients with damage to the vmPFC, the order of the inductions was not counterbalanced. The two opponents were actually audio recordings of community theater actors rather than real participants. The community theater actors were both males in their middle 50's (chosen for having similar voice quality, age, and gender) and the Research Assistant in the study was a female in her 20's. The age of the actors was selected to be similar to the age of the patient population in this study. Each audio recording was 4.5 min, with an 8-minute interval. This induction has effectively elicited empathy in healthy adults (42). For additional information on methods and pilot induction results see Ref. (43, 44).

#### Empathic Behavior

Empathic behavior was measured as the difference between the amount of money offered to the opponent in the UG following the empathy induction and the amount offered in the neutral condition. In the UG, the participant decided how much to offer the opponent out of \$10 on each of 20 rounds. The offers were summed across the 20 rounds, separately for each condition (empathy and neutral).

## Momentary Empathy and Emotional Responses

Empathy and other relevant emotions were measured through self-report momentary, state ratings that took place before and after each of the two conditions (neutral and empathy). Specifically, participants completed a questionnaire that assessed the participants' momentary (or state level) of empathy, personal distress, joviality, hostility, and sadness. Participants were asked to respond to the prompt, "Indicate to what extent you feel this way right now, that is, at the present moment," by rating each item on a scale from 1 (very slight or not at all) to 5 (extreme). This rating scale and prompt were adapted from the Positive and Negative Affective Schedule (PANAS) questionnaire (45). Furthermore, the items assessing joviality, hostility, and sadness were also adapted from this questionnaire and included sadness ("sad"; "downhearted"), hostility ("hostile"; "angry"), and joviality ("happy"; "joyful"). The items assessing emotional empathy and personal distress were drawn from a state measure of emotional empathy (23). These items included ("sympathetic"; "compassionate") and ("upset" and "distressed"). These questionnaires have been used in previous research studies to measure state empathy, personal distress, and basic emotions in healthy adults and patients with brain damage (42, 46).

## Patients Thoughts and Feelings About Empathy Induction

We examined written free responses from the participants about their thoughts and feelings involving the empathy induction. This questionnaire was completed at the end of the experiment after the participant had undergone both the neutral and empathy conditions, but prior to the debriefing session about the purpose of the study. In particular, participants responded to a question about their thoughts and feelings in response to the empathy induction in which they overheard their second opponent in the game talking about the anniversary of their son's death. Specifically the prompt was, "Please describe your thoughts and feelings (in a few words or a sentence) while hearing your second opponent talk with the Research Assistant. Please list these thoughts and feelings next to the bullets below. If there is not enough room, please use the lines below to describe further."

These free responses were coded by two raters (research assistants) who were blind to the group each participant was assigned to as well as the purpose of the study. Responses were coded as a "1" if the written text mentioned at least one of the following terms: "sorry for," "sad," "sympathy/sympathetic," and/ or "compassion/compassionate." Responses were coded as a "0" if the participant did not reference any of these terms.

## Believability/Manipulation Check

Participants completed four questions after the experiment, measuring the degree to which they believed they were playing against real opponents. The rating scale in response to these questions ranged from 1 = did not believe to 5 = believed extremely. These questions included the following: (1) "Did you believe that the first conversation you heard was a conversation between a Research Assistant and another person participating in the study?" (2) "Did you believe that the second conversation you heard was a conversation between a Research Assistant and another person participating in the study?" (3) "Did you believe that the first game was played against another person participating in the study?" (4) "Did you believe that the second game was played against another person participating in the study?" The responses across these four questions were averaged for each participant.

#### Trait Empathy Ratings

Participants completed a questionnaire designed to measure empathy as a trait, or a general tendency in one's daily life (27). In addition, the participants' family members also completed the same trait questionnaire about the participants, as a means of comparison. (Not all family members of the participants were available to complete the questionnaires. The final sample of family members included a total of 14, across the three groups.)

The Interpersonal Reactivity Index (IRI) (27) was used to assess trait empathy and is a well-validated, multidimensional measure of empathy that assesses both the emotional and cognitive aspects of empathy. Emotional empathy was measured using the Empathic Concern subscale and cognitive empathy was assessed through the Perspective Taking subscale. Each subscale ranges from 0 to 28 points, and higher scores indicate a greater tendency towards empathy in daily life. The IRI has adequate test/ retest reliability (range: *r* = 0.61–0.81) and internal consistency (range Cronbach's alpha: 0.68–0.79). An example item from the questionnaire is, "When I'm upset at someone, I usually try to 'put myself in his shoes' for awhile."

## Social Faux Pas Task: Assessing Accuracy of Detecting Others' Intentions

Theory of mind was measured with a standard task assessing one's ability to detect social faux pas from written scenarios, called the Social Faux Pas Task (47). In this task, participants read written scenarios about two characters engaged in a situation where someone says or does something that is socially inappropriate, or in other words, commits a social faux pas. Then, the participant answers a multiple choice question to determine whether they can detect what social faux pas was committed. In this task, there are also control scenarios to assess basic reasoning skills. A separate accuracy score is calculated for the 12 control and 12 theory of mind conditions for each participant.

## Statistical Analysis

#### Hypothesis Testing

Our primary variable of interest was the empathic behavior variable. We performed the Shapiro–Wilk test to assess the normality of the distribution of this variable. We found evidence that the NC group was not normally distributed (NC: S-W = 0.81, *p* = 0.04; BDC: S-W = 0.94, *p* = 0.59; vmPFC: S-W = 0.98, *p* = 0.97), and thus we have used non-parametric tests throughout this paper. We tested the degree to which the vmPFC group had lower empathic behavior than the comparison groups using a Kruskal–Wallis test. Next, we assessed the degree to which the vmPFC group had lower state empathy ratings than the comparison groups using a Kruskal–Wallis test. Based on our *a priori* hypotheses, planned comparisons (Mann–Whitney *U*

tests) were used to compare each group to the other two groups on these variables.

#### Sample Description

The mean and standard deviation of the variables believability, trait empathy, theory of mind, and state emotions other than empathy (e.g., sadness, personal distress, hostility, and joviality) are presented in **Table 3**. Because we did not have specific hypotheses about these variables, these results are descriptive in nature. The exploratory analyses included separate Kruskal–Wallis tests to compare the three participant groups on believability, theory of mind, trait empathy (patient and family ratings), and each of the state emotions (i.e., sadness, personal distress, hostility, and joviality). If the result was significant at *p* < 0.05, Mann–Whitney *U* tests were used to assess differences between the groups. For all tests, uncorrected *p*-values are listed. A total of 19 statistical tests were performed that were not testing *a priori* hypotheses. Consequently, we applied a false discovery rate correction for these 19 tests (false discovery rate level: 0.05). We also list the Benjamini–Hochberg *p*-value that resulted from this false discovery rate correction. All statistical tests were two-tailed and findings were considered to be significant at the *p* < 0.05 level. Nonparametric tests were used for all analyses. For our qualitative exploratory analysis of patients' thoughts and feelings in response to the empathy induction, we present the proportion of participants' responses from each group that were coded as a 1 and the participants' written responses. Statistics were not conducted on the thoughts and feelings responses because the results were qualitative in content.

## RESULTS

## Hypothesis Testing

#### Primary Behavioral Analysis: Empathic Behavior Towards a Suffering Individual

The primary analysis addressed the degree to which participants demonstrated empathy behaviorally by making larger offers in the UG in response to the empathy condition in comparison to the neutral condition. Patients with damage to the vmPFC did not make higher offers in the empathy condition than in the neutral condition, whereas both comparison groups made much higher offers [**Figure 2A**, group: *X*(2) = 9.56, *p* = 0.008; follow-up planned comparisons: vmPFC vs. BDC: *z*(14) = 2.73, *p* = 0.006; vmPFC vs. NC: *z*(14) = 2.37, *p* = 0.02; BDC vs. NC: *z*(14) = 1.06, *p*= 0.29]. The range of offers in each group included: vmPFC = −\$8 to 8.672 ; BDC = \$2 to 24; NC = \$3 to 16. In fact, of the eight vmPFC patients, four patients actually gave lower offers to the man who had lost his son, two had virtually zero change, and two had increases in response to the empathy condition. In sharp contrast, all 16 participants in the comparison groups gave higher offers in response to the empathy induction; in many instances, these were much higher (**Figure 2A**; Figure S1 in Supplementary Material for additional information).

<sup>2</sup>A fraction is indicated here because one patient with damage to the vmPFC was missing one offer, and thus a mean substitution approach was used to approximate the missing value based on the patient's other offers.

#### TABLE 3 | Assessments of state emotion, empathy, and theory of mind.


*Group labels include: vmPFC, ventromedial prefrontal cortex; BDC, group of patients with damage to areas of the brain not related to empathy; NC, healthy adult normal comparison group; M, mean; SD, standard deviation. State emotion rating change scores represent the effect of the empathy induction on each emotional state by subtracting out their emotional response to the neutral condition and their baseline response. IRI, Interpersonal Reactivity Index. Participants indicates participants' self-reported score on the questionnaire. Family indicates family member ratings of the participant. Difference score indicates participant score was subtracted from family member score—negative scores indicate family member rated the participant lower than the participant rated themselves; positive scores indicate family member rated the participant higher than the participant rated themselves. Theory of mind task represents accuracy on the theory of mind condition, control indicates accuracy on the control condition. Kruskal–Wallis tests were used to compare the three groups on each measure. Because the state empathy rating examined a specific hypothesis, planned comparison tests were used, with no correction. Nineteen statistical tests were performed that were not testing a priori hypotheses. Consequently, we applied a false discovery rate correction for these 19 tests (false discovery rate level: 0.05).*

FIGURE 2 | Group differences in empathic behavior, ratings, and believability. The three participant groups were compared on their empathic behavior, ratings, and the believability of the experiment. Graphs depict mean values and error bars are standard error of the mean. \**p* < 0.05. N.S., not significant. (A) Empathic behavior by group. Empathic behavior on the Ultimatum Game (UG) was depicted as a change score reflecting the difference in the amount of money given after experiencing an empathy or neutral condition (sum of offers: empathy – neutral condition). Positive numbers indicate that greater money was given in response to the empathy induction than the neutral condition. (B) Empathy ratings by group. An empathy rating change score was computed measuring empathic concern ratings before and after each induction condition: (After – Before Empathy Induction) – (After – Before Neutral Induction). Positive change scores indicate higher ratings on the empathy induction versus the neutral condition. (C) Believability. Participants completed four questions at the end of the experiment measuring the degree to which they believed they were playing against real opponents. Responses across the four questions were averaged. (Rating scale: 1–5; 1 = did not believe and 5 = believed extremely.) Nineteen statistical tests were performed that were not testing *a priori* hypotheses. Consequently, we applied a false discovery rate correction for these 19 tests (false discovery rate level: 0.05).

#### Emotional Response: State Empathy

We compared the degree to which there were differences in state self-rated empathy in response to the experimental conditions across the three groups. Overall, group differences were not significant (**Figure 2B**; **Table 3**, *X*(2) = 3.11, *p* = 0.21). Follow-up planned comparisons revealed that the vmPFC group did not significantly differ from the BDC group [*z*(14) = 1.70, *p* = 0.09] or from the NC group [*z*(14) = 1.23, *p* = 0.22]. Also, the BDC and NC groups did not differ significantly [*z*(14) = 0.43, *p* = 0.67].

We tested the degree to which the state empathy of the three groups was statistically equivalent using the two one-sided tests (TOST) procedure (48), with an alpha level of 0.05 and an effect size value of Cohen's *d* = 0.3 (indicating a small effect size). When comparing the vmPFC group to the BDC group, the equivalence test was non-significant [*t*(12.54) = 1.31, *p*= 0.89]. When comparing the vmPFC group to the NC group, the equivalence test also was non-significant [*t*(13.08) = 0.97, *p* = 0.83]. When comparing the BDC group to the NC group, the equivalence test was nonsignificant [*t*(13.90) = −0.22, *p* = 0.42]. Although there were no statistically significant differences between the groups on this measure, our study's small sample size prevented us from establishing statistically significant equivalence between the groups.

#### Exploratory Analyses Patients Thoughts and Feelings About Empathy Induction

We sought to further understand the degree to which patients were aware that the content of the empathy induction was about an empathy-eliciting situation. To further assess this question, we examined written free responses from the participants in response to a questionnaire that occurred at the end of the experiment, but prior to the debriefing session about the purpose of the study. In particular, participants responded to a question about their thoughts and feelings in response to the empathy induction in which they overheard their second opponent in the game talking about the anniversary of their son's death. Specifically, the prompt was, "Please describe your thoughts and feelings (in a few words or a sentence) while hearing your second opponent talk with the Research Assistant. Please list these thoughts and feelings next to the bullets below. If there is not enough room, please use the lines below to describe further." We present the full written responses of the patients with damage to the vmPFC in **Table 4**. The full written responses of the BDC and NC groups are presented in Table S1 in Supplementary Material.

Two raters who were blind to the group each participant was assigned to as well as the purpose of the study coded the written free responses of the participants. Responses were coded as a "1" if the written text mentioned at least one of the following terms: "sorry for," "sad," "sympathy/sympathetic," and/or "compassion/ compassionate." Responses were coded as a "0" if the participant did not reference any of these terms. In the group of patients with damage to the vmPFC, seven out of the eight patients' responses were coded as a "1" in response to the empathy condition. Similarly, in the BDC group, seven out of the eight patients' responses were coded as a "1" and in the NC group, all eight participants were coded as a "1." (There was perfect agreement among the raters in TABLE 4 | Written free responses about empathy induction by patients with damage to the ventromedial prefrontal cortex (vmPFC).


*Participants with damage to the vmPFC filled out a questionnaire after the experiment was completed about their thoughts and feelings in response to the audio recording designed to induce empathy. Specifically, participants responded to the prompt: "Please describe your thoughts and feelings (in a few words or a sentence) while hearing your second opponent talk with the Research Assistant. Please list these thoughts and feelings next to the bullets below. If there is not enough room, please use the lines below to describe further." We list the written comments of each participant with damage to the vmPFC.*

their coding of the written responses). Some examples of the free responses of the patients with damage to the vmPFC are, "Sad person since son's death; could not connect with wife's feelings now; desperately looking for help," and, "Sympathy for losing a loved one; compassion for what he is experiencing."

#### Manipulation Check

A manipulation check was used to determine the degree to which participants believed the experiment (i.e., whether participants believed that the opponents they overheard during the experiment through the intercom were actual participants). This manipulation check demonstrated that the groups did not significantly differ on the believability measure [*X*(2) = 0.87, *p* = 0.65; Benjamini–Hochberg *p*-value = 0.82]. For additional information about the believability results and questionnaire, see **Figure 2C** and Section "Materials and Methods."

#### State Emotion

There were no significant group differences after a false discovery rate correction in any of the emotions measured in response to the experimental conditions which included sadness, personal distress, hostility, and joviality [sadness: *X*(2) = 2.87, *p* = 0.24; Benjamini–Hochberg *p*-value = 0.59; personal distress: *X*(2) = 0.87, *p* = 0.65, Benjamini–Hochberg *p*-value = 0.97; hostility: *X*(2) = 4.17, *p* = 0.12, Benjamini–Hochberg *p*-value = 0.53; joviality: *X*(2) = 1.12, *p* = 0.57, Benjamini–Hochberg *p*-value = 0.79; (**Table 3**)].

#### Trait Empathy and Accuracy of Assessing Others' Intentions

Groups were compared on their self-reported trait empathy (cognitive—IRI Perspective Taking subscale; emotional—IRI Empathic Concern subscale), and their theory of mind performance (**Table 3**; see Materials and Methods). In addition, family members completed the trait questionnaire about the participants, as a means of comparison. This analysis revealed no significant differences between the groups in self-reported trait empathy by the participants after false discovery rate correction [IRI-EC: *X*(2) = 2.07, *p*= 0.36; Benjamini–Hochberg *p*-value = 0.68; IRI-PT: *X*(2) = 1.31, *p* = 0.52, Benjamini–Hochberg *p*-value = 0.79]. There were also no significant differences between the groups after false discovery rate correction in family members' reports of participants' trait empathy [**Table 3**; IRI-EC: *X*(2) = 3.16, *p* = 0.21, Benjamini–Hochberg *p*-value = 0.59; IRI-PT: *X*(2) = 3.89, *p*= 0.14, Benjamini–Hochberg *p*-value = 0.53].

Next, the groups were compared on their accuracy scores in the theory of mind task (Social Faux Pas Task) which measures one's ability to detect the motivations and intentions of others through written scenarios. This task includes a theory of mind condition (i.e., accuracy of determining others' intentions) and a control condition (i.e., accuracy of basic reasoning skills). When comparing the performance accuracy of the groups on the theory of mind condition, there was no significant effect of group after correction for false discover rate [Theory of mind condition: *X*(2) = 5.82, *p* = 0.05, Benjamini–Hochberg *p*-value = 0.48]. There were also no significant group differences in the control condition after correction for false discovery rate [Control condition: *X*(2) = 1.11, *p* = 0.58, Benjamini–Hochberg *p*-value = 0.79].

## DISCUSSION

For the first time, we experimentally demonstrated that patients with damage to the vmPFC behaved with little empathy in a financial context towards a man who is suffering. This corroborates clinical case studies reporting that patients with damage to the vmPFC behave with reduced empathy toward family members (40, 41, 49). Furthermore, we also advance the literature by demonstrating that patients with damage to the vmPFC obtained more money than comparison groups in the UG when witnessing another person suffering. This is in contrast to previous studies which found that patients with damage to the vmPFC achieved poorer financial outcomes than comparison participants in decision making games, such as the Iowa Gambling Task (11, 12). The findings of the present study are consistent with research suggesting that the vmPFC plays an important role in using contextual information to guide decision making (50, 51).

Patients with damage to the vmPFC showed significantly less empathic behavior towards a person who was suffering. Specifically, the patients with damage to the vmPFC did not give more money to the man who was suffering than to the man in the control condition. In stark contrast, the comparison groups gave more money to the man who was suffering than to the man in the control condition. However, behaving with less empathic behavior than the comparison groups actually benefited the patients with damage to the vmPFC financially, as they received higher rather than lower financial payoffs than comparison participants.

Determining whether a financial decision is advantageous or not depends not only on the financial outcome, but also on the social consequences that may result. For instance, imagine the situation in which your mother cannot afford her chemotherapy treatments. You decide to help pay for her chemotherapy treatments, even though this decision could negatively impact your financial situation out of concern for her well-being which may in turn result in increased relationship quality. Therefore, healthy adults may choose to forego financial gain in order to achieve greater social rewards. On the other hand, imagine if an individual acted in a manner similar to the patients with vmPFC damage in the current study where they decided to pay very little for the chemotherapy treatments. Although this would result in better financial outcomes for the patient, it could severely and negatively impact their relationship with their mother. Therefore, advantageous financial decision making in social contexts is likely to require making decisions that are likely to facilitate social relationships, even if finances are negatively impacted.

The lower empathic behavior of patients with damage to the vmPFC in the present study may contribute to their difficulties making and maintaining relationships that are often highlighted in case reports (8, 9, 52). Anderson and colleagues studied two cases of patients with damage to the vmPFC and noticed that they both had few friends, mentioning that in the case of Patient B the, "lack of friends was conspicuous," (52). Furthermore, there is other anecdotal evidence that patients with damage to the vmPFC have difficulty maintaining relationships, such as in the case of seminal patient EVR, who went through a divorce after 17 years of marriage (8), and in another case study of a patient who had already been divorced by 21 years of age (9). Because of the important role of empathic behavior in maintaining and nurturing relationships, it is likely that difficulty showing empathic behavior toward others could have a negative impact on one's personal relationships.

The results in the present study also provide new information about how patients with damage to the vmPFC perceive and experience empathy-eliciting situations. We report that patients with damage to the vmPFC did not significantly differ from the comparison groups in their experience of "in the moment" empathy in response to an empathy induction involving exposure to another person's suffering. However, it should be noted that our study's small sample size prevented us from establishing statistically significant equivalence between the groups. Consequently, at this time, we cannot determine whether the patients with damage to the vmPFC have lower or equivalent levels of state empathy relative to the comparison groups. Our exploratory free response analyses suggest that the majority of patients with damage to the vmPFC are aware that they were exposed to an empathy-eliciting situation, as seven out of the eight patients reported that they felt "sorry for," "sad," "sympathy/sympathetic," and/or "compassion/ compassionate" in response to the empathy induction. However, we note that despite this reported awareness of the empathic content, this was not sufficient for the patients with damage to the vmPFC to behave in an empathic manner. Taken together, our findings suggest that future studies are needed to tease apart this important question as to whether patients with damage to the vmPFC have lower state empathy than healthy adults in response to empathy-eliciting events.

The patients with damage to the vmPFC in our sample did not significantly differ from the comparison groups in their ability to accurately detect the intentions of others in a separate theory of mind task. A previous study of patients with damage to the vmPFC focused on patients' reports of trait empathy and found that there is evidence for lower reported cognitive empathy than healthy adults of comparable demographics (38). However, a key difference between the present study and study by Shamay-Tsoory and colleagues is that their sample of patients with damage to the vmPFC included a large proportion of closed head injury cases which could have had more diffuse brain damage. In contrast, the present sample does not include any closed head injury cases. Furthermore, in the present study, we did not find significant group differences in accuracy on the theory of mind task which would provide further support that their ability to discern others' intentions is relatively intact. Previous studies of theory of mind in patients with damage to the vmPFC have found mixed evidence about whether they have difficulty detecting others' intentions and motivations (53, 54).

It is relevant to discuss our findings in the context of important theories of vmPFC function [for review see Ref. (55)]. The somatic marker hypothesis proposes the role of the vmPFC as a secondary inducer, or a higher order emotional response that helps to guide decision making (17, 18). Roy and colleagues highlight an important role for the vmPFC in affective meaning (56). In particular, they suggest that the vmPFC plays a role in behavioral responses to higher order conceptual levels of emotion, rather than lower order simple emotional responses (56). The role for the vmPFC in insight and reflection has also been pointed out by Koenigs et al. (57) who suggested that this region plays an important function in reflecting on one's emotional state and how it may affect others. Our results suggest that the vmPFC is important for behaving in an empathic manner in response to a financial context in which another person is suffering. We find preliminary evidence that despite seven out of eight patients reporting that they are aware of the empathic context, they did not show empathic behavior, suggesting that they have difficulty using this type of information to guide their empathic behavior.

In the financial domain, previous studies have established that the vmPFC is critical for advantageous financial decisions, whether in the context of the Iowa Gambling Task (15, 16), or the UG (57). In both cases, patients with vmPFC damage act differently than normal healthy adults, and fail to use emotional information in an advantageous way to guide financial behavior. In the present study, the patients with damage to the vmPFC appear to be aware of the empathy-eliciting context but behave with lower empathy than the comparison groups. However, as a result, they also have greater payoffs in the game. Consequently, it suggests that the vmPFC may be important for using contextual information in a socially advantageous manner, such as showing empathy toward others in need, or regulating one's anger when someone rejects your offers in the UG. This interpretation is in line with Koenigs et al.'s (57) discussion of the important role for the vmPFC in self-reflection about one's emotions and the consequences of their behavior. Although the present study focuses on the financial domain, the vmPFC may serve similar functions in non-financial contexts. This is seen in their failure to use emotional context in a socially advantageous manner in moral scenarios, as patients with vmPFC damage exhibit utilitarian type behavior (58) and socially inappropriate behaviors (59). In summary, the present study adds to the growing literature on the role of the vmPFC in social decision making in financial and non-financial contexts.

This study has limitations. The measure of empathic feelings in this study was self-report which can be influenced by concerns for social desirability. To attempt to address this issue, we also collected ratings from the family members about the patients' empathy, as a form of corroboration. We found that the family members' ratings did not significantly differ across the participant groups, which provides support for the accuracy of the patient ratings. In the present study, we cannot directly address the question of whether the empathy that the patients with damage to the vmPFC felt in response to the empathy induction was similar to or more extreme than the level of empathy they may experience in their daily lives. To answer this question, future studies may compare patient ratings of empathy in real time in their daily lives vs. laboratory-based empathy inductions. Because patients with focal damage to the vmPFC are rare, our sample size is smaller than that of studies focusing on healthy adults. However, the size of our sample is consistent with other studies on patients with damage to the vmPFC [e.g., *N* = 7, (6); *N* = 8, (58)]. Because the inductions were not counterbalanced in the present study, there is the possibility of an order effect. In a different study of healthy younger and older adults, we counterbalanced the order of a similar empathy induction and neutral induction (this one used a series of notes rather than audio recordings) and found no significant effects of order. This suggests that in a similar context, there was no significant effect of order (46). However, in future studies, it would be useful to counterbalance the order of the conditions in order to specifically address this limitation. Characterization of the patients' emotional responses to empathy inductions through physiological (e.g., skin conductance, heart rate) measures would further add to our understanding of their momentary empathic experience in response to others' suffering.

In summary, the current study is the first to experimentally demonstrate that the vmPFC is critical for empathic behavior in a financial context towards those who are suffering. We show preliminary evidence that awareness of an empathy-eliciting event, where someone is suffering, is not enough to elicit empathic behavior in patients with damage to the vmPFC. Rather, it suggests that these patients do not appear to use this information to guide their behavior in a way that helps the suffering person. On the other hand, by behaving in a manner seemingly not influenced by the empathic context, patients with damage to the vmPFC have better financial payoffs than the comparison groups.

These findings have broad implications for the treatment of other populations suffering from difficulty behaving with empathy toward others who are suffering. It helps us understand how groups affected by changes to the frontal lobe might respond in financial contexts where they witness another person's suffering. Because decreased functioning of the frontal lobe is seen in many different populations ranging from healthy aging, to dementia, and brain injury, it has far reaching implications for financial decision making in social contexts for these groups (60). This is important for family members of patients with damage to the frontal lobe to be aware of because it may help them to have greater compassion for the patient.

In contrast to the behavior of the patients with damage to the vmPFC, if an individual puts too much weight on the emotional context of a situation, it could also have a negative impact on financial decisions and personal relationships. For instance, highly empathic caregivers or nurses may become too emotionally invested in their patients or loved ones which could lead to compassion fatigue and burnout (61–63). In future research studies, it would be useful to investigate the utility of interventions designed to help individuals strategize about making financial decisions that optimize both financial and social well-being. In conclusion, the present study characterizes the role of the vmPFC in an empathy-eliciting situation involving financial decision making towards an individual who is suffering.

## ETHICS STATEMENT

This study was carried out in accordance with the recommendations of the Declaration of Helsinki and the University of Iowa Institutional Review Board with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the University of Iowa Institutional Review Board.

## AUTHOR CONTRIBUTIONS

JB was involved in the methodological design, data collection and analysis, theoretical framework, and writing the manuscript.

## REFERENCES


DT and SP contributed to the methodological design, theoretical framework, and writing the manuscript.

## ACKNOWLEDGMENTS

The authors would like to thank Kelsey Carlson, Felice Loi, and Erika Holm-Brown for their assistance in the data collection process.

## FUNDING

JB was supported by the neuroscience training grant through the Neuroscience Program at the University of Iowa College of Medicine, which is funded by the Institute of Neurological Disorders and Stroke at the National Institute of Health (T32 NS00742). During manuscript preparation and editing, JB was also supported by the University of Nebraska Program of Excellence Fund, the Vada Kinman Oldfield Award in Alzheimer's Disease Research, and by the National Science Foundation, EPSCoR Research Infrastructure Award #1539067. SP was supported by the National Institute on Aging (5K23AG027837), the Mallinckrodt and Dana Foundations and by the Brain and Behavior Research Foundation (formerly, NARSAD Foundation). DT was supported by grant 2P50MH094258 from the National Institute of Mental Health, grant 220020387 from the James S. McDonnell Foundation, and a grant from the Spastic Paralysis Research Foundation of Kiwanis International.

## SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at https://www.frontiersin.org/articles/10.3389/fneur.2018.00288/ full#supplementary-material.


ventromedial prefrontal cortex. *J Int Neuropsychol Soc* (2006) 12(2):224–35. doi:10.1017/S1355617706060346


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2018 Beadle, Paradiso and Tranel. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

*Sona Patel1 , Kenichi Oishi2 , Amy Wright <sup>2</sup> , Harry Sutherland-Foggio2 , Sadhvi Saxena2 , Shannon M. Sheppard2 and Argye E. Hillis <sup>2</sup> \**

*1Seton Hall University, South Orange, NJ, United States, 2 Johns Hopkins Medicine, Baltimore, MD, United States*

Impaired expression of emotion through pitch, loudness, rate, and rhythm of speech (affective prosody) is common and disabling after right hemisphere (RH) stroke. These deficits impede all social interactions. Previous studies have identified cortical areas associated with impairments of expression, recognition, or repetition of affective prosody, but have not identified critical white matter tracts. We hypothesized that: (1) differences across patients in specific acoustic features correlate with listener judgment of affective prosody and (2) these differences are associated with infarcts of specific RH gray and white matter regions. To test these hypotheses, 41 acute ischemic RH stroke patients had MRI diffusion weighted imaging and described a picture. Affective prosody of picture descriptions was rated by 21 healthy volunteers. We identified percent damage (lesion load) to each of seven regions of interest previously associated with expression of affective prosody and two control areas that have been associated with recognition but not expression of prosody. We identified acoustic features that correlated with listener ratings of prosody (hereafter "prosody acoustic measures") with Spearman correlations and linear regression. We then identified demographic variables and brain regions where lesion load independently predicted the lowest quartile of each of the "prosody acoustic measures" using logistic regression. We found that listener ratings of prosody positively correlated with four acoustic measures. Furthermore, the lowest quartile of each of these four "prosody acoustic measures" was predicted by sex, age, lesion volume, and percent damage to the seven regions of interest. Lesion load in pars opercularis, supramarginal gyrus, or associated white matter tracts (and not control regions) predicted lowest quartile of the four "prosody acoustic measures" in logistic regression. Results indicate that listener perception of reduced affective prosody after RH stroke is due to reduction in specific acoustic features caused by infarct in right pars opercularis or supramarginal gyrus, or associated white matter tracts.

Keywords: prosody expression, stroke, right hemisphere, emotion, communication

## INTRODUCTION

A flat tone-of-voice is often interpreted as apathy, displeasure, sadness, or lack of empathy of the speaker, depending on the context. Yet, survivors of right hemisphere (RH) stroke (1–5) and people with certain neurological diseases—e.g., Parkinson's disease (6–8), frontotemporal dementia (9–13), schizophrenia (14, 15)—may have trouble modulating their tone-of-voice to express emotion, even

#### *Edited by:*

*Peter Sörös, University of Oldenburg, Germany*

#### *Reviewed by:*

*Daniel L. Bowling, Universität Wien, Austria Doriana De Marco, Istituto di Neuroscienze (CNR), Italy*

> *\*Correspondence: Argye E. Hillis argye@jhmi.edu*

#### *Specialty section:*

*This article was submitted to Applied Neuroimaging, a section of the journal Frontiers in Neurology*

*Received: 01 November 2017 Accepted: 22 March 2018 Published: 06 April 2018*

#### *Citation:*

*Patel S, Oishi K, Wright A, Sutherland-Foggio H, Saxena S, Sheppard SM and Hillis AE (2018) Right Hemisphere Regions Critical for Expression of Emotion Through Prosody. Front. Neurol. 9:224. doi: 10.3389/fneur.2018.00224*

**109**

when they feel joyful or empathetic. Affective prosody (changes in pitch, loudness, rate, and rhythm of speech to convey emotion) communicates the speaker's emotion and social intent. Thus, impairments in affective prosody can disrupt all daily interactions and interpersonal relationships, as well as influence social behavior (16).

## Neural Regions Supporting Affective Prosody

It has long been recognized that strokes involving the right frontal lobe, particularly posterior inferior frontal cortex, are associated with impaired expression of affective prosody (3, 17). Infarcts in the right temporal lobe are often associated with impaired recognition of affective prosody (3) or impaired recognition and expression (17). Previous studies have identified cortical areas important for expression of emotion through prosody, using either functional MRI (fMRI) of healthy participants (18–22) or lesion-symptom mapping in individuals with focal brain damage (3, 23). Several studies show activation in inferior frontal cortex, specifically during evoked expressions. However, the brain regions involved seem to be dependent on the type of emotion expressed by the speaker (22). Although most studies of affective prosody impairments have focused on cortical regions, one study showed that infarcts that affected the right sagittal stratum (a large bundle of white matter fibers connecting occipital, cingulate, and temporal regions to the thalamus and basal ganglia) interfered with recognition of sarcasm (24). Nevertheless, few studies have identified the role of specific white matter tracts in the neural network underlying emotional expression.

## RH Dorsal and Ventral Stream Regions for Affective Prosody

The majority of studies investigating emotional prosody have focused on the perception rather than the production of emotion in speech. It has been suggested that, similar to the wellestablished dual-stream model subserving language processing in the left hemisphere (25–27), prosody comprehension proceeds along analogous dual ventral and dorsal streams in the right hemisphere (28). Specifically, it is proposed that the dorsal "how" pathway is critical for evaluating prosodic contours and mapping them to subvocal articulation, while the ventral "what" pathway, which includes the superior temporal sulcus and much of the temporal lobe, maps prosody to communicative meaning. While the research investigating these pathways in affective prosody generation is sparse, it has been proposed that bilateral basal ganglia play an important role in modulation of motor behavior during the preparation of emotional prosody generation, while RH cortical structures are involved in auditory feedback mechanisms during speech production (29).

## Changes in Acoustic Features Associated With Impaired Affective Prosody

Recent advances in acoustic analysis of speech and voice allow characterization of the fundamental frequency (i.e., the high versus low quality of the voice; measures include the mean, range, peak, and variation), intensity (i.e., how loud or soft the voice is; measures include the mean, range, peak, nadir, variation within, and across bandwidths), speech duration (i.e., how fast or slow the speech is), and rhythm (rate, timing, and relative intensity of various speech segments, such as vowels, consonants, pauses, and so on). Any of these features might be affected by focal brain damage, and changes in one or more feature can influence the perception of the emotion or intent of the speaker.

Previous studies have identified changes in acoustic features that are responsible for abnormal affective prosody in Parkinson's disease (6, 8), schizotypal personality disorder and schizophrenia (15, 30), and frontotemporal dementia (10, 31). Almost all studies report that less pitch variability and slower rate of speech are associated with reduced prosody in these individuals. A reduction in the pitch variability is what is often referred to as "flat affect," i.e., a "flat" pitch contour or one that is not as variable. Some conditions such as Parkinson's disease result in a reduced vocal intensity as well, potentially due to an underlying motor control problem (32), resulting in a quieter voice. Taken together, the impact of these changes is a less variable and therefore a more monotone sounding voice. It has yet to be established whether abnormalities in specific acoustic features account for listener perception of impaired affective prosody after RH stroke.

In this study, we hypothesized that: (1) abnormal patterns of specific acoustic features correlate with lower listener rating of emotional expression and (2) these abnormal acoustic features are associated with infarcts of specific RH gray and white matter regions. Since no one-to-one map between each acoustic feature and a specific brain region exists, a standard set of acoustic parameters were investigated based on the features that are known to be affected in pathological conditions, including RH stroke.

## MATERIALS AND METHODS

#### Participants

A consecutive series of 41 acute ischemic RH stroke patients who provided written informed consent for all study procedures were enrolled. Consent forms and procedures were approved by the Johns Hopkins Institutional Review Board. Exclusion criteria included: previous neurological disease involving the brain (including prior stroke), impaired arousal or ongoing sedation, lack of premorbid competency in English, left handedness, <10th grade education, or contraindication for MRI (e.g., implanted ferrous metal). The mean age was 62.7 ± SD 12.5 years. The mean education was 14.4 ± 3.3 years. The mean lesion volume was 37.2 ± 67.0 cc. Participants were 41.5% women. Within 48 h of stroke onset, the participants were each administered a battery of assessments of affective prosody expression and recognition, but in this study we focused on affective prosody expression to test our hypotheses.

## Acoustic Analysis

The speech samples from each participant included a description of the "Cookie Theft" picture, originally from the Boston Diagnostic Aphasia Examination (33). This same picture is also used in the National Institutes of Health Stroke Scale (34, 35). The stimulus is shown in **Figure 1**. Participants were

instructed to describe the picture as if they were telling a story to a child. Participants were prompted to continue ("anything else that you can tell me?") once. Recordings were made using a head-worn microphone placed two inches from the mouth of the participant. All samples were segmented and converted into mono recordings for analysis in Praat (36). A total of 26 parameters were automatically extracted from the speech samples using customized scripts. The parameters included measurements related to fundamental frequency (*F*0), intensity, duration, rate, and voice quality. The full list of parameters is given in **Table 1** along with a short description of each measure. Because we had no *a priori* evidence to hypothesize that some of these features would be more affected than others, we included a standard list of features (measures of *F*0 and intensity and durations of various parts of speech) as well as a set of features that were either relative to certain frequency bands or parts of speech. We followed the same procedures followed in previous publications; for example, see Ref. (37) for details of the analyses.

#### Listener Rating

The emotional expression of the speech samples was rated by 21 healthy volunteers, using a 1–7 scale (from no emotion to very emotional). They were given several practice items with feedback. The mean score for the 21 listeners for each voice sample was used to identify the acoustic features related to the listener ratings of emotional expression.

#### Image Analysis

Participants were evaluated with MRI diffusion weighted imaging (DWI), fluid attenuated inversion recovery (to rule out old lesions), Susceptibility weighted imaging (to rule out hemorrhage), and T2-weighted imaging to evaluate for other structural lesions. A neurologist (Kenichi Oishi) who was blind to the results

Table 1 | Acoustic measures that were included in the analyses.


of the acoustic analyses identified the percent damage to each of seven gray and white matter regions that have previously been associated with deficits in expression of affective prosody and two control areas that have been associated with deficits in recognition but not expression of prosody (3, 13, 23, 24, 38). The seven regions of interest hypothesized to be related to prosody expression in the RH were: inferior frontal gyrus pars opercularis; supramarginal gyrus; angular gyrus; inferio-frontal-occipital fasciculus, superior frontal occipital fasciculus; superior longitudinal fasciculus (SLF); and uncinate fasciculus. The control areas that have previously been identified as critical for prosody recognition but not production (22, 24, 30) were: superior temporal gyrus and sagittal stratum. The procedure followed previous publications (39–41). In brief, the boundary(s) of acute stroke lesion(s) was defined by a threshold of >30% intensity increase from the unaffected area in the DWI (42, 43) then manually modified to avoid false-positive and false-negative areas by a neurologist (Kenichi Oishi). Kenichi Oishi was blinded to the results of the acoustic analyses to avoid bias in lesion identification. Then, the non-diffusion weighted image (b0) was transformed to the JHU-MNI-b0 atlas using affine transformation, followed by large deformation diffeomorphic metric mapping (LDDMM) (44, 45). The resultant matrices were applied to the stroke lesion for normalization. LDDMM provides optimal normalization to minimize warping regions of interest (37, 38). A customized version of the JHU-MNI Brain Parcelation Map1 was then overlaid on the normalized lesion map to determine the percentage volume of the nine regions (**Figure 2**), using DiffeoMap.2

<sup>1</sup>http://cmrm.med.jhmi.edu (Accessed: January 6, 2017).

<sup>2</sup>www.MRIstudio.org (Accessed: January 6, 2017).

Figure 2 | Representative individuals with acute infarction in the selected structures. The structures are color-contoured: inferior frontal gyrus, pars opercularis [pink (A,C)], superior temporal gyrus [cyan (D,F,G)], supramarginal gyrus [orange (A,B,C)], angular gyrus [chartreuse green (A,C)], inferior fronto-occipital fasciculus [yellow (F)], sagittal stratum [purple (E)], superior fronto-occipital fasciculus [blue (B)], superior longitudinal fasciculus [red (A)], and the uncinate fasciculus [green (G)]. Diffusion weighted images were normalized to the JHU-MNI atlas space and pre-defined ROIs were overlaid on the normalized images. Images are all in radiological convention: left side of the figure is the right side of the individual.

## Statistical Analysis

All analyses were carried out with Stata, version 12 (StataCorp3 ). Acoustic features (from the 26 listed in **Table 1**) that correlated with mean listener judgments of affective prosody were identified with Spearman correlations. An alpha level of *p* < 0.05, after correction for multiple comparisons (*n* = 26) with Bonferroni correction, was considered significant. Acoustic features that independently contributed to listener rating of emotional expression, after adjustment for other acoustic features and age, were identified with linear regression, separately for men and women speakers, and were used in further analyses as the "prosody acoustic measures." Then, percent damage to each ROI that independently predicted the lowest quartile of each of the identified prosody acoustic measures were identified using logistic regression. The independent variables included age, sex, education, and percent damage to (lesion load in) each of the nine ROIs (including two control regions). We included age and sex in all multivariable logistic regressions, along with percent damage to each of the five cortical regions of interest and the four white matter bundles of fibers, because age and sex can influence all acoustic features. Because we did not include healthy controls, we defined the lowest quartile of each prosody acoustic measure as abnormal. We chose this definition because we aimed to focus only on the most disrupted prosody for our analyses. Thus, the dependent variables were whether or not a patient's score on a particular acoustic measure fell in the lowest quartile of the distribution across patients, coded as 0 or 1.

#### RESULTS

## Abnormalities in Acoustic Features Associated With Listener Perception of Impaired Affective Prosody

Mean scores (and SDs) for each of the acoustic features for men and women are shown in **Table 2**. There was no significant difference between men and women in age [male mean = 62.0, female mean = 63.8, *t*(39) = 0.41, ns]. Listener judgments of prosody correlated with certain cues, namely the relative articulation duration, i.e., the relative duration of voiced segments to the total duration of speech segments excluding pauses (Durv/s) (rho = 0.63; *p* < 0.00001) and spectral flatness (SF) (rho = −0.55; *p* = 0.0002). None of the other acoustic features correlated with listener judgment of prosody in univariate analyses.

In multivariable analyses, mean listener rating (from 1 to 7) was best accounted for by a model that included Durv/s, SF, *F*o range, and *F*0 coefficient of variation (*F*0CoV) of fundamental frequency in both women [*F* (4, 12) = 6.58; *p* = 0.0048; *r*<sup>2</sup> = 0.69] and men [*F*(5, 18) = 5.13; *p* = 0.0056; *r*<sup>2</sup> = 0.52]. These identified acoustic measures that correlated with listener judgment of prosody were then considered the "prosody acoustic measures" used in further analyses. The only feature found to be independently associated with rating of emotional expression was Durv/s (*p* < 0.0001) for women, and SF (*p* = 0.007) for men, after adjusting for other variables (age and the other acoustic features, from the set of 26) (**Table 3**). Durv/s was positively correlated with perceived emotional rating in both women (rho = 0.71; *p* = 0.0015) and men (rho = 0.55; *p* = 0.0052), but the correlation was stronger in women. SF was negatively correlated with perceived emotional expression in both women (rho = −0.39; *p* = 0.13) and men (rho = −0.69; *p* = 0.0002), but the association was significant only in men. That is, women (and to a lesser degree, men) who used more voicing were rated as having higher emotional prosody, and men who had higher SF were rated as having lower emotional prosody.

#### Lesions and Demographics Associated With Abnormal Acoustic Features

As indicated above, speech samples with the lowest level of each of the four "prosody acoustic measures" were rated as having the lowest affective prosody by healthy listeners. The lowest quartile of SF was predicted by sex, age, lesion volume, and percent damage to the nine RH regions (*X*<sup>2</sup> = 27; *p* = 0.0081). Sex, lesion volume, damage to inferior frontal gyrus pars opercularis, inferior fronto-occipital (IFO) fasciculus, SLF, and uncinate fasciculus were the only independent predictors, after adjusting for the other variables. The lowest quartile of *F*0CoV was predicted by sex, age, lesion volume, and percent damage to the nine RH regions (*X*<sup>2</sup> = 33; *p* = 0.0005); age and damage to supramarginal gyrus and SLF were the only independent predictors. The lowest

<sup>3</sup>www.stata.com (Accessed: January 6, 2017).



Table 3 | Results of linear regression to identify "prosody acoustic measures" measures that contributed to listener rating of affective prosody.


quartile of Durv/s was predicted by sex, age, education, lesion volume, and percent damage to the nine RH regions (*X*<sup>2</sup> = 25; *p* = 0.02), but none of the variables were independent predictors of Durv/s, after adjustment for other independent variables. The more ventral control regions (STG and sagittal stratum) were not independent predictors of any of the prosody acoustic features in the logistic regression models.

#### DISCUSSION

There are two novel and important results of this study. First, we identified abnormal patterns of acoustic features that contribute to diminished emotional expression of RH stroke survivors, as rated by healthy listeners. The features that together best accounted for diminished emotional expression were: the relative duration of the voiced parts of speech (Durv/s), SF, *F*0 range, and *F*0CoV. The first two features, Durv/s and SF, are measures of rhythm; the latter two features, *F*0 range and *F*0CoV, relate to pitch. Several previous studies have shown that *F*0 range (31) or *F*0 CoV (3, 23, 46) are abnormal in neurological diseases associated with impaired prosody, but most studies have not compared these acoustic features to other acoustic features that might convey emotional expression. We found that Durv/s was particularly important in emotional expression of female stroke participants, and SF was particularly important in emotional expression of male stroke participants. SF (computed as the ratio of the geometric to the arithmetic mean of the spectral energy distribution) has been shown to be important in conveying happy and sad tone-of-voice (47); see also (48). Differences between sexes might reflect differences in which emotions were rated as less emotional in men versus women. It is possible that men were rated as less emotional mostly on the happy and sad stimuli (which depend on SF), whereas women were rated less emotional on emotions that depend more on less noise or breathiness (captured by Durv/s), such as angry and happy. Our study was not powered to evaluate each emotion separately, so this speculation will need to be evaluated in future research.

The variable of Durv/s has been less studied than the other prosody acoustic measures we identified, with respect to emotional communication. However, one study showed that vocal fold contact time (which underlies Durv/s) varied substantially between expression of different emotions (49), consistent with a role for the percentage of voiced speech segments in conveying emotion. Yildirim et al. (50) carried out acoustic analysis of transitions from neutral to happy, sad, or angry speech, and found that angry and happy speech are characterized by longer utterance duration, as well as shorter pauses between words, higher *F*0, and wider ranges of energy, resulting in exaggerated, or hyperarticulated speech (51); but they did not specifically evaluate Durv/s.

The second important finding is that the measures of acoustic features associated with impaired expression of emotion ("prosody acoustic measures") were associated with lesion load in right IFG pars opercularis or supramarginal gyrus, or associated white matter tracts, particularly right IFO fasciculus, SLF, and uncinate fasciculus. These findings are consistent with, but add specificity to, the proposal of a dorsal stream for transcoding acoustic information into motor speech modulation for affective prosody expression in the RH and a ventral stream for transcoding acoustic information into emotional meaning for affective prosody recognition (38, 52). The areas we identified that affected prosody acoustic measures, particularly IFG pars opercularis, supramarginal gyrus, and SLF (roughly equivalent to the arcuate fasciculus) are regions often considered to be included in the dorsal stream of speech production in the left hemisphere (53) and the dorsal stream of affective prosody production in the RH (29, 44). Because we focused on affective prosody expression, we did not provide evidence for the role of the proposed ventral stream. However, lesions in relatively ventral areas, including STG and sagittal stratum (white matter tracts connecting basal ganglia and thalamus with temporal and occipital lobes), which served as control regions, were not associated with impaired (lowest quartile) of prosody acoustic measures in multivariable logistic regression. Other studies are needed to evaluate the cortical and white matter regions associated with recognition of affective prosody. One study identified an association between damage to the sagittal stratum and impaired recognition of sarcastic voice (30).

An important role of right inferior frontal gyrus lesions in disrupting affective prosody expression has also been reported by Ross and Monnot (3). Furthermore, in an fMRI study of healthy controls, evoked expressions of anger (compared with neutral expressions) produced activation in the inferior frontal cortex and dorsal basal ganglia (22). Expression of anger was also associated with activation of the amygdala and anterior cingulate cortex (23), areas important for some aspects of emotional processing, such as empathy (31). The role of disruption to specific white matter tract bundles on affective prosody expression has been less studied than the role of cortical regions. One study showed that in left hemisphere stroke patients, deficits in emotional expression that were independent of the aphasic deficit were associated with deep white matter lesions below the supplementary motor area (which disrupt interhemispheric connections through the mid-rostral corpus callosum) (54). Here, we identified RH white matter tracts that are critical for expression of emotion through prosody, including IFO fasciculus, SLF, and uncinate fasciculus. Results indicate that affective prosody production relies on right IFO fasciculus, SLF, uncinate fasciculus, as well as supramarginal gyrus and inferior frontal gyrus pars opercularis.

Limitations of our study include the relatively small number of patients, which also limited the number of regions of interest we could evaluate. The small number of patients also reduces the power to detect associations between behavior and regions that are rarely damaged by stroke. Thus, there may be other areas that are critical for expression of emotion through prosody. We also did not analyze speech of healthy controls for this study, so

## REFERENCES


we defined as "abnormal" those who were rated as having lowemotional expression by healthy controls. Despite its limitations, this study provides new information on specific gray and white matter regions where damage causes impaired expression of emotion through prosody.

## ETHICS STATEMENT

This study was carried out in accordance with the recommendations of the Johns Hopkins Institutional Review Board with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the Johns Hopkins Institutional Review Board. Analysis of the voice data was also performed under a protocol approved by the Seton Hall University Institutional Review Board.

## AUTHOR CONTRIBUTIONS

SP responsible for acoustic analysis, design of the study, and editing of the paper. KO responsible for image analysis and editing of the paper. AW, HS-F, and SS responsible for data collection and analysis, editing of the paper. AH responsible for data analysis, design of the study, and drafting of the paper.

## FUNDING

The research reported in this paper was supported by the National Institutes of Health (National Institute of Deafness and Communication Disorders) through awards R01DC015466, R01DC05375, and P50 DC014664. The content is solely the responsibility of the authors and does not necessarily represent the views the National Institutes of Health. The authors declare no competing financial interests.


mimiking the feeding of a conspecific. *Front Psychol* (2016) 7:672. doi:10.3389/ fpsyg.2016.00672


network for affective prosody. *Front Hum Neurosci* (2010) 4:19. doi:10.3389/ fnhum.2010.00019


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2018 Patel, Oishi, Wright, Sutherland-Foggio, Saxena, Sheppard and Hillis. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# The Affective Nature of Formulaic Language: A Right-Hemisphere Subcortical Process

Diana Van Lancker Sidtis\* and John J. Sidtis

*Communicative Sciences and Disorders, New York University, New York, NY, United States*

Formulaic expressions naturally convey affective content. The unique formal and functional characteristics of idioms, slang, expletives, proverbs, conversational speech formulas, and the many other conventional expressions in this repertory have been well-described: these include unitary form, conventionalized and non-literal meanings, and reliance on social context. Less highlighted, but potent, is the intrinsic presence of affective meaning. Expletives, for example, signal strong emotion. Idioms, too, inherently communicate emotional connotations, and conversational speech formulas allow for empathetic bonding and humor. The built-in affective content of formulaic expressions, in combination with their other unique characteristics, is compatible with the proposal that brain structures other than those representing grammatical language are in play in producing formulaic expressions. Evidence is presented for a dual process model of language, whereby a right hemisphere-subcortical system modulates formulaic language.

#### Keywords: formulaic language, (LH) damage, PET, RH damage, neurolinguistic

Connotations, affect, attitudes, and emotional meanings inhere essentially in formulaic language—fixed, unitary expressions that are known to a language community. Expletives (Dammit, Good heavens) make this point easily: their purpose is to communicate anger, surprise, shock, disapproval, or excitement (1–4). Idioms engage emotional arousal, subtle or strong, positive or negative. The idiom he's out on a limb communicates worry, risk, failure, and anxiety, while a matched literal sentence, he's out in a boat, is neutral. Don't bite the hand that feeds you carries a warning and a criticism; He pulled the rug out from under us implies disappointment, dismay, and reproachful anger. As a standard ingredient of their meaning—e.g., Sleep with one eye open, In a nutshell, He's at the end of his rope, Just in the nick of time, Quit cold turkey, Shoot for the stars, You lucky dog, You're playing with fire, She has a snowball's chance in hell, I'll be there with bells on—formulaic expressions weave together affect and attitude, which may be empathetic, reproachful, suspicious, or encouraging. Similarly, conversational speech formulas (Okay!, Right!, Really? You're kidding!, Gotcha!; Whatever!; Go to hell; Knock on wood; It's all good; Shut your cakehole!) carry connotations of affirmation or rejection, assent or disapproval, cooperativeness or resistance, through their bonding and affiliative functions (5–7). Routinized speech formulas form a large part of daily talk, communicating "beliefs, wants, wishes, preferences, norms, and values." [(8), p. 239].

#### Edited by:

*Argye Hillis, Johns Hopkins Medicine, United States*

#### Reviewed by:

*Fiona Kumfor, University of Sydney, Australia Charalambos Themistocleous, Johns Hopkins Medicine, United States*

#### \*Correspondence:

*Diana Van Lancker Sidtis diana.sidtis@nyu.edu*

#### Specialty section:

*This article was submitted to Applied Neuroimaging, a section of the journal Frontiers in Neurology*

Received: *06 April 2018* Accepted: *25 June 2018* Published: *24 July 2018*

#### Citation:

*Sidtis DVL and Sidtis JJ (2018) The Affective Nature of Formulaic Language: A Right-Hemisphere Subcortical Process. Front. Neurol. 9:573. doi: 10.3389/fneur.2018.00573*

## NEUROLINGUISTIC BACKGROUND

The early impetus for recognizing the role of formulaic expressions (FEs) in speaking arose from observations in aphasia, using the term "automatic speech." Starting with J. Hughlings Jackson in the nineteenth century (9, 10), clinicians with exposure to aphasia noted that fixed, holistic, known utterances are well-preserved despite severe language impairment [e.g., (11– 17)]; these clinical observations were confirmed by systematic surveys (18–21). Early categories of "automatic" serial speech (counting and days of the week) have been greatly expanded to cover a very large domain. FEs are utilized to communicate in aphasic speech (22) and they play a key role in rehabilitation (23–25).

Examination of monologs from persons with left hemisphere (LH) damage and aphasia reveal high proportions of FEs, while right hemisphere (RH) damage is associated with significantly lower proportions (26, 27). Baldo et al. (28) also reported a trend toward fewer FEs in elicited responses in RH damaged speech than healthy speakers. Formal testing of persons with aphasia supported a preserved ability for FEs (29–31).

Persons with Alzheimer's disease (AD) speak with a preponderance of FEs throughout the progression of the disease; AD leaves the basal ganglia essentially intact for a considerable time (32). In contrast, Parkinsonian disease (PD) arises from impaired subcortical motor nuclei. Experimental studies confirmed that AD speakers' proportions of FEs are higher than healthy speakers, while PD speakers show deficient output (33–36).

## FUNCTIONAL IMAGING AND THE DUAL PROCESS MODEL OF LANGUAGE

The few functional imaging studies dealing with FEs have yielded contradictory results (This review focuses on production and does not include studies of novel metaphor). Using a precursor ( <sup>133</sup>Xe) of PET, Larsen et al. (37) studied subjects at rest or while

for the brain regions predictive of the respective expressive language measures in the linear regression model (47).

counting or reciting the weekdays. Rest values were subtracted from speaking values. For subjects who had the LH studied, there were significant task differences in two of four frontal regions. For subjects whose RH was studied, there were no differences in these regions. Interpretations were problematic because no direct leftright comparisons were possible, the normalization was different for left and right data sets, and task subtraction was employed.

Bookheimer et al. (38) used oxygen-labeled water with PET to study serial-months and the Pledge of Allegiance. Syllable repetition and an oral-motor task were included with a resting state. Using subtraction, the data from the non-propositional tasks were contrasted with data from the rest state. Of the 24 brain regions with blood flow increases, 14 were in the LH while 10 were in the RH. The results regarding functional lateralization were thus not definitive.

Using PET, counting and recitation of nursery rhymes were contrasted with spontaneous monologs (39). All tasks resulted in activation of left hemisphere frontal and temporal sites. This study relied on multiple, complex and simple additions and subtractions of images, lending complexity to interpretation.

Finally, in another PET study, healthy subjects produced animal names, vocalized syllables, and counting. Instead of subtraction, a partial least squares analysis was used (40). Three significant latent variables were identified: one for naming and syllables, with left anterior area predominating over right; a second for naming in bilateral anterior areas, and a third, associated with counting, involved RH and subcortical sites (41). Unlike the previous studies reviewed, these results corresponded to clinical observations, whereby even the most severely aphasic individuals can count.

We report a PET imaging study examining FEs, recently performed in our laboratory, using a complementary approach to activation methods: performance-based analysis. This method explores factors that contribute to cerebral lateralization for language (42, 43). The approach determines if there is a linear combination of brain regions that is predictive of performance during scanning. It is a fundamentally different approach to brain-behavior relationships as it does not rely on group mean differences or task contrasts. Rather, it identifies relationships between individual differences in performance and individual differences in brain activity. This method has consistently yielded functional profiles that are compatible with clinical observations (44).

Speech samples (monologs, syllables and words) produced during scanning were recorded for acoustic and linguistic analyses. From monologs, FEs were quantified as the proportion of FE words in the total word count. Based on previous studies (45, 46), multiple regions were measured for the inferior frontal area and the caudate, bilaterally. The results are presented in **Figure 1**. Using a multiple linear regression analysis, the predictors of speech rate showed that as syllable and word production rates increased, blood flow increased in the left inferior frontal region and decreased in the right caudate. In contrast, the predictive model for the proportion of FEs in the monologs was a complementary pattern of cortical-subcortical interaction. As the proportion of FEs in the monologs increased, blood flow increased in the right inferior frontal region and decreased in the left caudate (47). This laterality profile is consistent with the effects of RH damage on the expression of FEs.

## DISCUSSION

Formulaic expressions naturally carry an affective load. Idioms, proverbs, and other conventional expressions communicate a large range of positive and negative affects, implied within their non-literal meaning. In studies of persons with unilateral lesions and progressive neurological disease, it was observed that formulaic language relies on a cooperation between the cortical RH and subcortical nuclei. Performance based analysis of cerebral blood flow measured during formulaic and propositional speech identified predictive, complementary patterns corresponding with these two modes. Greater use of conversational speech formulas was associated with increased blood flow in the RH and reduced flow in the left caudate. Exemplars of propositional speech were significantly associated with the opposite pattern.

Known characteristics of the brain systems modulating formulaic as contrasted with grammatical language are compatible with the proposed dual model of language [e.g., (48– 51)]. The RH specializes (52, 53) in empathy (including "theory of mind") (54–59), affect and emotional experiencing (60–62), social-context based meanings and pragmatic competence

#### REFERENCES


(28, 63–68), diffuse lexical processes (69–71), personal familiarity (72, 73), and holistic configurations (74–76).

The basal ganglia stores and processes overlearned motor gestures. The characteristics of subcortical structures, shown to be important in FE production, include modulating routinized motor and verbal gestures (77, 78), including grammatical elements (79, 80) and recited speech (81, 82). Basal ganglia impairment interferes with normal production of FEs (36).

Both of these structures, RH and basal ganglia, in their intrinsic functionality are well-suited to the properties of FEs (see **Figure 2**).

In the dual processing language model, two distinctive modes of language competence exist: formulaic and grammatical (83–87). These language modes have different intrinsic characteristics and rely on disparate cerebral systems. Further studies can look toward uncovering the cerebral switching mechanisms that allow for smooth integration of these two modes in fluent speech. Recognition of the dual process of language competence has important implications for our understanding of first language acquisition, second language learning, and clinical rehabilitation of language disorders.

## AUTHOR CONTRIBUTIONS

DS wrote major portions of paper, partly designed and contributed to experimental study reported in the paper. JS wrote portions of paper, partly designed and contributed to experimental study reported in paper, analyzed functional imaging results.

## ACKNOWLEDGMENTS

This work was supported by the National Institute of Deafness and Communicative Disorders (R01 DC007658). The assistance of Phoebe Spetsieris and the PET imaging team at the Feinstein Institute, and Michele Burgevin, Vanessa Monserrat, and the members of the Brain and Behavior Laboratory and the Discovery Science Project at the Nathan Kline Institute are gratefully appreciated.


metaanalysis of previous data. Neuropsychologia (1989) 27:1165–78. doi: 10.1016/0028-3932(89)90099-7


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The reviewer CT and handling Editor declared their shared affiliation.

Copyright © 2018 Sidtis and Sidtis. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.