# THE IMPACT OF SENSORY, LINGUISTIC AND SOCIAL DEPRIVATION ON COGNITION

EDITED BY: Matthew Dye and Olivier Pascalis PUBLISHED IN: Frontiers in Psychology

#### *Frontiers Copyright Statement*

*© Copyright 2007-2017 Frontiers Media SA. All rights reserved. All content included on this site, such as text, graphics, logos, button icons, images, video/audio clips, downloads, data compilations and software, is the property of or is licensed to Frontiers Media SA ("Frontiers") or its licensees and/or subcontractors. The copyright in the text of individual articles is the property of their respective authors, subject to a license granted to Frontiers.*

*The compilation of articles constituting this e-book, wherever published, as well as the compilation of all other content on this site, is the exclusive property of Frontiers. For the conditions for downloading and copying of e-books from Frontiers' website, please see the Terms for Website Use. If purchasing Frontiers e-books from other websites or sources, the conditions of the website concerned apply.*

*Images and graphics not forming part of user-contributed materials may not be downloaded or copied without permission.*

*Individual articles may be downloaded and reproduced in accordance with the principles of the CC-BY licence subject to any copyright or other notices. They may not be re-sold as an e-book.*

*As author or other contributor you grant a CC-BY licence to others to reproduce your articles, including any graphics and third-party materials supplied by you, in accordance with the Conditions for Website Use and subject to any copyright notices which you include in connection with your articles and materials.*

> *All copyright, and all rights therein, are protected by national and international copyright laws.*

*The above represents a summary only. For the full conditions see the Conditions for Authors and the Conditions for Website Use.*

ISSN 1664-8714 ISBN 978-2-88945-354-2 DOI 10.3389/978-2-88945-354-2

### About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

### Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

### Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

### What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office: researchtopics@frontiersin.org

# **THE IMPACT OF SENSORY, LINGUISTIC AND SOCIAL DEPRIVATION ON COGNITION**

Topic Editors: **Matthew Dye,** National Technical Institute for the Deaf, United States **Olivier Pascalis,** Université Grenoble Alpes, France

Early experience plays a crucial role in determining the trajectory of cognitive development. For example, early sensory deprivation is known to induce neural reorganization by way of adaptation to the altered sensory experience. Neville and Bavelier's "compensatory theory'' hypothesizes that loss of one sense may bring about a sensory enhancement in the remaining modalities. Sensory deprivation will, however, also impact the age of emergence, or the speed of acquisition of cognitive abilities that depend upon sensory inputs.

Understanding how a child's early environment shapes their cognition is not only of theoretical interest. It is essential for the development of early intervention programs that address not just the early deprivation itself, but also the cognitive sequelae of such deprivation. The articles in this e-book all address different aspects of deprivation - sensory, linguistic, and social - and explore the impacts of such deprivation on a wide range of cognitive outcomes.

In reading these contributions, it is important to note that sensory, linguistic, and social deprivation are not independent factors in human experience. For example, a child born deaf into a hearing family is likely to experience delays in exposure to natural language, with subsequent limits on their linguistic competence having an effect on social interactions and inclusion: a child raised in environments where social interaction is highly limited is also likely to experience reductions in the quantity and quality of linguistic inputs. Future work will need to carefully examine the complex interactions between the sensory, linguistic and social environments of children raised in atypical or impoverished environments.

**Citation:** Dye, M., Pascalis O., eds. (2017). The Impact of Sensory, Linguistic and Social Deprivation on Cognition. Lausanne: Frontiers Media. doi: 10.3389/978-2-88945-354-2

# Table of Contents


Marjorie Dole, David Méary and Olivier Pascalis

*48 Auditory and Visual Electrophysiology of Deaf Children with Cochlear Implants: Implications for Cross-modal Plasticity*

David P. Corina, Shane Blau, Todd LaMarr, Laurel A. Lawyer and Sharon Coffey-Corina


Hannah Pimperton, Amelia Ralph-Lewis and Mairéad MacSweeney

### **II. Sensory Deprivation: Blindness**


*122 The Influence of Tactile Cognitive Maps on Auditory Space Perception in Sighted Persons*

Alessia Tonelli, Monica Gori and Luca Brayda

*131 Cerebral versus Ocular Visual Impairment: The Impact on Developmental Neuroplasticity*

Maria B. C. Martín, Alejandro Santos-Lozano, Juan Martín-Hernández, Alberto López-Miguel, Miguel Maldonado, Carlos Baladrón, Corinna M. Bauer and Lotfi B. Merabet

### **III. Linguistic and Social Deprivation**

*140 Visible Social Interactions Do Not Support the Development of False Belief Understanding in the Absence of Linguistic Input: Evidence from Deaf Adult Homesigners*

Deanna L. Gagne and Marie Coppola


Florin Tibu, Margaret A. Sheridan, Katie A. McLaughlin, Charles A. Nelson, Nathan A. Fox and Charles H. Zeanah

# Editorial: The Sensation-Cognition Interface: Impact of Early Sensory Experiences on Cognition

Matthew W. G. Dye<sup>1</sup> \* and Olivier Pascalis <sup>2</sup>

<sup>1</sup> Department of Liberal Studies, Rochester Institute of Technology, Rochester, NY, United States, <sup>2</sup> CNRS, Université Grenoble Alpes, Saint-Martin-d'Hères, France

Keywords: blindness, language deprivation, plasticity, cognitive development, spatial localization, auditory perception, visual perception, deafness

**Editorial on the Research Topic**

#### **The Sensation-Cognition Interface: Impact of Early Sensory Experiences on Cognition**

The fourteen articles in this special topic are linked by their consideration of how various kinds of experiential deprivation can affect cognitive development. Reflecting recent increased interest in multisensory processing, most submissions look at how the absence of one sensory modality influences processing in another. Codina et al. studied perceptual changes in the peripheral visual field induced by deafness and/or sign language. They compared performance in the central and peripheral vision of deaf signers, hearing signers and hearing non-signers in a forced choice task. Deaf participants performed better than the other groups for peripherally presented stimuli, suggesting that sign language acquisition alone does not change peripheral vision in the way that deafness does. Samar and Berger investigate the hypothesis of reallocation of attentional resources from the central to the peripheral visual field in the deaf, using a spatial attention paradigm requiring target localization. Comparing deaf participants with or without cochlear implants (CIs) with hearing participants, they surprisingly observed that deaf with no CIs show a reduced central attentional capacity which was not associated with enhanced peripheral attention. In the perception of faces, a left visual field bias typically exists which suggests a right hemisphere specialization. This bias is already observed in 5-year-old hearing children. Dole et al. found a reduced left visual field bias in deaf adults compare to hearing peers when using chimeric faces. This result was associated with increased looking time toward the mouth for deaf participants. Early profound deafness is therefore associated with differences in face scanning and could induce a change in hemispheric specialization possibly linked to speech reading. Pimperton et al. investigated whether deaf adults with CIs showed significantly better scores on a test of speechreading than hearing, hypothesizing that the age of implantation might influence this ability. They found that the deaf with CIs were better speechreaders than the hearing, with a significant positive correlation between age at implantation and speechreading performance: earlier implantation was associated with poorer speechreading scores. Aparicio et al. investigated the neural substrate of speech reading and speech with cued speech (CS) recruiting deaf individuals, CS users, and individuals with typical hearing. Their study explores the neural similarities and differences in processing oral language delivered in a visuo-manual or in an audio-visual modality. They found a common, amodal neural basis for the perception of both audiovisual speech and CS, but clear differences were observed in the posterior parts of the superior temporal sulcus for auditory and speechread information in audiovisual speech, and in the occipitotemporal junction for CS. Corina et al. recorded auditory and visual evoked potentials in hearing children and deaf children with CIs. They reported an atypical auditory P1 in the deaf children, even with early implantation and significant auditory experience, which they interpret as potentially reflecting an aberrant maturation of cortical function early in development. They found no differences in the visual evoked potentials.

#### Edited and reviewed by:

Michael Banissy, Goldsmiths, University of London, United Kingdom

> \*Correspondence: Matthew W. G. Dye mwddls@rit.edu

#### Specialty section:

This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology

Received: 29 July 2017 Accepted: 20 September 2017 Published: 06 October 2017

#### Citation:

Dye MWG and Pascalis O (2017) Editorial: The Sensation-Cognition Interface: Impact of Early Sensory Experiences on Cognition. Front. Psychol. 8:1742. doi: 10.3389/fpsyg.2017.01742 When analyzing the relationship between auditory P1 and visual N1 within participants, they found a significant correlation in the hearing children not in deaf children with CIs.

In addition to these studies of deaf children and adults, several contributions looked at cognition in blind children and adults. Cappagli et al. looked at auditory spatial localization in preschool children who were congenitally blind or had low vision. Their results suggest that some early visual experience is required for the development of domain-general spatial cognition. Gori et al. report a study that required early blind, late blind and sighted adults to report the shape made by a moving sound source and then to replicate that movement in a locomotion task. They report that early blind, but not late blind, individuals struggled to identify the auditory motion paths and, even when they could correctly do so, their ability to reproduce those shapes in the locomotion task was error prone. Tonelli et al. asked whether vision is necessary for the calibration of auditory space, or whether the same function can be subserved by touch. In blindfolded adults with typical vision, they report that haptic exploration of a 3D model of a room, accompanied by ambulatory exploration of the room while blindfolded, resulted in improved bisection of an auditory line. Subsequent visual exploration did not improve bisection any further. Finally, two reviews explore different aspects of cognition in blind individuals. Voss reviews the literature on auditory spatial cognition in the blind and suggests that data must be interpreted in terms of task demands. In particular, findings differ depending upon whether testing occurs in the horizontal or vertical plane, whether depth judgments are absolute or relative, and whether tasks are performed mono- or binaurally. He suggests that the specificity of effects likely represents the use of different auditory spectral cues. Martin et al. point out the importance of considering etiology in their review of ocular versus cortical/cerebral visual impairment (CVI). They point out that CVI is now a highly prevalent cause of early childhood blindness in developed countries, and that it is associated with a range of attentional and visual dysfunctions that mean rehabilitation strategies designed for those with ocular deficits are ineffective for individuals with CVI.

Together, these eleven articles on sensory deprivation deafness and blindness—reveal some striking theoretical and practical parallels. The cross-sensory calibration hypothesis of Gori et al. (2012) bears a remarkable similarity to the auditory scaffolding hypothesis proposed by Conway et al. (2009), and provides a potential unifying framework for understanding how a lack on input in one sensory modality affects development in another [see also the intersensory redundancy hypothesis of Bahrick and Lickliter (2000)]. There are also important methodological considerations that arise from both literatures.

### REFERENCES


Voss points out that early and late blind individuals may make use of different auditory spectral cues, and may make use of different reference frames. In addition, he also notes that transient visual deprivation (such as by blindfolding) results in effects not observed in early blind individuals. In the same way, whether or not a deaf child has a CI (and the age at which they receive one) may also results in different patterns of compensatory change, and the work of Bross et al. established that transient auditory deprivation results in visual changes not observed in those born profoundly deaf (Bross et al., 1980). Last of all, Martin et al.'s review of ocular versus cerebral etiologies for visual impairment mirrors discussion in the field of deafness about carefully considering the etiology of hearing loss in deaf individuals (Bavelier et al., 2006).

The last three contributions to this special topic reflect the effects of different kinds of deprivation on cognition linguistic and social deprivation. Gagne and Coppola ask whether observing social interactions is sufficient to support social cognition in deaf adults who have not acquired conventional language. They report that these Homesigners performed well on simple visual perspective taking tasks. However, they struggled on false belief tasks, as did unschooled signers and speakers of conventional languages. The authors suggest that language may be important for the development of theory of mind, but that it is not sufficient. Henner et al. examined how the age of entry to a signing school for the deaf (and, by proxy, age of sign language exposure) affects the ability to make syntactic judgments and perform an analogical reasoning task. Their data suggest that earlier entry into signing schools for the deaf is, in these children, associated with better ASL syntactic knowledge and improved analogical reasoning skills. Finally, Tibu et al. analyzed data from the Bucharest Early Intervention Project, looking at the effects of institutionalization on the development of ADHD symptoms. They report an analysis to show that the effect of social and communicative deprivation on the emergence of ADHD symptoms at age 12 years is mediated by impaired working memory. These studies show that it is not only sensory experience that shapes cognitive abilities; social and linguistic interaction also play an important role in the development of a child's ability to think and reason. Further research is needed for a better holistic understanding of how sensory, linguistic and social interactions are interrelated and work together to shape and dictate the direction of cognitive development.

### AUTHOR CONTRIBUTIONS

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.


Gori, M., Sandini, G., and Burr, D. (2012). Development of visuoauditory integration in space and time. Front. Integr. Neurosci. 6:77. doi: 10.3389/fnint.2012.00077

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Dye and Pascalis. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Does a Flatter General Gradient of Visual Attention Explain Peripheral Advantages and Central Deficits in Deaf Adults?

#### Vincent J. Samar<sup>1</sup> \* and Lauren Berger<sup>2</sup>

<sup>1</sup> NTID Department of Liberal Studies, Rochester Institute of Technology, Rochester, NY, USA, <sup>2</sup> PhD Program in Educational Neuroscience, Gallaudet University, Washington, DC, USA

Edited by: Olivier Pascalis, Université Grenoble Alpes, France

#### Reviewed by:

Chloe Farrer, Institut des Sciences Cognitives – CNRS, France Richard Palluel Germain, Université Grenoble Alpes, France

> \*Correspondence: Vincent J. Samar vjsncr@rit.edu

#### Specialty section:

This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology

Received: 20 November 2016 Accepted: 21 April 2017 Published: 16 May 2017

#### Citation:

Samar VJ and Berger L (2017) Does a Flatter General Gradient of Visual Attention Explain Peripheral Advantages and Central Deficits in Deaf Adults? Front. Psychol. 8:713. doi: 10.3389/fpsyg.2017.00713 Individuals deaf from early age often outperform hearing individuals in the visual periphery on attention-dependent dorsal stream tasks (e.g., spatial localization or movement detection), but sometimes show central visual attention deficits, usually on ventral stream object identification tasks. It has been proposed that early deafness adaptively redirects attentional resources from central to peripheral vision to monitor extrapersonal space in the absence of auditory cues, producing a more evenly distributed attention gradient across visual space. However, little direct evidence exists that peripheral advantages are functionally tied to central deficits, rather than determined by independent mechanisms, and previous studies using several attention tasks typically report peripheral advantages or central deficits, not both. To test the general altered attentional gradient proposal, we employed a novel divided attention paradigm that measured target localization performance along a gradient from parafoveal to peripheral locations, independent of concurrent central object identification performance in prelingually deaf and hearing groups who differed in access to auditory input. Deaf participants without cochlear implants (No-CI), with cochlear implants (CI), and hearing participants identified vehicles presented centrally, and concurrently reported the location of parafoveal (1.4◦ ) and peripheral (13.3◦ ) targets among distractors. No-CI participants but not CI participants showed a central identification accuracy deficit. However, all groups displayed equivalent target localization accuracy at peripheral and parafoveal locations and nearly parallel parafoveal-peripheral gradients. Furthermore, the No-CI group's central identification deficit remained after statistically controlling peripheral performance; conversely, the parafoveal and peripheral group performance equivalencies remained after controlling central identification accuracy. These results suggest that, in the absence of auditory input, reduced central attentional capacity is not necessarily associated with enhanced peripheral attentional capacity or with flattening of a general attention gradient. Our findings converge with earlier studies suggesting that a general graded trade-off of attentional resources across the visual field does not adequately explain the complex task-dependent spatial distribution of deaf-hearing

performance differences reported in the literature. Rather, growing evidence suggests that the spatial distribution of attention-mediated performance in deaf people is determined by sophisticated cross-modal plasticity mechanisms that recruit specific sensory and polymodal cortex to achieve specific compensatory processing goals.

Keywords: deafness, attention, cross-modal plasticity, cochlear implant, peripheral advantage, central deficit

### INTRODUCTION

Deaf people often show enhanced performance in the visual periphery for certain tasks, like motion detection and spatial localization (Parasnis and Samar, 1985; Bavelier et al., 2006; Pavani and Bottari, 2012), which typically involve auditory–visual integration in hearing people (Dye et al., 2009). This enhancement presumably compensates for the loss of cross-modal auditory information that normally helps individuals to visually orient to unexpected events in their complex changing environment (Parasnis et al., 2003; Bavelier et al., 2006). Neuroimaging and behavioral studies converge to support the hypothesis that peripheral performance enhancements in deaf people are specifically related to population differences in peripheral attentional control (Neville and Lawson, 1987; Bavelier et al., 2000; Bavelier and Neville, 2002). Several studies suggest that in the absence of attentional demands, deaf and hearing people do not differ in performance on peripheral psychophysical tasks like motion processing (Brozinsky and Bavelier, 2004), brightness discrimination (Bosworth et al., 2013), or visual contrast sensitivity (Finney and Dobkins, 2001).

Given prior evidence that peripheral enhancement in deaf people is attention dependent, Proksch and Bavelier (2002) proposed the influential hypothesis that early auditory deprivation alters the gradient of visual attention from the central to peripheral field. Using an interference-from-distraction search task to measure attentional resources, they reported that peripheral distractors interfered with visual search performance more for deaf than hearing participants, whereas central distractors interfered more for hearing participants. Based on these results, Proksch and Bavelier (2002) suggested that early auditory deprivation may expand peripheral attentional resources by drawing resources away from central vision to more equally distribute limited resources across visual space, essentially flattening the gradient of attention in deaf people relative to hearing people. However, they also acknowledged that it remains unclear whether peripheral enhancements and central deficits are linked or are determined by different mechanisms.

The altered gradient of attention proposal is a general hypothesis that offers a neurally plausible model (Pavani and Bottari, 2012) to explain both the peripheral advantages and central deficits reported in the literature across a variety of attention-demanding tasks, irrespective of specific task demands. However, there is currently no definitive evidence to support this proposal as a general mechanism. Prior to and since Proksch and Bavelier's (2002) study, peripheral enhancements and central deficits in deaf children and adults have been reported by several researchers in a variety of attention-demanding experimental tasks. Typically, peripheral enhancements are found in studies that use dorsal stream tasks (e.g., spatial localization or motion detection) and central deficits are found in studies that use ventral stream tasks (e.g., object identification). However, other than possibly Proksch and Bavelier (2002), we are unaware of any studies that have reported peripheral enhancements concurrently coupled to central deficits, either on the same or different tasks, within the same participants. Typically, studies report peripheral enhancements or central deficits, but not both. Furthermore, one recent divided attention study explicitly designed to test Proksch and Bavelier's (2002) altered gradient of attention proposal reported a central deficit but no peripheral enhancement (Dye, 2016). Most previous studies were not designed with methodological controls to rigorously test the proposal (e.g., controls for task demands at different eccentricities) and therefore their results have limited evidentiary value.

In the present paper we briefly review the gradient of attention construct and examine the limitations of the existing related literature on early deafness. We then describe the results of an experiment to test the altered gradient of attention proposal using a novel divided visual attention paradigm that overcomes some of the methodological limitations of previous studies. This paradigm involved an object identification task presented in central vision and a concurrent target localization task designed to measure an attentional localization performance gradient from near-central parafoveal locations to peripheral locations. This design allowed us to retain the attention-demanding advantages of a conventional dorsal/ventral stream divided attention task while simultaneously examining the gradient of attention unconfounded by task-specific processing differences. We compared the performance of hearing participants with the performance of prelingually deaf participants with and without cochlear implants (CI) to test specific literature-based predictions of the effect of reduced auditory input on the spatial distribution of attention.

### Gradient of Attention Construct

The gradient of attention construct refers to a continuous decrease in allocation of processing capacity as a function of increasing stimulus eccentricity away from the attended location (LaBerge and Brown, 1989). LaBerge and Brown (1989) defined a formal model that incorporates both a fixed structural acuity gradient and an independent attentional gradient that can be dynamically reshaped under different task demands to alter perceptual performance across the visual field. For example, their model allows for the center of attention to move away from fixation depending on task demands. However, in tasks involving unguided attention to locations symmetric about fixation and central processing demands or foveal loads, as is typical of many

studies of enhanced peripheral attention in deaf people, the center of vision is generally the focus of attention (LaBerge and Brown, 1989; Staugaard et al., 2016).

Previous research on hearing people supports this model. Spatial attention is generally resource limited, and attentional gradients diminish continuously from the focus of attention, with the spread of the gradient adjusting to match the range of possible target locations in distributed attention tasks (see Bush, 2012, for a review). Consistent with the proposal that early auditory deprivation drives a more equal distribution of attentional resources toward the periphery, experimental conditions that require people to spread their attention over a wider spatial range under exactly the same task requirements cause them to move some attentional resources away from the center of attention, altering the availability of attentional resources everywhere within the range, including at and near the center of attention (Greenwood and Parasuraman, 1999; Bush, 2012), thereby flattening the slope of the gradient.

Most recently, Staugaard et al. (2016) used Bundesen's (1990) Theory of Visual Attention to confirm that several independently estimable components of attention (visual short term memory capacity, visual perceptual threshold, visual processing speed) diminish monotonically with increasing target eccentricity away from central vision, independent of visual system structural gradients like the cortical magnification factor and of eccentricity dependent motor reactions. Staugaard et al. (2016) reported further that manipulating endogenous attention did not alter these attentional gradients, but cited Proksch and Bavelier's (2002) study to support the claim that long-term environmental factors may lead to a compensatory trade-off between attentional resources in peripheral vs. central vision.

### Early Deafness and the Gradient of Attention

Although some behavioral and neurophysiological studies of visual performance in deaf adults and children are consistent with an experience-driven altered general gradient of attention proposal, other than possibly Proksch and Bavelier (2002), studies have not demonstrated that peripheral enhancements are concurrently linked to central deficits as a limited attentional resource model would predict. Peripheral enhancements without concurrent central deficits have been shown in several behavioral and neuroimaging studies (e.g., Neville and Lawson, 1987; Loke and Song, 1991; Bavelier et al., 2000). Conversely, Dye (2016) showed central deficits on a divided visual attention task but equivalent concurrent peripheral performance. Similarly, Sladen et al. (2005) showed that deaf adults display greater interference than hearing adults from incompatible flankers at a parafoveal location, but equal interference at a central location, suggesting that a broader spread of visual resources in the deaf group was not accompanied by a central deficit in visual resources. Parasnis and Samar (1985) showed enhancement for reorienting to a peripheral stimulus when central stimuli compete for attention, and Shiell et al. (2014) showed peripheral enhancement for visual motion detection thresholds, but neither study tested performance centrally. Other attention studies have shown central deficits on continuous performance tests (CPT), but they did not test at peripheral locations to confirm an attentional tradeoff (Quittner et al., 1994; Mitchell and Quittner, 1996; Smith et al., 1998; Parasnis et al., 2003; Horn et al., 2005). Still other CPT studies failed to show central deficits for deaf children (Tharpe et al., 2002; Dye and Hauser, 2014). Bosworth and Dobkins (2002) showed no reliable peripheral advantages or central deficits for coherent motion detection thresholds for deaf participants, even when attention was cued to the target stimulus location. Thus, no consistent picture of concurrently coupled peripheral advantages and central deficits associated with auditory deprivation has emerged from the literature. Generally, attention studies comparing deaf and hearing participants have varied widely in experimental design, task demands, eccentricity of stimuli, and participant deaf group characteristics (e.g., chronological age, age of onset and etiology of deafness, CI use, controls for medical or developmental conditions). Hence, methodological limitations and participant group differences could potentially account for the lack of consistent results across studies (Bosworth and Dobkins, 2002; Dye et al., 2009).

### Divided Attention Studies and the Gradient of Attention

Previous research collectively indicates that compensatory changes in the distribution of attention across the visual field associated with auditory deprivation are best revealed by attention-dependent paradigms involving competing central and peripheral tasks, uncertainty about target location, and the presence of distractor stimuli (Bavelier et al., 2006; Dye and Bavelier, 2013), all conditions typical of real world environments. Experimentally, these conditions are most closely approximated in divided selective attention paradigms that require participants to perform a central ventral stream task (e.g., an object identification task) and a concurrent peripheral dorsal stream task (e.g., a localization or movement detection task) in the presence of spatially distributed distractors. Furthermore, studies have shown that performance on divided attention tasks, especially those that engage working memory and involve central identification tasks and peripheral localization or motion detection tasks, predict real-world daily life performance in a number of normal and clinical populations (Clay et al., 2005; Miloyan et al., 2013). Since the peripheral advantage in deaf people has been generally regarded as an adaptation to compensate for the loss of auditory information in real world settings (Parasnis, 1983; Parasnis et al., 2003; Bavelier et al., 2006; Pavani and Bottari, 2012), we would expect divided attention paradigms involving central ventral stream tasks and peripheral dorsal stream tasks to provide an ideal laboratory protocol for testing the altered gradient of attention proposal.

Few previous studies have searched for peripheral advantages and central deficits using divided attention paradigms. Bosworth et al. (2013) used a divided attention paradigm to compare deaf and hearing adults on static stimulus orientation discrimination

and motion perception performance tasks within central and peripheral regions while participants concurrently counted target shapes in a central RSVP task. They reported no peripheral advantages or central deficits for either orientation or motion tasks. Importantly, the performance of deaf subjects on the central RSVP task was significantly worse than the performance of hearing subjects regardless of whether the RSVP stimuli occurred during the motion or orientation tasks at peripheral or central locations. Thus, these results reveal a selective central deficit for object detection (a ventral stream task) in deaf adults compared with hearing adults in a divided attention paradigm, but equivalent group performance for detecting motion (a dorsal stream task) as well as for discriminating static orientation (a competing ventral stream task) at both central and peripheral locations. These results suggest that attentional control of central task performance may be independently determined by specific task demands rather than governed by a redistribution of general attentional resources across space. Therefore, this result does not support the altered gradient of attention proposal.

Dye et al. (2009) tested deaf and hearing adults and children on the Useful Field of View (UFOV) test, a divided selective attention paradigm that requires participants to localize a peripheral target among distractors and concurrently discriminate the identity of a central target. Deaf adults had shorter peripheral stimulus duration thresholds than hearing adults on the UFOV, but not on a simpler divided attention task not involving distractors. Dye et al. (2009) attributed these UFOV results to enhanced attention to peripheral stimuli due to auditory deprivation. However, they only measured peripheral thresholds on trials where both central identification and peripheral localization were correct, and did not independently identify thresholds at the central site. Consequently, relative group performance on the central identification task was unknown. Therefore, as Dye (2016) acknowledged, the performance enhancement observed in Dye et al. (2009) cannot be linked to a shift in the gradient of attention for deaf participants involving selective peripheral enhancement and concurrently deficient central performance.

Dye (2016) disentangled participants UFOV performance on the central identification and peripheral localization tasks by measuring separate peripheral and central thresholds concurrently to specifically test Proksch and Bavelier's (2002) altered gradient of attention proposal. Dye reported a deficit on the central identification task for deaf adults that only appeared under attentionally demanding competition from peripheral targets and distractors. However, contrary to expectation and to the previous results of Dye et al. (2009), deaf and hearing adults did not differ on peripheral performance. Thus, under demanding conditions of divided attention between central identification and peripheral localization tasks, deaf adults did not display superior peripheral performance despite an apparent reduction of attentional resources in central vision, suggesting, like Bosworth et al.'s (2013) results, that the central deficit might have been specific to Dye's (2016) foveated ventral stream object identification task.

## The Divided Gradient of Attention Paradigm (DGAP)

Dye's (2016) results converge with those of Bosworth et al. (2013) to suggest that peripheral advantages and central deficits in previous studies may have been caused by independent mechanisms rather than by an altered general gradient of attention. However, both of their paradigms have methodological limitations for testing the altered gradient of attention proposal. In general, the UFOV paradigm that Dye (2016) used completely confounds ventral and dorsal stream task demands with central vs. peripheral stimulus location, respectively. Because object identification and target localization are mediated by distinct ventral and dorsal stream mechanisms (Ungerleider and Haxby, 1994; Weisberg et al., 2012), a flattening of the performance gradient due either to a peripheral advantage alone, a central deficit alone (as in Dye), or even both concurrently, cannot distinguish between a redistribution of general attentional resources from central to peripheral locations on the one hand, and population differences in underlying neural control of specific task performance. Bosworth et al.'s (2013) paradigm does nominally overcome this confound in that their orientation and motion stimuli were presented at both central and peripheral locations while participants concurrently performed a foveated identification task. However, their central and peripheral motion and orientation stimuli subtended a full 5 ◦ of visual angle within three immediately adjacent regions that spanned a total of only 7.5◦ on either side of fixation. Therefore, participants' performance at the study's nominal peripheral locations actually integrated across broad regions of parafoveal space that were closely situated to the nominal central location which, itself, integrated across foveal and parafoveal regions. This design cannot provide sufficiently high resolution to measure performance at discrete, well separated locations from central to peripheral regions across the attentional gradient.

In the present study, we employed a new divided attention paradigm to test the altered gradient of attention proposal that preserves the ecologically sensible combination of a central identification task and an eccentric localization task while simultaneously dissociating central task performance from performance along the gradient of attention for the localization task. Participants performed a vehicle identification task that required them to distinguish cars from other vehicles on each trial and keep track of how many target vehicles they saw across a block of trials (**Figure 1**). The central identification task included many distinct vehicle exemplars and a sustained working memory component, establishing a demanding level of competition with the localization task. Participants simultaneously identified the location of a target stimulus (X) among symmetrically placed distractors (circles) presented unpredictably at either near-central parafoveal (1.4◦ from fixation) or peripheral (13.3◦ from fixation) locations (**Figure 1**). Previous studies have reported attentional advantages for deaf participants over a broad range of eccentricities from 2 ◦ to more than 20◦ (Parasnis and Samar, 1985; Pavani and Bottari, 2012). Our use of a near-central parafoveal location less than 1.5◦ from fixation, and a peripheral location well into

the range of previously reported peripheral advantage effects, allowed us to examine the linear slope of the localization gradient between these two discrete points for deaf and hearing groups when attention was spread broadly over a considerable angular distance, independent of their central identification task performance.

### Hypotheses

We developed two related hypotheses for performance on the DGAP to test the altered gradient of attention proposal. Hypothesis 1 is that deaf participants without CIs will show a set of related effects, namely worse central identification performance, better peripheral localization performance, and a flatter parafoveal-peripheral response gradient in the localization task, relative to hearing participants. This hypothesis is directly implied by the previous literature on attentional gradients and the altered gradient of attention proposal. Confirmation of only the central identification deficit would not support the altered gradient of attention hypothesis, but would be consistent with the alternative hypothesis that the central identification deficit is independent of attentional effects on competing tasks at other eccentricities.

Hypothesis 2 is contingent on confirmation of Hypothesis 1. Assuming Hypothesis 1 is supported for deaf participants without CIs, we predict a selective pattern of performance for deaf participants with CIs. Specifically, we hypothesize that deaf participants with CIs will not show inferior central identification task performance, superior peripheral localization performance, or a flatter localization gradient than the hearing group. This hypothesis is motivated by previous CPT studies that report that deaf children without CIs show central attentional deficits, but deaf children with CIs show central attentional skills that approach those of hearing children by about 18 months after implantation (Quittner et al., 1994; Smith et al., 1998). Accordingly, consistent with the altered gradient of attention proposal, previous authors have speculated that CI users, unlike CI non-users, should not display enhanced target or motion detection in the periphery since they can use auditory cues to support cross-modal integration (Kim et al., 2016). Note that if Hypothesis 1 is not supported, confirmation of Hypothesis 2 offers no support for the altered gradient of attention hypothesis since non-inferiority of performance would not then be selectively associated with restored auditory input.

The altered gradient of attention hypothesis also predicts that the hearing-typical attention gradient should be restored in CI users as a function of time-with-implant. Specifically, across CI-implanted individuals, we would expect increased time-withimplant to be associated with reduced central attentional deficits and with contemporaneously reduced peripheral advantages due to newly restored auditory experience. However, limited research on deaf children has failed to find a correlation between timewith-implant and central attentional performance (Smith et al., 1998). No one has tested the relationship in any attentiondependent tasks in deaf adult CI users. Therefore, we explored this relationship for our two tasks to determine whether it

conformed to the predictions of the altered gradient of attention hypothesis.

Most previous studies have not screened their deaf or hearing participants for known or hidden attention and executive function deficits that can influence group performance differences in attention studies. Considering that ADHD is a high prevalence, often hidden disorder in all groups and especially in the deaf population (Samar et al., 1998), this is a significant methodological shortcoming that we address in two ways. First, we tested only participants who reported no history of ADHD diagnosis. Second, we administered the Behavior Rating Inventory of Executive Functions–Adult Form (BRIEF-A) to all participants to control for variation in executive function and its signature disorder, ADHD. We have previously validated the BRIEF-A for use with deaf adult college students, and have shown that it is sensitive to the presence of ADHD in this population (Hauser et al., 2013).

### MATERIALS AND METHODS

### Participants

Twenty-five deaf and 25 hearing students at the Rochester Institute of Technology were recruited through flyers and RIT's on-line experiment participation system and were paid \$10 and/or given psychology course participation credit. They were screened for self-reported history of ADHD diagnosis, neurological disorders, severity and age of onset of hearing loss, and vision problems. Seven deaf and three hearing participants were eliminated because of a history of Usher Syndrome, neurological disorders or illness, ADHD diagnosis, or becoming deaf after 3 years old. The remaining 22 prelingually deaf young adults, 11 with cochlear implants (CI group) and 11 without cochlear implants (No-CI group), and 23 hearing young adults were included in the study.

**Table 1** presents psychometric and demographic characteristics for these groups. The groups had comparable overall scores on the BRIEF-A near the normative population mean (T-score = 50) and all individual subjects scored within approximately 1.6 SD of the mean, indicating no evidence of attentional or other executive function disorder in any group. Gender composition across groups was significantly different. The two deaf groups had somewhat different distributions of previous school types but were otherwise comparable. All deaf participants reported early severe to profound hearing loss, however, audiometric hearing loss data were not available.

### Measures

### Behavior Rating Inventory of Executive Functions–Adult Form (BRIEF-A, Roth et al., 2005)

The BRIEF-A is a 75-item self-report instrument with nine clinical scales: Inhibit, Shift, Emotional Control, Self-Monitor, Initiate, Working Memory, Plan/Organize, Task Monitor, and Organization of Materials- The instrument yields a Behavioral Regulation Index (BRI), a Metacognitive Index (MI), and a Global Executive Composite (GEC) that provides an overall measure of executive function. Hauser et al. (2013) have demonstrated psychometrically that the BRIEF-A is a reliable, unbiased diagnostic tool for use with deaf college students, with discriminant and predictive validity for ADHD diagnosis comparable to that for the hearing college population.

### Divided Gradient of Attention Paradigm (DGAP)

The DGAP consists of a central object identification task and a concurrent spatial localization task. The object identification task required participants to identify centrally presented vehicles as belonging to the class of cars or non-cars (other vehicles). Either cars or non-cars were designated as target vehicles and the other vehicle set as non-targets, counterbalanced across participants (**Figure 1**). At the end of each of four 80-trial blocks, participants reported how many target vehicles appeared during that block. The concurrent spatial localization task required participants to press a button to indicate whether an X target appeared on the left or right of fixation. On each trial, along with the central vehicle, the X appeared at one of eight locations, either parafoveally near the center of the visual field or peripherally, simultaneous with three symmetrically placed circle distractors (**Figure 1**). Reporting which side of fixation the target occurred instead of its specific location ensured identical motor responses for parafoveal and peripheral stimuli.

The central vehicle identification task was attentionally challenging, requiring participants to selectively attend to local stimulus features that define vehicle category membership over a large range of vehicle exemplars and to hold a running sum in working memory. Additionally, the large number of vehicles helped prevent participants from overlearning the vehicle stimulus set and therefore helped maintain sustained attention for local defining features near the center of vision.

The vehicles were selected from standard stimulus sets (Kovalenko et al., 2012; Moreno-Martínez and Montoro, 2012), and a few needed additional car stimuli were randomly acquired from arbitrary internet sites. A variety of car models, colors, and styles (e.g., sports car, SUV, sedan, convertible) and non-car vehicles (e.g., train, bicycle, skateboard, hot air balloon, rocket, plane, wagon, truck, baby carriage) were included. Twenty-four cars and 24 non-car vehicles were selected for a total of 48 distinct vehicles. The vehicles spanned a region of 2◦ centered at fixation on the monitor display.

The X target stimulus and the circles were symmetrically placed along the inter-cardinal directions at 45, 135, 225, and 315◦ . Parafoveal targets and distractors spanned 0.23◦ , centered 1.4◦ from fixation immediately adjacent to the vehicle. Peripheral targets and distractors spanned 0.77◦ , centered 13.3◦ from fixation. The size of the localization stimuli was adjusted to compensate for the cortical magnification factor (Virsu and Rovamo, 1979).

To construct the final stimuli, the orders of the 24 cars and the 24 non-cars were separately randomized. The first five cars and the first five non-cars were paired with an X appearing on the upper left inter-cardinal line at 1.4◦ from center (e.g., **Figure 1**, Trial n). Similarly, the next three sets of five cars and five non-cars were paired with X's appearing on the upper right, lower right, and lower left inter-cardinal lines, respectively.

#### TABLE 1 | Participant demographics by group.


#### Deaf demographics


<sup>∗</sup>Group comparison χ <sup>2</sup> p < 0.025.

Circle stimuli were placed symmetrically on the remaining intercardinal lines. This procedure resulted in 20 unique cars and 20 unique non-cars paired with parafoveal X (target) stimuli equally distribute over the four inter-cardinal positions. The same vehicle and target-position stimulus pairings were then reproduced with the peripheral X's to create a parallel set of 40 target stimuli equally distribute over the four peripheral intercardinal positions (e.g., **Figure 1**, Trial n+1). Thus, across these 80 stimuli, the attentional and processing demands associated with specific central vehicle images were perfectly matched between parafoveal and peripheral target sets.

These 80 stimuli were presented as the first and forth block of the four-block experimental session, in a different trial random order, with a total vehicle-class target count of 40 and non-target count of 40 for each of those two blocks. To avoid participants learning the expected vehicle target count per block after the first block, the number of target vehicles was reduced to 32 and the number of non-target vehicles was increased to 48 for block 2. Conversely, for block 3 the number of target and non-target vehicles was increased to 48 and reduced to 32, respectively. To reduce the number of target vehicles in block 2, four randomly chosen stimuli containing a target vehicle and an X at each of the four parafoveal positions and the corresponding four stimuli containing the same four target vehicles and X at each of the four peripheral positions were removed. To increase the number of non-target vehicles in block 2, the four unused non-target vehicles from the appropriate original 24 vehicle set were each paired with one parafoveal X and one peripheral X at one of the four inter-cardinal directions. This procedure maintained the equal distribution of parafoveal and peripheral locations and vehicle pairings within each vehicle-class stimulus set. For block 3, the corresponding procedure created the 48 target vehicle stimuli using the four unused target vehicles and reduced the number of non-target vehicle stimuli to 32. This procedure resulted in a completely balanced set of 320 stimuli across four blocks of 80 stimuli each. To counterbalance cars and non-cars as target vehicles, two sets of 320 stimuli were constructed using the appropriate target and non-target vehicles across all four blocks. Thus, half the participants counted cars as target vehicles with a distribution of 40, 32, 48, and 40 car counts across the four blocks and half counted non-cars as target vehicles with a distribution of 40, 32, 48, and 40 non-car counts across the four blocks.

### Procedure

This study was carried out in accordance with the recommendations of the RIT Human Subjects Research Office with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the RIT Human Subjects Research Office.

### Demographic and Psychometric Testing

All participants were screened for a history of ADHD, neurological disorders, and vision problems, and then responded to a computerized survey to report the demographic and deaf demographic information in **Table 1**. Participants whose both parents had high school or vocational degrees or lower were classified as low childhood socioeconomic status. Participants who had one or more parents with some college attendance or higher degrees were classified as high childhood socioeconomic status. Additionally, deaf participants reported their deaf cultural identity, age of onset of hearing loss, best language, types of schools attended, and parents' hearing status. Participants took the BRIEF-A immediately after the demographic survey.

### Experimental Protocol

Participants sat 58<sup>00</sup> from a SONY GDM-F500 21in monitor. The vehicle identification and spatial localization tasks were introduced separately in a practice session. Participants first saw a practice sequence of 12 cars and non-cars, flashed in the center of the screen. A fixation asterisk appeared for 500 ms, followed by the vehicle for 100 ms, followed by a 2 s inter-trial interval (ITI). Each participant was assigned either cars or non-cars as target vehicle, and reported their target count at the end of this practice block. After a minute rest, participants saw a practice sequence of 12 spatial target trials, with an inter-trial interval of 2 s, containing a fixation asterisk for 500 ms followed by an X and three symmetrically placed circles presented for 100 ms at either the parafoveal or peripheral location, but without a central vehicle. Participants practiced pressing a button with their right or left index finger to indicate the appearance of the X on the right or left of fixation, respectively. After another minute rest, the two tasks were combined for a third practice run of 12 trials, and participants responded to the X's with a button press on each trial and simultaneously kept track of the count of their target vehicles to report after the practice block.

Participants then completed the four blocks of 80 trials with a minute rest between blocks. Response times (RT) and accuracy on each trial were recorded for the localization task. Participants used a keyboard to enter their total target vehicle count at the end of each 80-trial block. They were told to try to be accurate in their vehicle count and to be both fast and accurate in localizing the X's.

### Analysis

For the central vehicle identification task a total percent correct accuracy score was computed for each participant as 100 minus the absolute value of the percent difference score, where percent difference score = 100<sup>∗</sup> ((vehicle count reported)–(vehicle count presented))/(vehicle count presented). This measure takes account of the fact that some participants reported fewer and some reported more than the total number of target vehicles that were presented. Like ordinary percentages, a 100% score means perfect performance. An 85% score means that the participant either incorrectly identified 15% of the targets as non-targets or 15% of the non-targets as targets, and so on. For the target localization task, the percent correct localization accuracy for target stimuli presented at each of the eight parafoveal and peripheral locations was computed as the number correct out of the total number of targets presented at that location across all four blocks. A total parafoveal localization accuracy score was computed as the average of the

percent accuracy across the four parafoveal locations. Similarly, a total peripheral localization accuracy score was computed as the average of the percent accuracy across the four peripheral locations. The percentage scores were arcsine transformed to correct for the inherent deviation from normality of percentage scores.

Trimmed mean correct RTs were computed for parafoveal and peripheral locations. For each participant, any trial RT exceeding 2 standard deviations around their mean of all correct trials within a given block was scored as an error and dropped from the analysis. Trimmed mean correct RTs were then computed for each of the eight locations across all four blocks. Parafoveal and peripheral localization mean RT's were then computed as the average of the four mean RTs at their respective eccentricities.

Preliminary correlational analyses were conducted to rule out the presence of statistically reliable speed-accuracy tradeoffs. Speed-accuracy correlations were computed both within and across tasks for each group separately.

We used planned comparisons to test the predictions of our hypotheses. Planned comparisons offer greater statistical power than unplanned omnibus tests, such as ANOVA, to test group differences when specific hypotheses based on predictions from the literature are planned in advance, and have been used in previous divided attention work to test the altered gradient of attention hypothesis (Dye, 2016). Accordingly, based on clear predictions from previous studies that report central deficits in stimulus identification for non-CI users (Quittner et al., 1994; Mitchell and Quittner, 1996; Smith et al., 1998; Parasnis et al., 2003; Horn et al., 2005; Dye, 2016), but not CI users (Quittner et al., 1994; Smith et al., 1998; Horn et al., 2005) we used a directional planned pairwise comparison to test the prediction from Hypothesis 1 that the No-CI group would perform worse than the hearing group on the central identification task. We also compared the central identification task performance of the CI group against that of the hearing group to test the prediction from Hypothesis 2 of non-inferiority for that group.

Since a peripheral advantage for the No-CI group is expected based on much previous literature (Bavelier et al., 2006; Pavani and Bottari, 2012), we conducted directional planned pairwise comparisons to test the prediction from Hypothesis 1 that the No-CI group would perform better than the hearing group on localization accuracy and localization RT at the peripheral location. We also compared the peripheral performance of the CI group against that of the hearing group as well to test the prediction from Hypothesis 2 of non-superiority for that group.

Hypothesis 1 also predicts that the parafoveal-peripheral localization gradient would be flatter for the No-CI group than for the hearing group. This is a specific form of interaction between these two groups and the two localization eccentricities (parafoveal, peripheral). We were not interested in testing all possible group by eccentricity interactions, only the predicted one. Therefore, we computed parafoveal-peripheral difference scores within participants to represent the gradient and then compared the No-CI group with the hearing group on these difference scores using a directional planned comparison following our prediction. A significant group difference in the direction of smaller parafoveal-peripheral difference scores for the No-CI group would support the specific predicted interaction in Hypothesis 1 of a flatter localization gradient. We examined the parallel interaction for the CI and hearing groups to test the prediction from Hypothesis 2 of no flatter localization gradient for the CI group.

For group and correlational analyses, we conducted both unadjusted comparisons and follow-up adjusted comparisons including age, gender, race/ethnicity (coded as white non-Hispanic vs. other race/ethnicity), and childhood SES as covariates. These common demographic variables are known to affect performance in object identification and localization tasks in particular and can be a significant source of artefactual findings in attentional studies using convenience samples (Scialfa et al., 1994; Hackman and Farah, 2009; McGugin et al., 2012; Clearfield and Jedd, 2013; McKone et al., 2013; Gruber et al., 2014). They have generally not been controlled in previous studies. Including both unadjusted and adjusted analyses allowed us to confirm that group differences in performance on the identification and localization tasks were not confounded by group sampling differences on these known demographic variables. In addition, to rule out possible general perceptualmotor processing differences among groups as an explanation for group differences within specific tasks, we included analyses that adjusted group performance on the central identification task for overall group performance differences on the localization task, and vice-versa. We also included analyses adjusted for BRIEF-A GEC scores to further control the potential influence of group or individual variation in overall executive function related to deafness (Hintermair, 2013). Finally, much evidence suggests that deaf and hearing groups may differ in their working memory skills (Hall and Bavelier, 2010). Because our central identification task relied heavily on working memory, we also conducted analyses that specifically controlled for group and individual differences in working memory using the BRIEF-A Working Memory subscale.

### RESULTS

Accuracy for the two tasks was generally high and well distributed for all groups, indicating that the groups successfully performed both tasks: central identification task, M(SD), No-CI: 84.7%(16.9%), CI: 88.3%(15.9%), Hearing: 94.1%(5.1%); localization task M(SD), No-CI: 95.4%(3.1%), CI: 93.0%(9.0%), Hearing: 94.9%(3.7%). There were no significant or marginal correlations between RT and accuracy within the localization task at parafoveal or peripheral sites, overall or for any group individually. Similarly, accuracy in the central identification task was not related to RT at parafoveal or peripheral sites overall or for any group. Thus, there was no evidence that any group differences in accuracy or RT could be attributed to a speedaccuracy tradeoff.

### Accuracy Group Comparisons

### Central Identification Accuracy Group Comparisons

As predicted by previous studies and Hypothesis 1, the unadjusted planned comparisons showed that the No-CI group

performed significantly worse than the hearing group on the central identification task, No-CI: M = 84.7%, SE = 3.6; Hearing: M = 94.1%, SE = 2.5; t(42) = −2.1, p = 0.0195. By contrast, the CI group mean was numerically intermediate between the No-CI and hearing group means and was not significantly inferior to the hearing group mean, CI: M = 88.3%, SE = 3.6; Hearing: M = 94.1%, SE = 2.5; t(42) = −1.1, p = 0.1412. Adjusting for age, gender, race/ethnicity, and childhood SES produced the same results as the unadjusted analysis, No-CI: least squares M = 83.7%, SE = 4.5%; Hearing: least squares M = 92.2%, SE = 3.4%; t(38) = −1.9, p = 0.0327, and CI: least squares M = 86.9%, SE = 4.5%; Hearing: least squares M = 91.9%, SE = 3.4%; t(38) = −1.1, p = 0.1368. **Figure 2** displays the adjusted least squares group means for the central identification task, plotted at 0◦ on the eccentricity axis. The same central deficit for the No-CI group was still obtained when the three groups were further equated on parafoveal localization accuracy, t(37) = −2.2, p = 0.0177, peripheral localization accuracy, t(37) = −2.2, p = 0.0184, or overall localization accuracy (average of parafoveal and peripheral accuracy), t(37) = −2.4, p = 0.0110, confirming that significantly worse performance of the No-CI group was not due to overall worse performance on both tasks together. The CI group continued to show no significant inferiority to the hearing group on the central identification task in analyses adjusted for parafoveal localization accuracy: t(37) = −0.9, p = 0.1761, peripheral localization accuracy, t(37) = −0.9, p = 0.1891, or overall localization accuracy (average of parafoveal and peripheral localization accuracy), t(37) = −0.8, p = 0.2094. In addition, the central deficit remained for the No-CI group after controlling further for the BRIEF-A GEC scores, t(36) = −2.2, p = 0.0189, as well as for BRIEF-A working memory scale scores, t(36) = −2.4, p = 0.0103, indicating that this deficit was not accounted for by group differences in overall executive functions or working memory in particular. The CI group continued to show no significant inferiority to the hearing group on the central identification task after controlling for BRIEF-A scores, t(36) = −1.1, p = 0.1360, as well as for BRIEF-A working memory scale scores, t(36) = −1.0, p = 0.1547.

To confirm that the central deficit on the object identification task in the planned comparisons for the No-CI group was robust against possible unknown violations of distributional assumptions due to small sample size, we conducted follow-up distribution free directional Wilcoxon two-sample tests. The results matched those of the planned comparisons, confirming the central identification deficit for the No-CI group and no significant deficit for the CI group: No-CI: z = −1.9, p = 0.0286, and CI: z = −0.3, p = 0.3769.

#### Peripheral Localization Accuracy Group Comparisons

Contrary to the prediction of Hypothesis 1, unadjusted planned comparisons of the No-CI group against the hearing group revealed no accuracy advantage at the peripheral location,

t(42) = −0.3, p = 0.6150. Adjusting for age, gender, race/ethnicity, and childhood SES produced the same results as for the unadjusted analysis, t(38) = −0.03, p = 0.5110. **Figure 2** displays the adjusted least squares group means for analysis of the localization task at the peripheral location, plotted at 13.3◦ on the eccentricity axis. Further adjusting for central identification accuracy did not cause a No-CI group advantage to emerge t(37) = −1.0, p = 0.8305. In addition, no peripheral advantage emerged for the No-CI group after controlling further for BRIEF-A GEC scores, t(36) = 1.8, p = 0.1227, as well as for BRIEF-A working memory scale scores, t(36) = 1.3, p = 0.1064, indicating that the lack of superior peripheral performance was not accounted for by group differences in overall executive functions or working memory in particular. Planned comparisons of the CI group against the hearing group also revealed no reliable performance superiority at peripheral locations in unadjusted analyses, t(42) = −0.6, p = 0.7372. Nor were there any reliable group superiorities in analyses adjusted for the demographic measures, t(38) = −0.4, p = 0.6469, nor when further adjusted for central identification accuracy, t(37) = −0.2, p = 0.5708, nor after controlling for BRIEF-A GEC scores, t(36) = 0.1, p = 0.4437, as well as for BRIEF-A working memory scale scores, t(36) = −0.02, p = 0.5065. **Figure 2** shows that the No-CI and hearing group peripheral localization task means were nearly identical, and all group means were within one standard error of the hearing group mean.

### Localization Accuracy Gradient Group Comparisons

Contrary to the prediction of Hypothesis 1, unadjusted planned comparisons of the No-CI group against the hearing group for the parafoveal-peripheral localization accuracy difference scores revealed no group differences in their performance gradients, t(42) = 1.2, p = 0.8861. The same result was obtained after adjusting for demographic measures, t(38) = 0.6, p = 0.7092. The localization task gradient for each group is defined by the line connecting the means between 1.4 and 13.3◦ in **Figure 2**. The figure shows that the No-CI group's localization task gradient was not flatter than the hearing group's gradient. Further adjusting for central identification accuracy did not cause a flatter gradient to emerge for the No-CI group, t(37) = 0.1, p = 0.5380. In addition, no reliably flatter gradient emerged for the No-CI group after controlling further for BRIEF-A GEC scores, t(36) = −0.1, p = 0.4663, as well as for BRIEF-A working memory scale scores, t(36) = −0.2, p = 0.4033, indicating that the lack of superior peripheral performance was not accounted for by group differences in overall executive functions or working memory in particular. Planned comparisons of the parafoveal-peripheral localization accuracy difference scores for the CI group against the hearing group also revealed no reliably flatter gradient in unadjusted analyses, t(42) = 0.4, p = 0.6692. Nor was there any reliably flatter gradient in analyses adjusted for the demographic measures, t(38) = 0.2, p = 0.5736, nor when further adjusted for central vehicle identification accuracy, t(37) = −0.1, p = 0.4680, nor after controlling for BRIEF-A GEC scores, t(36) = −0.04, p = 0.4826, as well as for BRIEF-A working memory scale scores, t(36) = 0.1, p = 0.5570. **Figure 2** shows that, like the group peripheral localization task means, the group parafoveal localization task means were all within one standard error of the hearing group mean. A full set of analyses on the parafoveal accuracy data alone, parallel to those performed for the peripheral accuracy data above, showed no significant group differences in unadjusted or adjusted analyses at the parafoveal locations. Therefore, as these analyses show, contrary to Hypothesis 1, the target localization gradients for the three groups were nearly parallel and the No-CI group's gradient in particular showed no evidence of flattening relative to the hearing group's gradient.

### Localization RT Group Comparisons

The RT data for the localization task were submitted to the same full set of analyses described above for the localization task accuracy data, including unadjusted analyses, analyses adjusted for demographic variables only, and analyses further adjusted for central identification accuracy and executive function. There were no significant differences or trends among groups to support the existence of a peripheral advantage or a flattened gradient of attention in speed of processing for the deaf participants under conditions of divided attention.

### Correlations between Time-with-Implant and Task Performance

The correlation between time-with-implant (chronological age minus age of implantation) and central identification task accuracy, consistent with Smith et al. (1998) for children 4–7 years old, was small and not significant, r = 0.04, t(9) = 0.11, p = 0.9114, even after adjusting for demographic variables, adjusted r = 0.07, F(1,5) = 0.08, p = 0.7855. Correlations between time-with-implant and accuracy at parafoveal and peripheral locations, or averaged over both locations did not reach significance, parafoveal localization accuracy: unadjusted r = 0.51, t(9) = 1.8, p = 0.1082, adjusted r = 0.46, F(1,5) = 1.7, p = 0.2453; peripheral localization accuracy: unadjusted r = 0.28, t(9) = 0.9, p = 0.4130, adjusted r = 0.52, F(1,5) = 2.3, p = 0.1892; overall localization accuracy, unadjusted r = 0.29, t(9) = 0.91, p = 0.3857, adjusted r = 0.55, F(1,5) = 2.5, p = 0.1730. We considered the possibility that individual differences in general factors (e.g., overall perceptual-motor skills) might commonly affect performance across the two tasks, which might create substantial error variance that masked subtler partial correlations between time-with-implant and performance on specific tasks. In fact, central identification accuracy was highly correlated with parafoveal and peripheral localization accuracy within the CI group (r = 0.74 and 0.75, respectively), suggesting the presence of a substantial common source of error variance. Therefore, we included central identification accuracy and localization accuracy as covariates to control their common variance in a regression analysis to predict time-with-implant, along with the demographic covariates. Because parafoveal and peripheral accuracy were highly correlated (0.84), they were averaged together to avoid multicollinearity problems and to act as an estimate of overall localization task accuracy. After controlling this joint variance, overall localization accuracy was positively correlated with time-with-implant, adjusted r = 0.60, F(1,4) = 9.8, p = 0.0354. Higher localization accuracy was

associated with longer time-with-implant over the range of 4 to 18 years. In addition, with localization task performance controlled, there was a trend for an independent negative correlation between central task performance and time-with implant, adjusted r = −0.44, F(1,4) = 5.3, p = 0.0828. Lower central task accuracy tended to be associated with longer timewith-implant over the range of 4–18 years. Parallel analyses between time-with-implant and RT produced correlations that were all small and did not approach significance.

### Correlations of Performance among Stimulus Locations within Groups

The dissociation of the No-CI group's central identification deficit from their localization performance in the group analyses raises the possibility that auditory deprivation is associated with a decoupling of performance across tasks or visual field locations within individuals. This possibility led us to examine the intercorrelations among central task performance and performance at the parafoveal and peripheral eccentricities across participants within each group for evidence of such decoupling. **Table 2** confirms this possibility. Controlling for demographic variables, accuracy at all eccentricities was weakly to strongly correlated within the hearing and CI groups, but was not correlated within the No-CI group. Similarly, localization RTs at the parafoveal and peripheral eccentricities were strongly correlated with each-other, with nearly identical values for the hearing and CI groups. The parafoveal versus peripheral RT correlation for the No-CI group was significantly smaller than the same correlation for the other two groups combined (Fisher test, p < 0.05).

### DISCUSSION

This study used a novel divided attention paradigm requiring central object identification and concurrent spatial localization at parafoveal and peripheral locations. Our goal was to test the proposal that a general graded trade-off of attentional resources from central to peripheral locations across the visual field, due to auditory deprivation, can adequately explain the spatial distribution of deaf-hearing performance differences on attention-dependent tasks that has been reported in the literature. We used groups with different levels of auditory experience to test the prediction from this proposal that concurrent central object identification deficits and related peripheral spatial localization advantages would appear for deaf participants without, but not with, CI, compared with hearing participants, and that these effects would be linked to a flattening of the spatial gradient of attention.

Our results showed that deaf young adults without CIs, but not with CIs, displayed a significant deficit in central identification accuracy compared with hearing young adults. This central deficit was reliable in unadjusted analyses and in analyses with several common demographic variables controlled, as well as analyses with localization task performance and executive function further equated across groups. This result is consistent with the finding of central attentional deficits in several previous CPT studies of children and adults (Quittner et al., 1994; Mitchell and Quittner, 1996; Smith et al., 1998; Parasnis et al., 2003; Horn et al., 2005), in Proksch and Bavelier's (2002) study using a central flanker interference task, and in previous divided attention studies involving a central identification task concurrent with a motion or orientation detection task (Bosworth et al., 2013) or a peripheral localization task (Dye, 2016).

Contrary to predictions, the three groups did not differ in peripheral localization accuracy or RT performance. Furthermore, neither deaf group's localization gradient was significantly different from the hearing group gradient in unadjusted analyses, nor in analyses adjusted for demographic factors nor further adjusted for overall group central identification accuracy and executive function. These results agree with those of previous divided visual attention studies, which showed a central identification deficit but failed to show a peripheral advantage for deaf participants for attentiondependent motion processing (Bosworth et al., 2013) or target localization amidst distractors (Dye, 2016). The fact that statistically equating groups on their central identification performance did not cause an overall localization performance advantage or a flattening of the localization performance gradient to emerge for the No-CI deaf group implies that this group did not trade-off central attentional resources for enhanced attentional resources in the localization task generally or at peripheral locations specifically. Thus, central identification task performance and localization performance across visual space were functionally and statistically dissociated in our study, suggesting independent attentional regulation of ventral and dorsal stream tasks. In support of this interpretation, Bosworth et al. (2013) used a different pair of ventral and dorsal stream tasks (RSVP shape identification and coherent motion detection, respectively) and also presented evidence that the central object identification group deficit was independent of equivalent group performance on motion detection in both central and peripheral regions. However, it is worth noting that Bosworth et al. (2013) also did not find a central deficit on their concurrent ventral stream orientation task, suggesting that the central deficit might not be general over all ventral stream tasks. Generally, our results combine with previous divided attention studies to suggest that central object identification deficits are independent of peripheral task performance.

It is unlikely that the central object identification deficit can be explained by general sensory or perceptual skill deficits in our deaf participants. Dye (2016) reported that performance on his central identification task in the absence of an attentionally demanding concurrent peripheral task did not result in a central deficit in deaf participants, indicating that the central deficit found in his UFOV task was not associated with an overall deficit in sensory or perceptual skills. Although we did not test central vehicle identification performance in the absence of our localization task, the facts that all three groups displayed nearly identical performance on the localization task at the parafoveal location on the immediate edge of the central vehicle stimuli in both unadjusted and adjusted analyses, and that the deaf group with CI did not show a central deficit in this or previous studies, converge with Dye's (2016) earlier finding to suggest that the

central object identification deficit we observed cannot be easily attributed to unknown deafness-related differences in overall sensory or perceptual skills.

It is possible that population differences in specific attention demanding cognitive processes involved in identification tasks could account for the central deficit. For example, working memory demands have been a component of the central task in most CPT studies (but not all – see Parasnis et al., 2003) and some divided attention studies, and it has been proposed that deaf and hearing people have different processing biases in working memory depending on auditory deprivation (Hall and Bavelier, 2010). Similarly, other work suggests that deaf people exhibit deficits on various executive function measures including the BRIEF (Hintermair, 2013). However, our analyses controlling for BRIEF-A working memory and GEC scores tend to rule out group differences in working memory or other executive functions as a specific factor determining the central deficit in our participants. Generally, the selective appearance of a central object identification deficit in prelingually deaf college adults without but not with CI, no history of ADHD diagnosis or neurological disorders, normal or corrected vision, with common demographic characteristics and executive function skills controlled, and with equivalent performance on a competing dorsal stream localization task, suggests a specific deficit in central attention-dependent processes.

Another important consideration is that like most previous studies, we were not able to control for factors such as specific etiologies of deafness or early language delay. Dye and Hauser (2014) recently showed that 6–13 years old deaf children born to deaf parents who acquired ASL from birth did not show a central deficit on the Gordon CPT. These results suggest that language delay and/or unknown neurological consequences of non-genetic causes of deafness may account for the central deficits shown in previous CPT studies of sustained attention. Since these factors may have been present in some of our deaf participants, it remains possible that they represent the mechanism underlying the central deficit we observed in the No-CI group. However, against this interpretation is the fact that the No-CI and CI groups had statistically equivalent distributions of ages of onset of deafness, best language, and parental hearing status, with the numerical balance tipped in favor of more No-CI participants having deaf parents.

It is important to consider the possibility that a general altered gradient of attention could still explain a central deficit without a concurrent peripheral advantage. Although Proksch and Bavelier (2002), reported that deaf participants showed weaker interference by central distractors and hearing participants showed weaker interference by peripheral distractors, these effects were not completely linked in their study, but tended to emerge separately as a function of target search load. Proksch and Bavelier (2002) used target search sets ranging from 1 to 6 stimuli. At the smallest and largest search loads, deaf and hearing participants showed equivalent distraction effects. The weaker central distraction effect for deaf compared with hearing participants first appeared at a search load of 2, but the weaker peripheral distraction effect for hearing participants did not occur until a search load of 4. Proksch and Bavelier (2002) argued that, assuming a more limited allocation of attentional resources to central vision, deaf participants depleted their central attentional resources at a lower load than hearing participants, but both groups had sufficient peripheral attentional resources to produce peripheral distractibility effects at that load. At a higher load, hearing participants depleted their more limited peripheral attentional resources before deaf participants did, selectively eliminating the distractibility effect for that group. However, if Proksch and Bavelier (2002) had measured distractor effects only at the smaller intermediate load, they would have seen a weaker central distractor effect for their deaf participants without a corresponding weaker peripheral distractor effect for their hearing participants, which would have appeared to suggest a central attentional deficit for deaf participants without a corresponding peripheral advantage.

One could argue therefore that, assuming attentional resources are redistributed from central to peripheral vision, the difficulty of our tasks was such that the No-CI deaf group's available central attentional resources might have been insufficient to match the hearing groups central identification performance, but that both groups' peripheral attentional resources might have been sufficient to maintain equivalent peripheral localization performance. Since we did not vary task difficulty, we might not have seen a peripheral advantage emerge for our No-CI group at a higher difficulty level. This argument might be consistent with data from studies that only sample performance at one central and one peripheral eccentricity, separated by significant distance. However, standard attentional gradient models (Greenwood and Parasuraman, 1999; Bush, 2012) predict that a general flattened attention gradient should still smoothly allocate very similar levels of attentional resources

TABLE 2 | Semipartial correlations of accuracy scores among central, parafoveal, and peripheral sites and of RT scores between parafoveal and peripheral sites within groups adjusted for age, gender, race/ethnicity, and childhood SES.


<sup>∗</sup>p = 0.05 or better; !p < 0.1.

at closely adjacent eccentricities everywhere along the gradient, for example at the closely adjacent central and parafoveal locations used in our study. Indeed, the hearing group's central identification and parafoveal localization accuracies were nearly identical, suggesting that the two tasks are equivalent in difficulty under conditions of similar allocation of attentional resources along the gradient at those closely adjacent locations. Given the dramatic deficit in central vehicle identification performance shown by the No-CI deaf group (approximately 10%), their sudden jump of approximately 11% to an equivalent performance level as the hearing group at the parafoveal eccentricity on the very edge of the central vehicle stimuli is not consistent with a flattened continuous gradient of attentional resources across those adjacent locations. Rather, their significant discontinuous local jump in performance, the lack of evidence for a flatter attentional gradient from parafoveal to peripheral locations, and the statistical dissociation of group differences across the two tasks near the center of vision suggest that the central identification deficit was specific to the processing demands of the identification task. These results therefore favor the proposal that central deficits and peripheral advantages may be independently determined by distinct mechanisms, an alternative interpretation suggested by Proksch and Bavelier (2002).

Dye (2016) attempted to explain the presence of a central deficit without a peripheral advantage in his data by proposing that the fine-scale resolution required for central object discrimination in his task may have biased participants to prioritize central processing at the expense of enhanced peripheral attention. Accordingly, deaf participants may have still drawn some resources from central vision to maintain equivalent peripheral performance but not enough to produce superior peripheral performance to hearing participants. This explanation appears to preserve a form of the altered gradient proposal since central resources are still redistributed toward the periphery. However, there are two problems with this explanation. The first is theoretical. It has been argued that deaf/hearing differences in performance on visual tasks may be better characterized as a generalized attentional difference rather than as resulting from deficient visual cognition (Bavelier et al., 2006). This position seemingly implies that deaf and hearing individuals have the same total available attentional resources and that the redistribution of visual attentional resources in deaf individuals is an adaptive zero-sum game since it results in local advantages and deficiencies but not a general deficiency. That is, the original altered gradient of attention hypothesis is a mechanism proposed to explain deaf-hearing differences in performance on visual tasks while preserving the proposal of total non-deficiency. However, given the presence of a reliable central deficit, Dye's (2016) explanation seems to imply that the total available attentional resources for allocation between central and peripheral eccentricities in tasks such as his may be smaller in deaf adults than in hearing adults, since the extra resources taken from central vision are hypothetically only sufficient to elevate peripheral performance to equivalent hearing population levels. This position therefore undermines the motivation for the original altered gradient of attention hypothesis. The second problem is that our present results did not reveal an altered gradient of attention in the localization task, or a statistical coupling of the central deficit to peripheral group performance, even allowing for a possible overall population deficit in total attentional resources.

It has also been proposed that in situations where deaf people must allocate attention across the visual field, they may shift the peak focus of their attention to peripheral locations (Dye and Bavelier, 2013). However, following standard models of the spread of the gradient of attention (LaBerge and Brown, 1989; Staugaard et al., 2016), this proposal still amounts to the claim that performance at most eccentric points in the visual field should not be equivalent in deaf and hearing people, and therefore, the slopes of the gradients of attention measured between identical eccentricities should generally not be the same, even if attention is focused at different points in space for the two groups. Therefore, our localization gradient data are inconsistent with a simple refocusing of the locus of attention to an eccentric location due to auditory deprivation.

It is also possible that we simply did not test for deaf-hearing differences far enough out in the visual field. However, Dye (2016) tested at 20◦ and did not find a peripheral advantage and peripheral advantages have been shown in other studies as close as 2–3◦ (Parasnis and Samar, 1985; Pavani and Bottari, 2012). Nevertheless, there is animal evidence that the gradients of attention for deaf and hearing groups can run parallel for a considerable distance out from central fixation and then diverge with a shallower slope for deaf individuals. Lomber et al. (2010) reported parallel and equivalent performance of deaf and hearing cats in a localization task over the range of 0–45◦ . However, farther from central vision (45–90◦ ), hearing cats showed a steep graded drop in localization performance while deaf cats showed a shallower, more even decrease, resulting in an increasingly larger perceptual advantage with increasingly extreme eccentricity. Since we tested over only part of the visual field (up to 13.3◦ ), we cannot be certain that our deaf No-CI participants would not show a peripheral advantage in the localization task farther out in the visual field. Indeed, as in previous studies, our No-CI group must presumably still have monitored their full visual field beyond the computer screen for unexpected environmental events while they were simultaneously monitoring the demanding events on the spatially limited screen in our experiment. It could be argued that the participants in our study, and in previous studies that have not shown peripheral advantages, broadly partitioned resources taken from central vision to cover both the eccentric regions of the computer display and the rest of their visual environment, and that the peripheral advantage would have shown up if we had tested farther out in the field. However, given the discontinuous jump in performance for the No-CI group specifically between the central identification task and the localization task at the 1.4◦ parafoveal location, an undetected and more remote peripheral advantage would still not be easily explained by a simple broadening of a general gradient of attention. Furthermore, Lomber et al.'s (2010) deaf cats showed no central deficit at 0◦ , suggesting that their peripheral advantage did not stem from a specific smooth but

broad tradeoff of fixed attentional resources across the visual field. Rather, using localized cortical deactivation, Lomber et al. (2010) showed that the performance advantage was caused by cross-modal reorganization of a specific region of auditory cortex. Superior motion detection in the same animals was shown to be caused by specific cross-modal reorganization of a different region of auditory cortex. Deactivation of the motion detection cortex did not alter the cats' localization performance anywhere in the visual field, indicating that their peripheral advantage in the localization task was not due to an enhancement of general attentional resources at peripheral locations but was task specific. Thus, their results suggest that the mechanism underlying deaf-hearing differences in visual attentional performance gradients is not a general redistribution of attentional resources across the visual field that affects all attention-demanding tasks, but rather a functionally specific cross-modal reorganization of multiple neurobiologically distinct systems that may independently regulate performance at different points in the visual field for specific purposes.

Lomber et al.'s (2010) results are consistent with growing evidence that functionally specific plastic changes occur at different loci within sensory and polymodal cortical systems in response to sensory deprivation (Bavelier and Neville, 2002), and that functionally adaptive attention-dependent changes occur within both dorsal and ventral stream systems (Weisberg et al., 2012), giving rise to specific patterns of deaf-hearing differences under different experimental conditions. This specificity appears to be a natural outgrowth of the highly differentiated organization of the attentional system and its interaction with cross-modal processes. The distribution of attentional resources within and across modalities and tasks is driven by competition modulated in part by specific task demands and stimulus features (Rapp and Hendel, 2003). Normally, cross-modal integration continuously interacts with the attentional system to increase the amount of convergent information available to the visual system for detecting and interpreting complex environmental events (Talsma et al., 2010). For example, many studies have shown that auditory stimulation substantially enhances basic visual perception, including regulating brightness perception, improving the speed and accuracy of target detection, increasing temporal resolution, facilitating motion perception, and modulating attentional processes (Shams and Kim, 2010). Furthermore, spreading responsibility for processing multiple events across modalities, especially those involving the same stimulus features, results in less competition for total limited attentional resources than performing the same tasks within that modality (Rapp and Hendel, 2003). Thus, cross-modal integration essentially extends the visual system's degrees of freedom for multitask processing while reducing multitask competition.

By the same token, the loss of auditory cross-modal integration may increase competition between tasks when they must be accomplished within the visual modality instead of cross-modally. Evidence suggests that under these circumstances of heightened intra-modal competition, the visual system adaptively adds capacity by mechanisms such as recruitment of multisensory or primary sensory cortex (Bavelier et al., 2006; Hirst et al., 2012; Karns et al., 2012). Added capacity heightens the internal differentiation of the visual attention system, essentially transforming cross-modal degrees of freedom from the auditory system into intramodal degrees of freedom in the visual system. This increased organizational complexity would permit more flexible, fine-grained coordination of attentional resources within the visual system across tasks and stimulus feature representations as well as at different eccentricities for different tasks, leading to a different attentional architecture to achieve the same goal-directed adaptive results that would ordinarily be accomplished by cross-modal integration mechanisms. In particular, Lomber et al.'s (2010) demonstration in deaf cats of a far, but not near, visual field flattening of the gradient for visual localization due to specific recruited mechanisms in auditory cortex provides evidence for an increased capacity to flexibly regulate attentional resources across distant local regions of the visual field. In general, current evidence supports the view that visual system mechanisms become increasingly differentiated in response to specific competitive behavioral and cognitive pressures associated with auditory deprivation (Bavelier and Neville, 2002; Lomber et al., 2010; Weisberg et al., 2012). These considerations generally suggest that deaf-hearing differences in the distribution of spatial attention are likely to be task and goal dependent and precisely regulated by multiple distinct neural subsystems rather than driven by a redistribution of general attentional resources that applies across tasks.

The finding that time-with-implant did not have a simple correlation with performance in either the central identification task or the localization task at any eccentricity for our adult participants extends Smith et al.'s (1998) previous correlational findings with children to adults and to an eccentric localization task. Smith et al. (1998) replicated Quittner et al.'s (1994) finding that CIs improved CPT performance in children relatively rapidly (within about 18 months), but also reported that, unexpectedly, time-with-implant did not correlate with CPT performance over a range of times-with-implant as brief as 6 months to 2 years. They argued that the lack of correlation is consistent with the claim that the relevant effects of the implant occur rapidly. Our CI participants had their implants for a minimum of 4 years and a maximum of 21 years. Hence our findings are also consistent with the assumption that adults with CI may have experienced a rapid reversal of central attentional deficits following implantation.

Given that restored attentional skill may occur less than 2 years after implantation, the lack of simple correlations between time-with-implant and central and peripheral task performance provides no evidence per se for or against the altered gradient proposal. However, our finding of independent semi-partial positive correlations raises the possibility that some aspects of cross-modal plasticity related to restored auditory input may operate on a longer maturational time scale. The general gradient of attention proposal does not appear to predict these correlations. If anything, it would predict that with increased time-with-implant, dorsal stream peripheral accuracy enhancements would be reduced and ventral stream central accuracy deficits would be improved, not the other way around. On the contrary, the independent semi-partial correlations we observed suggest that some aspects of dorsal and ventral stream

processes may be differentially affected by restored auditory input over long periods during childhood. It appears that performance on dorsal stream tasks may partially improve slowly over several years of CI use. Somewhat paradoxically, central ventral stream performance may independently partially decrease slowly with long-term CI use. This is an issue for future research. Note, however, time-with-implant strongly reflects age of implantation during childhood in our sample (r = 0.93) which ranged from 1.5 to 17 years old. Therefore, these effects may reflect separate interactions between the timing of restored auditory input and developmental stages or critical periods of specific ventral and dorsal stream processes. This interpretation is consistent with studies showing that plastic changes associated with sensory deprivation are shaped by specific experience interacting with the timing of critical periods and other maturational factors (Bavelier and Neville, 2002).

The correlations in **Table 2** suggest that auditory deprivation may decouple, and restored auditory input may recouple, overall performance across tasks and visual field locations. The No-Ci group had uniformly low, non-significant accuracy intercorrelations among tasks and eccentricities and a significantly lower correlation between parafoveal and peripheral RT than the other groups. Absent or reduced correlations for the No-CI group are not explained by less reliable general performance than the other groups. If that were true, we would likely expect overall worse performance and increased variability in that group compared with the other groups. While the No-CI group displayed worse central object identification performance, the three groups had closely equivalent RTs and high accuracy scores at both eccentricities in the localization task. In addition, the No-CI group's variance was at or numerically below that of the CI group on both tasks (Accuracy: p's = 0.8546 to 0.9884; RT: p's = 0.9586 to 0.9612). Another explanation, which would be consistent with the altered gradient of attention model, is that individual differences in the availability of residual auditory input might have induced within-group variability in the No-CI participants' gradient-of-attention slopes, whereas the restoration of auditory input might have reduced such withingroup variability in CI participants' gradients. However, the No-CI and CI group variances of the parafoveal-peripheral performance difference scores were numerically close and did not differ significantly (accuracy: p = 0.3975; RT: p = 0.4388), indicating no evidence that the No-CI group's absent or low correlations reflect individual variation in attention gradient slopes. We propose instead that the selective decorrelation of performance across tasks and sites is consistent with the claim that cross-modal plasticity following auditory deprivation increases the visual system's capacity to independently regulate attention on a finer scale across visual space in order to maintain optimal task performance under competing task conditions.

### REFERENCES


It is important to remember that, although we controlled several background factors that might impact group differences, it was not possible to match participants across groups on all potentially relevant personal characteristics and developmental factors. The heterogeneous nature of the deaf community, including the potential developmental impact of different causes of deafness, audiometric functional complexity, and educational, family, social, and linguistic history present significant challenges to research designed to experimentally characterize the behavioral and neurobiological consequences of auditory deprivation. Understanding the impact of these factors on attentional performance in the larger deaf community will require increased emphasis on experimental controls and sampling practices in future studies.

### CONCLUSION

Our results converge with previous divided visual attention studies that failed to demonstrate a trade-off of attentional resources from central to peripheral visual field locations in deaf people who grow up without access to functional auditory input. In the context of the larger literature that documents inconsistent reports of deaf-hearing advantages, deficits, and equivalencies, our results suggest that a general graded trade-off of attentional resources across the visual field cannot adequately explain the complex task-dependent spatial distribution of deaf-hearing performance differences. Rather, our results are consistent with growing neurobiological evidence that the spatial distribution of attention-mediated performance in deaf people is determined by sophisticated cross-modal plasticity mechanisms that recruit specific sensory and polymodal cortex, increasing intra-modal visual processing capacity and attentional control to achieve specific compensatory behavioral goals.

### AUTHOR CONTRIBUTIONS

LB and VS conceived and designed the research and analyzed data. LB recruited participants and collected the data. VS primarily wrote the paper with critical editing contributions for intellectual content from LB.

### FUNDING

LB was supported in part by the NSF Science of Learning Center, Visual Language and Visual Learning (VL2), at Gallaudet University (Grant SBE-1041725, Petitto, Scientific Director; Allen and Petitto, Co-PIs).

Bosworth, R. G., and Dobkins, K. R. (2002). Visual field asymmetries for motion processing in deaf and hearing signers. Brain Cogn. 49, 170–181. doi: 10.1006/ brcg.2001.1498

Bavelier, D., Tomann, A., Hutton, C., Mitchell, T., Corina, D., Liu, G., et al. (2000). Visual attention to the periphery is enhanced in congenitally deaf individuals. J. Neurosci. 20, RC93.


implant users. PLoS ONE 11:e0148466. doi: 10.1371/journal.pone. 0148466



**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The reviewer RPG and handling Editor declared their shared affiliation, and the handling Editor states that the process nevertheless met the standards of a fair and objective review.

Copyright © 2017 Samar and Berger. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Peripheral Visual Reaction Time Is Faster in Deaf Adults and British Sign Language Interpreters than in Hearing Adults

Charlotte J. Codina<sup>1</sup> \*, Olivier Pascalis <sup>2</sup> , Heidi A. Baseler <sup>3</sup> , Alexandra T. Levine<sup>4</sup> and David Buckley <sup>1</sup>

<sup>1</sup> Academic Unit of Ophthalmology and Orthoptics, School of Medicine and Biomedical Science, The University of Sheffield, Sheffield, UK, <sup>2</sup> Laboratoire de Psychologie et NeuroCognition, Université de Grenoble Alpes, Grenoble, France, <sup>3</sup> Centre for Neuroscience, Hull York Medical School, University of York, York, UK, <sup>4</sup> Department of Psychology, University of York, York, UK

Following auditory deprivation, the remaining sense of vision has shown selective enhancement in visual cognition, especially in the area of near peripheral vision. Visual acuity is poor in the far periphery and may be an area where sound confers the greatest advantage in hearing persons. Experience with a visuospatial language such as British Sign Language (BSL) makes additional demands on the visual system. To test the different and separable effects of deafness and use of a visuo-spatial language on far peripheral visual processing, we investigated visual reaction times (RTs) and response accuracy to visual stimuli, between 30◦ and 85◦ along the four cardinal and four inter-cardinal meridians. We used three luminances of static, briefly illuminated stimuli in visually normal adults. The cohort tested included profoundly congenitally deaf adults (N = 17), hearing fluent BSL users (N = 8) and hearing non-signing adults (N = 18). All participants were tested using a peripheral forced choice paradigm designed previously to test deaf and hearing children (Codina et al., 2011a). Deaf adults demonstrated significantly faster RTs to all far peripheral stimuli and exceeded the abilities of both signing and non-signing hearing adults. Deaf adults were significantly faster than BSL interpreters, who in turn were significantly faster than hearing non-signing adults. The differences in RT demonstrated between groups were consistent across all visual field meridians and were not localized to any one region of the visual field. There were no differences found between any groups in accuracy of detecting these static stimuli at any retinal location. Early onset auditory deprivation appears to lead to a response time visual advantage in far peripheral responses to briefly presented, static LED stimuli, especially in the right visual field. Fluency in BSL facilitates faster visuo-motor responses in the peripheral visual field, but to a lesser extent than congenital, profound deafness.

Keywords: deafness, reaction times, accuracy, British Sign Language, visual attention, peripheral vision

## INTRODUCTION

Human peripheral visual perception is affected by sensory, developmental, and environmental experience. The visual system has inherent plasticity, peripheral vision in particular showing a high capacity for plasticity and the potential for peripheral visual plasticity has been previously underestimated (Burnat, 2015). Both the peripheral retina and the magnocellular visual pathway

#### Edited by:

Tilo Strobach, Medical School Hamburg, Germany

#### Reviewed by:

Martina Manns, Ruhr University Bochum, Germany Robert Gaschler, FernUniversität Hagen, Germany

> \*Correspondence: Charlotte J. Codina c.codina@sheffield.ac.uk

#### Specialty section:

This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology

Received: 07 September 2016 Accepted: 09 January 2017 Published: 06 February 2017

#### Citation:

Codina CJ, Pascalis O, Baseler HA, Levine AT and Buckley D (2017) Peripheral Visual Reaction Time Is Faster in Deaf Adults and British Sign Language Interpreters than in Hearing Adults. Front. Psychol. 8:50. doi: 10.3389/fpsyg.2017.00050 have emergent, immature topographies that may facilitate high levels of visual plasticity throughout life. Far peripheral vision plays a crucial role in monitoring the environment, especially in the absence of sound.

Several visual changes have been noted in association with deafness. Bosworth and Dobkins (2002) showed that deaf adults performed significantly better to peripheral, but not central visual stimuli. Proksch and Bavelier (2002) used a visual search paradigm to report that deaf individuals had greater visual attentional resources in the visual periphery, and less in central vision. Neville and Lawson (1987a,b) found enhanced attention to the visual periphery in a motion decision task, coordinated with event-related potential (ERP) responses from the occipital cortex of deaf participants to peripheral stimuli. Bavelier et al. (2001) by means of fMRI, detected greater recruitment of the motion selective area V5/MT for deaf participants when they attended peripherally rather than centrally. Fine et al. (2005) found that fMRI responses to visual stimuli were uniquely represented in the auditory cortex of deaf participants, and this effect was not seen in adult participants who were children of deaf adults (CODAs), signing from birth. Bavelier et al. (2006) summarized that not all aspects of vision are improved in deaf individuals—deaf adults showing slower reactions in central visual cognitive tasks (Proksch and Bavelier, 2002). Bavelier et al. (2006) argue that selective visual changes occur which compensate for those aspects of vision that would normally benefit from the combined auditory and visual inputs. In line with this, Codina et al. (2011b) found altered retinal ganglion cell layer distribution to support peripheral vision, and increased retinal ganglion cell number and structural changes correlated with increased peripheral vision performance. Deafness therefore has a differential influence on both the structure and behavior of the remaining senses such as vision.

It is not only deafness that has been shown to promote a peripheral visual advantage in humans. Habitual playing of computer games has been linked with improved localization of a peripheral target amongst distractors (Green and Bavelier, 2003), and increased visual field (VF) area Buckley et al. (2010). Memmert et al. (2009) found specific visual attention improvements between athletes and non-athletes when the stimuli were most similar to their practiced sport. Muiños and Ballesteros (2013) reported that karate athletes were faster at localizing peripheral visual stimuli than non-athletes. They suggested that the rapidity of response in their athletes may be due to the suddenly appearing, peripherally attended, opponent maneuver. Different patterns of visual skills may result in specifically trained motor responses to peripheral visual stimuli. There is consensus amongst several authors that the visual differences found between athletes and non-athletes are not in the "hardware" of functional visual pathway changes, but in the "software," using perception and visual-cognitive processes more efficiently, employing skill utilizing strategies made effective with practice (Abernethy et al., 1994; Muiños and Ballesteros, 2013). Schubert et al. (2015) conducted a detailed investigation on the training effects of video gaming on visual attention. They found that video gamers showed increased visual processing speeds in the lower aspects of the VF. The video gamers did not change their attention allocation strategy, with high speed processing demonstrated in all display locations. However, the video gaming advantage was seen specifically in areas where non-experts performed less well, with higher speed visual processing and shorter minimal exposure duration needed to begin perceptual processing. These higher processing speeds in response to training might improve the temporal resolution of attention and allow attention to be moved between focally and peripherally presented stimuli.

What is particularly interesting about deaf individuals is that the altered sensory experience of deafness clearly brings about unique sensory and cognitive changes. However, most deaf persons are also immersed, to some degree, in a visual spatial language such as British Sign Language (BSL), which in itself places altered conceptual and sensory demands on the visual system, quite different to spoken language (see Bavelier et al., 2006 for a review).

During signed language conversation, fluent, signing individuals typically focus on the face of the person signing to them (Siple et al., 1978; Muir and Richardson, 2005; Agrafiotis et al., 2006). Taking the visuospatial nature of signed language into account, signed language is therefore likely to stimulate peripheral vision in a manner extraordinary to spoken language. Indeed, Swisher et al. (1989) demonstrated that deaf adults could understand American Sign Language (ASL) signs using peripheral vision only, between 45◦ and 61◦ eccentric to fixation, whereas hearing individuals could not identify large words presented at similar eccentricities. In signed language space is used both topographically and referentially (MacSweeney et al., 2008). Signing space extends from the navel to above the head and "neutral space" is the area in front of the signer's body at mid-lower chest level one where most of the BSL signs occur (British Deaf Association, 1992). The majority of "words" in ASL are produced below eye level (Teuber et al., 1980); therefore, it may be that the inferior field of vision is particularly stimulated by signed language experience.

Familiarization with visuospatial language does not seem to produce the same enhancements in peripheral vision as revealed in deaf adults: the visual advantages cited earlier in deaf individuals have not been found in hearing signing populations (Neville and Lawson, 1987a,b; Bosworth and Dobkins, 2002; Proksch and Bavelier, 2002; Fine et al., 2005). However, these experiments did not test as far in the periphery as we have tested here. Nevertheless, signing has produced distinct cortical and visual changes. Cortical adaptations have been observed in both hearing and deaf signers in response to language perception. Although a right hemisphere predilection and therefore left VF advantage is widely accepted in the general population (Paillard et al., 1981; Paillard and Amblard, 1985; Clarke et al., 2000), Bosworth and Dobkins (1999, 2002) demonstrated a left hemisphere lateralization and right VF advantage for motion processing in signers, whether deaf or hearing. Bavelier et al. (2001) found early exposure to ASL led to greater reliance on the left hemisphere motion selective area V5/MT. The left, language dominant hemisphere may become increasingly activated by motion processing in deaf and hearing signers, leading to a right VF advantage for the processing of visual motion (Neville

and Lawson, 1987a; Bosworth and Dobkins, 1999; Bavelier et al., 2001). Cardin et al. (2013) in an fMRI study of distinct deaf signers and deaf lip readers showed that cortical regions adjust to process the different signals—either speech reading or signed language and that functionally distinct cortical substrates separate deaf adults who sign from those who speech read.

Given the plasticity of the visual periphery to maximize its response to the pattern of visual skills required, one might expect BSL experience itself to influence far peripheral vision and RTs therein. In a previous paper (Codina et al., 2011a) in which we reported deaf and hearing children's peripheral visual performance development on a far peripheral vision task (30– 85◦ ), young deaf children (aged 5–8 years) were initially slower to respond to peripheral stimuli than hearing children, they performed similarly at ages 9–11 years, and were significantly faster than controls at ages 12–15 years. To the authors' knowledge, the RT advantage consistently observed in deaf adults has not been thoroughly investigated across the far peripheral field and neither has it been investigated in hearing sign language users. The aim of the current study was to investigate far peripheral visual sensitivity and RT in early onset deaf adults and BSL interpreters, to explore the different and separable effects of auditory deprivation and experience with a visuospatial language.

### MATERIALS AND METHODS

### Participants

All participants were emmetropic; the refractive error did not therefore affect the VF and glasses frames would have interfered with detection of peripheral stimuli. Inclusion criteria for all groups were: good visual acuity in either eye unaided, minimum 0.200 LogMAR units (equivalent to 6/9.5 Snellen acuity, using a standard illuminated ETDRS vision chart at 4 m), absence of epilepsy, and no known abnormal ophthalmological history self-reported during the study consent procedure.

### Deaf Group

Seventeen adults (11 males, 6 females, mean age 33.25 years, range 18–45) with profound binaural hearing loss were recruited by invitation from Grange Crescent Deaf Club in Sheffield, the University of Sheffield, personal contacts, word of mouth through other deaf participants, and from deaf individuals working at Lower Meadow Primary Academy and Allerton Grange School. Criteria for entering the study for deaf participants were: deafness was either present at birth or had onset before the age of 8 months, and was not due to any systemic or genetic disorder known to affect vision such as Usher's syndrome. Eleven participants reported BSL as their native language and 6 reported English. Five participants were left handed and 12 right handed. Nine participants reported being regular action video game players. Four out of the 17 deaf participants contracted deafness as a result of in-uterine rubella and were thus screened by full ophthalmic examination prior to entry into the study to ensure there were no visual deficits.

### Hearing Group

Eighteen participants with no hearing loss and no prior knowledge of any signed language took part in this study (9 males, 9 females, mean age 30.28 years, range 18–45). These participants were recruited through colleagues at The University of Sheffield. Six participants reported themselves as regular action video game players. One participant was left handed and 17 were right handed.

### BSL Interpreter Group

Eight participants, all trained and qualified full-time BSL interpreters, registered with ASLI (Association for Sign Language Interpreters), with a minimum of 6 years' experience formed this group (6 females, 2 males, mean age 39.1, range 27–62). Two participants in this group are CODAs and learned sign language as their first language. None of the interpreters reported being action video game players. One participant was left handed and seven were right handed.

### Eye of Testing

Time was a constraint for three participants in the deaf group and two participants in the BSL group who were teachers in one of the schools visited, therefore for these participants only the right eye was tested. For most of the results only the right eye data is presented in line with the work of other authors (Stevens and Neville, 2006; Codina et al., 2011a; Bjerre et al., 2014), as VFs are known to be highly symmetrical in normal subjects (Brenton et al., 1986) and no differences were found between our right and left eye data. We do investigate possible lateralization in the Results section, although the number of participants for which we have data for both eyes is less than that for which we have the right eye data [right eye (N = 17), both eyes (N = 14) for the deaf group, and right eye (N = 8), both eyes (N = 6) for the BSL group].

### Stimuli and Procedure

Informed, written consent was obtained from all participants prior to testing and the study procedures were approved by The University of Sheffield Psychology Department Ethics Committee and complied with the Helsinki Declaration. The methods of this study have been fully described elsewhere (Codina et al., 2011a). In brief, this peripheral vision task was to detect static, briefly illuminated LEDs, presented to the far visual periphery. As is shown schematically in **Figure 1** the VF test incorporated 96 LEDs (Nichia, 1.5 cds), implanted into a uniform gray hemisphere (0.5 m radius). This hemisphere contained 12 LEDs along each of the eight meridians that correspond to the four cardinal and four inter-cardinal directions for the right eye and left eye VFs (see **Figure 1**). The LEDs were positioned between 30◦ and 85◦ in 5◦ steps. An adjustable chin and forehead rest enabled a fixed viewing distance of 1 m and centralization of the participant's eye to the central fixation light behind which was a black and white camera for monitoring fixation. In total 224 LEDs were each very briefly illuminated (for 200 ms) in front of the participant's right eye or left at three different light intensities in a random order. Ninety-six dim stimuli at 83.47 cd/m<sup>2</sup> , 96 medium stimuli at 91.81 cd/m<sup>2</sup> , and only 32 bright

stimuli (at 40◦ , 55◦ , 70◦ , and 85◦ only) of 118.94 cd/m<sup>2</sup> intensity were presented in a random order. The brightest stimuli were easy to locate and therefore only tested at every third eccentricity to maintain participant interest and check compliance. The data from these stimuli are not reported here. The test was calibrated with an oscilloscope prior to each testing session to ensure uniformity of time period and degree of illumination after transportation. The participant responded by setting a joystick, positioned at chest height, to one of eight possible positions. The joystick was positioned either in front of the right hand or left hand according to self-reported handedness.

The test was carefully explained to each participant in either English or BSL in a lit room and the directions and response directions and instructions for the joystick were both explained and demonstrated to each participant. Participants were seated on an adjustable desk chair, facing the LED array test, their chin and head on rests, aligned and adjusted so that the participant's tested eye was centered 0.5 m behind the fixation target. The other eye was occluded with a patch. All external light sources were eliminated prior to testing and only low level artificial illumination mounted on the upper surface of the hemisphere was provided at a constant level of 1.2 cd/m<sup>2</sup> brightness for all test environments. Specifically written MATLAB (The MathWorks, Inc.) software with the Data Acquisition Toolbox controlled both the LEDs and logged the data from the joystick via National Instruments data acquisition hardware. Participants first completed a practice trial which consisted of 32 bright stimuli, where four stimuli were presented on every meridian all at eccentricities of 40◦ , 55◦ , 70◦ , and 85◦ , and on satisfactory completion of the practice the test was begun. Participants were asked to move the joystick to the position corresponding to the meridian of the stimulus LED. Participant fixation during stimulus presentation was observed by the experimenter through a small TV screen monitoring the camera at the fixation point; a stimulus would be repeated later in the sequence if fixation was not maintained. If the participant responded either with the exact matching meridian of the LED or adjacent meridians then this was recorded as a correct response and the RT recorded. We did also record the exactly correct data, when the response exactly matched the LED meridian, and this is only described in the Accuracy data section. Pilot studies had shown that with such peripheral presentations it is difficult to localize the exact position of a flashed LED, particularly for young children. The same procedure was adopted here as we wished to compare our pediatric data (Codina et al., 2011a) with our adult data. All other VF tests reported in the literature used yes/no (detection) responses (Rowe, 2016), and therefore our discrimination paradigm requiring a response accuracy of ±45◦ was a relatively difficult task.

## RESULTS

### RT data

No differences were found within any test between our right and left eye data; we therefore initially present data from the right eye only, in line with previous authors (Stevens and Neville, 2006; Bjerre et al., 2014). As not all targets were correctly localized by participants, their RT data were analyzed in two separate ANOVAs: by mean meridian RTs averaged across eccentricity and separately mean eccentricity RTs averaged across meridians. Only intermediate and dim stimuli results are presented as the brightest stimuli were used as a control measure.

### Meridian Data Averaged Across Eccentricity

The mean meridian RT data were analyzed by a three factor mixed measures ANOVA where the factors were group (deaf, hearing or BSL interpreter), stimulus intensity (intermediate and dim) and meridian (8 levels). **Figure 2** shows the mean RT data averaged across all stimuli for the right eye for each of the three groups. There was a significant main effect of group [F(2, 40) = 4.11, p = 0.03]; as can be seen from **Figure 2** the mean for the deaf group (mean 585.31 ms) was less than for the hearing group (mean 731.77 ms) with the BSL Interpreter group somewhere in

between (mean 627.39 ms). Bonferroni corrected post-hoc t-tests showed that deaf adults had significantly faster RTs than either the hearing group (t = 6.22, df = 33, p < 0.001), and the BSL interpreter group (t = 2.40, df = 24, p = 0.03). BSL interpreters also showed faster RTs than hearing non-signers (t = 3.29, df = 25, p = 0.003).

There was no significant main effect of stimulus intensity, or any interaction with the other factors therefore all graphs show data averaged across the intermediate and dim stimulus intensities. As can be seen in **Figure 3** the mean RTs at each meridian location for the right eye do show some variation and the main effect of meridian was significant [F(7, 280) = 17.67, p < 0.001], yet the interaction between meridian and group was not significant. No other effects or interactions were significant. Faster RTs are apparent for all three groups in the inferior temporal VF and the hearing and BSL interpreter groups appear closest to each other in this region. Bonferroni corrected post-hoc t-tests between groups for each meridian revealed significant differences at each meridian between deaf and hearing groups only and these results are shown in the Table within **Figure 3**.

#### Eccentricity Data Averaged across Meridian

**Figure 4** compares mean RTs for eccentricities averaged across meridians for the three groups as a function of eccentricity. A three factor mixed measures [group × stimulus intensity × eccentricity] ANOVA was conducted. There was again, a significant main effect of group [F(2, 40) = 3.87, p = 0.03]. There was a significant main effect of eccentricity [F(11, 440) = 2.28, p = 0.01]: increased eccentricity resulted in a slower RT for all three groups. There was a consistent RT ordering of deaf < BSL interpreters < hearing across all eccentricities. There was no interaction between eccentricity and group and no other interaction with eccentricity was significant. However Bonferroni corrected post-hoc t-tests showed that deaf adults had significantly faster RTs than the hearing group at 35◦ (t = 2.56, df = 33, p = 0.02), 60◦ (t = 3.26, df = 33, p = 0.003) and 70◦ (t = 3.25, df = 33, p = 0.003), though deaf vs. hearing results were close to significance at most eccentricities.

#### Native Language

To determine possible influences of the deaf participants' native language, it was considered as a fourth factor (Native BSL N = 11 and native English N = 6). The native language factor was not significant, nor did it affect the levels of significance for any other factor. This is an interesting finding, as BSL cannot be wholly responsible for the differences observed in the deaf group. We also tested for differences between action computer game players and non-computer game players within the deaf and hearing groups and found no significant effects or interactions with this factor. However, none of the computer game players would be classed as habitual players under Green and Bavelier's (2003) criteria.

#### Right and Left Visual Fields

Although no differences were found between right and left eye data, right and left visual field data were different. Lateralization differences have been found before when the right VF (comprised of right temporal and left nasal VFs) is compared to the left VF (left temporal and right nasal). **Figure 1** illustrates how the right VF (A,B,C for **both eyes**) and left VF (A',B',C' for **both eyes**) are comprised. **Figure 5** shows the mean RT data for each group for this data and a slight left VF RT advantage is observed in all groups. A one way ANOVA with between subjects factor of group (bootstrapped) was conducted on right and left VF data. A significant effect of group was found for the right VF only [F(2, 35) = 3.641, p = 0.037]. Post-hoc pairwise t-test comparisons (Bonferroni corrected and bootstrapped) within the right VF showed that deaf participants (n = 14, M = 589.43, SE = 32.2) were significantly faster than hearing (n = 18, M = 706.8, SE = 32.9), (p = 0.036). No other differences between groups were significant and no significant differences were found between right and left VFs within any group. Consistent with other published studies (Papadatou-Pastou and Sáfár, 2016) the deaf group had an atypically higher proportion of left handed participants (29% left handed) than the general population, therefore handedness was considered as a second factor in a separate ANOVA. Handedness was not significant, nor did it affect the significance levels of any other factor.

### Accuracy

Percentage correct response data were analyzed by a three factor mixed measures [Group × stimulus intensity × meridian] ANOVA. The overall effect of group was not significant, see **Figure 6A**, which shows that the accuracy is similar for each group. Meridian had a significant effect as expected, due to nasal and superior aspects of the VF being obscured by the nose and brows [F(7, 280) = 79.62, p < 0.001]. Stimulus intensity had a

FIGURE 3 | Mean RT in ms for the three groups (deaf, hearing and BSL interpreter) for the eight meridian locations. The table within the figure shows the results of the Bonferroni post-hoc analyses with df = 33 for each reported value. Significant differences were found at each meridian location only between deaf and hearing groups. For clarity, no error bars are shown, but the SEM was between 10 and 17 ms.

between each pairing (p < 0.001).

significant effect [F(2, 62) = 6.54, p = 0.003], yet there was no significant interaction of group with stimulus intensity and no other significant interactions.

For comparison an identical ANOVA examined data that was only considered exactly correct if the actual correct meridian was chosen by the participant. As expected the percentage of exactly

correct responses were found to be lower, yet the levels and factors of significance remained unchanged. **Figure 6B** shows that the groups also all performed very similarly in the percentage of exactly correct responses.

Overall then accuracy data showed no differences between the groups.

### DISCUSSION

Deaf participants reacted significantly faster to the peripherally presented stimuli when compared to the hearing group and BSL interpreter group. This pattern of results was found across all VF locations and up to the maximum eccentricities tested. Faster RTs in early onset deaf adults found in this study are consistent with the faster deaf RTs reported in the literature (Parasnis and Samar, 1985; Neville and Lawson, 1987a; Loke and Song, 1991; Stivalet et al., 1998; Bosworth and Dobkins, 2002; Proksch and Bavelier, 2002). Importantly, however, the current study demonstrates a greater advantage in the far peripheral VF, in a range of peripheral vision (30–85◦ ), which has not previously been investigated. Peripheral visual acuity is increasingly poor at increasing eccentricities, thereby sound confers the greatest advantage at this range of eccentricities. At far peripheral locations our study finds markedly speeded RTs in deaf participants as well as moderately speeded RTs in full time BSL interpreters.

The fastest RTs for all groups and the fastest RTs overall were displayed by the deaf group in inferior and temporal aspects of the VF. One might expect the greatest advantage for deaf individuals in these VF areas where the majority of "words" in signed language would occur (Teuber et al., 1980). However, the significantly speeded RTs were identifiable in all meridian locations and could not be localized to any particular region of the VF. The deaf RT advantage was not significant at all eccentricities, likely because of the increased standard error in eccentricity data. The significant differences in all **Figure 4** posthoc analyses revealed differences only between deaf and hearing groups and not between BSL and hearing or between BSL and deaf groups. This deaf advantage at these far eccentricities is consistent with Swisher et al.'s (1989) finding that deaf adults could reliably identify ASL signs 45◦ and 61◦ eccentric to fixation using peripheral vision only and with Buckley et al. (2010) report of significantly larger VFs in deaf adults, using kinetic stimuli at a similar range of eccentricities. Buckley and colleagues found that the areas of most significant increase were the inferior and temporal aspects of the VF—regions most stimulated by signed language but also the most expansible aspects of the VF. In deaf adults the RT advantage results from the combined effects of auditory deprivation and the cross-modal plasticity evidenced to this (Fine et al., 2005), as well as immersion into an entirely visual language. However, our results suggest that visuo-spatial BSL language immersion alone does not confer the same peripheral vision RT advantage that auditory deprivation does.

The visuo-cognitive influences of auditory deprivation and signed language exposure are likely to be distinct, yet segregating one from the other is difficult. As previously described, 11 of our deaf group reported BSL as their native language, yet analyses by native language showed no significant influence of native language on RT. However, even in the minority of predominantly aural deaf individuals (for example those married to hearing persons) cumulative exposure to sign language throughout life is still significant. Cardin et al. (2013) reported that after plastic reorganization in deafness, cortical regions adapt to process the different types of signal—either lip reading or signed language and that functionally distinguishable substrates are present at the cortical level between deaf who sign and deaf who lip read.

Interestingly, Emmorey et al. (2009) found an eye gaze fixation pattern difference between beginning and native signers: beginning signers fixated nearer to the signer's mouth so to perceive lip mouthings more clearly; whereas fluent signers fixated nearer to the interlocutor's eyes. Thus increased experience with signed language was related to a greater ability to perceive signed and mouthed information more peripherally. In our data, the hearing signers were all currently working BSL interpreters, having been fluent in BSL for a minimum of 6 years. They might have therefore adapted during the course of their BSL careers to move further away from lower face fixation as peripheral vision adapts to improve sensitivity to the most peripheral areas most utilized by signed language and facial expression.

In a previous paper (Codina et al., 2011b), we reported that the retinal nerve fiber layer in the eyes of early onset deaf adults was differentially distributed to support peripheral vision, particularly temporal aspects of the VF where the left and right VFs do not overlap and neural resources may be most influential. Fine et al. (2005) reported that the crossmodal plasticity within the auditory cortex responding to signed language was not present in either non-profoundly deaf individuals nor present in hearing signers. The results showing that facilitating this level of neural reorganization requires a dramatically altered sensory experience. That said,

it is perhaps only in profoundly deaf adults that increased neural circuitry to the remaining senses is expected, consistent with Fine et al. (2005) and Codina et al. (2011b), which might facilitate the RT decrease identified in the far periphery in this study. However, electrophysiological (Osorio et al., 2010) and brain imaging studies (Ballesteros et al., 2013) have found altered neural correlates in response to simple behavioral training in conceptual object priming, and this in itself may be evidence of compensatory neural activity. Our results are comparable with Buckley et al. (2010) where habitual video game players showed enhanced areas of peripheral visual sensitivity in comparison to non-video-game players, which were even larger in early onset deaf adults. This suggests that enhancement of peripheral vision may be partially mediated by visual attention, with additional compensatory improvement due to sensory deprivation.

Based on our previous research, it is likely that the reduced RTs identified in deaf adults were slow to develop (Codina et al., 2011a), and were perhaps facilitated by altered neural substrates and compensatory neural activity (Codina et al., 2011b). It is possible that similar neural changes may have occurred in hearing signers as well, although this has not yet been tested. However, the visual differences found between hearing signers and non-signers might be similar to the differences found between athletes and non-athletes, not so much in the "hardware" of functional visual pathway changes, but in the "software" efficiency of perceptive and attentional processes (Abernethy et al., 1994; Muiños and Ballesteros, 2013).

Recruitment of BSL interpreters was particularly difficult due to the national shortage of BSL interpreters at the present time (McAleer, 2006), and as a consequence, the mean age of the BSL interpreter group is slightly higher than for deaf and hearing groups. However, simple RT is known to increase with age (Der and Deary, 2006) and become more variable (Hultsch et al., 2002). Also of note was that none of the BSL interpreter group played computer games, as computer game playing has been shown to improve peripheral vision (Green and Bavelier, 2003; Buckley et al., 2010). Therefore, to find faster RTs in this slightly older and non-computer game playing group is a more striking result.

The RTs we report in our study are larger than those studies employing standard kinetic perimetry (Grobbel et al., 2016). Grobel and colleagues, with varying ages of adult participants, reported RTs of 391—522ms. However, theirs was a motion detection task, more suited to the peripheral visual pathway and required the simple press of a button, whereas our experiment utilized static eccentric stimuli and an 8-alternative forced choice task and was therefore conceptually more demanding.

All groups showed a slight left VF RT advantage, in line with the right hemisphere predilection for visual-spatial activity, widely reported in the literature (Paillard et al., 1981; Paillard and Amblard, 1985; Clarke et al., 2000) and reported in hearing non-signers by Bosworth and Dobkins (1999, 2002), Neville and Lawson (1987b). In a stochastic motion task within 15◦ of fixation, Bosworth and Dobkins (2002) reported that both deaf and hearing signers displayed the opposite pattern of results to hearing non-signers, finding a right VF RT advantage in deaf and hearing signers. We did not find this right VF advantage in the far periphery tested in our study. However, the significant RT reduction in the deaf group's right VF, in comparison with hearing controls, and the highly similar right and left VF RTs in our deaf group may indicate a sensory compensatory mechanism to advance the typical left RT advantage additionally to the right VF. Therefore, auditory deprivation rather than BSL exposure seems to influence the right VF RT. This is interesting in that lesion and neuroimaging studies have consistently reported that the neurobiology of signed language is very similar to spoken language, principally recruiting the left lateralized perisylvian network no matter which language is involved (MacSweeney et al., 2008).

In light of the markedly reduced RTs for the deaf group it is interesting to consider which particular aspects of visuomotor processing may be enhanced by auditory deprivation and training. Auditory deprivation may speed peripheral perception by use of compensatory cortical plasticity and exposure to a language stimulating the visual periphery may call into play

alternative visual attention allocation strategies which may, in turn speed the visuomotor response. When considering the visuomotor nature of this study's task it is of note that the deaf group contained a high number of left handed individuals and this finding is consistent with other studies (Papadatou-Pastou and Sáfár, 2016). Atypical handedness may contribute to VF laterality differences, though had no significant effect on our results. Bavelier et al. (2006) put forward four hypotheses in relation to deaf neural and attentional adaptations. They proposed that adaptation may be genetic; that areas V1 and V2 may be more susceptible to intramodal recruitment to visual attention; that multisensory associative cortical areas might reorganize to the remaining modalities such as vision; or that auditory cortex might reorganize to mediate other functions such as vision. Our results suggest that the most speeded responses, highlighting the highest visual attention in the far periphery in deaf adults, supersede the increased visual attention brought about by practice with a visual-spatial language alone, signifying different mechanisms of visual compensation.

### CONCLUSION

Deaf adults demonstrated significantly faster RTs than both hearing non-signers and hearing BSL interpreters to a range of far peripheral briefly presented static stimuli and this advantage was consistent across all VF locations up to 85◦ eccentric to fixation. BSL interpreters displayed faster RTs than hearing non-signing adults across the entire VF. Early onset deafness leads to visual compensation in the form of much faster peripheral vision RTs consistent with the cross-modal plasticity benefits to vision of auditory deprivation and use of a visual spatial language. The deaf RT advantage is most apparent in the right VF, where hearing responses are significantly slower. Fluency in BSL without deafness also leads to rapid responses to peripheral stimuli, although not to the same degree as deafness. Daily immersion in a visual-spatial language benefits visual responsiveness to stimuli in the peripheral VF.

### REFERENCES


### ETHICS STATEMENT

Ethical approval was granted by the University of Sheffield, Department of Psychology, Ethics Committee. Participants were invited by the experimenter by letter, email or by personal invitation to take part. The researcher gave every participant a participant information sheet that had been given ethical approval. This information was additionally given to the deaf participants in British Sign Language as required. Participants gave full written, informed consent before taking part in the experiment. Deaf adults were given the information verbally, in writing and in British Sign Language as each preferred. Participants were fully informed of the procedure in the language of their choosing. Participants all knew the test procedure and had the opportunity to ask questions under conditions of full lighting before any testing began.

### AUTHOR CONTRIBUTIONS

DB, OP, and CC designed the test equipment, were involved in piloting and refining the experiment and in the ethics process. CC collected and analyzed the data. AL and HB assisted in the interpretation of the data and further analysis. The paper was written by CC with contributions from all authors.

### ACKNOWLEDGMENTS

We are grateful to the Action on Hearing Loss who funded CC (née Westerman)'s Ph.D, to Jennifer Smedley for kind assistance in collecting data at Grange Crescent Deaf club, who we also extend our thanks to. We thank Richard France, Richard Stacey, and Ricky Greaves for help in recruiting deaf participants, to ASLI for kind assistance in recruiting BSL interpreters, to Richard Squires and staff of Allerton Grange School in Leeds. We also thank all of our participants, especially the BSL interpreters whose time was in high demand.

of testing, normative data, and repeatability. NeuroOphthalmology 38, 189–198. doi: 10.3109/01658107.2014.902971


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Codina, Pascalis, Baseler, Levine and Buckley. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Modifications of Visual Field Asymmetries for Face Categorization in Early Deaf Adults: A Study With Chimeric Faces

#### Marjorie Dole1,2 \*, David Méary<sup>1</sup> and Olivier Pascalis<sup>1</sup>

<sup>1</sup> Laboratoire de Psychologie et NeuroCognition, CNRS UMR 5105, Université Grenoble-Alpes, Grenoble, France, <sup>2</sup> Gipsa-Lab, Département Parole et Cognition, CNRS UMR 5216, Université Grenoble-Alpes, Grenoble, France

Right hemisphere lateralization for face processing is well documented in typical populations. At the behavioral level, this right hemisphere bias is often related to a left visual field (LVF) bias. A conventional mean to study this phenomenon consists of using chimeric faces that are composed of the left and right parts of two faces. In this paradigm, participants generally use the left part of the chimeric face, mostly processed through the right optic tract, to determine its identity, gender or age. To assess the impact of early auditory deprivation on face processing abilities, we tested the LVF bias in a group of early deaf participants and hearing controls. In two experiments, deaf and hearing participants performed a gender categorization task with chimeric and normal average faces. Over the two experiments the results confirmed the presence of a LVF bias in participants, which was less frequent in deaf participants. This result suggested modifications of hemispheric lateralization for face processing in deaf participants. In Experiment 2 we also recorded eye movements to examine whether the LVF bias could be related to face scanning behavior. In this second study, participants performed a similar task while we recorded eye movements using an eye tracking system. Using areas of interest analysis we observed that the proportion of fixations on the mouth relatively to the other areas was increased in deaf participants in comparison with the hearing group. This was associated with a decrease of the proportion of fixations on the eyes. In addition these measures were correlated to the LVF bias suggesting a relationship between the LVF bias and the patterns of facial exploration. Taken together, these results suggest that early auditory deprivation results in plasticity phenomenon affecting the perception of static faces through modifications of hemispheric lateralization and of gaze behavior.

Keywords: early deafness, hemispheric laterality, chimeric face, gender, eye movements, categorization task

### INTRODUCTION

Hemispheric specialization of cognitive function in typical adult brain is well documented. One well-known example is the lateralization of language in the left hemisphere, perisylvian areas dedicated to language processing being functionally (see Tzourio et al., 1998; Celsis et al., 1999, for examples) as well as anatomically (Geschwind and Levitsky, 1968; Foundas et al., 1995) left lateralized in the great majority of right-handed subjects. Conversely visuo-spatial and face

#### Edited by:

Guy Dove, University of Louisville, USA

#### Reviewed by:

Teresa Mitchell, University of Massachusetts Medical School, USA Martina Manns, Ruhr University Bochum, Germany

\*Correspondence: Marjorie Dole marjorie.dole@univ-grenoble-alpes.fr

#### Specialty section:

This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology

Received: 01 September 2016 Accepted: 05 January 2017 Published: 20 January 2017

#### Citation:

Dole M, Méary D and Pascalis O (2017) Modifications of Visual Field Asymmetries for Face Categorization in Early Deaf Adults: A Study With Chimeric Faces. Front. Psychol. 8:30. doi: 10.3389/fpsyg.2017.00030

processing abilities would be lateralized toward the right hemisphere. This right hemisphere dominance for face processing has been originally demonstrated, thanks to prosopagnosic patients – patients showing specific inability to recognize faces following brain damage. Indeed, although prosopagnosics generally suffer from bilateral lesions, a right hemisphere lesion seems sufficient to produce significant impairments in face recognition (De Renzi, 1986; De Renzi et al., 1991). The hypothesis of a right hemisphere advantage for face processing is also supported by fMRI results, showing a functional asymmetry in favor of the right hemisphere during face processing, particularly in the Fusiform Face Area (FFA; Badzakova-Trajkov et al., 2010; Rossion et al., 2012; Bukowski et al., 2013).

At the behavioral level, this right hemisphere dominance for face processing is thought to be the cause of a left visual field (LVF) bias, the fact that facial information present in the LVF is crucial for categorization and recognition (Levy et al., 1983; Luh et al., 1991; Burt and Perrett, 1997). Burt and Perrett (1997), for example, used chimeric faces (faces vertically split in two different halves) to assess right hemisphere advantage during the detection of variable face attributes, such as gender, age or facial expression. The stimuli presented to participants were composed of two average half faces (e.g., left half is an average of male faces whereas right half is an average of female faces) with the join down the center blended rendering it invisible to participants. The rationale beyond this image manipulation is that the left and right hemispheres receive respectively the right and left part of the image relative to the point of foveation. Gazing three degree to the right of a face will place the entire image in the LVF and this signal, conveyed through the right visual tract, will be first processed in the right hemisphere. Acuity drops drastically with eccentricity from point of foveation and we generally look directly at faces to access to more details. Nevertheless, when we fixate different locations in a face, the left and right hemisphere are processing only partly overlapping right and left parts of the face. Burt and Perrett (1997) found that participants' judgments of gender and expression were influenced to a greater extent by the information on the left of the face from the viewer's perspective. This finding has been largely replicated (Butler and Harvey, 2005; Butler and Harvey, 2008; Yovel et al., 2008; Bourne and Gray, 2011). To establish a straight relationship between hemispheric lateralization and the LVF bias obtained using chimeric faces, Yovel et al. (2008) used fMRI while participants performed a matching task. Participants were presented chimeric faces in the scanner, and also performed the same task outside the scanner. The resulting activity in the FFA was rightward asymmetric, and this asymmetry was positively correlated with the LVF bias obtained from the behavioral test ran outside the scanner. This confirmed that the LVF bias obtained using chimeric faces does, at least in part reflect right hemispheric specialization of face processing areas. Right hemisphere advantage for the processing of face could be related to the processing of configural information in face (Schiltz and Rossion, 2006; Maurer et al., 2007), right hemisphere being generally thought to process predominantly global information whereas left hemisphere would be specialized in the processing of local information (Fink et al., 1996, 1999; Lux et al., 2004).

The relative contribution of right hemisphere lateralization and attentional factors resulting from the scanning patterns in the LVF bias is, however, not fully understood. An increased LVF bias in trials in which participants spent more time looking at the left part of the face suggests a clear link between attentional factors resulting from scanning patterns and the LVF bias (Butler et al., 2005). However, this bias can be observed even with short presentation times (100 ms) that are preventing eye movements, ruling out the effect of purely attentional factors (Butler and Harvey, 2006, 2008). The presence or absence of a LVF bias would thus result from a complex interplay between bottom-up perceptual processing factors and top–down attentional factors which could be both lateralized, similarly to what has been suggested for written language (Selpien et al., 2015).

The LVF bias is also robustly found even with line drawing (Luh et al., 1991), or inverted stimuli (Butler and Harvey, 2005). Using chimeric faces, Aljuhanay et al. (2010) found that the LVF bias was present as early as 5 years of age. The root of the hemispheric asymmetry in face processing has been hypothesized to lie in the development of the hemispheric specialization. Infants recognize a face faster if it is initially presented in the LVF as opposed to the right visual field (de Schonen and Mathivet, 1990). This processing bias may represent the precursor of the asymmetry observed in adults.

If the brain asymmetry for face processing emerges during development, early deprivation or dramatic differences the infant experiences with the world should affect it. For example, 9- to 23-year-old participants treated for bilateral congenital cataracts after 7 weeks of age, who were deprived of patterned visual input during this duration, fail to develop some of the aspects associated with typical adult levels of face recognition such as the face composite effect suggesting impaired configural processing (Le Grand et al., 2001). This emphasizes the importance of early visual experience in the development of adult face processing abilities. Cross-modal interactions have also been found to affect visual development; for example early auditory deprivation has been shown to affect the development of some visual abilities. Several studies showed that deaf participants could detect targets at larger eccentricities, indicating larger visual field (Buckley et al., 2010; Codina et al., 2011). Better abilities have also been found in deaf participants for the detection of motion in the visual periphery (Armstrong et al., 2002; Bosworth and Dobkins, 2002; Stevens and Neville, 2006; Hauthal et al., 2013). The observation of enhanced processing in the periphery, particularly under attentional conditions, seems very reliable in the literature (Parasnis and Samar, 1985; Neville and Lawson, 1987; Bavelier et al., 2000, 2001; Bottari et al., 2010). Higher-level visual abilities have also been shown to be modified by early deafness, such as visual imagery (Emmorey et al., 1993, 1998) or the processing of faces (Bettger et al., 1997; Arnold and Murray, 1998). Using the Benton Test of Facial Recognition, Bettger et al. (1997) tested the recognition of individual faces in deaf participants. They obtained better scores than hearing non signers, but only in a difficult condition,

in which faces were shadowed. This enhanced processing in deaf people could thus concern very particular aspects of face processing; McCullough and Emmorey (1997) found that deaf and hearing participants differed only by the detection of subtle facial features. Feature analysis relates to configural face processing and de Heering et al. (2012) suggested an increased dependency on this mode of processing in deaf participants.

If visual processing is affected by early deafness, what about visual asymmetries? Several experimental studies examining hemispheric asymmetry in congenitally deaf individuals found that it differs from the one observed in hearing individuals (Szelag and Wasilewski, 1992; Szelag, 1996; Neville et al., 1997). As for the processing of sign language, an extensive amount of data show that it could activate the typical left-lateralized speech processing network (Pettito et al., 2000; MacSweeney et al., 2002, 2008). However, other studies suggest greater contribution of the right hemisphere for the processing of sign language than for spoken language (Neville et al., 1997, 1998; Emmorey et al., 2002). In addition damage to both left and right hemisphere lead to language deficit in sign language users (Corina and McBurney, 2001). Neville and colleagues proposed that greater recruitment of right hemisphere would be related to the visualspatial characteristics of sign language. However, recent results suggest a reduction of hemispheric lateralization in a spatial attention task (Cattaneo et al., 2014). A shift of hemispheric lateralization during the detection of motion has also been demonstrated, deaf subjects showing a left hemisphere advantage, whereas hearing subjects showed a right hemisphere advantage (Bosworth and Dobkins, 1999; Bavelier et al., 2001; Bosworth et al., 2013). The question of cerebral lateralization for sign language processing results from a complex interplay between language-related and other cognitive – visuo-spatial, gestural, motion-related – functions modulated by sensory experience that are still poorly understood.

With regard to face processing, few studies investigated hemispheric lateralization during the perception of faces in deaf people and the results are rather contrasted. Phippard (1977) presented briefly unfamiliar faces in either the left or right visual field and found no differences between deaf and hearing participants. More recently Letourneau and Mitchell (2013) found a typical LVF asymmetry during an identity judgment task. Neurophysiological studies suggest a reduced right hemisphere asymmetry in deaf participants compared with hearing ones (Weisberg et al., 2012; Mitchell et al., 2013). Mitchell et al. (2013) for example found a reduced asymmetry of the neural responses to neutral faces around 200 ms in deaf participants when compared with the hearing group. Other studies principally focused on the processing of emotional and/or linguistic facial expressions. Indeed, although facial expressions are a universal cue to recognize the emotional state of individuals, sign language users must also recognize facial expressions as linguistic markers which could affect the hemispheric lateralization. Concerning emotional expressions, Szelag and Wasilewski (1992) presented emotional (happy, sad) and non-emotional faces in the left or right visual field in deaf children. They found a LVF (right hemisphere) advantage for neutral and sad faces in normal hearing, and no hemispheric advantage for any kind of faces in deaf participants. More recently, Letourneau and Mitchell (2013) found a reduction of the LVF bias during an emotion judgment task in deaf participants. Corina (1989) investigated the LVF advantage for affective and linguistic facial expressions; they obtained a reduced LVF bias in deaf participants for both types of expressions, but this was strongly affected by the order of presentation. A following study (Corina et al., 1999) suggests that the cerebral lateralization for facial expression could depend on the functional role (linguistic/affective) of these expressions. Finally, McCullough et al. (2005), using fMRI, investigated cerebral asymmetries during the presentation of linguistic or emotional facial expressions. For emotional expressions, they found a right hemisphere lateralization in the STS in hearing controls, whereas activation was symmetrical in deaf participants. For linguistic facial expressions, activation was also right lateralized in hearing subjects, but left lateralized in deaf participants. Some modifications of the asymmetry for emotional and linguistic facial expressions were also observed in the fusiform gyrus, where hearings exhibited a slight rightward asymmetry for both types of expressions, whereas activity was leftward lateralized in deaf participants (see also Emmorey and McCullough, 2009).

Taken together, these results suggest differences of the functional hemispheric asymmetry for the processing of both linguistic and emotional expressions. However, it still unclear whether this plasticity extends to the core aspects of face processing. By presenting neutral faces, Weisberg et al. (2012) found a reduced activity in the right fusiform gyrus in deaf participants in comparison with hearing non signers, whereas no difference was observed in left fusiform. This could suggest reduced asymmetry in deaf participants. However, this study being not designed to investigate cerebral asymmetry, it remains difficult to draw firm conclusion about hemispheric lateralization in deaf participants.

Our review of the literature suggests the existence of some modifications in the cerebral asymmetries in deaf people, resulting either from auditory deprivation or/and their extensive use of sign language. Concerning facial processing, these modifications are less well established. To date, evidences for modifications of hemispheric lateralization for the processing of neutral faces are rather scarce and it is unclear how deafness affects the processing of invariant aspects of face. The present experiments are interested in examining the LVF bias in a population of deaf adults and non-signer hearings. We used a gender recognition task with chimeric faces. Assuming that LVF bias reflects right hemispheric dominance for face processing we predicted a reduced LVF bias in deaf participants. From the results of Weisberg et al. (2012), this reduced LVF bias would be linked to a reduced activity in the right fusiform gyrus during face processing. This hypothesis was tested in Experiments 1 and 2. In Experiment 2 we also measured eye movements during face scanning to investigate the consequences of early auditory deprivation on the visual attention toward face features in the gender recognition task.

### EXPERIMENT 1

fpsyg-08-00030 January 19, 2017 Time: 16:12 # 4

### Participants

Fourteen deaf adult participants (six females, mean age: 34.92, SD: 8.58) and 14 normal hearing controls (seven females, mean age: 31.27, SD: 8.56) selected to match deaf participants in gender, age, and handedness participated in the study. A two sample t-test confirmed that the two groups did not differ in age or handedness (both p > 0.05). All participants were right-handed according to the Edinburgh Handedness Inventory (Oldfield, 1971). Deaf participants were bilateral severe to profoundly deaf (80 dB hearing loss and greater) and all were prelingually deaf. None of the hearing participants were exposed to signed language. No participants reported any neurological or psychiatric illness, and all had a normal to corrected vision. Details concerning the characteristics of the deaf group can be found in **Table 1**. All participants signed written informed consent and were paid for their participation.

### Material and Procedure

Forty faces were presented in a randomized order to the participants: 10 female/male (Chimeric F/M), 10 male/female (Chimeric M/F), 10 blended female (Entire F) and 10 blended male (Entire M) (**Figure 1**). Stimuli used were previously described (see Burt and Perrett, 1997; Butler et al., 2005). Briefly, each chimeric face was composed of one blended male and one blended female face. Each blended face was composed of five different faces with the age of photographed people approximately matched. Additional features that could facilitate



FIGURE 1 | Example of the stimuli used in this experiment. (Top) Blended male and female faces. (Bottom) Chimeric female/male (Bottom left) and male/female (Bottom right) faces.

gender recognition such as earrings, make-up or beard, were absent. Before blending, all faces were rotated and aligned with respect to eyes and mouth. After blending, 10 blended female and 10 blended male images were selected to create 10 pairs of chimeric faces. The two blended faces composing a pair were aligned to match eye position across the pair. The first picture of the pair was composed with the left half of the blended male face and the right half of the blended female face, and the second picture of the pair was the mirror of the first image. Gradual change in shape and color from one image to the other across the vertical midline produced a seamless merger between the left and right halves of the chimeric faces rendering the vertical midline between the two halves imperceptible. Each face was then converted from color to gray-level.

The 40 stimuli were presented centrally on an Iiyama Vision Master Pro 513 screen. Screen resolution was 1024 × 768 pixel (40.5 cm × 30 cm). Participants sat 62 cm away from the screen, their chin relying on a chin rest. Stimulus size was 396 × 522 pixels. Participants were required to indicate by key press on the keyboard if the face was feminine or masculine. All responses were made using the right hand only and the positioning of male and female labels was counterbalanced between participants. All participants performed five practice trials in order to ensure good comprehension of the instructions. Participants were given

enough time to provide their answer, but were encouraged to answer as quickly as possible. The image was displayed until the participants gave their answers. Instructions were given in writing and orally for the hearing participants and in writing and either orally or in French Sign Language depending on preference for the deaf participants.

### Results

Statistical analyses were run in R (R Development Core Team, 2008). Rapid analysis of the percentage of correct gender classification for the average female and male faces showed that it was nearly perfect in both groups (deaf: 99.6%; hearing: 98.6%). The mean response time of hearing participants seemed faster than that of the deaf participants (deaf: M = 1070 ms, SD = 258 ms; hearing: M = 920 ms, SD = 241 ms) but this difference was not significant (two-sample t-test, t<sup>26</sup> = −1.58, p = 0.12).

Response time increased in both groups when judging chimeric faces (deaf: M = 1768 ms, SD = 330 ms, paired t-test, t<sup>13</sup> = −3.37, p = 0.005; hearing: M = 1332 ms, SD = 555 ms, t<sup>13</sup> = −6.32, p < 0.001). This increase in response time reflects the increased difficulty in judging the gender of chimeric faces. We also found a significant interaction between Group and the Type of Face (Entire, Chimeric) [F(1,42) = 7.70, p = 0.01], indicating that the increased response times for the Deaf group was particularly important for chimeric faces and not for entire faces.

For the analysis of responses, a score of 1 was given if the participant's answer represented a LVF bias (i.e., female for chimeric F/M stimuli and male for chimeric M/F stimuli) and 0 otherwise. The average score over the 20 chimeric faces × 100 was used as an index of LVF bias with value above 50% representing a LVF bias while value below 50% represents a RVF bias. A boxplot showing the median, 1st and 3rd quartiles, and individual data points for both groups is given in **Figure 2**. First, we built a generalized linear model of LVF index. Formally the model was written LVF-50 = β<sup>1</sup> + β2G<sup>j</sup> + εij where G represented the group and was coded j = 0 for hearing and j = 1 for deaf. We subtracted 50 from the LVF values to center the result with respect to chance level (50%). According to the model the intercept term represents the amount of LVF bias in the hearing group and the second term β<sup>2</sup> represents the change in LVF bias in the deaf group. The intercept was significant (β<sup>1</sup> = 17.8 %, t<sup>26</sup> = 3.59, p = 0.001) indicating the presence of a LVF bias in the hearing group. This LVF bias was not significantly reduced in the deaf group (β<sup>2</sup> = −6.42 %, t<sup>26</sup> = −0.93, p = 0.35). One-sample t-test on the LVF values in the group of deaf participants showed a significant LVF bias at the group level (M = 11.2%, t13 = 2.46, p = 0.028).

To test the existence of a LVF bias at the level of the participant we considered the 2 × 2 contingency table formed by the Female or Male responses of the subject to the Female or Male chimeric faces (according to the left part of chimera). Filled data points in **Figure 2** represent the participants for which the χ 2 statistics for 1 degree of freedom was significant at the p-value α = 0.05 (bilateral test). The number of participants with LVF bias in the hearing and deaf groups were respectively 6 and 3 (out of 14).

statistics. Two participants showed a RVF bias (filled dots < 50%).

One participant in each group showed a RVF bias. To estimate the probability of having 0, 1,. . ., N significant test under the null hypothesis in a group of N = 14 participants we simulated LVF score assuming n = 20 draws per participants and a normal distribution of the error with µ = 10 and σ = 3.6. Note that in theory, with p = 0.05, σ = √ np(1-p) = 2.24. However because the observed SD in the hearing group was larger we used the observed value which was more conservative. Over 100,000 tests the probability to obtain exactly 3 or 6 significant LVF in a group of 14 individuals was 0.23 and 0.014 respectively. Finding 3 or less significant comparisons occurred in 81 % of the cases. Finding 6 or above occurred only in 1.7% of the cases. To sum-up on the result of this simulation under the null hypothesis of no LVF bias in the population average, finding 3 individuals (out of 14) with a significant LVF bias (as in our group of deaf participants) is likely (p = 0.23), in contrast, finding 6 individuals (as in our group of hearing participants) is very unlikely (p = 0.017). Although it is an indirect way of testing the presence of a larger bias in the hearing group, these results point in the same direction than the linear model analysis of the LVF bias: that it is stronger on average and more frequent at the individual level in the hearing population.

### Discussion

to a χ 2

This experiment was designed to determine if modifications of the hemispheric lateralization for face processing happened in early deaf participants during a gender categorization task with chimeric faces. The results confirmed the presence of a LVF bias in hearing participants that was not significantly reduced in the group of deaf participants. However, the number of participants with LVF bias in the deaf group was not as large

as in the hearing group suggesting that the LVF bias might be reduced for some individuals in the population of early deaf adults.

A crucial question when investigating hemispheric dominance by the mean of chimeric faces is the relationship between the LVF bias and the scanning patterns of the participants. Number of studies showed indeed that the left side of the face is investigated first, and for more time, than the right side (Phillips and David, 1997; Butler et al., 2005; Guo et al., 2012). Gazing first at the left of the face would make sense because the left part of the face projects to the right hemisphere when the fixation point is centered on the face. Faster processing of faces in the right hemisphere could then lead to early saccades toward information coming from the left visual hemifield, that is, to the left part of faces.

Some authors argue that the LVF advantage not only reflects the right hemispheric dominance for face processing, but could also arise from the habitual scanning patterns of the participants. Evidence for an effect of the habitual scanning pattern comes from Arabic or Hebrew subjects (right-to-left reading patterns) who show, when compared with English or French readers, a reduced LVF bias (Heath et al., 2005). Moreover when the eye movements are made impossible, the LVF bias is noticeably reduced, although still present (Butler and Harvey, 2006). This LVF advantage could thus arise from the interplay between scanning pattern and hemispheric dominance (see also Butler and Harvey, 2005). This is crucial because it highlights the presence of plasticity arising from different scanning habits during development.

Experiment 2 was designed to confirm the results of experiment 1 but we also recorded eye movements in addition to participant's responses while deaf and hearing participants performed the gender categorization task with chimeric and normal faces as in Experiment 1. Early deafness has been found to affect the pattern of eye movements in an anti-saccade task (Bottari et al., 2012) and in a task involving judgment of faces' emotional valence (Watanabe et al., 2011). In addition to the analysis of differences in scanning patterns between deaf and hearing participants in our face categorization task, we also looked for differences in initial fixations and overall exploration of face side and features in an attempt to relate LVF bias to scanning strategy.

### EXPERIMENT 2

### Participants

Fourteen deaf participants (six female, mean age = 34.92, SD = 8.58), and 14 control hearing subjects (seven female, mean age = 30.84, SD = 9.79), contributed to this second experiment. All deaf participants already participated in Experiment 1. Ten out of the 14 control participants of Experiment 1 also took part in this second study.

### Material and Procedure

Sixty faces were presented to the participants: 15 chimeric F/M faces, 15 chimeric M/F faces, 15 average male faces, and 15 average female faces. The stimuli were designed identically to the previous experiment and were presented for 2 s on the screen. Other methodological aspects were identical to Experiment 1. Face image size was 497 × 653 pixels.

Eye movements were recorded from both eyes using a Eyelink 1000 system (SR Research Ltd., Mississauga, ON, Canada) with a 250 Hz sampling frequency. We used a chin-rest to limit head movements. The test phase was preceded by a calibration phase during which participants were instructed to fixate a 0.3◦ black circle on a gray background which appeared sequentially at five different positions on the screen. During the test phase, a drift correction was made every five trials, in order to realign gaze and screen space, and correct for small head movements. Each trial began by a fixation point. In order to control for starting position effects (Arizpe et al., 2012) the fixation point was placed at the top of the image for half of the trials and at the bottom of the image for the other half.

### Data Analyses

#### Gender Categorization Task

As in Experiment 1, the percentage of correct gender classification for the average female and male faces showed that it was nearly perfect in both groups (deaf: 99.3%; hearing: 99.3%). The mean response time of hearing and deaf participants were almost identical (deaf: M = 1082, SD = 166 ms; hearing: M = 1090 ms, SD = 180 ms). The smaller SD in both groups, compared to experiment 1, suggested that inter-individual variability was reduced in this second practice with the gender-recognition task.

Although participants seemed more trained to the task, response time was still increased in both groups when judging chimeric faces (deaf: M = 1266 ms, SD = 304 ms, paired t-test, t<sup>13</sup> = −2.99, p = 0.01; hearing: M = 1354 ms, SD = 360 ms, t<sup>13</sup> = −4.32, p < 0.001). As in Experiment 1 the average score over the 30 chimeric faces × 100 was used as an index of LVF bias. The boxplot is given in **Figure 3**. Running our GLM on this second dataset we found (β<sup>1</sup> = 13.3%, t<sup>26</sup> = 2.28, p < 0.001) indicating the presence of a LVF bias in the hearing group. This LVF bias was not significantly reduced in the deaf group (β<sup>2</sup> = −11.19%, t<sup>26</sup> = -1.35, p = 0.18). However, the difference from 50% was not significant in the deaf group indicating an absence of LVF bias at the group level (one-sample t-test: M = 52.14%, t<sup>13</sup> = 0.37, p = 0.71).

We tested the LVF bias at the level of the participant as in Experiment 1. Filled data points in **Figure 3** represent the participant for which the χ 2 statistics for 1 degree of freedom was significant at the p-value α = 0.05 (bilateral test). The number of participants with LVF bias in the hearing and deaf group were respectively 6 and 2 (out of 14). One participant in the hearing group and two participants in the deaf group showed a RVF bias. Simulated LVF score under null hypothesis assuming n = 30 draws per participants and a normal distribution of the error with µ = 15 and σ = 3.6 indicated that the probability of obtaining exactly 2 or 6 significant LVF score in a group of 14 individuals were 0.28 and 0.003 respectively. Finding three or less significant comparisons occurred in 77% of the cases. Finding six or above occurred only in 0.4% of the cases (p = 0.004).

#### Test–Retest Reliability

dots < 50%).

Because all deaf participants involved in Experiment 2, and 10 of the hearing participants, were also tested in Experiment 1 we estimated the test–retest reliability for the LVF bias. In practice we regressed the LVF score in Experiment 2 using the LVF score in Experiment 1 as a predictor. The coefficient of fidelity rxx was equal to 0.735 and the regression results gave LVF<sup>2</sup> = −0.012 + LVF1\*0.898. The slope of the regression was significant (t<sup>22</sup> = 5.08, p < 0.001) while the intercept was not different from 0 (t<sup>22</sup> = −0.1, p = 0.92). Thus, the measures from the two gender recognition tasks were clearly related. We then examined the fidelity within both group. For the deaf participants we found rxxDeaf = 0.77 (LVF<sup>2</sup> = −0.06 + LVF<sup>1</sup> ∗ 0.95; t<sup>12</sup> = 4.9, p = 0.0012). For the hearing participants we found rxxHearing = 0.56 (LVF<sup>2</sup> = 0.18 + LVF1\*0.65; t<sup>8</sup> = 1.91, p = 0.09). Reliability of the LVF measure was high in the group of deaf participants and medium in the group of hearing participants. However the smaller number of subjects participating in both experiments in the hearing group make it difficult to draw a firm conclusion on the difference in test–retest reliability between both groups.

Further analysis showed that 5 out of the 6 hearing participants who had a significant visual field bias in Experiment 1, and participated in Experiment 2, showed a similar significant bias in Experiment 2. In the deaf group, 2 out of 4 participants showed a visual field bias in both experiment. Given these results we recomputed the LVF measure of the participants using the results from both experiments when it was possible. These values (see Supplementary Table S1), which presumably best render the participants' visual bias, were used to examine the relation between scanning strategy and visual field bias in the following section.

#### Eye-Tracking Data

Raw position signal from the eye-tracker was processed offline. Saccades and fixations were parsed using an algorithm adapted from Engbert and Kliegl (2003). We set the minimum amplitude for saccades to 0.5◦ of visual angle. Only trials where participants gazed at the face and with the initial fixation located on the fixation point were included in the analyses. The AOI for the mask, used to identify fixation location, were defined post-hoc using the full distribution of fixation locations (see Supplementary Figure S1 for details and **Figure 4** for an illustration of the final mask). Five areas of interest were constructed. Supplementary Figure S1 shows the overall proportion of fixations within each AOI. The goal of the analyses was to test whether hearing and deaf participants differed with respect to face scanning and whether individual differences in face scanning could be related to the LVF bias.

#### **General oculomotor behavior**

In order to check for the existence of differences between hearing and deaf participants with respect to basic aspects of oculomotor behavior, we first analyzed the distribution of fixation duration. The dataset was composed of N = 7,751 fixations and was best fitted by a lognormal model with µ = 5.39 (219 ms) and σ = 0.51. The mean and standard deviation in the two groups were 5.38 (n = 4,158, SD = 0.49) and 5.39 (n = 3,593,

mean proportion of looking time.

SD = 0.54) for the hearing and deaf group respectively. No differences were found between the hearing and deaf groups concerning fixation durations. Saccadic reaction time (i.e., the time between stimulus onset and the onset of a saccade away from the fixation point) was similar in both groups (hearing: M = 180 ms, SD = 41 ms; deaf: M = 200 ms, SD = 87 ms, twosample t-test, t<sup>26</sup> = −0.75, p = 0.46). The increased SD in the deaf group was due to one subject (D2) who showed very large saccadic RT (M = 471 ms, SD = 181 ms). No differences were found between the hearing and deaf groups regarding parameters related to saccades (amplitude or velocity). The Supplementary Table S1 shows the extracted parameters for each participant. Knowing that oculomotor behavior was comparable in hearing and deaf participants we analyzed in more detail the scanning pattern of each participant.

#### **First saccade**

We focused on the first saccade and the subsequent landing position for the first fixation on the stimulus image. To quantify the differences in initial visual attention to the left and right part of the face, we simply divided the number of first fixations landing on the left part of the face by the total number of initial fixations made by participants (i.e., the number of valid trials) and subtracted 0.5 from this ratio. A score of 0.5 thus indicates that all initial fixations were on the left part of the face while a score of −0.5 indicates a right initial fixation bias (IFB). The distribution of our IFB was highly heterogeneous (M = 0.16, SD = 0.39). As shown in the Supplementary Table S1, most participants had a large positive bias toward the left part of the face (IFB > 0.25, n = 14). However, four participants showed a large bias toward the right part of the face (IFB < -0.25). We used a linear model of the form IFB = β<sup>1</sup> + β2G<sup>j</sup> + εij where G represented the group and was coded j = −0.5 for hearing and j = 0.5 for deaf. According to the model the intercept term represents the amount of IFB bias in the whole group and the second term β<sup>2</sup> represents the change in IFB bias due to group. The intercept was significant (β<sup>1</sup> = 0.16, t<sup>26</sup> = 2.86, p = 0.008) confirming the previously observed initial bias toward the left part of faces during face perception tasks. This bias was not influenced by group (β<sup>2</sup> = 0.03, t<sup>26</sup> = 0.28, p = 0.77).

Next, we classified each landing position according to the AOI shown in **Figure 4**. Complementary analyses showed that starting position strongly influenced landing position but this was independent of group and had no influence on the overall IFB (see Supplementary Table S2). Density maps for the location of the initial fixation points depending on starting position are shown in the Supplementary Figure S2.

#### **Relative total looking time in AOI**

In addition to initial fixations we classified all the fixation durations in our sample according to the predefined AOIs. **Figure 4** shows the mean ratio of looking time to each AOI for the hearing and deaf group as a function of the stimulus type (chimeric, average).

We built a linear model of the ratio of looking time (RLT) of the form RLT = β<sup>0</sup> + βjAOIj + βjkAOI<sup>j</sup> \* G<sup>k</sup> + εijk where AOI was coded j = 0, 1, 2, 3, or 4 for the face, left eye, right eye, nose, and mouth areas respectively. Group (G) was coded 1 for hearing participants and 2 for deaf participants. According to the model the intercept term represents the ratio of looking time in the face AOI for the hearing participants. The values of β<sup>j</sup> represent the change from this baseline ratio for the left eye, right eye, nose, and mouth AOI for the hearing group. Finally, values of βjk represent the change from the ratio in the hearing group to the ratio in the deaf group for each AOI. **Table 2** summarizes the results. The ratio of looking time for hearing and deaf participants differed for the face, left eye, right eye, and mouth AOI. Deaf participants spent less time in the eye areas than hearing participants, but they spent more time to the face and mouth area. Framed within the classical ANOVA format we found a main effect of AOI [F(4,270) = 9.64, p < 0.001] and an AOI x Group interaction [F(4,270) = 4.06, p < 0.0014]. Complementary analyses showed that stimulus type (average, chimeric) had no effect on the repartition of visual attention toward the faces (see Supplementary Figure S3).

Next we calculated an index of visual bias toward the left part of the face similar to the IFB used for the analysis of the first fixation. The distribution of total fixation bias (TFB) was more homogeneous (M = −0.004, SD = 0.12). As shown in the Supplementary Table S1, most participants had small


TABLE 2 | Result of linear model of the ratio of looking time within AOI.

Residual standard error: 0.1336 on 270 degrees of freedom. Multiple R<sup>2</sup> : 0.1791. Adjusted R<sup>2</sup> : 0.1517. F-statistic: 6.545 on 9 and 270 DF. p-value < 0.001. Bold values indicate significant results.

fixation bias (0.25 < TFB > −0.25). We used the linear model TFB = β<sup>1</sup> + β2G<sup>j</sup> + εij where G represented the group and was coded j = −0.5 for hearing and j = 0.5 for deaf. According to the model the intercept term represents the amount of TFB bias in the whole group and the second term β<sup>2</sup> represents the change in TFB bias due to group. The intercept was non-significant (β<sup>1</sup> = −0.0048, t<sup>26</sup> = −0.195, p = 0.85) indicating that the previously observed initial bias toward the left part of faces during face perception tasks is limited to the initial part of exploration. We found no effect of group (β<sup>2</sup> = 0.008, t<sup>26</sup> = −0.16, p = 0.87).

#### Post hoc Analysis

We analyzed the mean values of LVF index computed using the two experiments (given in Supplementary Table S1) using one sample t-test for the hearing and deaf group separately. The LVF bias was significantly different from 0 in the hearing group (M = 15.6, t<sup>13</sup> = 2.81, p = 0.046) but not in the deaf group (M = 6.78, t<sup>13</sup> = 1.38, p = 0.189).

Finally, we checked the correlation between the LVF bias and the measures of visual fixation bias (IFB and TFB) and relative time to AOI for the first fixation as well as for the whole set of participants. The correlation of LVF with IFB and TFB were non-significant. The relative time spent to halve faces was not predictive of the response bias in the gender categorization task. Instead, the visual exploration parameters that were most related to the LVF bias were the proportion of fixation time to the left eye (r = 0.44, t<sup>26</sup> = 2.54, p = 0.017) and the mouth area (r = −0.41, t<sup>26</sup> = −2.31, p = 0.028). Note that the proportion of fixation time to the left eye area was positively related to LVF while the proportion of initial fixation to the mouth area was negatively related to LVF.

### Discussion

Overall it was more systematic to find a LVF bias at the group level for hearing participants than for deaf participants. If the between-group analysis of behavioral results in Experiments 1 and 2 did not show a global reduction of LVF bias, the number of participants presenting a significant LVF bias was, however, greater in the hearing group than in the deaf group. It suggests some changes in the hemispheric asymmetry, at least in a part of the population of early deaf adults.

The analysis of the fixation patterns revealed interesting findings. First we did not find any significant differences in the left/right repartition of fixations between the two groups, either for the location of the first fixation or for the overall fixation time. However group differences were found in attention to the eye and mouth areas, deaf participants being more attentive to mouth area but less to the eyes than hearing participants. We found a positive correlation between the LVF bias and the relative time spent looking at the left eye, highlighting the importance of this region in the lateral bias. This latter result may explain the reduced LVF bias in deaf participants who spent less time in the eye areas than hearing participants, but spent more time to the face and mouth area. This result makes sense because the left eye is an informative location to decide on the gender of the hemiface. Paying more attention to the mouth area focuses the attention toward the center of the face, thus leading to smaller LVF bias.

### GENERAL DISCUSSION

An established fact in perceptual asymmetries is that for many aspects of face identity processing (perception of age, attractiveness, gender or expression) typical individuals attend to information on the right side of the face, leading to a LVF bias (Burt and Perrett, 1997; Butler and Harvey, 2005). This left bias is thought to reflect a right hemisphere advantage for face processing (Yovel et al., 2008). In deaf people, auditory deprivation and use of sign language seem to affect hemispheric lateralization (Bosworth and Dobkins, 1999; Bavelier et al., 2001). To date, very few studies have found modifications of lateralization using face stimuli in deaf people, and only for the processing of facial expressions (McCullough et al., 2005; Letourneau and Mitchell, 2013). The present study is the first to specifically investigate visual field asymmetries for the processing of facial identity using chimeric faces in deaf people. Using a gender categorization task we found that it was less frequent to find a significant LVF in a group of early deaf participants than in hearing controls. This suggests modifications of cerebral lateralization in deaf people for the processing of invariant aspects of faces, suggesting that early deafness, together with the extensive use of signed language, affects not only the processing of facial expressions, but also the core mechanisms underlying face recognition. Our results are in agreement with those obtained by Szelag and Wasilewski (1992), who found using a divided visual field task a reduced LVF bias in congenital deaf children. In their study this absence of visual field advantage seemed to come from a more variable asymmetry in deaf children, with approximately half of deaf children showing a leftward asymmetry and the other half showing a rightward asymmetry. This variability is also present in our experiments where a clear LVF bias was found in fewer deaf participants than in the hearings. This suggests a greater variability in face hemispheric lateralization in deaf people, potentially resulting from an increased role of the left hemisphere relatively to the right.

This variability may be explained by the heterogeneity of the deaf sample; as shown in **Table 1** our deaf participants differ in terms of etiology of deafness, learning age and daily use of sign language, lip reading abilities or the daily use of hearing aid. This heterogeneity may have influenced our results, and can explain a greater variability in terms of brain specialization. In particular the principal language used in the daily life (oral vs. signed) as well as the age of acquisition of sign language could greatly influence the development of visual field asymmetries. It would be of interest to study more directly the impact of sign language in visual field asymmetries for face processing. Another important point to consider is lip-reading ability, as shown by our post hoc analysis; there is a negative correlation between fixation time on the mouth area and the amount of left visual bias. It indicates that paying more attention toward the mouth reduces the LVF asymmetries, because it draws attention toward of the center of the face.

The question that arises from our present results is why deafness would affect face processing. One possibility is that during infancy, children have to learn to link auditory and visual inputs to form one unique perceptual object. To identify a person

in everyday life, we rely indeed not only on visual processing of faces, but also on the processing of vocal information. In the absence of the auditory modality, visual processing of face should thus become more salient for communication and social interactions. Another possibility is that face processing is influenced by the use of sign language whereby facial expressions convey information about the emotional state of individuals, but also carry linguistic information. Thus, deaf signers have to pay attention to face for both affective and linguistic inputs; it seems thus possible that they develop particular processing mechanisms that allow them to maximize the ability to gather information from faces. Shifts of cerebral lateralization for the processing of facial expressions from right to left have been observed in deaf participants for the processing of facial expressions (Emmorey and McCullough, 2009) and some results show that right hemisphere activation could be reduced for the processing of neutral faces (Weisberg et al., 2012). Interestingly enough, Weisberg et al.'s (2012) results seem to indicate joint effects of auditory deprivation and extensive use of sign language on cerebral activation. To disentangle these two effects, they also tested a group of hearing signers. Activation in the right middle fusiform gyrus for this group was at an intermediate level between deaf signers and hearing non-signers, suggesting a combined effect of the sensory deprivation and use of sign language. However, this study was not specifically designed to test cerebral asymmetry for face processing in deaf, thus additional studies are needed to evaluate more precisely how asymmetry for face processing is modulated by auditory deprivation.

One limitation of our study is that we did not test a group of hearing signers; therefore the question of whether the modifications of visual field asymmetries observed in this study are related to auditory deprivation, expertise with sign language, or a combination of both, remains open. To disentangle the relative influences of sensory deprivation and plasticity resulting from the use of sign language, it will be necessary to investigate asymmetry using chimeric faces in a population of native hearing signers.

One purpose of the present study was to relate the amount of LVF in deaf participants with the left/right scanning behavior of the participants. Previous studies suggest indeed that the LVF bias results not only from a right hemisphere advantage during face processing, but also from the scanning pattern of participants favoring the inspection of the left side of the face. The left side of the face would be investigated first, and longer than the right side (Phillips and David, 1997; Butler et al., 2005; Guo et al., 2012). The LVF bias has been found to be reduced in people who have a reversed scanning pattern such as Hebrew or Arabic readers (Vaid and Singh, 1989; Heath et al., 2005). The reduction of the LVF bias in deaf people could thus come from a reduction of cerebral asymmetry in face areas, from a scanning pattern favoring more the right side of the face, or both. Early deafness has been found to affect the pattern of eye movements (Watanabe et al., 2011; Bottari et al., 2012), even in non-linguistic or non-emotional tasks, suggesting that the habitual gazing pattern toward faces is altered in deaf people. In our study, we found no left/right difference in the scanning patterns of our participants, but there was a difference in the bottom/up repartition of fixations. While deaf participants showed the classical fixation pattern eyesmouth-nose like hearing controls, the proportion of fixations on the mouth was increased in deaf participants as costs of attention to the eyes. This suggests a tendency in deaf participants to favor more the information contained in the mouth than hearing participants even in non-communication situations. Alterations of gazing behavior in communication situation have been suggested before (Emmorey et al., 2009). In this study the authors found that in a communication situation, beginning ASL signers fixated more the mouth than native deaf ASL signers who fixated preferentially the eyes. Interestingly enough, other results indicate that these alterations of the gazing behavior could extend to the perception of static faces (Letourneau and Mitchell, 2011; Watanabe et al., 2011). Watanabe et al. (2011) used static faces in early deaf and hearing participants and found an increased fixation time on the eyes in the deaf group relatively to the hearing group. This seems at odd with our results, however, this discrepancy can be explain by cultural bias; Watanabe et al.'s (2011) results have been obtained in Japanese participants which makes the comparison difficult to draw as East Asian observers fixate less the eyes than Western Caucasians (Blais et al., 2008; Miellet et al., 2013). In agreement with our results, another recent study (Mitchell et al., 2013) using composite neutral faces showed an increased attention to the bottom of the faces in deaf participants. Taken together, these results suggest that the use of lipreading and attention toward facial expression does affect profoundly the gazing behavior on faces for deaf participants, and extends toward non-communication situations with static and neutral faces.

### CONCLUSION

This study suggests that early auditory deprivation and/or expertise with sign language affect the processing of faces, by altering hemispheric lateralization and modifying visual attention taken to static faces. These results emphasize the need of more detailed investigations about face perception in early deaf people and the relation between hemispheric lateralization and gazing behavior, as well as the relative influences of auditory deprivation and the use of sign language in this plasticity for face processing.

### ETHICS STATEMENT

The experiment was approved by the local ethics committee ("Comité d'éthique des centre d'investigation clinique de l'interrégion Rhône-Alpes-Auvergne", no. 2014-A00088-39). Subjects signed informed consent before participating in the experiments.

### AUTHOR CONTRIBUTIONS

Designed the experiments: MD, OP, and DM. Performed the experiments: MD. Analyzed data: MD and DM. Wrote the manuscript: MD, DM, and OP.

### ACKNOWLEDGMENT

fpsyg-08-00030 January 19, 2017 Time: 16:12 # 11

This research is supported by ANR Plasticity and Multimodality in Oral Communication for the Deaf. The authors would like to thank G. Leroy for his help collecting the data.

### REFERENCES


### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fpsyg. 2017.00030/full#supplementary-material


experience and age of acquisition. Brain Lang. 57, 285–308. doi: 10.1006/brln. 1997.1739


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Dole, Méary and Pascalis. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

fpsyg-08-00030 January 19, 2017 Time: 16:12 # 12

# Auditory and Visual Electrophysiology of Deaf Children with Cochlear Implants: Implications for Cross-modal Plasticity

David P. Corina1,2 \*, Shane Blau<sup>1</sup> , Todd LaMarr<sup>1</sup> , Laurel A. Lawyer<sup>1</sup> and Sharon Coffey-Corina<sup>1</sup>

<sup>1</sup> Cognitive Neurolinguistics Laboratory, Center for Mind and Brain, University of California at Davis, Davis, CA, USA, <sup>2</sup> Department of Linguistics, University of California at Davis, Davis, CA, USA

#### Edited by:

Matthew William Geoffrey Dye, Rochester Institute of Technology, USA

#### Reviewed by:

Davide Bottari, University of Hamburg, Germany Pascale Sandmann, University of Cologne, Germany

> \*Correspondence: David P. Corina dpcorina@ucdavis.edu

#### Specialty section:

This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology

Received: 15 August 2016 Accepted: 10 January 2017 Published: 01 February 2017

#### Citation:

Corina DP, Blau S, LaMarr T, Lawyer LA and Coffey-Corina S (2017) Auditory and Visual Electrophysiology of Deaf Children with Cochlear Implants: Implications for Cross-modal Plasticity. Front. Psychol. 8:59. doi: 10.3389/fpsyg.2017.00059 Deaf children who receive a cochlear implant early in life and engage in intensive oral/aural therapy often make great strides in spoken language acquisition. However, despite clinicians' best efforts, there is a great deal of variability in language outcomes. One concern is that cortical regions which normally support auditory processing may become reorganized for visual function, leaving fewer available resources for auditory language acquisition. The conditions under which these changes occur are not well understood, but we may begin investigating this phenomenon by looking for interactions between auditory and visual evoked cortical potentials in deaf children. If children with abnormal auditory responses show increased sensitivity to visual stimuli, this may indicate the presence of maladaptive cortical plasticity. We recorded evoked potentials, using both auditory and visual paradigms, from 25 typical hearing children and 26 deaf children (ages 2–8 years) with cochlear implants. An auditory oddball paradigm was used (85% /ba/ syllables vs. 15% frequency modulated tone sweeps) to elicit an auditory P1 component. Visual evoked potentials (VEPs) were recorded during presentation of an intermittent peripheral radial checkerboard while children watched a silent cartoon, eliciting a P1–N1 response. We observed reduced auditory P1 amplitudes and a lack of latency shift associated with normative aging in our deaf sample. We also observed shorter latencies in N1 VEPs to visual stimulus offset in deaf participants. While these data demonstrate cortical changes associated with auditory deprivation, we did not find evidence for a relationship between cortical auditory evoked potentials and the VEPs. This is consistent with descriptions of intra-modal plasticity within visual systems of deaf children, but do not provide evidence for cross-modal plasticity. In addition, we note that sign language experience had no effect on deaf children's early auditory and visual ERP responses.

Keywords: cross-modal plasticity, cochlear implants, deaf children, intramodal plasticity, developmental p1, developmental n1, ERP

## INTRODUCTION

fpsyg-08-00059 January 30, 2017 Time: 18:52 # 2

Congenital deafness leads to significant language delays in children acquiring spoken language. Cascading effects of impoverished linguistic knowledge impact a wide range of psychological and cognitive behaviors including self-regulation (Calderon and Greenberg, 2011), working memory (Pisoni and Geers, 2000), and reading (Perfetti and Sandak, 2000). About two to three out of every 1,000 children in the United States are born with a detectable level of hearing loss in one or both ears (Centers for Disease Control and Prevention, 2010). Cochlear implants (CIs) have become a popular treatment option for deaf children. These devices deliver electrical stimulation to the auditory nerve, bypassing malfunctioning peripheral auditory mechanisms. Deaf children who receive a cochlear implant early in life and engage in intensive oral/aural therapy often make great strides in spoken language acquisition. However, even under optimal conditions and the best efforts of clinicians, there is a great deal of variability in language outcomes (Tobey et al., 2012).

The interplay of factors contributing to this lack of success is poorly understood (Svirsky et al., 2000; Geers et al., 2008; Peterson et al., 2010). One increasing concern is that under conditions of deafness, the auditory system is subject to crossmodal plasticity (CMP), (Kral and Sharma, 2012; Sharma and Mitchell, 2013). In CMP, primary sensory cortices that are associated with a deprived modality can become colonized by the remaining modalities (Bavelier and Neville, 2002). In the case of deafness, the processing demands of an intact sensory system, such as vision, may recruit nascent auditory cortex making it less available for speech processing. The extent to which this has negative effects on auditory processing after implantation, may be referred to as maladaptive CMP.

Early studies of animal models of deafness provide evidence supporting the idea that CMP is present in humans (Allman et al., 2009; Meredith and Allman, 2009). For example Lomber et al. (2010) demonstrate supranormal enhancements in peripheral vision localization and visual motion detection in deaf cats. These enhanced functions are isolated to anatomically distinct auditory regions: primary auditory field (PAF), associated with increased visual peripheral target detection, and the dorsal zone of the auditory cortex (DZ). Critically, the causal relationship between visual and auditory function was demonstrated by selective cooling of auditory association regions resulting in a loss of the supranormal abilities. However, recent evidence suggests that responsiveness to visual input in DZ is in fact quite limited and importantly doesn't come at a cost of auditory functionality (Land et al., 2016).

Recent work has reported evidence of CMP in pre- and postlingually deaf adults with CIs which has been suggested to be maladaptive. In studies that have reported maladaptive CMP, the research often makes use of a neural marker of visual processing (e.g., P1 or N1 evoked potentials), and relates this signal to a behavioral processing deficit such as identification of speech in noise (Doucet et al., 2006; Buckley and Tobey, 2011; Sandmann et al., 2012; Campbell and Sharma, 2016; Kim et al., 2016). The inference is then made that the altered visual response is causally related to the auditory speech processing and, by association, that auditory cortical regions are vulnerable to reorganization (Sharma et al., 2015).

There are several weaknesses in this line of reasoning, the foremost of which is that a high-level auditory function like the recognition of words is a multi-component process that encompasses many distinct processing stages. This involves not only fundamental elements of acoustic processing, but mechanisms of speech segmentation, phonemic identification, and lexical recognition, as well as other cognitive properties such as attention to the stimuli, and in the context of multimodal testing, integration of visual speech information. Thus behavioral performance draws on many cognitive systems that extend beyond the function of primary auditory cortex alone.

A second weakness is that the methods that are used to assert that auditory cortex capabilities have been usurped by visual processing rely on source localization of ERP signals such as sLORETA (Pascual-Marqui, 2002; see for example Sharma et al., 2015). Such methods are known to have limitations with neural data, particularly where there may be simultaneously active sources. A strong or superficial source may obscure weak or deep sources, and nearby sources of similar orientation tend not to be separated but interpreted as one source located roughly in between (Wagner et al., 2004). Caution is further warranted in the context of EEG data collected in the presence of CIs, as it is unclear how device-generated signal and noise may impact spatial resolution and source localization solutions.

Furthermore, few published studies have directly evaluated physiological measures of auditory and visual function in a pediatric population with congenital deafness. The use of pediatric populations is especially important as children in their formative years of language development may be at greatest risk of developing maladaptive CMP. This also highlights the active role of the language acquisition process, as the question has been raised as to whether language input itself may play a role in maladaptive CMP. For instance, Giraud and Lee (2007) assert that "exposure to sign language in the first 3 years of life locks the language system into a vision-only configuration that prevents possible future acquisition of auditory language," suggesting children exposed to visual language input should be at greater risk for maladaptive CMP.

The present study addresses these concerns by collecting both auditory and visual evoked potentials (VEPs) in a pediatric population in the early and middle stages of language development, including children exposed to sign language and those enrolled in aural/oral-only programs. We begin with an experiment designed to elicit a cortical auditory evoked potential, the auditory P1, which has been described as biomarker of primary auditory cortex development in deafness (Sharma et al., 2015). Next, we turn to a visual experiment designed to elicit VEPs where the onset and offset of a patterned peripheral visual display results in a characteristic biphasic P1–N1 complex. Comparing these measures allows us to examine whether visual processing modulates lower-level auditory function in children with CIs. We further explore whether a subject's language experience (exposure to oral versus signed language) interacts with the expression of these auditory and visual markers.

### EXPERIMENT 1: AUDITORY PROCESSING

### Materials and Methods

fpsyg-08-00059 January 30, 2017 Time: 18:52 # 3

### Participants

Twenty-six congenitally deaf children with severe-to-profound sensorineural hearing loss, ages 2.0–8.5 years (X = 4.10), who received CIs, served as subjects. Twenty-five normally hearing children, ages 2.4–8.3 years (X = 5.2), served as controls. **Table 1** presents the subject characteristics and demographics of the deaf children involved in the present study. Shown in **Table 1** is the child's age at time of testing, age of first implant (in days) (X = 701.5, range 287–1581), gender, cochlear implant(s) (bilateral, unilateral), time in sound (in days for the first implant) (X = 1006.65, range 239–2098), and language inventory scores (spoken and sign language production). Auditory data from two subjects (S13 and S22) was corrupted and therefore not used in the analysis of the auditory results.

### Behavioral Testing

Caregivers completed a modified 92 item MacArthur language inventory for English (Fenson et al., 1994) and American Sign Language (Anderson and Reilly, 2002).

### ERP Testing

Children were fitted with a 22-channel electrode cap. Most children sat in an appropriately sized chair, while some younger participants sat on their parent's lap during recording. In all cases, an experimenter sat to the right of the child. During the auditory testing, children were seated in front of Dell Latitude 620 laptop computer and watched a silent cartoon or played an iPad game (sound muted) while auditory stimuli were presented.

Auditory stimuli were presented in an oddball paradigm designed to elicit a P1 cortical auditory evoked potential. The stimuli consisted of either a synthesized speech syllable (/ba/) which served as a standard (85%) or a frequency modulated tone (600–1200 Hz) which served as a deviant (15%), both of which were 100 ms in duration. Auditory stimuli were presented free-field at 65 db for deaf children and 60 db for hearing controls using AUVIO 05A13 speakers located approximately 45◦ degrees to left and right of the subject and powered by a NuForce Icon amplifier driven by the laptop's audio output. Stimuli presentation was jittered between 2 and 4 s to reduce

TABLE 1 | Characteristics of deaf subjects in the present study, including age, gender, age at first implantation, whether bilaterally or unilaterally implanted, time since first implantation measured in days (Time in Sound; TIS), and scaled words/signs produced, gathered from a parental report of language production.


expectancies. A total of 202 trials per subject were presented, lasting approximately 4.5 min.

#### Data Recording and Analysis

fpsyg-08-00059 January 30, 2017 Time: 18:52 # 4

ERPs were collected using a Biosemi Active Two recording system (Biosemi B. V., Amsterdam, Netherlands). Recordings were taken at 22 electrode sites, using standard 10/20 system. Three additional external electrodes were used to record data from left and right mastoids and the third was placed below the left eye to monitor eye movements. The eye electrode was used to assist in eliminating trials where blinks or horizontal eye movements occurred and trials where participants were looking away. Voltage offsets between each active electrode and CMS (common mode sense -the online reference) were below 20 µV, before the start of data collection. Offsets were checked again at the end of the recording session.

Sampling rate during recording was 512 Hz. Offline, continuous data was downsampled to 256 Hz, and bandpass filtered at 0.1–30 Hz. Data from scalp and eye electrodes were rereferenced offline to the average of left and right mastoids. Initial analysis of the EEG data was performed using the ERPLAB plugin (Lopez-Calderon and Luck, 2014) for EEGLAB (Delorme and Makeig, 2004). Independent Component Analysis (ICA) using the Infomax algorithm implemented in the EEGLAB Toolbox was used to remove both eye movement and cochlear implant artifacts. ICA analysis was performed on both auditory and visual data in order to reduce eye blink artifact. Between one and two components were removed due to eye blink. Four subjects (2–3 year olds) did not have any obvious eye blink components in their visual data. In these cases, no components were removed. In all subjects, additional artifact rejection was performed automatically, removing all trials where voltage exceeded ±100 µV, in all channels that were used in analysis. For the auditory data 12.6% (range 0–47.6%, SD 10.85) of deaf subjects trials were rejected, while 15.07% (range 0.7–32.9%, SD 10.1) of hearing children's trials were rejected. T-test indicated that the numbers of auditory trails rejected across groups did not differ (t = 0.83, p = 0.42). For visual data 10.72% (range 0–42.6%, SD 10.09) of deaf subjects trials were rejected, while 8.6% (range 0–28.8%, SD 8.4) of hearing children's trials were rejected. T-test indicated that the number of visual trails rejected across groups did not differ (t = 1.13, p = 0.26).

Auditory data was collected with CIs functioning. In 11 children with CI, we were unable to establish contact at lateral temporal or parietal sites due to the location of the implanted receiver/stimulator. In these cases, we eliminated the affected channels prior to data analysis. ICA analysis was used to remove CI artifact from the deaf participant data. Between 1 and 5 components per subject were removed in auditory data set. Auditory data reported here represent responses only to the standard /ba/ stimuli. For all analysis, automatic peak detection [most positive (P1) or negative peak (N1)] was taken using ERPLAB's, ERP measurement tool (Lopez-Calderon and Luck, 2014).

#### Statistical Analysis

Statistical analyses of peak amplitude and latency values used mixed effects models which were estimated using the lme4 package (Bates et al., 2015) in R (R Core Team, 2016). Mixed effect models offer many advantages over traditional ANOVAs, including simpler post hoc testing, better modeling where assumptions of sphericity are violated (i.e., unequal variances across subjects), and analyses that are robust in cases with missing data and where cells are not completely balanced (Gueorguieva and Krystal, 2004). Increasingly mixed effect models are being used to evaluate EEG activity (see, for example Payne et al., 2015).

The group-level models included factors of Group, Age, and Gender.

All models were initially estimated with the maximum available fixed effects structure with factors iteratively assessed for significance. Individual factors were removed by excluding the factor with the lowest t-value and refitting the model until only factors with a t-value above 2 remained. Each model was also fitted with by-subject and by-site (frontal midlines sites Fz and Cz) random intercepts.

### Results

In control subjects, the electrophysiological response to auditory standards produced a positive peak between 100 and 200 ms, followed by a negative peak around 300 ms post stimulus. This was most prominent over fronto-central sites (**Figure 1**). This morphology is consistent with cortical auditory evoked potential P1–N1 complex. Deaf children showed more variable responses both in latency and in waveform morphology. To quantify the observed patterns across groups we measured both peak amplitude and latency of the most positive peak between 70 and 175 ms at frontal midlines sites Fz and Cz for all subjects.

### Auditory P1 Amplitude

A main effect of Group indicated that hearing controls showed a larger P1 compared to the deaf children with CIs (t = −2.424, p = 0.019; Hearing X = 8.36 µV, Deaf X = 5.73 µV). This effect is illustrated in **Figure 1**. No other factors were significant.

### Auditory P1 Latency

We observed a main effect of Age (t = −2.29, p = 0.03) and a significant Age × Group Interaction (t = 2.00, p = 0.05). The Age × Group interaction is depicted in **Figure 2**. The scatter plot shows that while hearing children show expected age-related changes (latencies decrease with age), this pattern is not observed in the deaf children with CIs (Hearing, r = −0.46, t = −3.53, p = 0.001; Deaf, r = 0.09, t = 0.59, p = 0.56). No other factors were significant.

### Discussion

The auditory experiment was successful at eliciting an identifiable P1 auditory evoked potential in the majority of control children and deaf subjects with CIs. The P1 auditory evoked potential reflects the sum of the accumulated synaptic delays and neural conduction times as an auditory signal travels from the ear to the primary auditory cortex. Gilley et al. (2008) report that the cortical generator of the P1 is the auditory cortex in normal hearing children. However, the amplitude of the P1 has been shown to be sensitive to

stimulus level (Bertoli et al., 2011), thus the amplitude difference between hearing and deaf children observed here may reflect the reduced perceived signal intensity in the children with CIs.

P1 latency has been used as a biomarker of auditory system maturity. The latency of the P1 has been shown to decrease with age in normal hearing children (Eggermont, 1988; Liegeois-Chauvel et al., 1994; Eggermont et al., 1997; Sharma et al., 1997, 2002). In our data, hearing control children show these expected age-related changes while this pattern is not observed in deaf children. In previous work with deaf children implanted with CIs prior to 3.5 years old, Sharma et al. (2002) showed normal P1 latency and morphology by 7–8 months post implant. Our data in part support this observation, however, we do note that four of the children who received a cochlear implant prior to 3.5 years and have had at least 8 months experience with their CI show a longer than expected P1 latency based upon the published norms (Sharma et al., 2002). These data suggest that even with early implantation and adequate experience, some children with CIs will nevertheless exhibit atypical P1 latencies, potentially reflecting an aberrant maturation of cortical function.

### EXPERIMENT 2: VISUAL PROCESSING

To investigate visual function in our cohort, subjects were asked to watch a silent cartoon presented in the center of a laptop screen, while a checkerboard pattern was intermittently displayed in the peripheral surround. The appearance of the checkerboard results in a robust visual "onset" evoked potential. Similarly, the disappearance of this patterned display often yields a secondary visual "offset" evoked potential. Using this paradigm we investigated the visual responsivity of hearing controls and deaf children with CIs. To the extent that CMP is evident in our deaf sample, we might expect to see VEPs that are qualitatively different from hearing controls.

### Materials and Methods Participants

The same deaf and hearing subjects participated in the visual experiments as in Experiment 1. Deaf children had their CIs turned off during the visual experiment. Both auditory and visual testing was done for all subjects at the same time (one ERP testing session for each subject).

#### Procedures

Following participation in the passive auditory task, subjects watched a silent cartoon presented in the middle of the screen against a dark gray background. A radial black and white checkerboard (24 checks/6 annular rings, subtending 21.24◦ visual angle) intermittently replaced the background and lasted for 2, 3, or 4 s. There were a total of 60 trials, which lasted approximately 6 min. This peripheral visual stimulus was designed to elicit a pattern-onset and a pattern-offset VEP. Based on previous reports of differences between deaf and hearing subjects observed during visual processing we focused on the expression of the P1 and N1 visual components (Buckley and Tobey, 2011; Sandmann et al., 2012; Campbell and Sharma, 2016).

### Data Recording and Analysis

EEG procedures and analysis are identical to that of Experiment 1.

#### Statistical Analysis

Statistical analyses of peak amplitude and latency measures followed the same procedures used in Experiment 1. Here, the Group-level model included fixed effects of Pattern (onset/offset), Group, Age, and Gender, as well as by-subject and by-site (O1, O2, Pz, and fronto-central sites Cz and Fz.) random effects. Post hoc testing used Tukey's HSD, corrected for multiple comparisons, implemented by the lsmeans package (Lenth, 2016) in R.

### Results

Visual inspection of the data revealed a P1, peaking at approximately 165 ms followed by an N1, peaking about 250 ms, in both onset and offset of the checkerboard pattern. The effects were most robust at posterior parietal and occipital sites (**Figures 3** and **4**). To quantify observed VEP differences we examined the window of 70–200 ms post stimulus to characterize effects related to the P1 component. The window of measurement used for the N1 was 175–325 ms. Analysis of the visual data included posterior electrode sites O1, O2, Pz, and fronto-central sites Cz and Fz.

Automatic peak detection [most positive (P1) or negative peak (N1)] was taken using ERPLAB's measurement tool (Lopez-Calderon and Luck, 2014).

### Visual P1 Amplitude

Examining data from the visual VEP responses we find a main effect of Pattern, showing overall larger responses to pattern onsets than offsets (t = −2.58, p = 0.01; Hearing: X onsets = 8.83 µV, X offsets = 6.55 µV; Deaf: X onsets = 7.88 µV, X offsets = 7.26 µV).

No other factors were significant predictors of P1 amplitude, and post hoc testing showed no significant difference between Group for onset amplitude (t = 0.97, p = 0.33) or offset amplitude (t = −0.66, p = 0.51).

#### Visual P1 Latency

Looking at P1 latencies, we find no significant differences between onset and offset within Groups (t = −1.18, p = 0.24; Hearing: X onsets = 144.47 ms, X offset = 140.59 ms; Deaf: X onsets = 143.55 ms, X offsets = 144.35 ms). Post hoc testing revealed no significant differences between Group for pattern offset (t = −0.73, p = 0.47) or onset latencies (t = 0.18, p = 0.86) (**Figure 5**, left panel).

#### Visual N1 Amplitude

Examining visual N1 amplitude, we observed a trend in Pattern which showed that N1 was larger to offset relative to onset in both groups (t = −1.88, p = 0.06; Hearing: X onsets = −2.70 µV, X offset = −4.09 µV; Deaf: X onsets = −2.06 µV, X offsets = −2.47 µV). Post hoc testing showed no significant amplitude difference between Groups for either onsets (t = −0.57, p = 0.57) or offsets (t = −1.50, p = 0.14).

### Visual N1 Latency

Assessment of N1 latency showed a significant difference in Pattern for the control group with offset latencies longer than onset latencies (t = −1.95, p = 0.05; X onsets = 251.82 ms, X offset = 260.45 ms). A significant Pattern × Group interaction (t = −2.64, p = 0.008) showed that in deaf subjects, offset latencies were shorter than onset latencies (X onsets = 248.41 ms, X offsets = 240.78 ms). Post hoc testing further revealed a significant Group difference for offset latencies (t = 3.73, p = 0.0003), but not onsets (t = 0.66, p = 0.51). These differences are illustrated in **Figure 5**, right panel.

### Discussion

The visual data reveal no robust group differences in P1 VEP amplitude or latency. Previous studies examining the P1 VEP in adult deaf and hearing participants have reported mixed results. In early work conducted by Neville and Lawson (1987a), there were no reported differences in P1 VEP latency or amplitude in deaf adult subjects compared to hearing controls in a peripheral motion detection task. Armstrong et al. (2002) reported no group differences between deaf and hearing adult subjects in P1 amplitude or latency in response to sinusoidal gratings presented in the fovea and peripheral visual field. Doucet et al. (2006) recorded VEP to shape-changing stimuli in deaf adults with CIs and hearing controls. No group differences were found for either P1 latency or amplitude. Using the same shapechanging stimuli used by Doucet et al. (2006) and Campbell and Sharma (2016) reported no differences in early P1 latency and amplitude in a comparison of deaf children with CIs and hearing controls.

In contrast, Sandmann et al. (2012) reported VEPs to parametrically varied flashing checkerboard stimuli in a heterogeneous group of post-lingual deafened adults (mean age 54, range 38–70 years) who received CIs as adults. They reported reduced P1 amplitudes and shorter latencies in the CI group relative to hearing controls, which they interpreted as indexing a different degree of visual cortex recruitment in CI users compared to controls. They speculated this reduced latency may reflect shorter, more efficient visual information processing. Hauthal et al. (2014) examined VEPs in congenitally deaf and hearing subjects to reversing checker-board stimuli that were systematically modulated in luminance ratio. These participants showed similar modulation of VEP amplitudes (N85, P110) and latencies (P110) to the luminance modulation. However, compared to hearing subjects, deaf participants showed shorter N85 latencies and larger P110 amplitude. These findings are taken to suggest an indication of more efficient neural processing of visual information in the deaf.

Bottari et al. (2011) reported that in response to a visual warning signal, deaf subjects showed a decreased latency in the C1 (45–95 ms) and differential P1 morphology compared to controls. On the other hand, responses to visual targets resulted in longer P1 latencies in deaf compared to hearing controls and P1 amplitude in deaf subjects was correlated with reaction time performance on their task. Bottari et al. (2011) suggest changes in the P1 dynamics in the deaf may thus reflect stronger exogenous attention capture in deaf compared to hearing subjects.

In contrast to the P1, the N1 data show a stronger group difference for offset VEP responses. Specifically we see a shorter N1 latency in the deaf subjects compared to the hearing subjects. Previous research on later visual components, including the N1, show that these components are more consistently modulated by deafness than the visual P1.

Neville and Lawson (1987a) reported larger attention-related N1 modulations over occipital regions and left-temporal and parietal regions in deaf subjects compared to hearing controls. Armstrong et al. (2002) reported deaf participants showed larger N1 amplitudes to central and peripheral movement stimuli. Buckley and Tobey (2011) examined the N1 VEP response to peripheral movement targets in two groups of deaf subjects with CIs. These subjects either had pre- or post-lingual onset of severe-to-profound hearing loss. They found that larger N1 amplitude was associated with lower speech perception scores in prelingually deaf subjects. This pattern was not observed in subjects with post-lingual deafness. Campbell and Sharma (2016) reported some evidence of larger N1 amplitude and earlier latencies in deaf children with CIs, a pattern similar to our own data which we observe in offsets only. It is interesting to note that in the present experiment, group differences were observed in response to a static visual image display, rather than the more commonly used dynamic movement stimuli.

It is well known that attention may modulate N1 VEP (Clark et al., 1995; Mangun, 1995). We find it noteworthy that N1 latency effects were observed to the offset of the visual stimulus. This may indicate that deaf children with CIs are more attentionally vigilant to visual stimulus, where attentional capture may be triggered

for both the appearance and disappearance of a visual stimulus. Further enhancements may be evident for dynamic movement stimuli.

### RELATIONSHIP BETWEEN AUDITORY AND VISUAL DATA

To date studies of cross-modal interactions in deaf individuals with CIs have typically reported broad correlations between VEPs and behavioral measures of speech understanding (e.g., Doucet et al., 2006; Buckley and Tobey, 2011; Sandmann et al., 2012; Kim et al., 2016). In the present study, we have the ability to examine more directly the relationship between auditory and visual activity. Specifically, we questioned whether there was a relationship between auditory P1 latency and visual N1 offset latency within subjects. Recall, in our data the deaf subjects showed the expected lack of developmental progression in their auditory P1 latencies, suggesting an aberrant maturation of cortical function. Here, we ask whether the degree of variance associated with auditory P1 latencies in our CI subjects is accounted for by visual reactivity as indexed by our visual-offset N1 measure.

### Materials and Methods

### Participants

The data obtained from deaf and hearing subjects who participated in Experiments 1 and 2 were included in this analysis.

### Procedures

We examine visual data from two electrode sites, site O2 where we observed our largest N1 latency difference, and further examine the visual response at Cz. Central site Cz was chosen as this site is typically associated with a robust auditory P1–N1 response and is thought to reflect synchronous neural activation of structures in the thalamic-cortical segment of the central nervous system in response to auditory stimulation (Vaughan and Ritter, 1970; Wolpaw and Penry, 1975; Naatanen and Picton, 1987; Woods, 1995). Some caution is warranted in the comparison of the N1 response recorded from site O2 and the N1 response recorded at site CZ, as these may reflect difference sources (Coch et al., 2005).

### Statistical Analysis

In an effort to establish a relationship between auditory and visual evoked responses, we constructed two models of auditory P1 latency that included factors of Group, Age, Gender, and N1 latency. This latter measure contained each subject's N1 latency value at electrode site Cz in the first model, and in the second model, at electrode site O2 where the largest response was observed. As before, both models also included random intercepts for each subject and electrode site.

### Results

In evaluating these models, no significant effects beyond those already discussed were observed (all p-values >0.20). A post hoc analysis using Pearson's product-moment correlation provides further confirmation of the lack of relationship between auditory and visual latencies observed in deaf subjects at site O2 (r = −0.17, t = −1.19, p = 0.24) and at site Cz (r = −0.04, t = −0.27, p = 0.79). However, in hearing subjects, a significant correlation between auditory and visual latencies is observed at Cz (r = 0.31, t = 2.20, p = 0.03), but not O2 (r = 0.15, t = 1.06, p = 0.29). These differences are illustrated in **Figure 6**.

are collapsed across channels Cz and Fz. In the left panel, VEP data from site O2, in the right panel, VEP data from site Cz. The solid line shows the linear regression for each group, with standard error represented by the gray band. Hearing children exhibit a significant positive relationship between auditory and visual latencies responses recorded at site Cz.

### Discussion

The finding of a significant correlation between auditory evoked P1 latency and visually evoked N1 recorded at Cz in the hearing children was unexpected. Both early auditory and visual evoked potentials are known to undergo developmental changes, including reductions in latencies from birth through early childhood (Barnet et al., 1980; Lippe et al., 2009). If maturation alone were driving these correlative effects, we would reason that this relationship should be most robust when signals are recorded from sites that maximally capture the sensory effect of interest (e.g., auditory; Cz, visual; O2). However, in the present study we captured a correlation between visual and auditory signals recorded from central site Cz. It is possible that this reflects a more robust coupling between sensory areas in typically developing hearing children, one that is not observed in children with CIs. As noted, the N1 signals recorded at O2 and Cz may reflect different generators (Coch et al., 2005) and the apparent within subject latencies differences across O2 and Cz in our data reinforce this possibility. Additional work is needed to fully characterize these patterns.

### EFFECTS OF LANGUAGE EXPOSURE

The role signed language exposure may play in maladaptive CMP is both controversial and understudied. We wished to explore the relationship between sign exposure, and other demographic variables, and response latency differences observed in the models above.

### Participants and Materials

The data obtained from deaf subjects who participated in Experiments 1 and 2 were included in this analysis. We included subject Age, Gender, Signs and Words Produced (derived from our modified MacArthur inventories), Age of Implantation (of the first implant, in cases of bilateral implantation), and Time in Sound (measured by days since implantation).

### Statistical Analysis

We constructed two new models using only the deaf CI data, and modeled auditory P1 (sites Fz, Cz) and visual N1 response latencies (sites Fz. Cz, Pz, O1, and O2) with factors for Age, Gender, Signs and Words Produced (derived from our modified MacArthur inventories), Age of Implantation (of the first implant, in cases of bilateral implantation), and Time in Sound (measured by days since implantation). Measures of language production, as well as age of implantation, and time in sound, were highly correlated with chronological age. To avoid colinearities in the data, these were each residualized against Age. Both models also included random effects for subject and site, as in previous analyses.

### Results

We found no differences in response latencies as a function of language exposure, signs produced, words produced, age of implantation, or time in sound, in either of the latency models (all p-values were >0.25).

### Discussion

While some researchers have questioned whether a deaf child's mode of language exposure may differentially affect visual and auditory neural systems, with visual sign language experience fundamentally altering auditory language system (Giraud and Lee, 2007), we observed no differences in the auditory and visual responses of deaf children who have been exposed to sign language and those who have elected an oral-based rehabilitative strategy.

### GENERAL DISCUSSION

fpsyg-08-00059 January 30, 2017 Time: 18:52 # 11

Several findings emerge from these studies. Replicating previous reports, measures of the auditory P1 in deaf children with CIs show morphological patterns that differ from hearing controls. In the present data, the deaf children's auditory P1 amplitudes were reduced, which may reflect differences in perceived intensity of the stimuli. Auditory P1 latencies were also reduced, and did not show evidence of expected maturational changes observed in hearing controls. Even though the majority of our subjects received a CI before the age of 3.5 years, we observed that some children with CIs will nevertheless exhibit atypical P1 latencies, potentially reflecting atypical maturation of cortical function.

Data from the visual experiment revealed robust latency differences in the N1 components. Deaf children with CIs showed shorter N1 latencies compared to hearing controls in response to the offset of a patterned checkerboard. This distinction in visual responsivity may reflect the plasticity of the visual system of deaf children who have experienced a delay in auditory habilitation.

We evaluated the presence of cross-modal reorganization by examining the relationship between the auditory P1 and visual N1 responses in deaf subjects and hearing subjects. Research has suggested that under conditions of auditory deprivation, regions of auditory cortex may become responsive to visual information at the expense of auditory processing. Evaluating the relationships of the auditory P1 to visual evoked activity in occipital site (O2) and central site (Cz) showed no systematic relationship between evoked-potential latencies across these two sensory domains in the deaf subjects. We observed that variability associated with auditory P1 latencies was not effectively modulated by a high-contrast visual pattern in deaf children. Our data would indicate that auditory cortex does not become responsive to the low level visual signals induced by the stimuli used here as a result of early auditory deprivation in children who have received a cochlear implant early in life. The lack of a trade-off between auditory and visual processing at this level accords with physiological data from deaf cats that show functional changes in visual processing does not come at the cost of auditory function (Land et al., 2016). These data help to delimit the neurophysiological interactions that may be evidenced in the face of auditory deprivation.

It is interesting to note that we did observe a relationship between auditory P1 latencies and visual N1 latencies recorded at central site Cz but not at occipital site O2 for the hearing children. These data may reflect a coupling between auditory and visual sensory systems that is present in typically developing hearing children that is not observed in the deaf children. Further work is needed to understand the development and scope of such interactions.

Finally, we assessed whether the observed latency differences in the auditory P1 and visual N1 components were affected by early exposure to a signed language, age at first implantation, or time since implantation. None of these factors were shown to influence auditory P1 or visual N1 latencies. We especially note that we observed no differences in the auditory and visual responses of deaf children who have been exposed to sign language and those who have elected an oral-based rehabilitative strategy. These data directly challenge claims that exposure to a visual language in the formative stages of language acquisition in deaf children with CIs locks the language system into a visiononly configuration that prevents possible future acquisition of auditory language (Giraud and Lee, 2007).

## ETHICS STATEMENT

Statements involving human subjects. This study was carried out in accordance with the recommendations of UC Davis Human Subject Institutional Review Board following the Human Research Protection Program Plan. The protocol was approved by the Human Subject Institutional Review Board. Written informed consent was acquired from the parents of all subjects prior to beginning the experiment. This study involved hearing and deaf children 2–8 years old. Parents were informed that they could stop testing at any time and their children were not required to participate. All subjects gave written informed consent in accordance with the Declaration of Helsinki.

### AUTHOR CONTRIBUTIONS

DC and SC-C designed the experiment. TL and SB created inhouse behavioral measures. DC, SC-C, TL, and SB collected the data. LL did the statistical analysis of the data. SC-C analyzed the ERP data. DC, SC-C, and LL drafted the manuscript.

## FUNDING

This work was supported by NIH NIDCD Grant 3R01DC014767- 01 to DC.

### ACKNOWLEDGMENTS

The authors would like to thank Daniel D. Gonzalez for programming the experiments and Eva Gutierrez-Sigut for her work on an earlier version of this study. We would like to thank the children and families who participated in this research and especially the CCHAT Center, Sacramento, CA, USA; Weingarten Children's Center, Redwood City, CA, USA; the Learning Center for the Deaf, Framingham, MA, USA. The Fremont School for the Deaf, Fremont, CA, USA; and the Hearing Speech and Deafness Center, Seattle, WA, USA.

### REFERENCES

fpsyg-08-00059 January 30, 2017 Time: 18:52 # 12


plasticity in cochlear implant users. Brain 135, 555–568. doi: 10.1093/brain/ awr329


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Corina, Blau, LaMarr, Lawyer and Coffey-Corina. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# The Neural Basis of Speech Perception through Lipreading and Manual Cues: Evidence from Deaf Native Users of Cued Speech

Mario Aparicio<sup>1</sup> \*, Philippe Peigneux<sup>2</sup> , Brigitte Charlier<sup>1</sup> , Danielle Balériaux<sup>3</sup> , Martin Kavec<sup>3</sup> and Jacqueline Leybaert<sup>1</sup>

<sup>1</sup> Laboratory of Cognition, Language and Development, Centre de Recherches Neurosciences et Cognition, Université Libre de Bruxelles, Brussels, Belgium, <sup>2</sup> Neuropsychology and Functional Neuroimaging Research Unit (UR2NF), Centre de Recherches Cognition et Neurosciences, Université Libre de Bruxelles, Brussels, Belgium, <sup>3</sup> Department of Radiology, Clinics of Magnetic Resonance, Erasme Hospital, Brussels, Belgium

#### Edited by:

Olivier Pascalis, Université Grenoble Alpes, France

#### Reviewed by:

Jean-Luc Schwartz, Centre National de la Recherche Scientifique, France Veronique Coizet, Joseph Fourier University, France

> \*Correspondence: Mario Aparicio maparici@ulb.ac.be

#### Specialty section:

This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology

Received: 14 September 2016 Accepted: 07 March 2017 Published: 30 March 2017

#### Citation:

Aparicio M, Peigneux P, Charlier B, Balériaux D, Kavec M and Leybaert J (2017) The Neural Basis of Speech Perception through Lipreading and Manual Cues: Evidence from Deaf Native Users of Cued Speech. Front. Psychol. 8:426. doi: 10.3389/fpsyg.2017.00426 We present here the first neuroimaging data for perception of Cued Speech (CS) by deaf adults who are native users of CS. CS is a visual mode of communicating a spoken language through a set of manual cues which accompany lipreading and disambiguate it. With CS, sublexical units of the oral language are conveyed clearly and completely through the visual modality without requiring hearing. The comparison of neural processing of CS in deaf individuals with processing of audiovisual (AV) speech in normally hearing individuals represents a unique opportunity to explore the similarities and differences in neural processing of an oral language delivered in a visuo-manual vs. an AV modality. The study included deaf adult participants who were early CS users and native hearing users of French who process speech audiovisually. Words were presented in an event-related fMRI design. Three conditions were presented to each group of participants. The deaf participants saw CS words (manual + lipread), words presented as manual cues alone, and words presented to be lipread without manual cues. The hearing group saw AV spoken words, audio-alone and lipread-alone. Three findings are highlighted. First, the middle and superior temporal gyrus (excluding Heschl's gyrus) and left inferior frontal gyrus pars triangularis constituted a common, amodal neural basis for AV and CS perception. Second, integration was inferred in posterior parts of superior temporal sulcus for audio and lipread information in AV speech, but in the occipitotemporal junction, including MT/V5, for the manual cues and lipreading in CS. Third, the perception of manual cues showed a much greater overlap with the regions activated by CS (manual + lipreading) than lipreading alone did. This supports the notion that manual cues play a larger role than lipreading for CS processing. The present study contributes to a better understanding of the role of manual cues as support of visual speech perception in the framework of the multimodal nature of human communication.

Keywords: audiovisual speech perception, lipreading, manual gestures, deafness, Cued Speech, fMRI, MT/V5

## INTRODUCTION

fpsyg-08-00426 March 28, 2017 Time: 16:37 # 2

There is increasing evidence that sensory-deprived individuals make adjustments to their sensory loss in order to interact effectively within their environment. These adaptations are linked to changes occurring at multiple regions of the brain (Bavelier and Neville, 2002). For people who are deaf from birth or lost their audition early in life, neural plasticity of the regions classically associated with auditory and speech sound processing is related not only to lack of auditory experience but also to the timing and nature of language experience (Cardin et al., 2013). Among the children born deaf, the majority is born to hearing parents, and only 5% have deaf parents. The modality in which language is conveyed can be very different from one deaf child to another: from auditory-oral (listening, talking, and lipreading and facial expressions, known as speechreading), to visual communication strategies like Cued Speech (CS, supporting perception of spoken language with hand shapes that disambiguate lipreading, see below), and/or Sign Language (SL, sign for each language concept, with a grammar of its own). Recently, Olulade et al. (2014) suggested that the nature of language experience (signed vs. oral) has an impact on the development of gray matter volume in the cerebral regions processing language measured in deaf adults, but this point remains to be confirmed.

The timing of language experience can also vary among deaf children. Some of them have daily access to a fully perceivable linguistic input through SL or CS during the first year(s) of life when cerebral plasticity is as its greatest (Kuhl, 2004) while others only have partial access to auditory input (via the cochlear implant) and/or a late access (after the age of 4 years) to visual languages like SL and CS. For instance, deaf children who use SL from early in infancy outperform deaf children who are late learners in tests of SL proficiency and even in tests of English proficiency (Mayberry et al., 2002, 2011). Those children who are exposed to SL only at a later age show long-term language deficits (Emmorey et al., 1995; Lyness et al., 2013). Delayed acquisition of SL as a first language is related to structural changes in the visual cortex (less gray-matter concentration in V1/V2 and V3a/V7 in late signers), and this effect is independent of auditory deprivation (Pénicaud et al., 2013).

Here we tested, for the first time to our knowledge, the language neural processing of congenitally deaf adults who were exposed from early years to spoken language fully perceivable through the visual modality. Since spoken language has evolved to be primarily heard, not seen, critical features like voicing or nasality cannot be perceived by eye only. Among systems dedicated to make spoken language entirely visible to deaf persons, CS is the most widespread one.

Cued Speech (Cornett, 1967) is a visual communication system used with and among deaf and hard-of-hearing persons. It is a system which delivers consonant-vowel (CV) dyads in the spoken language using a small number of handshapes and locations as a supplement to lipreading (see **Figure 1** for the English and French versions of CS). Handshapes and the location in space where the hand is placed combined with the mouth movements of speech make all the syllables and phonemes of spoken language appear distinct from each other. Consonants that have similar mouth movements (like /p/, /b/ and /m/) are coded with different handshapes (1, 4, and 5, respectively). Consonants that are coded by the same handshape (like /p/, /d/ and /Z/) are easily distinguished by lipreading. Vowels indistinguishable on the lips (for instance French /y/ and /u/) are coded at different hand locations, and the same location is used to code vowels different on the lips.

Each time a speaker pronounces a CV syllable, he/she adds manually information about the word's phonological structure that is not entirely visible on the lips, by producing a handshape at a particular location in relation to the head and upper body (see **Figure 1**). Take the example of a speaker producing the syllable /pa/. From the lips, the receiver perceives /pa/, /ba/, or /ma/. When the manual cue is added (i.e., handshape n◦ 1 representing /p, d, j/, produced at the side of the face representing /a, oe, o/ vowels), the uncertainty is reduced, and the syllable /pa/ remains the only possibility. Indeed, /ba/ and /ma/ are eliminated on the basis of the information read from the handshape, and /da/ and /ja/ are incompatible with the information read on the lips. From this example, it clearly appears that CS handshapes and hand locations are not themselves interpretable as syllables or phonemes. The integration of manual and labial information is mandatory to perceive an unambiguous syllable. Deaf users of CS are thus afforded access to the words in a spoken language in which sublexical features are entirely specified solely by visible articulatory gestures, i.e., manual cues and mouth movements.

Cued Speech was created in the 1960s with the aim of allowing deaf children to accurately perceive spoken language and improving their literacy skills (Cornett, 1967; Cornett and Daisey, 2001). However, the overall effect of CS on early spoken language development extends beyond this (Leybaert et al., 2010, 2013, 2015). The empirical evidence collected in English, French, Spanish, Farsi and even Amharic (national language of Ethiopia) shows that congenitally deaf children who were exposed to CS from their earliest months by their parents and other caregivers can reach levels of mastery of spoken (phonology, lexical, morpho-syntactic) language and written language (word reading, reading comprehension, spelling) within range of age-matched hearing peers when tested at school age. Children with late and less intensive exposure (i.e., at the age of 5–6 years, and at school only) do not demonstrate the outstanding phonological and reading abilities of the early CS-users, confirming the existence of a sensitive period for language acquisition via the visual modality (Nicholls and McGill, 1982; Périer et al., 1990; Charlier and Leybaert, 2000; Leybaert, 2000; LaSasso et al., 2003; Torres and Moreno-Torres, 2006; Koo et al., 2008; LaSasso, 2010; Movallali, 2011; Heracleous et al., 2012; Colin et al., 2013; Rees and Bladel, 2013; Shull et al., 2016).

It is this evidence for comparable language abilities reached by congenitally deaf individuals who are early CS-users and NH individuals which raise our interest concerning how the human brain processes linguistic information when conveyed by handshapes and speech-based mouth actions, compared to when conveyed by AV speech. To put this research into perspective, we summarize below critical points about auditory speech processing stream and the way in which lipread signals

FIGURE 1 | The French and English Cued Speech codes. Complete diagram of hand shapes and hand positions of French and English CS. In CS, the speaker holds one hand near the mouth while speaking to complement lipreading with a manual cue. A cue in CS is made of two parameters: hand shape (left) and hand position of execution around the mouth (right). For example, syllables as /pa/, /ba/ or /ma/ cannot be distinguished using lipreading because they provide similar visual information. In CS, the syllables /pa/, /ba/ or /ma/ can be easily distinguished by simply using three different hand shapes.

are integrated with the auditory speech stream. Next the neural basis of visual language perception (SL and speechreading) in deaf participants is discussed. We address the commonalities and differences between SL and CS. Finally, the knowledge about time course of manual and mouth movements articulation in CS is summarized, introducing our three research hypotheses concerning the comparison of neural activation of CS and AV speech.

### Neural Basis of Audiovisual Speech Processing

For hearing people, speech perception is a multimodal phenomenon. It is known since long that vision is of great help for hearing in noise and adverse conditions (Sumby and Pollack, 1954; Erber, 1969). The neuroanatomy and neurophysiology of audiovisual (AV) interactions in the human cortex have been abundantly explored in the last 15 years (see Campbell, 2008 for a review; Calvert et al., 1997, 2000; Calvert and Campbell, 2003). The acoustic speech signal projects posteriorly from Heschl's convolutions within lateral temporal cortex to further superior temporal regions (secondary auditory cortex). The mid-posterior superior temporal sulcus (pSTS) appears to be a prominent site for AV speech integration (Binder et al., 2000; Callan et al., 2004; Hickok and Poeppel, 2007; Szycik et al., 2007), with yet more posterior regions around the temporo-parietal sulcus being implicated specifically in lip movement perception (see Bernstein and Liebenthal, 2014).

Neuroanatomical studies displayed three types of AV interactions. First, there are direct connections between sensory cortices (Besle et al., 2008). Second, associative areas and particularly the pSTS play a crucial role in AV speech perception (e.g., Beauchamp et al., 2004, 2010). Third, parieto-frontal areas related to speech production are involved through the dorsal route (see Jones and Callan, 2003; Skipper et al., 2005, 2007; Okada and Hickok, 2009). Electrophysiological studies revealed that the influence of visual speech in cortical auditory processing can occur within 100 ms of signal onset, suggesting that lipreading exerts an early effect on auditory signal resolution (Colin et al., 2002; Van Wassenhove et al., 2005; Stekelenburg and Vroomen, 2007; Besle et al., 2008; Arnal et al., 2009; Peelle and Sommers, 2015).

### Neural Activity Related to Lipreading

Since two decades now, the cortical substrates for lipreading in hearing and deaf participants have been widely investigated with neuroimaging techniques. In hearing people, silent lipreading engages activation of pSTS and middle temporal gyrus, and inferior frontal regions (Calvert et al., 1997, 2000; Bernstein et al., 2002; MacSweeney et al., 2000, 2002; Paulesu et al., 2003; Campbell, 2008; Bernstein and Liebenthal, 2014). These regions are similar to those engaged when speech is heard. Activation of left pSTS is correlated with hearing participant's lipreading skills (Hall et al., 2005).

Some deaf individuals become better lipreaders than normally hearing individuals, i.e., deaf individuals may be better than hearing persons to extract information about spoken language structure from visible lip movements, jaws, and face (Bernstein et al., 2001; Mohammed et al., 2006; Auer and Bernstein, 2007). The variability among deaf individuals is partly explained by their practice and knowledge of oral language. Indeed, those deaf individuals exposed daily to CS from their early years become very proficient lipreaders (Aparicio et al., 2012), likely because precise lipreading is a mandatory component in CS perception.

Variability among deaf lipreaders also appears in neuroanatomical studies. MacSweeney et al. (2000) asked deaf and hearing participants to silently lipread numbers (from 1 to 9) in a scanner. They found that temporal activation was more dispersed on different sites and less intense in the group of deaf participants. They suggested that coherent exposition to AV speech may play an important role for the structuration of temporal cortex for visual speech. In a second study, MacSweeney et al. (2002) found that the cingular cortex is a structure more activated during lipreading in deaf people than in hearing ones. A conjunction analysis of the data of these two studies revealed posterior activation of cingular cortex (BA23/30, related to visuo-spatial functions) in the deaf adults, and a bilateral activation of superior temporal areas (BA22/42) in the hearing adults. The deaf group showed activation of superior temporal gyrus (BA22) on the right side, extending into the tip of Heschl's gyrus (BA42, part of the secondary auditory cortex), suggesting a predominantly right lateralized network in deaf people.

Capek et al. (2008) realized a further study in which congenitally deaf adults who were native signers and proficient speechreaders and hearing non-signing controls searched for a lipread target (the word "yes") embedded in lists of silently spoken unrelated words from an open list. Participants pushed a button only when they identified the target word. The control condition was a speaker at rest. The results showed a strong activation of temporal superior gyrus, and in regions located in the Heschl's gyrus, especially on the left side, in deaf as well as in hearing participants. Inferior frontal gyrus was also activated in both groups, which may reflect the involvement of mirror neurons in lipreading (Campbell et al., 2001; Paulesu et al., 2003; Pekkola et al., 2005). Lipreading ability was assessed outside the scanner, with the Test of Adult Speechreading (TAS, Mohammed et al., 2006). In deaf participants, the neural activations during lipreading were positively correlated with the TAS score. Deaf participants showed greater activation than hearing participants in the left temporal cortex, including the planum temporale and Heschl's gyrus. Compared to the two previous studies, the Capek et al.'s (2008) one differed on a number of variables (nature of the baseline task, task requirement, stimulus type, and size of the group of deaf participants). These variables can possibly explain the differences in the results obtained (see Capek et al., 2008, p. 1239). The conclusion is particularly relevant to our study: the authors assumed that if the superior temporal cortex is not used to process auditory speech, it may be recruited to process visual speech, to a larger extent than in hearing participants in whom AV speech perception is dominant (see also MacSweeney et al., 2002; Capek et al., 2010 for related data).

Part of the activation induced by lipreading must be related to visual movement detection and to the perception of biological movement, especially in the inferior and posterior regions

of the temporal cortex (Zeki et al., 1991). But most of the activation in superior temporal regions is related to lipreading itself. Therefore, the observation of more activation related to lipreading in deaf individuals than in hearing people in the pSTS suggests the following interpretation. The pSTS is an AV integration site in hearing people, but cannot play this role in deaf individuals. The activation of pSTS could be sensitive to the dominant language modality. This multimodal region could have developed sensitivity to visual speech for deaf individuals, and to auditory speech, and secondary to visual speech for hearing ones. Given that our early CS-users participants did not hear during their early years, and have been intensively exposed to lipreading + manual cues in daily communication, one may expect to observe a strong activation of pSTS during CS perception.

### Neural Activity Related to Sign Language

Sign language is the preferred means of communication for most of deaf persons. Deaf persons use visible actions from the hands, the head and the trunk to communicate meanings using phonological, lexical and morpho-grammatical rules. The articulators are visible gestures, and language perception is in the visuo-spatial modality. SLs are adapted to the human's processing abilities in the visual modality, as are spoken languages in the auditory modality (Christiansen and Chater, 2016). For instance, signs take longer to be articulated, but the mean duration of utterances is similar in SL and in English for ASL-English bilinguals (Bellugi and Fischer, 1972).

Similarities between sign and spoken language processing have been abundantly demonstrated. Lesion-based, neuroimaging, and neurophysiological studies have provided strong evidence for the importance of left perisylvian regions during production and perception-comprehension of signed as well as spoken languages (Emmorey, 2001; Capek et al., 2008; MacSweeney et al., 2008; Corina et al., 2013). The left inferior frontal gyrus is involved in both sign and speech production while the left superior temporal gyrus and sulcus, in addition to the left inferior frontal gyrus, are involved in sign and spoken language perceptioncomprehension.

Differences related to the visual signal delivered by the articulatory gestures have been revealed between AV seen speech (lipreading) and SL perception (Emmorey, 2001; MacSweeney et al., 2008; Corina et al., 2013; Leonard et al., 2013). The kinematic characteristics of SL and AV speech are very different. There is more movement in the moving image of sign than speech. Different parts of the visual recognition system are sensitive to movements of particular body-parts (mouth specific vs. hand specific regions in inferior temporal cortex, see Pelphrey et al., 2005). The articulators in AV speech and SL thus have different timing, dynamic, and visibility, and their perception elicit different brain activity. SL perception induces greater activation than AV speech in movement processing regions of the posterior temporal gyri, bilaterally, while AV speech perception generates greater activation than SL in auditory processing regions in superior temporal cortices, including the planum temporale (Petitto et al., 2000; MacSweeney et al., 2002).

Of particular relevance for the present study is the fact that some signs require mouth movements in addition to manual movements. Some of these mouth movements allow distinguishing minimal pairs of signs in SL (Sutton-Spence and Woll, 1999; Capek et al., 2008). Activations corresponding to mouth movements, distinct from those related to hand movements have been found during SL perception (Capek et al., 2008). Compared to manual only signs, signs including mouth movement elicit more activation in middle and posterior portions of the superior and middle temporal gyri, and in the inferior and middle frontal gyri, bilaterally. The manual only signs elicited activation in the right occipito-temporal cortex, and the fusiform gyrus. As the moving hand adopting several distinct configurations around the face is a important articulator in CS (see **Figure 1**), one may expect to find more activation in the temporal posterior regions, and perhaps in the temporo-occipital posterior inferior regions in CS than in AV speech.

### Similarities and Differences between CS and SL

Cued Speech, such as cued American English, has similarities and differences with SL. Like SL, CS is conveyed in the visual modality and can be used for social communication, as evidenced by interactions within the family unit, and among "cuers" in social events. Similarly to SL, the use of voice is not needed in order to communicate in CS: "cuers" can achieve 100% speech perception when manual cues and mouth movements are presented, without sound (Nicholls and McGill, 1982). Finally, for SL as for CS, presentation of a single phonological parameter of the lexeme cannot, on its own, generate a lexical item (similar handshapes and hand locations occur with multiple lexemes in CS, as similar handshapes do in SL). Even more relevant to the present study is the fact that some signs requiring mouth actions in SL cannot be interpreted unambiguously at the lexical level when presented in isolation (see Capek et al., 2008). This is also the case in CS where the manual cues cannot, on their own, generate access to a lexical item. Accurate perception of CS thus necessarily involves the integration of manual and labial information in order to achieve a specific lexical representation. That is, for CS as well as for SL, individual sublexical features of articulation must be integrated to allow access to a specific lexical representation.

There are also differences between CS and SL. First, CS is not a language, but rather a visual mode of communication of spoken language. CS is isomorphic to speech: it is a visual representation of the syllables and phonemes of spoken language while SLs have their own phonology based on manual articulatory parameters of hand location, handshapes, movement, and palm orientation (Stokoe, 1960). Second, CS handshapes are produced at a much more rapid rate than SL. The production of handshapes at different locations around the face follows the rate of spoken speech, meaning that the CS receiver must decode a rapid succession of changing handshapes and hand placements, in a space located from the speaker's eyes to the throat (Attina et al., 2004, 2006). In order to formulate more precise predictions about activations related to CS perception, it may be useful to know

what has been revealed so far about the time course of mouth and hand movements in CS. This is the topic of the next section.

## What Do We Know about Perception and Production of CS?

Attina et al. (2004) were the first to examine the precise temporal organization of CS production of syllables, words and sentences. Natural production of CS is characterized by a temporal anticipation of manual gesture over mouth opening: the hand movement begins up to 240 ms before the acoustic onset of a CV syllable, and the target position corresponding to the vowel is reached during the mouth production of the consonant, well before the vowel lip target. At the receptive level, deaf CSusers anticipate the linguistic target on the basis of "reading" the manual gesture: perception of the hand gives the first input for the selection of the possible phonemes pronounced, and the lips follow with the solution. Deaf people seem to extract first phonological information when a manual cue is produced, reducing the potential number of words compatible with the lipread signal. Attina et al. (2004) data suggest that manual cues, as opposed to lipread information, can be the primary source of phonological information for deaf early CS users. Lipread information would then disambiguate the information provided by the manual cues (Attina et al., 2004, 2006; Troille et al., 2007; Troille, 2009).

### Predictions Concerning the Neural Activation for AV Speech versus CS Perception?

In this study, we compare the neural activations created by processing of spoken language produced and perceived through two different modalities: AV and CS. We control for language experience by testing only native users: NH participants with the AV material, and congenitally deaf participants with the CS material. We address three research questions. First, what are the similarities and differences between the processing of CS by early CS-users and the processing of AV speech by NH participants? This comparison is designed to explore common regions of activation for spoken language, independent of modality. Our second research question concerns the neural basis for integration of manual cues and lipread information. In CS, integration is mandatory and concerns two types of dynamic visible information, i.e., the movements of labial and manual articulators. In AV speech, integration concerns two modalities that are congruent in terms of their articulatory origins: the heard and seen results of movements of the oral articulators. We expected to find the pSTS as site of integration for AV speech, and we wanted to document brain regions critical for CS integration. Our third research question concerned the relative activation created by manual cues only and lipreading cues only compared to the activation created by the combined movements of lips and hands in CS. Some authors have suggested that the manual component of CS delivers more useful information than the lipread component to get access to the lexicon (Alegria et al., 1999; Alegria and Lechat, 2005; Attina et al., 2006; Troille, 2009; Bayard et al., 2014, 2015). We hypothesize that the cortical activations may reveal greater activation for unisensory manual than labial movements, but also indicate specific locus/loci for integration of manual and labial information, different from those reported for AV integration.

## MATERIALS AND METHODS

## Ethics Statement

All participants gave written informed consent to participate in this study, which was approved by the Ethics Committee of the ULB Erasme Hospital, and conducted in accordance with the Declaration of Helsinki (BMJ 1991; 302: 1194).

### Participants

Two groups of participants were recruited. The CS group consisted of 14 participants (3 males, 11 females), with a mean age of 25.0 years (age range = 18–33 years). All participants but one were congenitally profoundly deaf, with a binaural hearing loss > 90 dB (computed on 250, 500, 1000, and 2000 Hz) in their better ear. The remaining participant had a severe hearing loss (i.e., between 71 and 90 dB at the better ear). All deaf participants were equipped with hearing aids since they were between 6 months and 2 years of age, and none had a cochlear implant.

The NH group consisted of 15 normally hearing Frenchspeaking participants (six males, nine females), with no knowledge of CS. Their mean age was 25 years 2 months (age range = 20–37 years).

Since native language involves a different brain network than second languages learned later in life (Dehaene et al., 1997), only participants who were native French speakers were selected. A participant was considered a native language user if he/she had received consistent, age-appropriate speech stimulation from fluent users of French before the age of 3 years (Locke, 1997). Currently, this criterion for native CS user can only be fulfilled within the deaf community, since nearly all NH people with an experience in French CS learned it later in their life. Consequently, only neural activity from deaf CS participants who were exposed at an early (i.e., prelingual) age is an appropriate comparison for the patterns of neural activity observed in native French speaking hearing participants.

The deaf participants of the CS group in our study selfreported French as their native language in a questionnaire completed prior to enrolling in the study. They had been exposed to French CS, at home from their parents before the age of 3 years, and at school via teachers and or via transliterators from spoken French to French CS. Participants also reported that French CS was the language most commonly used during their childhood/adolescence, although most of them also learned SL informally during this period in contacts with deaf peers. The CS users reported that they still use CS often today, in daily communication with their family or other deaf persons. They also used oral French to communicate with NH individuals. The NH transliterator gave qualitative feedback about deaf participants' CS comprehension: all of them could easily understand normal cued French conversation and were good at lipreading. All CS

participants had finished secondary school (high school), and 50% (n = 7) had either completed a post-graduate program or were in one at the time of the testing.

All deaf and hearing participants were right handed, with no known neurological or behavioral disorder.

### Experimental fMRI Design (Procedure)

In an event-related paradigm, a baseline condition and three experimental conditions were presented randomly intermixed to each group of participants. For deaf participants, the experimental conditions consisted of stimuli presented in CS (lipreading + manual cues), in lipreading alone, or in handshapes alone. For NH participants, the experimental conditions consisted of stimuli presented in AV, in auditory speech alone, or in lipreading alone. For both groups, the baseline condition consisted of a motionless face.

### Experimental Task and Conditions

In the experimental conditions, all participants watched videos of a hearing female French speaker (who was a professional CS transliterator) saying a randomized list of 45 bisyllabic frequent words (New et al., 2001) (see Annex for the full list of words used in this experiment) under six conditions (three for the NH group and three for the CS group; see below). All videos were recorded indoor (in a room). The female speaker was so positioned that her back was in contact with a white wall. She looked straight to the camera. Her full face and torso were shown in all videos in order to present a naturalistic display of CS.

In the NH group, participants performed the detection task under three different experimental conditions: (1) the speech AV condition, in which the speaker pronounced the word while the same word was presented orally through MRcompatible earphones, (2) the auditory alone (A) condition, in which the word was presented aurally but the speaker's face remained motionless, and (3) the lipreading alone (L) condition, in which the speaker pronounced the word but no auditory information was provided. In the CS group, participants performed also the detection task under three different experimental conditions: (1) the Cued Speech labial + manual (CSLM) condition, in which both lip movements and manual cues were visible, (2) the CS manual (CSM) condition, in which only hand movements were provided, but the speaker's face remained still (no lips movements), and (3) the CS labial (CSL) condition, equivalent to lipreading, in which the speaker pronounced the word, but no manual movements were provided.

For each condition, one video corresponded to one word. Altogether, there were 270 videos (i.e., 45 for each condition). Importantly, there was no manipulation of videos from one condition to another. For example, for the CSM condition, we did not create CS manual videos removing the lips information from the CS labial + manual videos. Instead, we recorded 45 videos in which the speaker had to produce the 45 words using only hand movements. Seemingly, in the CSL condition (lipreading condition), we recorded 45 videos in which the speaker pronounced the word without hand movements. The dynamic of both, hand movements produced in CSM videos and lip movements produced in CSL videos, have a great likeness when compared to the CSLM condition.

The same words were used in all conditions but in different orders between conditions. In each of the three conditions, the target word "papa" (i.e., daddy) was included in the list of stimuli. In order to ensure focused attention to the stimuli, participants were asked to press a button when they detected "papa" (i.e., daddy).

### Control Task – Baseline Condition

In order to control for attention, motor response parameters, and perception of a face and a body, a second task was designed. In this control task, all participants saw a still picture of the speaker's full face and torso on screen, as in the experimental task (45 trials). They had to press a button when a small red circle was superimposed on the speaker's chin. The group conditions are summarized in **Table 1**.

### Procedure

Before the presentation of each video, a white cross was displayed in the middle of the screen, indicating that the stimulus material (i.e., the video) would appear randomly within 1900–2800 ms. Then, the video containing the stimuli was displayed. The length of each video was also randomly assigned within 1900 and 2800 ms but, for all six conditions, they had the same mean of 2370 ms and the same standard deviation of 220 ms. In order to do this, we recorded each video with a length of 3 s, but leaving always a silent space at the end of at least 1200 ms in which the speaker's face and torso remained present but still. This allowed us to adjust the required length for each video (by cutting the necessary time needed for each video). After the presentation of each video stimulus, a question mark appeared on the screen for 2000 ms, reminding the participant to press a button on the MR-compatible keypad (fORP; Current Designs) only if the participant had perceived the target word (i.e., "papa"), in one of the experimental conditions, or the target circle in the still control task. The target word or red circle appeared eight times within each condition.

Presentation of the stimuli was randomized across the three experimental and the control conditions. Presentation of the stimuli and recording of the participant's responses were realized using Cogent Graphics (running on MatlabTM 6.1) developed by John Romaya at the LON at the Wellcome Department of Imaging Neuroscience. All participants practiced the tasks outside of the scanner beforehand to minimize task-learning



effects. fMRI data acquisition had an approximate duration of 30 min.

### fMRI Data Acquisition

fpsyg-08-00426 March 28, 2017 Time: 16:37 # 8

Data were acquired on a Philips Achieva 3-T (Philips Medical Systems, Best, the Netherlands) scanner using a T2<sup>∗</sup> sensitive gradient echo (EPI) sequence (TR = 2130 ms, TE = , 40 ms, FA 90◦ , SENSE acceleration factor 2.5, matrix size 64 × 64 × 32; voxel size: 3.06 mm × 3.06 mm × 3 mm). Thirty-two contiguous 3-mm thick transverse slices were acquired, covering the whole brain. An approximate number of 840 EPI volumes per participant was acquired across the four conditions. Additionally, an anatomical image was obtained using a T1-weigthed sagittal 3D MP-RAGE sequence (TR 1960 ms, TE 4.60 ms, TI 1040 ms, flip angle 8◦ , FOV 250 mm × 250 mm, matrix size 320 × 320 × 160, interpolated voxel size: 0.78 × 0.78 × 1.0 mm). The MR scanner was equipped with the Quasar imaging gradients and an eight channel SENSE head coil.

### fMRI Data Analysis

Functional MRI data were pre-processed and analyzed using Statistical Parametric Mapping (SPM8) software (Wellcome Department of Cognitive Neurology, London, UK) implemented in MATLAB 7.8 (Mathworks Inc., Sherbom, MA, USA). The first five functional EPI volumes were discarded to avoid magnetic saturation effects. The remaining EPI images were realigned (Woods et al., 1992), spatially normalized into standard stereotactic MNI space (Woods et al., 1999), and smoothed spatially at 8 mm full-width half-maximum (FWHM) (Worsley et al., 1997).

Data were analyzed using a mixed-random effects (RFX) model aimed at showing a stereotypical effect in the population from which the subjects were drawn (Penny and Holmes, 2003). For each subject, a first-level, intra-individual analysis aimed at modeling data to partition observed neurophysiological responses into components of interest, confounds, and errors, using a general linear model (Friston, 2003). The regressors of interest were built using stick functions corresponding to the four conditions (45 stimuli each). These regressors were secondarily convolved with a canonical hemodynamic response function. Movement parameters derived from realignment of the functional volumes (translations in x, y, and z directions and rotations around x, y, and z axes) were included as covariates of no interest in the design matrix. High-pass filtering was implemented in the matrix design using a cut-off period of 128 s to remove low drift frequencies from the time series. Serial correlations were estimated with a restricted maximum likelihood (ReML) algorithm using an intrinsic autoregressive model during parameter estimation. Effects of interests were then tested by linear contrasts (e.g., CSLM – still, Speech AV – still, CSLM vs. Speech AV, etc.), generating statistical parametric maps [SPM(T)].

Individual summary statistic images were further spatially smoothed (6 mm FWHM Gaussian kernel) and entered in a second-level analysis in which participants were treated as a RFX. At this second level, one-sample t-tests were used to assess the contrasts between two conditions in the CS and NH groups separately. Two-sample t-tests were used for a direct comparison of the same contrasts between CS and NH participants. Additionally, conjunction null analyses (Price and Friston, 1997; Friston et al., 2005) were used to identify the brain areas commonly activated between conditions or between CS and NH groups in contrasts of interest. This method tests whether individual effects are jointly significant, under the null hypothesis that all-but-one of the effects are significant, hence an "AND" over several individual hypotheses. ReML estimates of variance components were used to allow possible departure from the sphericity assumptions in RFX conjunction analyses (Friston et al., 2002).

In order to explore the role of MT/V5 region to the experimental conditions (see below, p 21), psychophysiological interaction (PPI) analyses (Friston et al., 1997; Gitelman et al., 2003) were computed to test the hypothesis that experimental conditions modulate functional connectivity between neural activity in left or right MT/V5 areas and other brain regions involved in CS and AV processing. First, the time course of activity within MT/V5 area was extracted separately at left and right MT/V5 coordinates for each individual. To do so, the CSLM (respectively speech AV) vs. still contrast effect (corresponding to the summary statistic images entered in the RFX analysis) was computed at the individual level, and the local maximum of activation determined in a volume within the probabilistic map of MT/V5, as identified in a previous cytoarchitectonic analysis (Malikovic et al., 2007). This peak value was selected, unless it was identified outside of the brain structure of interest upon visual inspection of the individual normalized anatomical T1 image and verification of localization in SPM toolbox Anatomy atlas (Eickhoff et al., 2005), in which case the maximum value that fitted the anatomical location was selected. Second, a new linear model was generated for each individual level, using three regressors. One regressor represented the task condition (CSLM [respectively speech AV] vs. still). The second regressor was the average activity in a sphere (radius 4 mm) centered on the coordinate of the participant-specific peak value. The third regressor represented the interaction of interest between the first (psychological) and the second (physiological) regressor. To build this regressor, the underlying neuronal activity was first estimated by a parametric empirical Bayes formulation, combined with the psychological factor (i.e., task condition) and subsequently convolved with the hemodynamic response function (Gitelman et al., 2003). The design matrix also included the movement parameters caused by subject movements. A significant PPI indicated a change in the regression coefficients between any reported brain area and the reference region related to the task condition. Individual summary statistic images obtained at the first level (fixed effects) analysis were then spatially smoothed (6 mm FWHM Gaussian kernel) and entered into a second-level (RFXs) analysis using one sample t-tests to test for condition-specific effects within CS or NH group separately.

Additionally, we performed an integration analysis, in which we searched for brain areas more strongly activated in a biarticulatory condition (e.g., CSLM) than in the mono-articulatory conditions (e.g., CSM and CSL), during which only one of

the two components was manipulated. This criterion of superadditivity (Beauchamp, 2005) was implemented as a t-contrast between the bi-articulatory condition and the sum of the two mono-articulatory conditions [e.g., CSLM > (CSM + CSL)]. For the CS group, the null hypothesis was thus a weighted contrast [2 −1 −1] such as [2<sup>∗</sup> (mean activity during CSLM)] vs. [mean activity during CSM + mean activity during CSL]. For the NH group, the null hypothesis was [2<sup>∗</sup> (mean activity during speech AV)] vs. [mean activity during speech auditory + mean activity during speech visual]. This comparison was masked inclusively by contrasts computed in the mono-articulatory conditions (e.g., CSM > still and CSL > still). This procedure indicated regions mainly devoted to integration while controlling for movement-related activity from manual cues and lipreading. This latter aspect is important since lip movements may have influenced activation results in simple subtraction analyses (e.g., CSLM-still).

In all of the analyses presented above, the resulting set of voxel values for each contrast constituted a map of the t-statistic [SPM(T)], at p < 0.001 threshold (uncorrected for multiple comparisons). Statistical inferences were then obtained after corrections at the voxel level using Gaussian random field theory (Worsley et al., 1996), p corr < 0.05 corrected for multiple comparisons in the whole brain volume, unless otherwise specified.

Anatomical localization of local maxima and clusters was assessed with reference to the MNI space, using standard anatomical atlases (Collins et al., 2002). MT/V5 is a functional area that was located using a probabilistic map in the Anatomy SPM Toolbox (Eickhoff et al., 2005; Malikovic et al., 2007).

### RESULTS

### Behavioral Data

The target detection of the word "papa" was accurate in both groups for all experimental conditions. In the CS group the mean performance was 100%, 99.1% (SD = 3.34) and 96.4% (SD = 5.8) in the CSLM, CSM and CSL conditions respectively. In the NH group, the mean performance was 100%, 95% (SD = 11.3) and 91% (SD = 19.5) in the speech AV, speech auditory and speech visual conditions respectively. The mean performance for the detection of the small circle in the control still task was 91% (SD = 12.9) in the CS group and 95% (SD = 11.7) in the NH group. The mean global performance was 98.5% in the deaf-CS group and 95.3% in the NH group. There was no significant difference between groups (t = 1.07; n.s.).

### fMRI Data

### Brain Activation during CS Processing: CSLM-Still (Deaf)

In CS participants, CSLM perception elicited higher bloodoxygen level dependent (BOLD) responses than the control (still) condition bilaterally in the occipito-temporal junction including the MT/V5 area, and in the middle and superior temporal lobe with a more extended activation in the left than in the right hemisphere. The activation in the superior temporal lobe did not include the primary auditory cortex (Heschl's gyrus). Other areas activated within this contrast (CSLM-still) were the left inferior parietal lobe, the premotor area, and the inferior frontal gyrus, pars triangularis (BA 45).

### Brain Activation during Speech AV Processing: Speech AV-Still (Hearing)

In NH participants, speech AV perception elicited greater BOLD responses than the control condition in bilateral superior and middle temporal gyri, including primary auditory cortex (Heschl's gyrus), as well as in bilateral inferior parietal lobe and left inferior frontal gyrus, pars triangularis (**Figure 2** and **Table 2**).

### Overlapping Brain Activations: A Conjunction Analysis of CSLM and Speech AV Processing

A conjunction analysis revealed a common activation pattern for AV speech (NH group) and CSLM (CS group) bilaterally in the middle and superior temporal gyrus (excluding Heschl's gyrus) and in the left inferior frontal gyrus pars triangularis (**Figure 3** and **Table 3**), indicating that, as in speech AV perception, CSLM perception is associated with neural activation in the secondary auditory cortices despite a complete absence of auditory input.

### Differences in Brain Activation between CSLM and Speech AV Word Processing: A Two Sample t-test Analysis

In comparing the speech AV and CSLM networks, we found greater BOLD responses in the CSLM (CS group) than in the speech AV (NH group) condition bilaterally in the occipitotemporal junction (MT/V5) and neighboring structures, with a greater extension in volume in the left hemisphere. Greater activations for speech AV than CSLM was found in the right middle temporal gyrus and the left superior temporal gyrus including Heschl's gyrus (**Figure 3** and **Table 3**).

### Is MT/V5 Involved in Speech Processing? Psychophysiological Interaction Analysis (PPI) in MT/V5 for CSLM and Speech AV Word Processing

The results shown above indicate that besides large commonalities, the neural basis of speech perception in CSLM in CS participants is shifted toward posterior regions of the brain as compared to speech AV in NH participants, with activation peaks in the occipito-temporal junction and MT/V5. Admittedly, activation in MT/V5 is associated with visual motion processing (Zeki et al., 1991; Tootell et al., 1995; Zacks et al., 2001), which was present in the CSLM but not in the control still condition. To test further whether activation in MT/V5 was associated specifically with processing of CSLM, a PPI analysis was performed with MT/V5 (left and right) as source area and CSLM vs. still condition as modulatory parameters. Results revealed increased connectivity during CSLM perception as compared to the still condition between the left MT/V5 and temporal lobes in both hemispheres, as well as with the posterior inferior temporal and fusiform gyrus bilaterally (**Figure 4** and **Table 4**).

Increased connectivity between left MT/V5 and temporal regions typically activated for speech processing suggests that

FIGURE 2 | Word perception in CSLM and speech AV. Higher activations during CSLM and speech AV perception of words as compared to the still control condition. All activations are displayed at p < 0.05 whole brain corrected, superimposed on the SPM MRI template. CSLM, Cued Speech (oral + manual) in deaf early users; AV, audiovisual speech in hearing speakers.

#### TABLE 2 | Brain activation during CSLM and speech AV word processing, related to Figure 2.


Brain regions more activated in the word perception task (speech AV or CSLM) than in the still control task. Only peak activations for distinct anatomical structures are reported within each cluster. L, Left; R, Right; BA, Brodmann Area; x, y, z are standard stereotaxic coordinates in the MNI (Montreal Neurological Institute) system. All activations reported at p < 0.05 whole brain corrected. CSLM, Cued Speech (oral + manual). AV, audiovisualin hearing speakers.

left MT/V5 interacts with regions playing a role during CSLM perception. Right MT/V5 was associated more strongly with CSLM in more posterior brain regions, including the middle occipital and fusiform gyri bilaterally, the left MT/V5 and superior parietal lobe, the right inferior occipital and posterior middle temporal gyri, and the right inferior parietal lobe.

Similar PPI analyses conducted in the NH group revealed a higher connectivity during speech AV than still condition between the left MT/V5 and a small left middle temporal region (**Figure 4** and **Table 4**).

### Where Does Integration of Manual Cues and Lipreading in CSLM Take Place? Super-Additivity Analysis of Speech AV and CSLM Word Processing

We performed analyses to identify the brain regions responding more strongly to bi-articulation (CSLM in the CS group; speech AV in the NH group) than to the sum of the mono-articulation stimulus presentation (e.g., CSM and CSL separately). The integration analysis in the NH group [speech AV – (speech auditory + speech visual)] revealed greater activity in the bilateral posterior superior and middle temporal gyri (**Figure 5** and **Table 5**). This result is in agreement with findings from previous studies of spoken language that have shown that integration between lipreading and auditory speech takes place in the pSTS (Callan et al., 2004; Szycik et al., 2007; Beauchamp et al., 2010, but see Hocking and Price, 2008). It is hypothesized that the particular sensitivity of this area for multimodal speech integration arises from a correlation between the dynamic aspects of seen lipreading and heard auditory speech (Calvert and Campbell, 2003).

In the CS group, the integration analysis [i.e., CSLM ≥ (CSM + CSL)] revealed a supplementary activation in the left occipito-temporal junction only, around the MT/V5 area (**Figure 6** and **Table 6**). This is the same location as the activation peak in the CSLM ≥ still condition (see **Figure 2** and **Table 2**). This finding further suggests that the left occipito-temporal junction, including MT/V5, supports the integration of lipread and manual speech features, above a mere visual processing of motion input.

### Importance of Labial and Manual Information? Comparison of Two Conjunction Analyses Within CS: CSM – CSLM and CSL – CSLM

Finally, we wondered whether one source of information is more important in CSLM perception for deaf CS users, as for AV speech in hearing people. Indeed, the primary processing of information in speech AV decoding comes from the auditory source, whereas the contribution of visual speech to the final percept directly depends on the ambiguity of the auditory information (Summerfield, 1987). In CSLM production,

#### TABLE 3 | Commonalities (conjunction analysis) and differences (two sample t-test) between CSLM (CS group) and speech audiovisual (NH group) processing, related to Figure 3.


<sup>1</sup>Regions commonly activated in both conditions: [speech AV – still control condition] and [CSLM – still control condition] perception of words. <sup>2</sup>Brain regions more activated during CSLM perception of words [CSLM vs. still control condition] compared to speech AV perception of words [speech AV – still control condition]. <sup>3</sup>Brain regions more activated during speech AV perception of words [speech AV vs. still control condition] compared to CSLM perception of words [CSLM – still control condition]. L, Left; R, Right; x, y, z are standard stereotaxic coordinates in the MNI (Montreal Neurological Institute) system. Activations of speech AV = CSLM and speech AV > CSLM are reported at p < 0.05 whole brain corrected. Activations of CSLM > speech AV are uncorrected for multiple comparisons; p < 0.001. Speech AV = audiovisual speech in hearing speakers; CSLM, Cued Speech (oral + manual); CSM, Cued Speech (manual only); CSL, Cued Speech (oral only).

it has been demonstrated that the manual cues movement anticipates the lip movements, and that this anticipation is used by the CS-perceivers to reduce the uncertainty about the phonemes pronounced (Attina et al., 2004, 2006; Troille, 2009). This suggests the possibility that manual cues are dominant in CSLM processing, while lipreading would have a (secondary) role associated to the disambiguation of the percept.

A conjunction analysis between CSLM and the CSM conditions was performed and revealed widely overlapping

#### TABLE 4 | Psychophysiological Interactions from left and right MT/V5 in NH and CS groups, related to Figure 4.


L, Left; R, Right; BA, Brodmann area; x, y, z are standard stereotaxic coordinates in the MNI (Montreal Neurological Institute) system. All activations reported at p < 0.05 whole brain corrected. AV, audio-visual; CS, Cued Speech.

neural activity, especially in the bilateral occipito-temporal junction, the left superior and middle temporal and inferior parietal lobes (**Figure 7** and **Table 7**).

The number of overlapping voxels in this conjunction represented 94% of the voxels activated in the CSLM network. Conversely, the conjunction analysis between CSLM and CSL conditions revealed a low overlap with only 25% of the CSLM network activated (mostly in the left middle temporal gyrus). Although this observation of higher overlap with the CSLM network was made at the descriptive level only, these conclusions are reinforced by subtraction analyses that failed to reveal significant differences between CSLM and CSM conditions, whereas extended, higher activation in the CSLM than in the CSL condition was found in the bilateral occipito-temporal junction. The great overlapping activity of CSLM and CSM might be linked to the common kinematic signature created by the movements in hands. However, our results may also suggest that the processing of manual cues might play a primary (dominant) role in CSLM decoding, while lipreading may play a second role. This makes CSLM even more akin to speech AV processing : similar analyses computed in the NH group revealed a high similarity between processing of speech AV and speech auditory conditions (68% overlap), but not between speech AV and lipreading (10% overlap), and subtractions showed larger differences between speech AV and lipreading conditions (**Figure 8** and **Table 8**).

### DISCUSSION

The current study compared the fMRI activations of CS processing by deaf participants with that of speech AV processing by hearing participants, with identical spoken words, presented either in CS (i.e., lipreading and manual cues) for deaf CS participants (CS group) or audio-visually for normally hearing participants (NH group). Both groups of participants had French as their native language. Our study is the first neuroimaging investigation of deaf people who had full sensory access to the phonemic and syllabic content of spoken speech, through the visual modality.

compared to the still control condition (top) and speech visual perception of words (lipreading) compared to the still control condition (middle) in AV hearing French speakers. Integration analysis of speech auditory and speech visual (lipreading) in speech AV: higher activations during speech AV [speech AV – still] compared to the sum of [speech A – still] and [speech visual – still] (bottom), masked by contrasts of the uni-articulatory conditions (speech A > still and speech visual > still) in speech AV hearing French speakers. AV, audiovisual speech in hearing speakers; A, auditory; p < 0.05 whole brain corrected.



<sup>1</sup>Brain regions more activated during speech auditory (only) perception of words compared to still control condition. <sup>2</sup>Brain regions more activated during speech visual (lipreading) perception of words compared to still control condition in AV hearing speakers. <sup>3</sup>Brain regions activated during AV integration in hearing speakers: [speech AV > (speech A + speech visual)] (speech A > still) (speech visual > still). Activations of speech A-still and speech visual-still are reported at p < 0.05 whole brain corrected. Activations of speech AV integration are uncorrected for multiple comparisons; p < 0.001. AV, audio-visual; A, auditory; LR, lipreading. L, Left; R, Right; BA, Brodmann area; x, y, z are standard stereotaxic coordinates in the MNI (Montreal Neurological Institute) system.

Our first research question concerned the similarities and differences between the processing of CSLM by early CSusers and the processing of speech AV speech by hearing participants. Our results show that the perception of oral language delivered through CSLM activates secondary auditory cortices in the MTG and STG together with IPL. This activation represents most of the overlapping activation regions found during CSLM and AV speech processing (see **Figure 3** and **Table 3**). In AV speech processing, MTG and STG are parts of a structure in the auditory ventral stream (Hickok and Poeppel, 2004) that serves as an interface between sound based representations of speech (Binder et al., 2000), visual speech (see Bernstein and Liebenthal, 2014 for a review) and widely distributed conceptual representations (Damasio, 1992). However, in CSLM, activity in MTG and STG cannot be linked to any kind of acoustic processing. Since earlier studies showed a very accurate CS perception and comprehension in deaf early CS users (Leybaert and Charlier, 1996; Alegria et al., 1999; Leybaert and D'Hondt, 2003; Aparicio et al., 2012; Colin et al., 2013), activation of the secondary auditory cortex might be linked to an interface between phonetic visual speech codes conveyed by lip movements/manual cues and conceptual representations.

Differences in brain activation between CSLM and AV speech showed that the neural basis of speech perception in CSLM is shifted toward posterior regions of the brain (parieto-occipital regions) as compared to AV. Some studies have reported more posterior language processing in less experienced language users, deaf (Mayberry et al., 2011) and not deaf (Brown et al., 2005; Gaffrey et al., 2007). However, our CS participants were exposed to visual representation of spoken language as their native language. Our deaf participants had all a good level of comprehension of CS (see participants). This suggests that activations presented here are not the consequence of poor language processing.

Interestingly, the common component of the speech AV and CSLM brain networks is located in the posterior portion of the STG. In contrast, the anterior part of STG (including the auditory primary cortex) is only activated in AV speech processing (see **Figure 3** and **Table 3**). We surmise here a functional subdivision of the superior temporal lobe for speech processing, with the anterior part supporting speech processing based on sounds (i.e., phonetic codes), and the posterior portion involved in speech processing integrating information provided by different modalities (visual, auditory, and even somatosensory). Indeed, previous studies have shown that the multisensory posterior

#### TABLE 6 | Brain activation during processing of CSM, CSL (lipreading) and integration of CSLM in deaf-CS group, related to Figure 6.


<sup>1</sup>Brain regions more activated during CSM perception of words compared to still control condition. <sup>2</sup>Brain regions more activated during CSL perception of words compared to still control condition. <sup>3</sup>Brain regions activated during integration of CSLM: [CSLM > (CSM + CSL)] (CSM > still)(CSL > still). L, Left; R, Right; BA, Brodmann area; x, y, z are standard stereotaxic coordinates in the MNI (Montreal Neurological Institute) system. Activations in CSM– still and CSL – still are reported at p < 0.05 whole brain corrected. Activations of CSLM integration are uncorrected for multiple comparisons; p < 0.001. CSLM, Cued Speech (oral + manual); CSM, Cued Speech (manual only); CSL, Cued Speech (oral only).

STG can be involved either in acoustic-phonetic processing (Hickok and Poeppel, 2007), in phonological mediation for lipreading (Paulesu et al., 2003) or in simultaneous auditorytactile stimulation (Beauchamp et al., 2008). Moreover, activation of STG also appeared during SL processing in deaf people suggesting that neural activity in posterior STG is linked to linguistic processing beyond auditory representation (Petitto et al., 2000; MacSweeney et al., 2008; Xu et al., 2009). The present findings also indicate that the multisensory posterior STG could be activated by the processing of language cues within a single modality (here: visual), at least when these visual cues provide meaning for the individual as it was the case with the words presented in this experiment. Given previous findings showing that the posterior STG is activated when participants observe symbolic gestures (Xu et al., 2009), our data support the hypothesis that this region responds to meaningful manuallipread stimuli in deaf CS-users as well as vocal-auditory stimuli in hearing AV participants. Our results thus provide converging evidence with these previous neuroimaging studies, but also add the new information about involvement of posterior STG in the processing of CSLM. Finally, there is a greater activation of occipital gyrus in CSLM compared to AV. This might reflect the greater kinematic information supplied by the hands and lip movement in CS.

It should be noted that activation in posterior STG during CSLM processing could also be explained by the fact that our CS participants were all deaf. These two interpretations are not easy to disentangle. Olulade et al. (2014) showed that cortical plasticity resulting from deafness depends on language experience for auditory and visual areas. If normally hearing participants who are CS-users (parents, teachers, educators of deaf CS-users) also showed activation of posterior STS, the hypothesis that multi-signal integration inside the visual modality occurs in this region of the brain would be reinforced.

Our second research question concerned the neural basis of integration of manual cues and lipread information. In CSLM, the integration is mandatory and concerns two types of dynamic information in the visual modality (i.e., visible movement of the labial and manual articulators). In contrast, in AV speech, integration concerns information in two different modalities: auditory and visual. Our integration analysis suggests that left MT/V5 plays a role in the integration of lipread and manual CS information. Note that this region has been already linked to speech processing (Jones and Callan, 2003; Sekiyama et al., 2003) and to processing of visual movements in SL (Capek et al., 2008). In addition, it has been shown that use of a visuospatial language like SL impacts the recruitment of the

during CSLM perception of words [CSLM – still] compared to CSL (lipreading) perception of words [CSL – still]. CSLM, Cued Speech (oral + manual); CSM, Cued Speech (manual only); CSL, Cued Speech (oral only); p < 0.05 whole brain corrected.

MT/V5 during motion visual processing. Deaf and hearing signers show a greater activation of the left MT/V5 than the right, while hearing non-signers show the opposite pattern (Bavelier et al., 2001; Bavelier and Neville, 2002), indicating that lateralization of MT/V5 is sensitive to language experience. Given that speech input in CS is primarily perceived through visual occipital networks, one may surmise that integration in the MT/V5 area is an early step in CSLM processing. This conception seems logical considering that manual cues and lips cues need to be processed before unambiguous decoding of speech (Alegria and Lechat, 2005). Importantly, our integration analysis, unlike the other analyses (e.g., subtractions) conducted in this study, controls for perception of movements related to CS production, since the comparison is made between CSLM and the sum of manual cues and lipreading. However, we cannot firmly conclude that integration occurring in MT/V5 is linguistic because some (or even all) of this integration might be non-linguistic, or even attentional (O'Craven et al., 1997). To support the hypothesis of a linguistic integration in MT/V5, future studies should compare these activation patterns with

#### TABLE 7 | Commonalities and differences between conditions within deaf CS group<sup>∗</sup> , related to Figure 7.


<sup>1</sup>Regions commonly activated in CSLM [CSLM – still] and CSM [CSM– still] perception of words. <sup>2</sup>Regions commonly activated in CSLM [CSLM – still] and CSL (lipreading) [CSL – still] perception of words. <sup>3</sup>Brain regions more activated during CSLM perception of words [CSLM– still] compared to CSL perception of words [CSL– still]. L, Left; R, Right; BA, Brodmann area; x, y, z are standard stereotaxic coordinates in the MNI (Montreal Neurological Institute) system. All activations reported at p < 0.05 whole brain corrected. CSLM, Cued Speech (oral + manual); CSM, Cued Speech (manual only); CSL, Cued Speech (oral only); LR, lipreading. <sup>∗</sup>There is no cluster significantly more activated when comparing CSLM to CSM in either way (i.e., CSLM – CSM or CSM-CSLM).

those coming from a control group that does not use or understand CS, presented with the same set of stimuli. In relation to the role of MT/V5 during CSLM perception, our PPI analyses yielded increased connectivity between left MT/V5 and speech processing regions located in the left temporal lobe. This suggests a cooperative activity between these areas in the treatment of CS components.

In addition, PPI analysis showed a functional interhemispheric differentiation in CS users: left MT/V5 interacts with left temporal lobes linked to speech processing whereas right MT/V5 interacts mostly with occipital areas, including left MT/V5, linked to visual perception and motion processing.

Our third research question concerned the relative activation created by CSM and by CSL compared to the activation created by CSLM. If the manual cues play a leading role in the processing of CSLM, one could expect a greater overlap between the activation created by the CSM and CS conditions, compared to a lower overlap between the activation created by CSL and CSLM. The conjunction analyses between CSLM and CSM conditions, on the one hand, and CSLM and CSL conditions on the other hand may be associated to a dominant role for the manual cues. Indeed, the almost complete overlap between CSLM and CSM networks is compatible with the view that the manual cues first provide phonological information, and lipreading intervenes subsequently in order to further disambiguate the linguistic message. This conclusion is in good agreement with three types of behavioral data. First, temporal analyses of CS production have shown that manual cues are produced temporally in advance to the lips (Attina et al., 2004, 2006; Troille, 2009). Second, when lipreading and manual cues are incongruent (e.g., pronouncing with the lips the phoneme /v/ with handshape 1 coding /d/, /p/ /j/ phonemes), most of the answers from the perceiver are related to the manual cues and not to lipreading, especially when the participant is an early CS user (Alegria and Lechat, 2005; Bayard et al., 2014, 2015). Third, deaf people who are early CS-users often succeed in daily natural communication with other CS users by producing manual cues alone, without lipreading. In this case, the full meaning of the message would be completed by the context (Weill, 2011). While the set of manual cues was initially originally created with the aim of disambiguating lipreading (Cornett, 1967), the present data support the hypothesis that it is lipreading which disambiguates manual cues, thus presenting a topsy-turvy vision of CS (Attina et al., 2004). One possible reason for this phenomenon is that in speech processing the perceiver takes a "preference" for decoding from those elements that are perceptually most distinguishable (i.e., the manual cues in CS for deaf participants or audio in AV for hearing participants). In case of deaf CS perceivers, it must also kept in mind that manual CS cues are executed and visually available to the perceiver a very short time before lip movements (Attina et al., 2004). At this step one could not also exclude that prior hand cues may reduce ambiguity before mouth shapes processing.

Interestingly, manual cues were associated with increased activity in left superior and middle temporal gyrus (**Figure 6** and **Table 6**), a brain zone typically activated during the processing

of natural languages like spoken languages and signed languages. However, manual cues are completely artificial gestures, not resulting from any evolutionary process. This suggests that invented manual gestures that convey linguistic information may become processed in the same areas as other articulatory gestures previously integrated in the human language through the natural evolutionary process of communication.

These findings should be investigated in further studies to better understand the degree of dependency of language processing on speech features. For example, in order to better determine the brain areas subtending speech processing, one could conduct a similar study of speech perception, which would use a control condition containing pseudospeech movements through manual cues and lips, instead of a still control

TABLE 8 | Commonalities and differences between conditions within NH group<sup>∗</sup> , related to Figure 8.


<sup>1</sup>Regions commonly activated in speech AV [speech AV – still] and auditory [speech A– still] perception of words. <sup>2</sup>Regions commonly activated in speech AV [speech AV vs. still control condition] and speech visual (lipreading) [speech visual – still] perception of words. <sup>3</sup>Brain regions more activated during speech AV perception of words [speech AV– still] compared to speech visual perception of words [speech visual – still]. L, Left; R, Right; x, y, z are standard stereotaxic coordinates in the MNI (Montreal Neurological Institute) system. All activations reported at p < 0.05 whole brain corrected. P < 0.05 whole brain corrected. AV, audio-visual; A, auditory; <sup>∗</sup>There is no cluster significantly more activated when comparing speech AV to speech A (auditory alone) in either way (i.e., speech AV-speech A or speech A-speech AV).

condition. The pseudospeech would be phonologically plausible but meaningless. This condition would enable us to dissociate brain areas linked to phonological processing from those linked to lexical processing in visual CS. Another interesting study would be to investigate the neural correlates of incongruent lip and manual cue movements (for example, a mouthed syllable /va/ accompanied by handshape 1 [/p, d, j/]), i.e., a McGurk-like effect experiment (Alegria and Lechat, 2005; Bayard et al., 2014, 2015). In CS, this would increase our understanding of the integration of visual speech features in deaf participants. Finally, we are interested in examining how manual cues are integrated with AV speech in deaf people fitted with a cochlear implant (Bayard et al., 2014, 2015). As mentioned, the articulatory movement of the hand precedes the mouth opening and the emission of sound and may therefore predict aspects of the lipread and the auditory signals, especially when the AV signal is ambiguous (e.g., visual /k/). The amount and nature of visual information extracted from the hand may initiate the speech-processing system, in which an abstract representation is activated through visual inputs, up to the point of explicitly registering auditory input. If this speculation appears to be correct, it means that processing of CS gestures could help deaf children fitted with a cochlear implant to discriminate, identify or interpret the new arriving sounds. Integration across modalities would allow the STG to help individuals with cochlear implants in discrimination, identification or interpreting the ambiguous auditory cues delivered by the cochlear implant.

At a more general level, our data about the neural processing of CS increase our general knowledge on how do deaf native users of CS process visual speech information. Compared to normally hearing individuals, deaf native users of spoken language have fewer areas of anatomical differences than do deaf native users of ASL (Olulade et al., 2014). The neural processes involved in CS and in AV speech seem, to a certain extent, also similar.

### CONCLUSION

In this article, we report results from the first neuroimaging study of CS processing, a mode of communication in which the syllables and phonemes of a spoken language are conveyed solely through the visual modality in the absence of either speech or hearing. First, we found that activation patterns in the secondary auditory cortex (i.e., temporal lobes) in visual CS perception and speech AV perception confirm the existence of a common language brain system for spoken languages, regardless of the sensory input modality. Second, our PPI and integration analyses suggest that MT/V5, a region classically associated with visual motion processing, exerts an active influence on the integration of hands and mouth. However, based on our analysis, we cannot conclude that this integration is merely linguistic. Finally, findings from our study suggest that the manual cues may dominate in the speech perception of skilled CS.

### AUTHOR CONTRIBUTIONS

MA, PP, and JL worked in the conception of the research and the redaction of the manuscript. BC has collaborated in the creation of the stimulus material. DB and MK have collaborated for the MRI data acquisition

### ACKNOWLEDGMENTS

We are grateful to all of our deaf and hearing participants for their enthusiastic collaboration and willingness to come to Brussels. We thank the staff of medical imaging of the Hospital Erasme in Brussels for their help. We want to thank Carol LaSasso (Professor in the Department of Hearing, Speech

and Language Sciences at Gallaudet University in Washington D.C) for assisting us with the English translation. This research was funded by the Fonds National de la Recherche Scientifique (FNRS, Belgium), grants 2.4621.07, 2.4539.11, and 7020836.

### REFERENCES


### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fpsyg. 2017.00426/full#supplementary-material



Children, eds C. LaSasso, K. L. Crain, and J. Leybaert (San Diego, CA: Plural Publishing), 3–25.



**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Aparicio, Peigneux, Charlier, Balériaux, Kavec and Leybaert. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Speechreading in Deaf Adults with Cochlear Implants: Evidence for Perceptual Compensation

Hannah Pimperton<sup>1</sup> , Amelia Ralph-Lewis<sup>1</sup> and Mairéad MacSweeney1,2 \*

1 Institute of Cognitive Neuroscience, University College London, London, UK, <sup>2</sup> Deafness, Cognition and Language Centre, University College London, London, UK

Previous research has provided evidence for a speechreading advantage in congenitally deaf adults compared to hearing adults. A 'perceptual compensation' account of this finding proposes that prolonged early onset deafness leads to a greater reliance on visual, as opposed to auditory, information when perceiving speech which in turn results in superior visual speech perception skills in deaf adults. In the current study we tested whether previous demonstrations of a speechreading advantage for profoundly congenitally deaf adults with hearing aids, or no amplificiation, were also apparent in adults with the same deafness profile but who have experienced greater access to the auditory elements of speech via a cochlear implant (CI). We also tested the prediction that, in line with the perceptual compensation account, receiving a CI at a later age is associated with superior speechreading skills due to later implanted individuals having experienced greater dependence on visual speech information. We designed a speechreading task in which participants viewed silent videos of 123 single words spoken by a model and were required to indicate which word they thought had been said via a free text response. We compared congenitally deaf adults who had received CIs in childhood or adolescence (N = 15) with a comparison group of hearing adults (N = 15) matched on age and education level. The adults with CI showed significantly better scores on the speechreading task than the hearing comparison group. Furthermore, within the group of adults with CI, there was a significant positive correlation between age at implantation and speechreading performance; earlier implantation was associated with lower speechreading scores. These results are both consistent with the hypothesis of perceptual compensation in the domain of speech perception, indicating that more prolonged dependence on visual speech information in speech perception may lead to improvements in the perception of visual speech. In addition our study provides metrics of the 'speechreadability' of 123 words produced in British English: one derived from hearing adults (N = 61) and one from deaf adults with CI (N = 15). Evidence for the validity of these 'speechreadability' metrics come from correlations with visual lexical competition data.

Keywords: speechreading, deaf, cochlear implants, compensation, lipreading

#### Edited by:

Matthew William Geoffrey Dye, Rochester Institute of Technology, USA

#### Reviewed by:

Jacqueline Leybaert, Université libre de Bruxelles, Belgium Jerker Rönnberg, Linköping University, Sweden

> \*Correspondence: Mairéad MacSweeney m.macsweeney@ucl.ac.uk

#### Specialty section:

This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology

Received: 13 September 2016 Accepted: 16 January 2017 Published: 07 February 2017

#### Citation:

Pimperton H, Ralph-Lewis A and MacSweeney M (2017) Speechreading in Deaf Adults with Cochlear Implants: Evidence for Perceptual Compensation. Front. Psychol. 8:106. doi: 10.3389/fpsyg.2017.00106

### INTRODUCTION

fpsyg-08-00106 February 4, 2017 Time: 17:58 # 2

The perceptual compensation hypothesis refers to the idea that sensory deprivation within one sensory modality will stimulate compensatory perceptual improvement in another sensory modality (Ronnberg, 1995). Individuals who are deaf have compromised, and sometimes minimal, access to the sounds that make up a spoken language via the auditory modality. However, when a speaker produces speech, visual, as well as auditory, information about speech sounds is available to the observer. This raises the possibility that deaf individuals may show spontaneous perceptual compensation in the domain of speech perception, with their greater reliance on the visual elements of speech in everyday life resulting in superior speech perception skills in the visual-only modality. If this perceptual compensation hypothesis is correct, we would predict that deaf individuals would show superior speechreading (visual-only speech perception) skills to hearing individuals at a group level. However, evidence regarding whether there exists a speechreading advantage for deaf individuals has been mixed.

A body of work by Ronnberg et al. (1983) with individuals who had acquired hearing loss in adulthood found no evidence for superior speechreading skills in these adults compared with hearing adults (Lyxell and Ronnberg, 1989, 1991). The results of these studies led Ronnberg to conclude that "daily dependence on lipreading in a variety of social situations does not seem to suffice as a trigger for the development of speech-reading skill" (Ronnberg, 1995). Tye-Murray et al. (2007) examined speechreading of phonemes, words and sentences in older adults with mild-moderate hearing loss acquired in adulthood and compared their performance to older adults without hearing loss. In a visual-only condition they found no significant advantages for the adults with hearing loss on phonemes or sentences, but did find that they displayed a significant advantage over the adults without hearing loss in terms of their visual recognition of single words.

In contrast with the findings on adults with acquired hearing loss, studies with groups of adults who have congenital or early onset deafness have been more consistent in demonstrating significant speechreading advantages compared to hearing adults. Bernstein et al. (2000) examined the ability of adults with normal hearing (N = 96) and with severe to profound early onset (94% experienced onset ≤ 4 years) deafness (N = 72) to speechread consonant-vowel nonsense syllables, words and sentences. The adults with early onset deafness showed enhanced speechreading ability relative to the hearing adults on all three types of speechreading stimuli, indicative of superior visual phonetic perception in the deaf adults. Auer and Bernstein (2007) replicated this finding of a significant speechreading advantage for adults with early onset deafness. They compared the performance of a large group (N = 112) of adults with early deafness (onset < 4 years) with that of a group of hearing adults (N = 220) on a sentence-level speechreading task. They found significant advantages for the deaf adults who identified 43.55% of the target words correctly compared to only 18.57% for the hearing group. They concluded that "the need to rely on visual speech throughout life and particularly for the acquisition of spoken language by individuals with early onset hearing loss, can lead to enhanced speechreading ability." Similar results were reported by Mohammed et al. (2006) using the Test of Adult Speechreading, a speechreading test that assesses speechreading skill at different levels of linguistic complexity and that was designed specifically to give deaf and hearing individuals an equal chance to demonstrate their speechreading skill by not requiring spoken or written responses. They found significant speechreading advantages for a group of 29 profoundly deaf adults (age of onset < 5 years) over a comparison group of 29 hearing adults. In a study of Brazilian Portuguese-speaking adults, Oliveira et al. (2014) found similar advantages for deaf adults over hearing adults in terms of their performance on a range of speechreading tasks, and consistent advantages for those deaf adults with prelingual onset as compared to those with post-lingual onset.

A range of skills are likely to underpin this speechreading advantage in those born deaf. In particular it is clear that individual differences in cognitive skills play an important role in speechreading skill (for review see Rönnberg et al., 2013). For example, Rönnberg et al. (1999) reported a case study of speechreading 'expert' – MM. They report that MM's speechreading skill was associated with high cognitive skills, such as phonological skills and working memory capacity.

Better visual speech understanding in individuals with congenital or early onset deafness, compared to hearing individuals, but not in those with later onset of deafness is consistent with work on perceptual compensation in blind individuals. Gougoux et al. (2004) found superior pitch discrimination skills in early blind adults (blinded < 2 years old) compared to sighted adults but no evidence of these enhancements to listening skills in late blind adults (blinded > 5 years old). They also reported a significant negative correlation between age of blindness onset and pitch discrimination performance, with those who were blind from an earlier age showing superior performance on the pitch discrimination task, and argued that "cerebral plasticity is more efficient at early developmental stages" (Gougoux et al., 2004). Subsequent studies have controlled for the influence of musical experience by including sighted controls closely matched on musical experience and have still provided consistent evidence regarding the enhancement to pitch discrimination associated with earlier onset of blindness (Wan et al., 2010).

### Speechreading in Cochlear Implant Users

In the majority of the studies reviewed above that demonstrated a speechreading advantage in adults born severely to profoundly deaf, the participants either used hearing aids or no hearing device (Bernstein et al., 2000; Mohammed et al., 2006; Auer and Bernstein, 2007). Thus, these individuals would have had minimal access to the auditory speech signal meaning that their dependence on visual speech to access spoken

language would have been high. Aparicio et al. (2012) have demonstrated that this visual speech signal can be enhanced by the use of cued speech (CS) which requires the user to pay more attention to the lips. They tested deaf CS and non-CS users and hearing participants on a sentence to picture speechreading test. Deaf participants who were native CS users were better speechreaders than deaf participants who were non-CS users. Furthermore, the two groups of deaf participants were better speechreaders than the hearing participants. This study demonstrates that different language and communication experiences in deaf individuals can lead to differences in speechreading skill.

Another way to increase the clarity of the speech signal to a deaf person is of course to increase access to the auditory input. Over the last two decades increasing numbers of profoundly deaf children and adults have received cochlear implants (CI); devices which convert acoustic stimuli into electrical signals and directly stimulate the auditory nerve to provide deaf individuals with access to sound (American Speech-Language-Hearing Association, 2004). For individuals who are congenitally deaf but are implanted in early childhood, or for those who receive a CI following an acquired hearing loss, it is often the case that the CI gives them sufficient access to speech sounds for them to be able to recognize speech in auditoryonly conditions, although there is considerable variability in speech perception outcomes even within these populations (American Speech-Language-Hearing Association, 2004). This raises the question of whether the superior access to auditory speech that deaf CI users experience impacts on their ability to perceive visual speech. It is possible that a lesser degree of dependence on the visual perceptual elements of speech for understanding spoken language means that the grouplevel deaf speechreading advantage may not be evident for CI users.

However, it is important to recognize that a CI uses a maximum of 22 electrodes to replace the function of around 16,000 hair cells and as a consequence conveys highly impoverished information about speech sounds compared to a normally functioning human cochlea (Giraud et al., 2001; Nittrouer et al., 2012). The reduced spectral information conveyed by the CI is particularly problematic in terms of its impact on auditory speech perception in the presence of background noise (Srinivasan et al., 2013). This suggests that despite the increased access to auditory speech that a CI can bring, CI users might continue to make greater use of visual speech information than hearing individuals and thus may display a speechreading advantage.

The study by Oliveira et al. (2014) described above included some deaf participants with CIs but they were grouped together with participants without CIs, so it was not possible to differentiate whether the individuals with CIs displayed a group advantage relative to the hearing controls in terms of their speechreading skills. A small number of studies have reported data comparing speechreading skills of groups of deaf individuals who have received a CI with hearing individuals. Rouger et al. (2007) assessed speechreading performance in a group of postlingually deafened adults using a task in which participants had to identify and repeat bisyllabic words presented in a visual only format. The participants completed the assessment both prior to receipt of a CI, immediately after switch on and in the years subsequent to implantation. They found that the deaf adults (N = 97) showed significantly higher speechreading performance than a comparison group of hearing adults (N = 163) when they were assessed prior to cochlear implantation. This advantage maintained in the months and years following cochlear implantation despite these deaf adults substantially increasing their auditory-only word recognition abilities. Additionally, Rouger et al. (2007) reported on a small sample (N = 8) of participants who had experienced sudden onset deafness less than a year before they received a CI and who still showed significantly superior speechreading performance compared to the hearing participants prior to, and following, cochlear implantation. They argued on the basis of this that "a high level of speechreading ability can be acquired rapidly during a period of auditory deprivation," a position in stark contrast to that of Ronnberg (1995).

As part of a study looking at audiovisual spoken word training, Bernstein et al. (2014) reported scores on a sentencelevel lipreading task (Auer and Bernstein, 2007) for a sample of pre- or perilingually profoundly deaf adults (N = 28) with CIs, the majority of whom received their CI in adulthood (>19 years), and a sample of hearing adults (N = 43). As was the case in the original Auer and Bernstein (2007) study, Bernstein, Eberhardt and Auer found a significant advantage for the deaf group over the hearing group in terms of their ability to identify words from sentences presented in a visual-only format; the average percentage words correct for the CI group was 39.4%, compared to 8.1% for the hearing group.

Huyse et al. (2013) compared the performance of congenitally, profoundly deaf children with CIs (N = 31; M age = 10 years, SD = 0.47) with that of hearing children (N = 31; M age = 10 years, SD = 0.5) on a task that required them to identify vowel-consonant-vowel nonsense syllables. They reported no significant differences between the groups in terms of their identification performance when the syllables were presented in a visual-only format, indicating no speechreading advantage for these deaf children with CIs over their hearing peers. This finding is consistent with the results from a study by Kyle et al. (2013) which compared the speechreading skills of a more audiologically diverse group of deaf children (severely to profoundly deaf, and using CIs, hearing aids or no device) with those of hearing children using a speechreading test that assessed visual speech recognition at multiple levels of linguistic complexity (words, sentences, and short stories). They found that the deaf and hearing children performed very similarly on this test; there were no significant differences between the groups on any of the three subtests. It is possible that increased experience of and attention to visual speech over a period of years is necessary for the development of superior visual speech perception skills observed in adults. Alternatively, it may be the case that the language skills of deaf children limit their performance on speechreading tasks, making it harder for them to demonstrate an advantage in their visual speech perception skills than it is for deaf adults with more experience of the spoken language the tests are conducted in.

In this study we therefore focused on deaf adults and aimed to test whether the group-level speechreading advantage demonstrated by Rouger et al. (2007) in post-lingually deafened adults who received their CIs in adulthood, and by Bernstein et al. (2014) in pre- and perilingually deafened adults who received their CIs in adulthood, could be replicated in a group of congenitally deaf adults who received their CIs in childhood or adolescence. We predicted that although these adults may on average have experienced greater access to auditory speech sounds than adults with equivalent levels of deafness without CIs they would still have experienced, and be continuing to experience, a much greater dependence on visual speech information than hearing individuals and hence would show evidence of perceptual compensation and demonstrate grouplevel advantages in their speechreading skills compared to hearing adults.

As mentioned above, auditory speech perception outcomes following implantation are highly variable and are impacted by a number of different variables. For post-lingually deafened adults, factors identified as predictors of auditory speech perception following implantation include duration of pre-implant deafness and residual hearing pre-implant (Blamey et al., 1992; van Dijk et al., 1999). For prelingually deaf children, age at implantation, residual hearing pre-implant, non-verbal ability, and exposure to an oral education have been identified as factors related to variation in speech perception outcomes following implantation (O'Donoghue et al., 2000; Svirsky et al., 2004; Geers et al., 2008). Under the framework of perceptual compensation, it would be predicted that individual variability in auditory speech perception with a CI may relate to individual variability in visual speech perception, with those individuals getting the least auditory speech access via their CI relying the most on visual speech on a day to day basis and hence showing the greatest enhancements in their visual speech perception skills.

In the present study we focus on one variable that has been consistently associated with variability in auditory speech perception outcomes following CI; age at implantation. Age at implantation effects on speech perception outcomes have been discussed in the context of sensitive periods for the development of the central auditory system. Sharma et al. (2002) have argued that the first 3.5 years of life is a period of maximal plasticity of the central auditory system. They found evidence from electrophysiological recordings of cortical auditory evoked potentials (CAEPs) that, for congenitally deaf children, implantation after 3.5 years is associated with an increased risk of developing atypical CAEPs following implantation, with these atypical CAEPs particularly likely with implantation after the age of 7 years (Sharma et al., 2002). These findings suggest that receipt of an implant earlier in life may be associated with better auditory speech perception as a consequence of increased plasticity of the central auditory system. Additionally, earlier recipients will also have experienced an increased number of years accessing auditory speech via the CI than later recipients. Taken together these factors may contribute to earlier implanted individuals showing a reduced reliance on, and therefore less well-developed, visual speech perception skills than later implanted individuals.

One study has addressed this question of whether there is evidence of a relationship between age at implantation and speechreading ability and has done so in children with CIs: Bergeson et al. (2005) compared the visual, auditory, and audiovisual speech perception performance of a group of earlier implanted (≤4 years 5 months) children with those of a group of later implanted (>4 years 5 months) children both before and in the years following implantation. They found that overall the earlier implanted children showed better speech perception performance than the later implanted children when sentences were presented in auditory-only conditions, but that this advantage was reversed when the sentences were presented in visual-only conditions with the later-implanted group showing superior performance in that context. These findings in children with CIs are consistent with the perceptual compensation hypothesis in indicating that a more protracted period of deafness, with onset in early childhood, may be associated with superior visual speech perception skills. In the present study we sought to test this hypothesis in adults with CIs who received their implant at highly variable ages (2–19 years). The majority of children who are eligible for a CI today are now receiving one before the age of 3 years (Raine, 2013) meaning that opportunities to address questions about the implication of variability in age of implantation that spans beyond the sensitive period for the development of the central auditory system are increasingly limited; thus this sample presents a unique opportunity.

To summarize, in the present study we tested the following hypotheses:


### MATERIALS AND METHODS

### Participants

Sixty one native English-speaking hearing participants provided data for this study. All reported normal hearing and normal or corrected-to-normal vision. The participants were either undergraduate students participating for course credit or volunteers from the wider community who had responded to adverts to take part in the study. All provided informed consent prior to participation in the study. An additional 13 participants consented to participate in the study but were excluded from the current dataset as a result of not completing all items in the task.

Fifteen congenitally deaf participants with CIs participated in this study. Age at implantation ranged from 2 to 19 years (M = 8.27, SD = 5.05). All reported normal or corrected-tonormal vision and profound deafness.

To faciliate comparisons between the speechreading performance of CI and hearing participants, each of the 15 CI participants was individually matched to a hearing participant from the larger hearing sample (N = 61) on the basis of age and education level, with these 15 matched hearing participants forming the hearing comparison group (HCG). There were no significant differences in the distribution of ages between the CI (median = 23, range = 22–26) and HCG (median = 23, range = 20–31) groups (U = 122, Z = 0.41, p = 0.71, r = 0.07). The groups also did not differ significantly in terms of the distribution of highest education level achieved [χ 2 (2, N = 30) = 0.16, p = 0.92, ϕ<sup>c</sup> = 0.07].

### Materials and Procedure

fpsyg-08-00106 February 4, 2017 Time: 17:58 # 5

One hundred and twenty three words were selected as the target words for this experiment (see Supplementary Table S1 for full list). All words were either concrete nouns or colors. Information on the visual speech lexical competition experienced by the speechread target words was sourced from the Phi-Lex database (Strand, 2014). The measure used was a continuous measure of visual lexical competition (ConV). This measure reflected the overall competition in the reference lexicon for the target word based on the similarity of the response distributions of its constituent phonemes (from a forced choice visual only phoneme identification task) to those of phonemes in every other word of the same pattern type in the lexicon. For further details of how this measure was derived, see Strand (2014). This variable was available for 86 out of the 123 words in the study.

A video of each word being spoken by a female model was made using a Sony Handycam (HDR-CX115). The word was spoken aloud at a normal conversational volume during the recording and the videos were subsequently edited to mute the volume such that the participants saw a natural production of the word but without any sound. The same model produced each word. The model maintained a neutral facial expression in the production of every word and the camera distance, lighting and background conditions were consistent for each word (see **Figure 1**).

Four different randomized orders of the 123 videos were produced and participants were randomly assigned to complete one of the four orders. The videos were presented using Opinio, a web-based survey tool, and participants completed the task via the internet on their personal computers. Participants were instructed that they would see silent videos of a model saying a single word and that they could only view each video once. They were required to click to play each video and then write the word they thought they had seen in a free text response box before moving onto the next video.

When scoring the responses as correct or not relative to the target, participants were given one point for an item if the response either directly matched the target or if they had produced a homophone of the target (e.g., had written 'I' for the target 'eye'). This meant they could score a maximum of 123 on the task.

Prior to completing the speechreading task the participants provided demographic information via the web-based tool. Additional audiological information was collected from the participants with a CI via a paper-based response form prior to their completion of the speechreading task.

FIGURE 1 | Screenshot of visual speech model who produced all stimuli.

## RESULTS

### Overall Speechreading Performance of Hearing Participants

The mean number of words identified correctly by the 61 hearing participants was 22.38 (SD = 9.94; range = 2–48) out of 123. This was equivalent to a mean proportion correct of 0.18 (SD = 0.08; range = 0.02–0.39). The mean number of words identified correctly by the hearing participants was significantly above the floor of 0 [t(60) = 17.56, p < 0.001, d = 2.25] and significantly below the ceiling of 123 [t(60) = −79.08, p < 0.001, d = −10.12].

### Comparison of Speechreading Performance for the CI Participants and the Hearing Control Group (HCG)

The mean number of words identified correctly by the CI participants was 40.80 (SD = 16.81; range = 9–62) and by the matched HCG was 24.20 (SD = 10.40; range = 4–46) (see **Figure 2**). This was equivalent to a mean proportion correct of 0.33 (SD = 0.14) for the CI group and 0.20 (SD = 0.08) for the HCG. Levene's test indicated unequal variances between the two groups. Therefore the unequal variance Welch t-test

was used. This showed that deaf CI users scored significantly higher than the hearing control group on the speechreading task, t(23.35) = 3.25, p = 0.003, d = 1.19. The mean number of items correct for the matched HCG used for this group comparison (N = 15; M = 24.20, SD = 10.40) did not differ significantly from that of the remaining hearing participants (N = 46; M = 21.78, SD = 9.83), [t(59) = 0.82, p = 0.42, d = 0.24], suggesting that the level of performance of these matched hearing participants was representative of the performance level of the wider hearing sample.

### Relationship between Speechreading Performance and Age at Implantation

Within the CI group there was a significant positive correlation between age at implantation and score on the speechreading task (r = 0.61, p = 0.02, 95% CI = 0.29–0.84), with those participants who received their CIs at a later age showing higher speechreading scores than the earlier implanted participants (see **Figure 3**). A regression analysis predicting speechreading task score with age and education level entered at Step 1 and age at implantation at Step 2 indicated that age at implantation accounted for significant unique variance in speechreading performance, accounting for 30% of the variance over and above the 11% accounted for by age and education level (**Table 1**).

### Individual Item Accuracy and Relationship with Word Properties

For the 61 hearing participants, the proportion correct for individual items ranged from 0.77 (for 'rabbit') to 0.00 (for 'bull'; 'duck'; 'fan'; 'jacket'; 'mat'; 'milk'; 'shorts'; 'skirt'; 'trousers'; 'van'; 'wall'; 'wheel'). Supplementary Table S1 presents data on the proportion correct responses for each of the 123 items individually for the hearing sample (Supplementary Table S1). These data provide a metric of 'speechreadability' of the 123 words when produced in British English.

There was a high degree of concordance between the hearing participants and the deaf participants in terms of the relative success rates on the individual items [r(121) = 0.79, p < 0.001, BCA bootstrapped 95% CI = 0.70–0.85]. For the 15 deaf participants with CIs, proportion correct for individual items ranged from 0.87 (for 'elephant'; 'fish'; 'lorry'; 'orange'; 'phone'; 'rabbit') through to 0.00 (for 'ball'; 'bee'; 'bull'; 'duck'; 'fan'; 'hand'; 'hen'; 'ladder'; 'mat'; 'milk'; 'pan'; 'peg'; 'red'; 'ring'; 'shorts'; 'wall'; 'wheel'; 'wing'). Supplementary Table S2 presents data on the proportion correct responses to each of the 123 items for the participants with CIs.

Relationships between the proportion of correct responses to an item (PropCorr) and word properties of that item were examined for the 86 words that had continuous visual lexical competition data (ConV) available for them. The distribution of the PropCorr and ConV variables showed deviations from normality so bias corrected and accelerated bootstrapped 95% confidence intervals (1000 bootstrap samples) are presented for the correlation coefficients. The number of correct responses for the items showed a significant negative correlation with ConV for both the hearing participants [r(84) = −0.39, p < 0.001, BCA bootstrapped 95% CI = −0.55 to −0.18] and the CI participants [r (84) = −0.50, p < 0.001, BCA bootstrapped 95% CI = −0.64 to −0.34] indicating that words that had fewer visual (visemic)

lexical competitors were more successfully identified by both groups of participants.

### DISCUSSION

The first aim of this study was to compare the single word speechreading ability of hearing adults and deaf adults with CIs in order to test the hypothesis that prelingually deaf adults who received an implant in childhood or adolescence would show a speechreading advantage. We found that the mean score on the speechreading task was significantly higher for the adults with CIs than for the matched comparison group of hearing adults. The performance of the matched HCG did not differ significantly from that shown by the larger sample of hearing participants who were not selected for the HCG, suggesting HCG performance was representative of the broader hearing sample. This finding of a speechreading advantage for the deaf adults with CIs in this study is consistent with the findings of Rouger et al. (2007) and Bernstein et al. (2014) who found evidence of a deaf speechreading advantage for adults with later age at implantation or later age at onset of deafness. These consistent findings in adults with CIs suggest that even with the greater access to the auditory elements of speech that a CI provides, these adults are still substantially more dependent on visual speech than hearing adults and consequently have developed compensatory superiority in their ability to use visual speech information to understand spoken language. It would be interesting in future studies to contrast this group with deaf adults without CI to further understand the extent of this compensation.

In terms of the locus of this compensation, the finding in this study of an advantage for the adults with CIs on a single word speechreading task in which there was no sentential context suggests that the deaf speechreading advantage is not exclusively driven by an enhanced ability to use sentence-level contextual information to facilitate identification of individual words. This is not to say that the use of top down sentence-level processing

TABLE 1 | Results of regression analyses predicting speechreading score with age and education level entered at Step 1 and age at implantation at Step 2.


<sup>+</sup>Dichotomised into < or ≥ undergraduate degree. Age at implantation accounted for unique variance in the speechreading score.

to aid speechreading is not something that deaf adults develop the capacity to use to effectively support speechreading in real word contexts, but rather that they also have enhanced skills in domains that support the type of context-free lipreading performance assessed in this task. This suggests that they may be better at visually perceiving individual phonemes. Alternatively, they may be better either at perceiving rapid sequences of phonemes or at using coarticulatory information in phoneme sequences to disambiguate visual phonetic information (e.g., influences of voiced vs. voiceless consonants on preceding vowel length; although the phonemes /t/ and /d/ are visually perceptually identical in isolation they have a differential effect on the articulation of the middle vowel when in a word final position; contrast 'beat' and 'bead.') Future studies should aim to test this by comparing the performance of hearing adults and deaf adults on a speechreading task in which they have to speechread both individual phonemes (e.g., '/f/') and also non-words which use the phonotactics of the ambient spoken language (e.g., 'mip,' 'niddy').

The second aim of this study was to test the hypothesis that, within the CI group, later implantation would be associated with superior speechreading skills. We found a significant positive correlation between age at implantation and speechreading performance, indicating a speechreading advantage for those implanted at later ages. This result is consistent with the hypothesis that a greater duration or degree of dependence on visual information for speech perception leads to improved visual speech perception skills and therefore with the idea of perceptual compensation in the domain of speech perception. The finding of superior visual speech perception skills in later implanted adults is also consistent with the findings of Bergeson et al. (2005), who reported superior visual-only speech perception in later-implanted children (in contrast to superior auditory speech perception skills in earlier-implanted children). However, it is important to acknowledge this was a preliminary study in adults with CIs with the small sample size meaning that the confidence intervals for this correlation are wide and therefore replication of this result in a larger sample would be of value. A further limitation of this study was the lack of detailed audiological information available for the CI participants. We had no objective audiological measures for the CI users. Thus it was not possible to determine whether the earlier-implanted participants did have superior auditory speech perception through their implant than the later-implanted participants, as was the case in the Bergeson et al. (2005) study with children, and whether this related to their visual speech perception skills.

It was also the case that we did not have extensive information regarding the language skills of the CI participants. It is possible that variability in underlying spoken language knowledge within the CI group may have influenced their performance on the speechreading task and hence the relationship between age at implantation and speechreading score. However, this would be dependent on the later implanted participants showing superior spoken language skills to the earlier implanted participants, a situation which is the reverse of that typically observed. Additionally, the language demands in this speechreading task were relatively low. Participants identified single words, all of which were early acquired concrete nouns, meaning that the contributions of existing language knowledge to task performance are likely to have been minimized.

The speechreading task used in this study was completed remotely via the internet. Unfortunately this meant that we were unable to collect measures of participant's other cognitive skills known to be important to individual differences in speechreading skill, such as phonological skills and working memory capacity (see Rönnberg et al., 1999, 2013). Future studies should attempt to include these measures. Remote data collection also meant that we were unable to monitor participants' attention while they undertook the task. We reasoned that if participants showed item-level response patterns that were consistent with the predictions of models of spoken word recognition in the visual modality, this would support the validity of this task as a measure of speechreading. Activation-competition models of spoken word recognition in the auditory modality (e.g., the Neighborhood Activation Model; Luce and Pisoni, 1998) have posited that hearing a spoken word elicits simultaneous activation of multiple words in the mental lexicon that are perceptually similar to the target word and compete with the target word for recognition, the result being that words with more perceptually similar 'neighbors' are recognized with less accuracy than words that have fewer such neighbors in auditory word recognition paradigms (Dirks et al., 2001). Analogously, research examining spoken word recognition in the visual modality has shown equivalent effects of visual lexical competition experienced by the target word on recognition accuracy suggesting that activation-competition models of word recognition extend beyond the auditory modality (Feld and Sommers, 2011; Strand and Sommers, 2011). For both the CI and hearing groups in the present study, visual lexical competition showed a significant negative relationship with recognition accuracy for the words; those words with greater visual lexical competition were correctly recognized less often than those with lower visual lexical competition. This suggests that the single word speechreading task was measuring a consistent construct across the two participant groups, and supports the validity of this task as a measure of visual speech perception.

### CONCLUSION

The results of this small scale study provided two strands of evidence that are consistent with perceptual compensation in the domain of speech perception. First, prelingually deaf adults with CIs showed a significant advantage in terms of their performance on a visual-only speechreading task compared to a comparison group of hearing adults matched on age and education. Second, age at implantation within the CI group showed a significant positive relationship with performance on the speechreading task; participants who received their implants later in life showed superior visual-only speech perception skills. We argued that in both cases these patterns of findings resulted from increased dependence on visual speech information leading to compensatory improvements in perception of speech via the visual modality.

### ETHICS STATEMENT

fpsyg-08-00106 February 4, 2017 Time: 17:58 # 9

This research was approved by the University College London Research Ethics Committee. Online informed consent was obtained from each participant.

### AUTHOR CONTRIBUTIONS

HP and MM designed the study. AR-L collected the data. HP analysed the data. HP and MM wrote the paper.

### REFERENCES


### FUNDING

This research was supported by a Wellcome Trust Senior Research Fellowship (grant number 100229/Z/12/Z) awarded to MM.

### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fpsyg. 2017.00106/full#supplementary-material



**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Pimperton, Ralph-Lewis and MacSweeney. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# The Impact of Early Visual Deprivation on Spatial Hearing: A Comparison between Totally and Partially Visually Deprived Children

Giulia Cappagli<sup>1</sup> \*, Sara Finocchietti<sup>1</sup> , Elena Cocchi<sup>2</sup> and Monica Gori<sup>1</sup>

<sup>1</sup> Unit for Visually Impaired People, Istituto Italiano di Tecnologia, Genova, Italy, <sup>2</sup> Istituto David Chiossone per Ciechi ed Ipovedenti, Genova, Italy

The specific role of early visual deprivation on spatial hearing is still unclear, mainly due to the difficulty of comparing similar spatial skills at different ages and to the difficulty in recruiting young blind children from birth. In this study, the effects of early visual deprivation on the development of auditory spatial localization have been assessed in a group of seven 3–5 years old children with congenital blindness (n = 2; light perception or no perception of light) or low vision (n = 5; visual acuity range 1.1–1.7 LogMAR), with the main aim to understand if visual experience is fundamental to the development of specific spatial skills. Our study led to three main findings: firstly, totally blind children performed overall more poorly compared sighted and low vision children in all the spatial tasks performed; secondly, low vision children performed equally or better than sighted children in the same auditory spatial tasks; thirdly, higher residual levels of visual acuity are positively correlated with better spatial performance in the dynamic condition of the auditory localization task indicating that the more residual vision the better spatial performance. These results suggest that early visual experience has an important role in the development of spatial cognition, even when the visual input during the critical period of visual calibration is partially degraded like in the case of low vision children. Overall these results shed light on the importance of early assessment of spatial impairments in visually impaired children and early intervention to prevent the risk of isolation and social exclusion.

#### Edited by:

Olivier Pascalis, Université Grenoble Alpes, France

#### Reviewed by:

Charlotte Joy Codina, The Sheffield College, UK David Méary, Université Grenoble Alpes, France

> \*Correspondence: Giulia Cappagli giulia.cappagli@iit.it

#### Specialty section:

This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology

Received: 04 October 2016 Accepted: 13 March 2017 Published: 10 April 2017

#### Citation:

Cappagli G, Finocchietti S, Cocchi E and Gori M (2017) The Impact of Early Visual Deprivation on Spatial Hearing: A Comparison between Totally and Partially Visually Deprived Children. Front. Psychol. 8:467. doi: 10.3389/fpsyg.2017.00467 Keywords: auditory perception, blindness, child development, spatial hearing, visual impairment

## INTRODUCTION

The effects of early sensory deprivation on the acquisition of perceptual and cognitive skills have been extensively studied in order to measure the weight that each sensory modality has on the development of those skills (Bruner, 1959; Freedman, 1971; Mistretta and Bradley, 1978; Grubb and Thompson, 2004). Nonetheless, while the impact of early sensory loss on specific perceptual skills is usually easier to detect in adulthood when sensory systems have been already developed, the shortterm as well as the long-term consequences of impaired perceptual functions in childhood are more difficult to predict. This is mainly due to the difficulty of comparing similar spatial skills at different ages and to the difficulty of assessing the spatial performance of congenitally and totally blind children at an early age. With this study, we aim to increase knowledge about the development of

spatial hearing in visually impaired children by comparing spatial localization skills of totally blind and low vision participants in two tasks assessing the ability to encode static and dynamic sound sources separately.

Considering early visual deprivation, although it is often commonly accepted that the complete absence of vision from birth produces a strengthening of the remaining intact senses especially in adulthood (Collignon et al., 2009; Kupers and Ptito, 2014), a critical review of the available literature shows that non-visual modalities are not always able to fully compensate for the lack of visual experience, especially during childhood (Pasqualotto and Newell, 2007). In other words, although compensatory mechanisms for spatial perception have been demonstrated in blind individuals (Lessard et al., 1998; Voss et al., 2004; Doucet et al., 2005; Collignon and De Volder, 2009), an early and pervasive visual impairment might delay the development of specific auditory spatial skills (Zwiers et al., 2001; Lewald, 2002; Eimer, 2004; Gori et al., 2014; Cappagli et al., 2015; Kolarik et al., 2016). Neuroimaging studies tend to confirm the idea that the lack of vision sharpens the remaining modalities (James, 2013) by showing that the aforementioned superior abilities of blind individuals are supported by the recruitment of the otherwise unused visual cortex areas by the remaining non-visual modalities (Burton et al., 2002; Amedi et al., 2003; Noppeney et al., 2005; Collignon et al., 2009; Kupers et al., 2011; Thaler et al., 2011) that may have a functional role (Cohen et al., 1997; Amedi et al., 2004; Gougoux et al., 2005; Merabet et al., 2009). While the above studies suggest that nonvisual sensory modalities can successfully compensate for the lack of vision, they don't help to explain the poorer perceptual abilities reported by studies assessing spatial performance in blind individuals. Despite the effort in including all the results reported on the topic in a single comprehensive theory, to date the role of visual input in the development of spatial cognition is still unclear due to the high controversy of the results supporting the mentioned scientific hypotheses. The cross-sensory calibration hypothesis proposed by Gori et al. (2008, 2014) and supported by experimental data (Postma et al., 2008; Cappagli et al., 2015; Finocchietti et al., 2015; Vercillo et al., 2016) attempts to provide a comprehensive explanation by stating that during the early development vision calibrates other senses to process spatial information because vision is the most robust sense to perceive the spatial properties of the world. The spatial properties of the surrounding environment are indeed best discovered with vision because it provides an immediate and complete representation of multiple and simultaneous stimuli in the environment (Thinus-Blanc and Gaunet, 1997). As a consequence, the complete development of spatial cognition should be more compromised when a visual impairment occurs at an early age and is pervasive compared to when the visual impairment occurs later in life and affects only partially the visual status of the individual. The comparison between the effects of total blindness versus degraded vision on spatial perception at an early age provides experimental evidence concerning the essential role of visual experience in shaping space perception and cognition. For this reason, in our study we explicitly compared the performance of totally blind and low vision children in different spatial tasks and evaluated how the residual vision correlates with the spatial performance.

Moreover while the development of spatial cognition has been extensively studied in sighted children (Vasilyeva and Lourenco, 2012), it is not clear how visually impaired children represent the surrounding space by using the spatial maps constructed through the remaining intact senses, especially touch and hearing. Recent findings suggest that sighted children acquire spatial capabilities thanks to the reciprocal influence between visual perception and execution of movements (Bremner et al., 2008): children monitor the success of an action through a sensory-motor feedback by matching expected and observed changes of visual information. Indeed self-generated movements commonly help to perceive the space acoustically because they convey the proprioceptive sensation corresponding to the movement of the ears toward sound sources (Aytekin et al., 2008). Visually impaired children not only lack the visual input necessary to establish the sensory-motor feedback that typically promotes spatial development, but also manifest a general delay in the acquisition of important locomotor and proprioceptive skills which may cause them to accumulate much less spatial experience compared to their sighted peers (Fraiberg and Fraiberg, 1977; Warren, 1977; Landau et al., 1984). To perceive space, visually impaired children typically use hearing and touch. The case of hearing is particularly interesting because the auditory sense is not only the main channel for providing distal information (Spencer et al., 1989; Ungar, 2000) but also it might be superior to all other sensory alternatives because it provides spatial information in both active and passive conditions and it does not necessarily involve direct contact with objects (Wanet and Veraart, 1985; Jacobson, 1998). At the same time, the use of hearing to perceive distal information might be particularly difficult for visually impaired children because in this case they do not have any sensory feedback about sonorous objects in the far space. On the contrary, the haptic-proprioceptive system can provide accurate spatial data only within the scope of the body itself (Ungar, 2000), and therefore a blind person must actively move in the environment to sequentially touch all the stimuli embedded in the space. For this reason, we developed an auditory localization task which assesses the ability of children to localize by touch an auditory sound source in near space. Moreover, since it is not clear whether early visual deprivation differently impacts on the ability to localize static versus moving sound sources in adulthood (Lewald, 2013; Finocchietti et al., 2015), we employed two different tasks in which children were asked to indicate the spatial position of a static or dynamic sonorous stimulus. According to the cross-sensory calibration hypothesis proposed by Gori et al. (2008, 2014), we expected to observe a more evident spatial impairment in totally blind children, because low vision children could have benefit from the visual calibration of audition to encode spatial information. Since the available data on adults has not suggested whether we should expect a difference in the encoding of static and dynamic audio sound sources in early childhood, we didn't postulate any specific hypothesis regarding the assessment of these skills in our sample.

### MATERIALS AND METHODS

fpsyg-08-00467 April 10, 2017 Time: 11:26 # 3

To increase knowledge about how the absence of vision can impact on the creation of spatial representations, and specifically for which spatial aspects the development of spatial cognition differs in blind and sighted children, we employed two auditory spatial tasks which assess the ability of children to localize a static (Static Sound Localization Task) or a moving (Dynamic Sound Localization Task) sound source. For all tasks, we compared the spatial performance of sighted and visually impaired groups of children.

### Task and Procedure

We developed a haptic setup made of a vertical surface (50 cm × 50 cm) covered by non-adjacent tactile sensors (2 cm × 2 cm) that can register the position of the contact and provide accurate information about spatial errors (**Figure 1**). The haptic setup consisted of 25 blocks of sensors, each block

hand. Localization error was calculated for each trial by extracting the length of the vector that connected the actual and the perceived position of the loudspeaker (mm). (B) Dynamic Sound Localization Task: In the Dynamic Sound Localization Task, the sound moved across loudspeakers from a starting point (blue dots on the right) toward one of the four end-point positions (red dots on the right) and the participant responded by touching the end-point of the motion trajectory, that is the last active loudspeaker. Localization error was calculated for each trial by extracting the length of the vector that connected the actual and the perceived position of the loudspeaker (mm).

containing 16 tactile sensors (4 × 4) and a single loudspeaker in the middle. Therefore the total number of loudspeakers mounted on the setup was 25. The participant sat on a chair in front of a table which supported the haptic setup used to run the experiment. Distance from the setup to the trunk was maintained at 40 cm by positioning the chair in order to make the device easily reachable with the dominant hand for all the participants. While listening to the sound presented, the participant kept the dominant hand fixed on the starting point on the table that was approximately the position corresponding to the right limit of the haptic device (the left limit for left-handed).

The Static Sound Localization Task (**Figure 1**, upper panel) required the participant to indicate the position of a single sound source on a vertical surface. We selected three target loudspeakers sufficiently distributed on the surface (left, center, right) and judged equally difficult to reach in a previous pilot study: each target was sampled five times for a total of 15 trials. On each trial, a sound coming from one out of three target loudspeakers (red dots) was presented and the participant responded by touching the perceived sound source with the index finger of the dominant hand. The Dynamic Sound Localization Task (**Figure 1**, bottom panel) required the participant to indicate the end-point of a dynamic sound source that moved in the horizontal and vertical plane. We selected four motion trajectories centered on the setup (yellow rows: up-to-down, down-to-up, left-to-right, and rightto-left): each motion trajectory was sampled four times for a total of 16 trials. On each trial, the sound moved across loudspeakers from a starting point (blue dots) toward one of the four end-point positions (red dots) and the participant responded by touching the end-point of the motion trajectory, that is the last activated loudspeaker.

For both tasks, the auditory stimulus was a 'meow' sound registered and implemented in Matlab (R2013a, The MathWorks, USA). In the static condition of the task, a single sound was played at a time. In the dynamic condition of the task, while the first sound was playing the second sound started in order to create an audio motion. The sound has been chosen in order to make the task more entertaining for children. Children were instructed to listen the sound produced by the kitten and try to catch him. For both tasks, localization error was calculated for each trial by extracting the length of the vector that connected the actual and the perceived position of the loudspeaker (mm). Spatial accuracy indicated by localization error was computed for each participant and for each group of children.

### Participants

Fourteen sighted participants (mean age: 3.6, 10 males) and seven visually impaired participants (N = 5 low vision children, mean age: 4.4, 4 males; N = 2 blind children, mean age: 3.5, 2 males) have been enrolled in the study.

Visual acuity values are represented in LogMAR. The main exams used for the functional assessment of visual abilities are light sources method and Early Treatment Diabetic Retinopathy Study with Lea Hyvarinen symbols chart (Lea Symbols <sup>R</sup> 15- Line Translucent ETDRS-Style Distance Chart). The distance from the chart was 3 m and assessment was performed with both eyes open using a backlit screen. The visual deficit of visually impaired participants has been interpreted according to the International Statistical Classification of Diseases and Related Health Problems (ICD) – 10th revision. The currently available version of the 10th revision of the International Statistical Classification of Diseases and Related Health Problems (ICD) defines visual impairment categories primarily on the basis of recommendations made by a World Health Organization (WHO) Study Group in 1972 (World Health Organization, 2003). The term 'visual impairment' in category H54 of the ICD classification, comprises category 0 for mild or no visual impairment, category 1 for moderate visual impairment, category 2 for severe visual impairment, categories 3, 4, and 5 for blindness, and category 9 for unqualified visual impairment. The term 'low vision' is used for visual acuity less than 0.5– 1.3 LogMAR in the better eye with best correction and includes categories 1 and 2. The term 'blindness' is used for complete (no light perception) or nearly complete (visual acuity less than 1.3 LogMAR to light perception) vision loss. The participants in our study are defined as 'low vision' and 'blind' according to these definitions, except two children classified as 'low vision' who have a visual acuity of 1.7 LogMAR. In each case the visual deficit was of peripheral origin.

The cognitive level of all the visually impaired children was assessed with "The Reynell-Zinkin Scales: Developmental Scales for Young Visually Handicapped Children" and considered appropriate for their participation in the study, according to the total scores and the cut-offs proposed by the authors. Clinical details of the visually impaired children enrolled in the study can be found in **Table 1**. Sighted participants reported no visual impairment and a visual acuity better than 9/10. None of the sighted and visually impaired participants had additional sensory disabilities, including hearing disabilities tested with classical audiometer tests during the periodic neuroophthalmological assessment.

Before entering the experimental room, all children with normal or residual vision were blindfolded so they had no chance to see the experimental setup. Before starting the test, each child was asked to familiarize with the experimental setup by exploring it with hands for 30 s. The study was approved by the ethics committee of the local health service and parental or adult informed written consent for the study was obtained in all cases.

### RESULTS

The average results of blind and low vision children are presented in **Figure 2** for the Static Sound Localization Task (black bars) and the Dynamic Sound Localization Task (red bars). As a measure of spatial accuracy, we plotted the localization error calculated as the average length of the vector that connected the correct and the perceived position of the loudspeaker (mm) for all trials and for each group.

When asked to localize a static sound source, totally blind children performed significantly worse than sighted children (t = −2.26, p < 0.05) but equally to low vision children (t = −1.68, p = 0.1). On the contrary, when asked to localize

TABLE 1 | Clinical details of visually impaired children.


The table shows the clinical details for each visually impaired participant: visual status, visual acuity, pathology and age at diagnosis. Residual vision has been measured with different methods (light sources method, ETDRS Optotype and ERG, ERP, PEV) and is expressed in LogMAR. NPL (no perception of light).

Sound Localization Task (∗∗p < 0.01).

a dynamic sound source, blind children performed significantly worse compared to both sighted (t = −2.4, p < 0.05) and low vision (t = −4.28, p < 0.01) children. Moreover, low vision children showed a better performance than sighted children for the localization of a dynamic sound source (t = 4.94, p < 0.01) suggesting that early visual deprivation might impact differently the ability to localize fixed and moving auditory sound sources. To quantify the association between visual acuity and different aspects of spatial perception, we correlated the visual acuity expressed in LogMAR and the spatial accuracy expressed as mean localization error (mm) of all participants. The LogMAR scale is calculated as log (MAR) = log (1/V) = −log (V) and it represents vision loss, so higher values indicate poorer vision while lower values indicate better vision. The correlational analysis further confirmed our hypothesis, by indicating that a significant positive correlation exists when considering the dynamic sound condition: the localization error decreases with increasing residual vision in the localization of moving sound sources (r = 0.92, p < 0.01), while the same is not evident for the localization of static sound sources (r = 0.39, p = 0.4). Even if a bigger sample of visually impaired participants would be necessary to draw any general conclusion, this result highlights the important role of early visual experience in the development of auditory spatial cognition. The case of low vision children is particularly interesting because it suggests not only that a poor but early visual experience is sufficient to develop the ability to localize static and moving auditory sound sources, but also that degraded vision might

represents the condition in which compensation mechanisms manifest.

### DISCUSSION

The acquisition of spatial hearing is of fundamental importance for visually impaired children, because it constitutes a good indicator of the ability to independently navigate in the environment and the propensity to engage in positive social interaction with peers. Indeed while for sighted individuals the visual feedback represents the most important incentive for actions and thus for the development of mobility and social skills (Gori et al., 2016), visually impaired individuals strongly rely on auditory landmarks to encode spatial and social information. To our knowledge, to date there have been no previous studies on spatial hearing in young visually impaired children that directly assessed the role of residual vision on the development of auditory localization in static and dynamic conditions. The present experiment aimed at assessing the effects of total versus partial early visual deprivation on the development of auditory spatial localization abilities in childhood. By comparing the performance of totally blind and low vision children in static and dynamic auditory localization conditions, we demonstrated that visual experience is fundamental for the development of specific auditory spatial skills. Indeed, while total blindness from birth strongly compromises the ability to localize both kinds of sound sources, a congenital but not total visual impairments leads to compensatory mechanisms that allow the individual to correctly perceive the position of static and moving sound sources in the surrounding space. These results reveal that vision has a pivotal role in guiding the maturation of space cognition in the brain even when visual acuity is poor, as indicated by the case of low vision children, and suggest that visual calibration of spatial perception in the first years of life is crucial for normal spatial inference ability.

Scientific research on the development of auditory localization skills in visually impaired children has provided contrasting results, not only because spatial hearing has been studied within the framework of broader research on the cognitive and motor skills development (Warren, 1977; Hueg et al., 2014) but also because the impact of blindness severity was not always primarily considered. Indeed it has been demonstrated that the onset of blindness has a strong impact on spatial performance in adulthood: for example, late blind individuals who lost vision later in life after a normal visual experience during the first years of life perform equally or even better than sighted participants when asked to identify sound sources in horizontal space (Abel et al., 2002; Cappagli et al., 2015; Finocchietti et al., 2015), to determine the relative distance of two sounds presented in far-auditory space (Voss et al., 2004) and to focus auditory attention in the periphery (Fieger et al., 2006). While several hypotheses have been advanced to explain the superior auditory spatial performance of late blind individuals, like the effect of practice with auditory cues (Cappagli et al., 2015), very little is known about the effects of residual vision on the spatial performance in early childhood.

One of the main issues could have been that studies performed on visually impaired children under 3 years of age do not employ psychophysical procedures but they frequently use the sound of familiar voices or toys to gather information about auditory localization abilities in blind children (Hueg et al., 2014). For example, while studies on older children with visual disabilities demonstrated that they have a good spatial hearing in terms of horizontal and vertical sound localization (Ashmead et al., 1998) but a worse performance in more complex auditory tasks (Cappagli et al., 2015; Cappagli and Gori, 2016; Vercillo et al., 2016), studies on infants with severe congenital blindness indicated that they have a developmental delay in sound localization abilities (Fraiberg, 1977) and motor responses to sound (Fraiberg et al., 1966; Adelson and Fraiberg, 1974) but also that they can compensate for the lack of vision with good manipulatory and walking skills which allow the exploration of sounding objects in the environment (Fazzi et al., 2011).

Although it is difficult to compare the studies that investigate the effect of vision loss on auditory spatial skills because they use different methodologies, an important aspect such as the onset of the visual impairment has not always been considered when investigating auditory localization skills in blind individuals. For example, many studies mixed data from children with no visual experience with those of children with partial visual experience in the first period of life (Ashmead et al., 1998). Our study aimed to compare spatial localization skills of totally blind and low vision young children in two tasks assessing the ability to encode static and dynamic sound sources separately. Our study led to two main findings: first, we showed that early visual experience has an important role in the development of spatial cognition, since totally blind children performed overall more poorly than sighted and low vision in all spatial tasks performed; as a second, we pointed out that visual calibration of spatial perception in the first years of life is crucial for the development of normal auditory spatial representation even if vision is degraded, since low vision children performed equally to or better than sighted children, respectively, in the static and dynamic auditory spatial tasks. Further studies will be necessary to further confirm the relevance of our results, since our sample comprised only two congenitally blind children due to the difficult nature of recruiting young children with a total visual deficit. Overall these results shed light on the importance of early assessment of spatial impairments in visually impaired children and early intervention to prevent the risk of isolation and social exclusion. Indeed, it has been shown that early blind children have difficulties not only in auditory and haptic spatial skills (Marmor, 1977; Pasqualotto and Newell, 2007; Röder et al., 2007; Gori et al., 2010; Cappagli et al., 2015) as well as locomotor and mobility skills (Sampaio et al., 1989; Stack et al., 1989; Bigelow, 1992; Perez-Pereira and Conti-Ramsden, 2001), but also in engaging in positive social interaction (Sacks and Kekelis, 1992; Sacks et al., 1992; Rettig, 1994; Guralnick et al., 1996a,b,c; McConnell and Odom, 1999). For this reason, the early assessment of spatial abilities in visually impaired children is fundamental to develop adequate intervention programs to restore or rehabilitate impaired aspects of spatial perception.

### ETHICS STATEMENT

fpsyg-08-00467 April 10, 2017 Time: 11:26 # 7

The study was approved by the ethics committee of the local health service (ASL3 3 Genovese) and parental or adult informed written consent for the study was obtained in all cases before testing the children.

### REFERENCES


### AUTHOR CONTRIBUTIONS

GC collected and analyzed all the data, wrote the article and provided an explanatory hypothesis. EC helped recruiting and motivating visually impaired children. SF helped programming the experiments. MG provided support throughout the study.

### ACKNOWLEDGMENT

The research has been supported by the European ABBI project (FP7-ICT-2013-10-611452).


with visual impairment and normal vision. Res. Dev. Disabil. 35, 2015–2025. doi: 10.1016/j.ridd.2014.04.022


action. Proc. Natl. Acad. Sci. U.S.A 104, 4753–4758. doi: 10.1073/pnas.0607 158104


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The reviewer DM and the handling Editor declared their shared affiliation, and the handling Editor states that the process nevertheless met the standards of a fair and objective review.

Copyright © 2017 Cappagli, Finocchietti, Cocchi and Gori. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Auditory Spatial Perception without Vision

### Patrice Voss\*

Cognitive Neuroscience Unit, Department of Neurology and Neurosurgery, Montreal Neurological Institute – McGill University, Montreal, QC, Canada

Valuable insights into the role played by visual experience in shaping spatial representations can be gained by studying the effects of visual deprivation on the remaining sensory modalities. For instance, it has long been debated how spatial hearing evolves in the absence of visual input. While several anecdotal accounts tend to associate complete blindness with exceptional hearing abilities, experimental evidence supporting such claims is, however, matched by nearly equal amounts of evidence documenting spatial hearing deficits. The purpose of this review is to summarize the key findings which support either enhancements or deficits in spatial hearing observed following visual loss and to provide a conceptual framework that isolates the specific conditions under which they occur. Available evidence will be examined in terms of spatial dimensions (horizontal, vertical, and depth perception) and in terms of frames of reference (egocentric and allocentric). Evidence suggests that while early blind individuals show superior spatial hearing in the horizontal plane, they also show significant deficits in the vertical plane. Potential explanations underlying these contrasting findings will be discussed. Early blind individuals also show spatial hearing impairments when performing tasks that require the use of an allocentric frame of reference. Results obtained with late-onset blind individuals suggest that early visual experience plays a key role in the development of both spatial hearing enhancements and deficits.

#### Edited by:

Matthew William Geoffrey Dye, Rochester Institute of Technology, USA

#### Reviewed by:

Maeve Mary Barrett, Georgetown University Medical Center, USA Andrew Joseph Kolarik, University of London, UK

### \*Correspondence: Patrice Voss

patrice.voss@mcgill.ca

#### Specialty section:

This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology

Received: 01 September 2016 Accepted: 30 November 2016 Published: 20 December 2016

#### Citation:

Voss P (2016) Auditory Spatial Perception without Vision. Front. Psychol. 7:1960. doi: 10.3389/fpsyg.2016.01960 Keywords: spatial hearing, vision disorders, blindness, auditory perception, critical period (psychology)

## INTRODUCTION

Our sense of vision provides us with the most detailed information about the spatial configuration of our environment. This visual dominance stems in part from the brain receiving high-resolution spatial information directly from the retina that is coded topographically throughout the visual pathway. While other modalities extract spatial information in a similar manner (e.g., tactile, vestibular, and proprioceptive modalities), they are body-centric and do not provide reliable information beyond personal and peripersonal space (i.e., beyond the reach of any limb). However, there are exceptions to this rule, such as when using sensory-substitution devices to translate visual information into tactile input that can be perceived, for instance, on either the tongue or the back (e.g., Bach-y-Rita et al., 1969; Chebat et al., 2007). The auditory system, like the visual system, also provides relevant spatial information regarding more distant regions of space. Localization information, however, is based on the detection and interpretation of auditory spatial cues that vary in their usefulness (for reviews, see Middlebrooks and Green, 1991; Schnupp et al., 2010).

Consequently, vision has often been thought to be essential for many aspects of spatial cognition and perception, and it has been often suggested that the absence of visual input might constitute a significant detriment to the ability to form accurate spatial representations. Two opposing views have emerged from early experimental findings (for review, see Rauschecker, 1995). The first view provides support for a perceptual deficit hypothesis whereby in the absence of visual input, individuals may develop cognitive spatial deficits in other sensory modalities (Axelrod, 1959; Jones, 1975). This hypothesis was supported by a large body of animal work that illustrated the importance of visual feedback in auditory spatial learning (Knudsen, 1985; King et al., 1988; Withington-Wray et al., 1990; Knudsen et al., 1991; Heffner and Heffner, 1992) and for the normal development of acoustic spatial maps in the superior colliculus (Knudsen, 1988; Withington, 1992; King and Carlile, 1993).

The opposing point of view supports a sensory compensation hypothesis whereby blind individuals develop exceptional perceptual abilities within their remaining sensory modalities to compensate for the visual loss (Rice, 1970; Miller, 1992). Pre-existing anecdotal support comes from, among others, both Denis Diderot (1749) in his Lettre sur les Aveugles and William James who dedicated a full chapter to this question in his 19th-century essay "The Principles of Psychology" (James, 1890). Experimental support was also provided by several animal (Rauschecker and Korte, 1993; King and Parsons, 1999) and human (Niemeyer and Starlinger, 1981; Muchnik et al., 1991) studies that reported enhanced sound localization abilities following prolonged visual deprivation. Subsequent studies provided evidence that corroborated the compensation hypothesis by demonstrating superior spatial hearing abilities in early blind individuals (Lessard et al., 1998; Röder et al., 1999; Leclerc et al., 2000; see also Voss et al., 2010 for a review), and supported the view that blind individuals can develop heightened abilities in their remaining sensory abilities. This hypothesis further gained traction with the growing body of evidence showing that these enhanced spatial hearing abilities are subserved by crossmodal plasticity (for reviews, see Collignon et al., 2009; Voss and Zatorre, 2012). Spatial hearing tasks have been shown to elicit significant activation within the visual cortex of early blind individuals (Weeks et al., 2000; Gougoux et al., 2005; Renier et al., 2010; Collignon et al., 2011), and individual localization abilities have been shown to strongly correlate with the magnitude of visual cortex activity (Gougoux et al., 2005; Voss et al., 2008, 2011). How auditory input comes to be processed in the visual cortex of the blind remains unclear, however, there is a growing body of animal tracer (Falchier et al., 2002, 2010; Clavagnier et al., 2004) and neuroimaging evidence (Klinge et al., 2010; Collignon et al., 2011, 2013) suggesting that corticocortical pathways between auditory and visual cortices may underlie the crossmodal processing. Despite this body of evidence supporting the compensation hypothesis, a clearer picture emerges when we take a closer look at the specific conditions under which enhanced spatial hearing abilities are observed.

### DIMENSIONS OF SPACE

Our spatial environment can be divided into distinct dimensions. With regards to spatial hearing, it is typically divided into the horizontal, vertical, and depth planes. The most studied dimension is the horizontal plane, likely due in part to its relevance for aurally localizing objects for navigation and wayfinding purposes. In the horizontal plane, the blind have been shown to possess similar spatial hearing abilities to the sighted in the frontal hemifield (e.g., within the region approximately overlapping the visual field). In contrast, however, the blind display more accurate localization in peripheral auditory space, particularly for sound sources straddling the interaural axis (Röder et al., 1999; Voss et al., 2004; Desprès et al., 2005).

A seminal study identified a marked difference in spatial hearing ability between early blind and sighted individuals when having to localize sounds under monaural listening conditions (e.g., with one ear occluded). Lessard et al. (1998) showed that the blind are significantly better at monaurally localizing sounds coming from sources ipsilateral to the occluded ear (see also Gougoux et al., 2005). This monaural superiority, combined with more accurate localization abilities in peripheral auditory space, point toward the better utilization of a specific set of localization cues by early blind individuals. Doucet et al. (2005) and Voss et al. (2011), using distinct but complementary methodologies, showed that a higher sensitivity to spectral cues likely underpins the superior localization abilities of the early blind in the horizontal plane. Spectral cues result from the location-specific head-dependant filtering of the incoming sound by the outer ear (Shaw, 1966). The resulting spectral profile is altered by the pinna in a manner that is specific to the direction of the incoming sound wave. While it has also been shown that the blind display higher sensitivity to binaural sound location cues compared to sighted individuals (Nilsson and Schenkman, 2016), such cues are absent in monaural listening conditions and are not always reliable in peripheral auditory space (Jin et al., 1999), suggesting that these binaural cues are unlikely to underlie the spatial hearing enhancements observed on the horizontal plane in blind individuals.

The spectral cue hypothesis, however, is challenged by spatial hearing findings in the vertical plane. Although localization ability in the vertical plane is also believed to rest primarily on spectral cues (Middlebrooks and Green, 1991; Blauert, 1997), early blind individuals have been shown to be worse than sighted individuals when localizing sound targets in the vertical mid-sagittal plane (Zwiers et al., 2001; Lewald, 2002). Voss et al. (2015) recently attempted to resolve this discrepancy by comparing the ability of early blind individuals to localize sounds in both the horizontal and vertical plane. The results confirmed both sets of previous findings: on average, the blind are better at localizing sounds monaurally in the horizontal plane and display deficits when localizing in the vertical plane. The novel finding, however, was that performance in both tasks was inversely correlated for the blind: those who displayed the highest accuracy in the horizontal plane were also the ones with the largest deficit when localizing in the vertical plane. Such a correlation was not observed in sighted individuals. This

finding not only argues against the idea of generalized auditory spatial perceptual enhancements in the blind, but also suggests the possibility of a trade-off in the localization proficiency between the two auditory spatial planes, such that learning to use monaural cues for localization in the horizontal plane comes at the expense of using them to localize in the vertical plane. What remains unclear, however, is why such a trade-off occurs. From an ecological perspective, the enhancements observed in the horizontal plane may result from their greater relevance for navigational and wayfinding tasks. With regards to underlying mechanisms, one potential explanation may stem from the type of spectral information that is being used for each specific plane. For instance, it has been previously argued that localization in the vertical plane relies primarily on spectral notch cues, whereas localization in the horizontal plane appears to depend on the analysis of covert spectral features. A reliable cue to estimate source elevation is provided by the center frequency of a spectrum notch, which has been shown to increase systematically from about 5 to 14 kHz with corresponding increases in elevation (Hebrank and Wright, 1974; Bloom, 1977; Butler and Belendiuk, 1977). In contrast, it has been suggested that the most reliable spectral cue for determining horizontal position comes from covert peak analysis, which requires the comparison of spectral features across several source locations (Musicant and Butler, 1984; Rogers and Butler, 1992). Furthermore, these cues have been shown to be particularly helpful for resolving source locations in peripheral auditory space (Musicant and Butler, 1984; Humanski and Butler, 1988). It is, therefore, possible that blind individuals may have learned to place greater emphasis on the analysis of covert spectral cues given their importance for establishing horizontal source position, whereas sighted individuals may have learned to pay more attention to spectral notch cues for vertical localization.

Auditory depth perception has not been as extensively studied as localization, but there are nonetheless some emerging trends (for a review, see Kolarik et al., 2016). Our ability to sense depth is essential for estimating the distance that separates us from auditory sources. When having to make relative depth judgments, early blind individuals have been shown to be more accurate that sighted individuals (Voss et al., 2004), likely due to a better use of level and direct-to-reverberant ratio (DRR) auditory cues (Kolarik et al., 2013a). In contrast, the blind have been shown to be worse when having to perform absolute distance judgments (e.g., estimate the distance that separates the observer from the source; Wanet and Veraart, 1985; Kolarik et al., 2013b). Why this discrepancy between relative and absolute judgments exists is not clear. In sighted individuals, the ability to accurately aurally perceive absolute depth is much poorer than the ability to localize sounds and to localize depth visually (Loomis et al., 1998). Furthermore, the presence of visual cues has been shown to substantially improve auditory depth estimation accuracy (Anderson and Zahorik, 2014). In the absence of vision, the ability to estimate the absolute depth of sound sources might be compromised due to the lack of visual calibration of auditory spatial representations. The ability to perform relative depth judgments, however, depends primarily on the comparison of auditory cues (level or DRR) and therefore is likely not compromised by the lack of visual feedback. Therefore, spatial inferences resulting from the processing of auditory cues might be less affected by visual loss than the development of auditory spatial maps.

Evidence from echolocation studies, however, supports the idea the developing auditory spatial maps does not require visual input (for review, see Kolarik et al., 2014). Blind individuals often make use of these cues when navigating in unfamiliar environments by either passively listening to or actively creating reflecting sounds (e.g., by tapping a cane or making clicking noises). Research has shown that blind individuals are not only more sensitive to non-generated echo cues (Dufour et al., 2005; Kolarik et al., 2013a) but have also developed superior abilities to use self-generated sounds to localize objects in the environment (Rice et al., 1965; Schenkman and Nilsson, 2010, 2011).

### FRAMES OF REFERENCE

An alternative perspective with which we can examine the role played by vision in spatial hearing is to examine the frame of reference that is best suited to or necessary to carry out a spatial hearing task. In spatial cognition research, a reference frame defines a means of representing the locations of entities in space. The two dominant reference frames are the allocentric and the egocentric frames of reference (for reviews, see Kosslyn, 1987; Paillard, 1991; Klatzky, 1998). Egocentric frames of reference use the body or body parts as the center of the environment, whereas allocentric frames of reference are centered on external objects or on the environment itself. Multiple reports in the spatial cognition literature have suggested that, in the absence of vision, individuals primarily rely on egocentric frames of reference to carry out spatial tasks (Millar, 1994; Cattaneo et al., 2008; Coluccia et al., 2009; Corazzini et al., 2010; Pasqualotto and Proulx, 2012).

Gori et al. (2014) were among the first to provide evidence of an allocentric deficit related to spatial hearing in the blind. Early blind individuals were shown to be severely impaired when having to perform an auditory spatial bisection task in the horizontal plane. Participants had to determine whether the spatial location of a third sound source was closer to one or the other of the first two presented sound source locations. This task requires a spatial judgment that might be more anchored to an allocentric frame of reference that depends on two external auditory landmarks. In contrast, more traditional sound localization tasks can be resolved by using an egocentric frame of reference, since subjects need no other reference point other than their own position in space. This allocentric spatial hearing deficit was subsequently confirmed by several follow-up studies in both blind adults and children (Vercillo et al., 2015, 2016), suggesting the existence of dichotomic spatial hearing abilities in the early blind that depend on the frame of reference that is best suited to carry out a spatial task. Interestingly, however, Vercillo et al. (2015) also showed that blind echolocators were not impaired when having to perform a spatial bisection task, which suggests that the development of echolocation abilities may improve auditory spatial representations or the use of allocentric frames of reference.

### INSIGHTS FROM SPATIAL COGNITION STUDIES

fpsyg-07-01960 December 17, 2016 Time: 17:38 # 4

Support for an allocentric deficit in the early blind is provided by spatial cognition research. The majority of studies investigating the wayfinding and spatial navigation abilities of blind individuals, in particular, have provided consistent findings (for review, see Thinus-Blanc and Gaunet, 1997). For sighted individuals, wayfinding and spatial navigation have been shown to rely on both egocentric and allocentric frames of references (Millar, 1994; Klatzky, 1998; Shelton and McNamara, 2001). It is estimated that about half the population spontaneously uses an egocentric frame of reference while the other half uses an allocentric frame of reference (Iaria et al., 2003). While the early blind tend to perform tasks requiring an egocentric frame of reference as well as sighted controls (Millar, 1994; Tinti et al., 2006; Fortin et al., 2008), they generally show difficulties when the use of an allocentric frame of reference is required (Thinus-Blanc and Gaunet, 1997; Schmidt et al., 2013). Overall, the allocentric deficit observed in spatial hearing tasks is in good agreement with the findings available in the spatial cognition literature. It should be noted, however, that a recent review does provide arguments that cast doubt over the idea of a general allocentric deficit in the blind (Schinazi et al., 2016), although these concerns apply primarily to contexts not related to spatial hearing.

### INSIGHTS FROM BLIND INDIVIDUALS WITH EARLY VISUAL EXPERIENCE

The study of the effects of late-onset of blindness may provide additional valuable information on the mechanisms that govern the development of spatial hearing in the absence of vision. The spatial hearing of late blind individuals' was shaped by the unique combination of visual calibration during development and prolonged blindness in adulthood. The limited available evidence suggests that the spatial hearing abilities of late-blind individuals lie somewhere in between those of early blind and sighted individuals (for a review, see Voss, 2013). Unlike the early blind, there is little evidence to suggest that late blind individuals have enhanced spatial hearing abilities relative to sighted individuals. However, the evidence from studies assessing localization abilities in the horizontal plane suggests that late blind individuals may also make better use of spectral cues to localize in peripheral space (Voss et al., 2004; Fieger et al., 2006). However, there is no evidence of enhanced monaural localization abilities in the late blind, suggesting that a different explanation likely underlies their ability to localize in peripheral space (Voss et al., 2008, 2011). Overall, it would seem that late-blind individuals do not benefit from many of the spatial hearing enhancements observed in the early blind, nor do they exhibit any perceptual deficits either (e.g., Finocchietti et al., 2015). Research into how late blind individuals localize sounds on the vertical plane and their performance on spatial bisection tasks is lacking and may provide a more complete picture of spatial hearing abilities in this population. A study by Pasqualotto et al. (2013) provided evidence that late-blind individuals employ an allocentric frame of reference when completing spatial tasks whereas early blind individuals employed an egocentric frame of reference. Deficits in using allocentric frames of reference to complete spatial tasks may not appear in late-blind individuals as they can encode spatial information through auditory channels while simultaneously benefitting from the calibration obtained via previous visual experience (Ruggiero et al., 2009; Iachini et al., 2014).

### CONCLUSION AND FUTURE DIRECTIONS

A complex relationship exists between spatial hearing and vision. Early theories proposed that blind individuals could either develop superior spatial hearing abilities to compensate for visual loss or, in contrast, demonstrate spatial hearing deficits due to a lack of calibration from the visual system. Currently available evidence suggests that both propositions are likely true. Early blind individuals are as accurate, if not more accurate, than sighted individuals when having to localize sound sources on the horizontal plane but show deficits when localizing sound sources on the vertical plane or when estimating the absolute distance of auditory cues. In fact, recent evidence suggests a trade-off in the localization proficiency of early blind individuals between the horizontal and vertical planes (Voss et al., 2015), such that the more accurate an individual is in one plane, the worse that same individual is in the other plane. Why this trade-off occurs remains unclear, but it might stem from the greater ecological benefit of being accurate in the horizontal plane. When spatial hearing tasks require the use of an egocentric frame of reference, early blind individuals display comparable or superior perceptual abilities. However, they display significant deficits when the use of an allocentric frame of reference is required (e.g., for a spatial bisection task).

Evidence obtained with late-blind individuals suggests that while the presence of visual input early in life prevents the development of spatial hearing deficits, it also limits the emergence of perceptual enhancements. Several aspects of spatial hearing, however, remain to be investigated in the late blind. In particular, data is lacking concerning their ability to localize sounds in the vertical plane and to perform auditory spatial bisection tasks. Predictions can still nonetheless be made based on available evidence. For instance, a trade-off in localization proficiency between the horizontal and vertical planes would probably not be observed given that late-blind individuals do not exhibit better monaural localization abilities (Voss et al., 2006, 2008, 2011). Similarly, based on the lack of evidence supporting allocentric deficits in the late blind (Pasqualotto and Proulx, 2012), they should not show deficits when performing auditory spatial bisection tasks.

Despite the evidence accumulated in recent years, it remains unclear to what extent the described perceptual consequences of early blindness translate to real-world settings. Most of the presented findings have been observed under experimental conditions with limited ecological validity. To properly ascertain the real-world abilities of individuals with complete blindness, there is a need to evaluate more ecologically relevant and useful

metrics. The ability to track dynamic sounds in space, for instance, and the ability to localize sounds in noisy environments appear to be important elements to investigate. While a few studies have started to investigate these metrics (Lewald, 2013; Finocchietti et al., 2015), it remains difficult, however, to draw any reliable conclusions given the limited amount of data available. Similarly, most spatial hearing experiments have been performed in anechoic environments, which are rarely found outside of the laboratory. As highlighted earlier, the fact that blind individuals are more sensitive to echoic cues (Dufour et al., 2005) means that their performance on spatial hearing tasks in an echoic environment maybe enhanced compared to an anechoic environment. Although there is evidence that blind individuals can better extract speech information from noise more efficiently than sighted controls (Rokem and Ahissar, 2009), their ability to localize sounds in the presence of background noise has not been thoroughly investigated.

Blindfolding sighted subjects might put them at a disadvantage compared to blind individuals and also might artificially inflate the difference in performance between them. Tabry et al. (2013) showed that blindfolding reduces localization accuracy in sighted individuals, and this was particularly true for the horizontal plane (compared to the vertical plane) and when

### REFERENCES


having to localize sounds via head pointing (compared to hand pointing). As such, great care should be taken when designing experimental procedures for assessing certain spatial dimensions to reduce the impact of methodologically induced biases on the results.

Finally, the finding that short-term visual deprivation (as little as a few hours) in sighted individuals can improve auditory localization (Lewald, 2007) is at odds with the lack of documented improvements observed in late-blind individuals (Voss, 2013). While the spatial hearing benefit of transient visual deprivation is consistent with data investigating other sensory abilities (Facchini and Aglioti, 2003; Landry et al., 2013; Pagé et al., 2016), it is currently the only study having looked at spatial hearing. Further studies are required to ascertain the underlying processes involved in spatial hearing enhancements caused by visual loss and to further our understanding of the effects of transient sensory deprivation.

### AUTHORS CONTRIBUTIONS

The author confirms being the sole contributor of this work and approved it for publication.



**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Voss. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

fpsyg-07-01960 December 17, 2016 Time: 17:38 # 7

# Shape Perception and Navigation in Blind Adults

#### Monica Gori<sup>1</sup> \*, Giulia Cappagli<sup>1</sup> , Gabriel Baud-Bovy2,3 and Sara Finocchietti<sup>1</sup>

<sup>1</sup> Unit for Visually Impaired People, Istituto Italiano di Tecnologia, Genoa, Italy, <sup>2</sup> Robotics, Brain and Cognitive Science Department, Istituto Italiano di Tecnologia, Genoa, Italy, <sup>3</sup> The Unit of Experimental Psychology, Division of Neuroscience, IRCCS San Raffaele Scientific Institute, Vita-Salute San Raffaele University, Milan, Italy

Different sensory systems interact to generate a representation of space and to navigate. Vision plays a critical role in the representation of space development. During navigation, vision is integrated with auditory and mobility cues. In blind individuals, visual experience is not available and navigation therefore lacks this important sensory signal. In blind individuals, compensatory mechanisms can be adopted to improve spatial and navigation skills. On the other hand, the limitations of these compensatory mechanisms are not completely clear. Both enhanced and impaired reliance on auditory cues in blind individuals have been reported. Here, we develop a new paradigm to test both auditory perception and navigation skills in blind and sighted individuals and to investigate the effect that visual experience has on the ability to reproduce simple and complex paths. During the navigation task, early blind, late blind and sighted individuals were required first to listen to an audio shape and then to recognize and reproduce it by walking. After each audio shape was presented, a static sound was played and the participants were asked to reach it. Movements were recorded with a motion tracking system. Our results show three main impairments specific to early blind individuals. The first is the tendency to compress the shapes reproduced during navigation. The second is the difficulty to recognize complex audio stimuli, and finally, the third is the difficulty in reproducing the desired shape: early blind participants occasionally reported perceiving a square but they actually reproduced a circle during the navigation task. We discuss these results in terms of compromised spatial reference frames due to lack of visual input during the early period of development.

Keywords: shape, perception, audio perception, blindness, motor, navigation

### INTRODUCTION

While navigating, sighted individuals rely on both visual and non-visual sensory information. Multisensory integration plays a critical role in the representation of space development. Sensory information converges onto multisensory areas of the parietal cortex for a representation of the internal and external space (Sereno and Huang, 2014) and this is critical for successful navigation.

Thanks to multisensory integration, it is possible to update the position of the body in the space and to orient ourselves in the environment (Loomis et al., 1993; Thinus-Blanc and Gaunet, 1997; Fetsch et al., 2009; Butler et al., 2010; Prsa et al., 2012; Schmidt et al., 2013).

During navigation, vision is important because it provides information about both the traveler's motion and the layout of near and far space (Blasch and Welsh, 1980; Foulke, 1982; Strelow, 1985;

#### Edited by:

Olivier Pascalis, Université Grenoble Alpes, France

#### Reviewed by:

Martina Manns, Ruhr University Bochum, Germany Famira Racy, Adler University, Canada

> \*Correspondence: Monica Gori monica.gori@iit.it

#### Specialty section:

This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology

Received: 02 August 2016 Accepted: 03 January 2017 Published: 17 January 2017

#### Citation:

Gori M, Cappagli G, Baud-Bovy G and Finocchietti S (2017) Shape Perception and Navigation in Blind Adults. Front. Psychol. 8:10. doi: 10.3389/fpsyg.2017.00010

Loomis et al., 1993). The visual system also provides more accurate information on distance than the auditory system (Loomis et al., 1998). In the absence of vision, such as in blindness, navigation capabilities may result compromised. Some studies support this view, showing impaired skills in blind individuals, associated with inferential navigation (Seemungal et al., 2007) and lower sensitivity to changes in perspective structure when walking without vision (Rieser et al., 1986). They also show slower walking speed, cautious posture, shorter stride length and longer duration of stance compared to sighted and late blind individuals (Nakamura, 1997). On the other hand, some other skills related to spatial navigation are intact even when visual input is missing (for a review Thinus-Blanc and Gaunet, 1997; Seemungal et al., 2007).

During development, vision plays a key role in aligning neural representations of space in the brain (King, 2009, 2014). Multisensory integration areas (including regions underlying audiovisual spatial processing) are significantly reorganized when visual input is absent. Early visual deprivation impacts on the topographic organization of the auditory receptive fields of superior colliculus neurons (King, 2009). Bias in auditory localization has been shown in owls reared with distorting prisms (Knudsen and Knudsen, 1985) and total visual deprivation of young ferrets has been shown to cause disordered development of auditory spatial maps (King and Carlile, 1993). Comparable (but transitory) effects have also been demonstrated in humans (Recanzone, 1998; Zwiers et al., 2003). In the case of blindness, compensatory mechanisms can be adopted to improve spatial skills (Merabet and Pascual-Leone, 2010). This plasticity allows the visual cortex in the congenitally blind individual to become colonized to some extent by the auditory and somatosensory systems (e.g., Sadato et al., 1996; Weeks et al., 2000). As result, a strong and reliable response to sound alone in the primary visual cortex has been observed in blind individuals, using fMRI (Roder et al., 2002; Amedi et al., 2007; Bedny et al., 2011; Lane et al., 2015) and EEG (Kujala et al., 1995; Focker et al., 2012).

Hearing can compensate for the absence of vision by providing inputs for spatial perception in the near and far space because it covers a larger spatial field compared to other senses (Kolarik et al., 2016) and by using allocentric perception of the surrounding space (relative to external objects) rather than egocentric (relative to the observer; Loomis et al., 2001). Psychophysical evidence suggests that some tactile and audio skills in congenital blind individuals result enhanced (e.g., Lessard et al., 1998; Roder et al., 1999; Goldreich and Kanics, 2003; Gougoux et al., 2004; Tinti et al., 2006; Fortin et al., 2008), such as the ability to localize a sound source in the horizontal plane (Lessard et al., 1998; Gougoux et al., 2004; Doucet et al., 2005) or discriminate between different sounds (Gougoux et al., 2004). Studies of animals confirm this view by suggesting that sound processed by neurons in auditory cortical areas can be enhanced following visual deprivation (Korte and Rauschecker, 1993; Petrus et al., 2014).

On the other hand, recent psychophysical works have pointed out that some forms of auditory perception in visually impaired individuals result compromised, raising some doubts about the degree and the limits of cross-modal plasticity in the case of sensory loss. Blind individuals are impaired in complex skills such as sound localization in the mid-sagittal plane (Zwiers et al., 2001; Lewald, 2002; Voss et al., 2015) and tasks requiring a metric representation of the auditory space (Gori et al., 2014; Finocchietti et al., 2015). This impaired audio space perception in blind individuals can hamper the compensation during navigation provided by audition in the absence of vision.

In everyday life, it is fundamental for blind individuals to decode static sounds (e.g., a telephone) and dynamic sounds (e.g., people walking around while speaking), but especially to navigate toward them and engage in interaction. Understanding how static and dynamic sounds are perceived and how they are interpreted during navigation is therefore of fundamental importance. The studies reported above suggest that both impaired auditory representations and impaired navigation strategies might affect everyday interaction of visually impaired individuals. One of the more compromising issues that they encounter is the association between decoded audio signals and navigation strategies to reach the target. What are the internal processes that link auditory perception and navigation in blind individuals? Does past visual experience influence their development? Previous studies have shown that lack of visual experience impacts on navigation skills (Rieser et al., 1986) and on complex auditory perception (Gori et al., 2014; Finocchietti et al., 2015). Starting from this evidence, we hypothesize that the integration between auditory perception and motor responses in blind individuals could be compromised, giving rise to impairments in navigation. Moreover, given that vision is fundamental for space development (King, 2009, 2014), we also hypothesize that prior visual experience should shape this integrative audiomotor process, producing different performances in early and late blind individuals. These two hypotheses were addressed by investigating the relative role of auditory perception, navigation and past visual experience on path reproduction and on path recognition abilities. We evaluated the ability of sighted and blind individuals in recognizing a sonorous stimulus in a localization task involving auditory static sources and dynamic geometrical auditory pathways. To evaluate the ability to reproduce dynamic audio pathways and to reach static sounds we analyzed the movement pattern on both tasks. To evaluate the integration between auditory perception and motor responses we asked participants to report the shape perceived and afterward to reproduce the perceived sonorous pathway by walking. To highlight the role of past visual experience on the development of navigation and auditory perception, we compared the performance of sighted, early blind and late blind participants.

### MATERIALS AND METHODS

### Participants

Ten visually impaired and 10 sighted individuals participated in the experiment. In the group of blind participants, seven were congenitally blind and three were acquired blind (see **Table 1** for clinical details). All the participants had a similar level of education (at least an Italian high school diploma, indicating 13 years of school). All the early blind participants were blind

#### TABLE 1 | Clinical details of the early blind (EB) and late blind (LB) participants.


The table shows age at the time of testing, gender, pathology, and age at which they became completely blind.

at birth. None of the participants had any history of hearing impairment. Blind participants were contacted from a list of participants that had taken part in our previous experiments. Sighted participants were contacted from a list of participants that had asked to take part in our studies.

### Materials and Procedures

Each participant was asked to reproduce four shapes in different orientations for a total of 30 shape combinations, to categorize all the shapes previously heard, and to walk toward 30 static sound sources, for a total of 90 trials per participant (see **Figures 1** and **2**). The four shapes considered in the study were a square with an area of 4 m<sup>2</sup> , a triangle with an area of 2 m<sup>2</sup> , a triangle with an area of 3 m<sup>2</sup> , and a circle with an area of 3.14 m<sup>2</sup> . The sound was a single pulse at 500 Hz, intermittent sound at 180 bpm, as previously used by our group (Finocchietti et al., 2015). The experiment was performed in a dark room, with low light allowing the experimenter to see, but at the same time not perceivable by the blindfolded participants. When the participants arrived, we briefly outlined the experiment and they were asked to read and sign the written informed consent form. For the blind participants, the ethical documentation was read by the experimenter. Before the testing, all participants were blindfolded, guided into a motion recording room and positioned at the starting point. The room had a rectangular floor (300 × 200 cm) that was defined by the recording space that our motion tracking system could cover (Vicon Motion Systems Ltd. UK). Ten different landmarks and relative connections between each other were marked with colored tape on the floor (**Figures 1** and **2**). A spherical marker was mounted on a hat positioned on the head of both the participant and the experimenter for motion tracking. In addition to the marker, the experimenter also had an audio source positioned in the hat. Three experimenters instructed the participant and performed all the experiments (MG, GC, and SF). The experimenters had been previously trained to perform the task as uniformly as possible, so that the movement velocity was consistent across trials, positions, and groups. The participants were randomly assigned to one of the three experimenters. The participant was positioned at the starting point, corresponding to the initial position of the first shape (**Figure 2** on the top left, shape 1). Each trial was structured in three phases as follow:

### Phase 1

The experimenter navigated following the first shape reported in **Figure 2**. After having heard the audio motion (**Figure 1A**), the participant had to report which shape was produced by the experimenter. The participant knew in advance that the navigation could follow a square, circular, or triangular path.

### Phase 2

After having reported the shape perceived, the participant had to navigate to reproduce the path previously heard and return to the initial position (**Figure 1B**). The participant was always positioned facing the first motion direction of the geometric shape.

### Phase 3

The experimenter moved toward a new starting position (reported in **Figure 2** as the starting point in the second circular shape, red dot) and produced a 1 s static sound. The participant had to move toward the sound source (thus navigating from the circular open to the circular closed red dot) to start the next trial. In this phase the participant was not oriented by the experimenter toward the static sound but had to orient himself/herself by rotating his/her body and then walking, starting from the final orientation and position reached with the previous navigation trial. All the shapes in **Figure 2** were reproduced in the same order. The static sounds to be reached have been reported for the first 10 shapes. Before starting a new trial, the participant was always oriented frontally toward the direction of the first segment of the shape (indicated by the arrow in **Figure 2**).

### Data Analysis

Kinematic data were post-processed and analyzed using Matlab (R2013a, The MathWorks, USA). Spatial accuracy and localization error during navigation was computed for each participant and for each spatial position (see **Figure 1B** as example of motion tracks for three participants). The area of the shape produced was computed by considering the limits of the trajectories performed by the participant. The end-point was calculated for both shape reproduction and static sound reaching. Each end-point position (x\_pos, y\_pos) was computed as the average of the last 10 samples and normalized on the final position (the end of the shape or the end of the linear trajectory toward the static sound), in order to avoid movement errors. Area values were averaged based on the number of participants for each of the 30 shapes and normalized by dividing it by the actual shape area (**Figure 3**). For each shape, the localization error was calculated as the Euclidean distance (in mm) between the position reached by the participant at the end of reproduction and the correct final position reached by the experimenter (**Figure 4**). For the static sound, the localization error was calculated as the Euclidean distance (in mm) between

FIGURE 1 | Representation of the protocol. (A) Experimenter moved along a path by moving an audio source. (B) The participant, after hearing to the audio information, had to reproduce the path. (C) The dimension of the surface where the shapes were produced was 200 cm × 150 cm. (D) A motion tracking system was used to record the motion trajectories. Three examples trajectories are reported for an early blind, late blind and sighted participant.

the final position reached by the participant and the position where the static sound source was positioned (**Figure 5**). Correct sound localization was defined as the difference from the experimenter and participant categorization was used for further analysis. Shape recognition data were recorded by collecting the verbal responses of the participants. Data were normally distributed, confirmed by visual inspection of Q–Q plots. Data are presented as mean and standard error (SE). Absolute area and normalized area and localization error for shape reproduction trials were analyzed with three repeated-measure ANOVA with shape (circle, square, triangle1, triangle2) as within-participant factor and group (EB, LB, sighted) as between-participant factor.

reported with symbols on the abscissa. (C) Normalized area (mean ± SD) for early blind (dark gray), late blind (gray), and sighted participants (light gray) for all the shapes considered.

green) for all the shapes considered. (B) Average errors made, calculated by merging all the 30 shapes for each participant group. (C) Average of the errors made, for the three groups and the four categories of shapes.

The localization error for static sound was analyzed with a one-way ANOVA with group (EB, LB, sighted) as factor. The percentage of correct responses and velocity of movement were analyzed with a one-way ANOVA with group (EB, LB, sighted) as factor. Olejnik and Algina (2003) generalized Eta (ηG) was used to compute effect size. Tukey HSD was used to test significance of multiple comparison post hoc tests. Data from late blind individuals, even if only collected in three subjects, were analyzed separately from sighted and early blind to show a specific role of visual experience on this task. Visual inspection of the distribution of the residuals against the fitted values and of quantile–quantile plots indicated that the basic requirements of ANOVAs (homoscedasticity and normality) were satisfied.

### RESULTS

### Navigation Responses

This section reports the results of the navigation skills of early blind, late blind and sighted individuals in reproducing the audio shapes and in reaching static sounds. **Figure 3A** reports the areas for all the shapes and **Figure 3B** the average area reported for the three groups. Sighted and late participants (in yellow and red) performed the task correctly while early blind participants (in blue) show a general compression of the surrounding space (correct area for each shape is reported with symbols on the abscissa). Statistical analysis (ANOVA) revealed a very large group effect [F(2,17) = 2701.4, p < 0.01, η<sup>G</sup> = 0.89]. The shape effect and interaction were also statistically significant but had a much smaller effect [F(3,51) = 137.9, p < 0.01, η<sup>G</sup> = 0.06 and F(6,51) = 104.4, p < 0.01, η<sup>G</sup> = 0.10, respectively]. On average, the area of the shapes reproduced by the EB was 55% the size of the reference area (**Figure 3C**). In contrast, the control and LB groups reproduced shapes that were only slightly smaller than the reference shapes (95 and 93%, respectively, **Figure 3C**). Tukey HSD post hoc analyses between the group levels for significance indicated that the average area reported between early blind participants were significantly lower (in blue, M = 1.54, SE = 0.07) than that one of the late blind (in yellow, M = 2.6, SE = 0.06, p < 0.001) and sighted group (in red, M = 2.68, SE = 0.04, p < 0.001), which did not differ statistically from each other. These results suggest that early blind individuals show some navigation impairments compared with sighted and late blind individuals when they have to reproduce a previously heard shape but not when they have to reach a static sound. Furthermore, the localization error to close the shape (i.e., on reaching the initial starting point) was higher for all the shapes in early blind than in sighted and late blind individuals (**Figure 4**). **Figure 4A** reports the error in closing the shapes created. This error represents the difficulty of localizing the initial starting point. On average (**Figure 4B**), sighted and late blind individuals (in yellow and in red) are also significantly more precise in this task compared to early blind individuals (in blue). Statistical analysis (ANOVA) showed a Main effect for participant group [F(2, 17) = 65.409, p = 0.001, η<sup>G</sup> = 0.69]. In early blind individuals the error to close the shape is higher for all the shapes considered (**Figure 4C**). Again, both the shape and interaction are statistically significant but the effect size much smaller [Shape: F(3,51) = 11.715, p < 0.001, η<sup>G</sup> = 0.09; Group × Shape interaction: F(6,51) = 5.10, p < 0.001, η<sup>G</sup> = 0.08]. Post hoc tests (Tukey HSD) also indicated that the early blind group (M = 234.9, SE = 16.2) differed from the late blind group (M = 165.7, SE = 9.8, p < 0.001) and the sighted group (M = 176.6, SE = 13, p < 0.001). **Figure 5** represents the error associated with the localization of static audio sources. Interestingly, early blind individuals showed no deficit in reaching static sounds (**Figure 5**). In contrast to the shape reproduction task, all participants localized the static audio information in a similar way (**Figures 5A,B**) [one-way ANOVA, F(2,77) = 0.206, p = 0.815]. These results suggest a navigation problem in early blind individuals only when they are asked to reproduce an auditory geometric path, not when they are asked to reach a static audio source.

### Audio Shape Recognition

This section reports the results of geometrical auditory shape recognition and shape motor reproduction. **Figure 6A** reports the percentage of correct responses for the square, circular, and triangular shapes listened. Early blind individuals fail to correctly perceive the shapes, especially triangles and squares, for which barely 30% of responses were correct [**Figure 6A**,

one-way ANOVA, F(2,17) = 126.4933, p < 0.001]. A Tukey post hoc test revealed that it was statistically significantly higher for the early blind group (Squares: M = 30, SE = 5.8; Circles: M = 57.6, SE = 4; Triangles: M = 26.62, SE = 3.1) compared to the late blind (Squares: M = 77.8, SE = 5.2; Circles: M = 88.8, SE = 8; Triangles: M = 75.9, SE = 2.7, p < 0.001) and the sighted group (Squares: M = 83.3, SE = 10.4; Circles: M = 87.5, SE = 4; Triangles: M = 87.9, SE = 3.4, p < 0.001). **Figure 6B** reports the probability of perceiving the three categories of shapes and **Figure 6C** indicates the probability of reproducing the three categories of shapes for the three groups of participants, compared with the real probability that the specific shape occurred (in gray). The primarily motor response of early blind individuals is the circle, with a probability of over 0.6.

While sighted and late blind individuals are very good at recognizing all the different shapes (**Figure 6**), early blind individuals' performance is worse for all the three shapes considered. The evidence that they better recognize the circles (**Figure 6A**) is probably due only to their tendency to frequently report the circle shape (**Figure 6B**). Interestingly, if we compare their perceptual response (**Figure 6B**) and their navigation reproduction (**Figure 6C**) we can observe a similar pattern. Contrarily to sighted (in yellow) and late blind (in red) individuals, who perceived and reproduced all the shapes correctly (gray bar), early blind (blue) individuals tended to report and to reproduce more circles. Results suggest that the probability of perceiving circles was different among the three groups for the perceptual circular responses [**Figure 6B**, one-way ANOVA, F(2.17) = 104.18615, p < 0.001] and motor circular responses [**Figure 6C**, one-way ANOVA, F(2.17) = 126.4933, p < 0.001]. A Tukey post hoc test (Tukey HSD) revealed that it was statistically significantly higher for the early blind compared to the late blind (p < 0.001) and to the sighted group (p < 0.001) for both perceptual and motor tasks.

### Perceptual vs. Navigation Response

This section reports the results on associating perceptual and motor responses in the three groups. To evaluate this, we reported in **Figure 7** the matrices of confusion for both the perceptual task (upper line) and the motor task (lower panel). While identical patterns can be observed for sighted and late blind individuals in both tasks, early blind individuals show a different pattern of perceptual and motor responses, suggesting a mismatch between perceptual audio shape recognition and navigation shape reproduction. In late blind and sighted individuals, responses show a red equality line, suggesting that the shapes reported and reproduced are the same as the shape presented. The results of early blind individuals did not show the red equality line, but there were more responses associated with the circular shape in both the perceptual and the motor responses. This is particularly true for the square shape (**Figure 7**, left column). Early blind individuals are more likely to report perceiving a square when they actually hear a square. On the other hand, although they intended to reproduce a square, they actually reproduced a circular shape (see the first column of the matrix). This mismatch points to a third problematic aspect, i.e.,

FIGURE 7 | Matrices of confusion: association between motor and perceptual responses. (A) Matrices of confusion for the motor responses and the three categories of shapes used. (B) Matrices of confusion for the perceptual responses and the three categories of shapes used. Responses of early blind participants are reported in the first column, of late blind in the second column, and of sighted in the last column. The level of probability is associated with the colored scale presented: red means high probability and blue low probability of reporting the specific shape, middle colors indicate intermediate probability.

the inability of early blind individuals to navigate and reproduce the desired path.

### DISCUSSION

To highlight possible differences among the three groups, we also report in **Figure 8A** the walking speed of participants for all the different shapes and groups. **Figure 8B** reports the average velocity for the three groups. Results suggest that walking speed was different among the three groups [one-way ANOVA, F(2.17) = 59.00275, p < 0.001]. A Bonferroni post hoc test revealed that the walking speed to reproduce a shape was statistically significantly higher for the early blind (M = 5.6, SE = 0.04) compared to the late blind (M = 5.3, SE = 0.04, p < 0.001) and to the sighted group (M = 5.3, SE = 0.01, p < 0.001). The speed of the sighted and late blind group was not significantly different (p = 0.78). The different walking speed among groups could be another aspect that might negatively affect their navigation capabilities.

In this work, we have investigated how spatial representation of the environment is shaped by sensory experience by studying cross-modal plasticity mechanisms in early and late blind individuals during navigation tasks. The lack of visual experience in blind individuals impacts on navigation skills (Rieser et al., 1986) and on complex auditory perception (Gori et al., 2014; Finocchietti et al., 2015). Starting from this evidence, we hypothesize that integration of auditory and motor responses could be compromised in blind individuals. This could give rise to impairments in navigation. Since vision is fundamental for space development (King, 2009, 2014), we also hypothesize that visual experience in the first period of life could be important for audio-motor integration development. If this is true then a different performance in early and late blind

individuals should be observed. Both hypotheses have been confirmed by our study. Our first hypothesis was supported by audio shape recognition and navigation deficits observed in early blind participants. Our second hypothesis was supported by the fact that the deficit was specific for early and not late blind individuals, who perform all the tasks in a similar way to sighted individuals. In general, our study indicates that plasticity mechanisms in blind individuals might not fully compensate for lack of vision: some sensory-motor skills not directly dependent on visual experience are correctly acquired by blind individuals; some other sensory-motor skills directly dependent on visual experience are compromised, probably because their acquisition requires the development of dedicated neural networks which rely on visual input.

### Audio Shape Recognition

In support of our first hypothesis, firstly, we observed a deficitary pattern of auditory spatial analysis or representation of dynamic auditory geometric cues for spatial navigation in early blind participants. Previous works, however, have shown superior processing of audio motion (e.g., Lewald, 2013; Jiang et al., 2016) for artificially moving stimuli positioned in front of early blind individuals. We think that the performance observed in our auditory task can be explained with two main considerations: Firstly, the stimulus presented in our study needed to be integrated for longer spatial and temporal windows than the stimuli used in previous experiments. Secondly, the task required the participant to build a complex metric representation of the space by memorizing and comparing the segments presented in each geometric figure. Both these aspects may require an Euclidian representation of space that has been shown to be compromised in blind individuals (Gori et al., 2014; Finocchietti et al., 2015; Gori, 2015; Vercillo et al., 2015, 2016). Finally, another aspect to be considered is that the task required the participant to construct a spatial representation not only by discriminating basic directional changes (left and right) but also by sequentially updating angular patterns derived from the shape. Early blind individuals could have a deficit in recognizing changes in audio movement direction. Early blind individuals tend to perceive and reproduce a circular path even when presented with angular shapes. The circular is represented with a continuous sound without the pauses associated with angular paths. More studies would be necessary to clarify the difficulty of early blind individuals in processing changes in audio movement direction. As previously mentioned, vision plays a dominant role in aligning neural maps of space in the midbrain during development (King, 2009). Auditory maps are usually shaped to match the visual field representation in the superior colliculus (King, 2014). The deficit observed here could be attributed to the lack of visual input on the natural integration process that permits us to associate signals provided by the eyes and by the ears about a common stimulus source. Interestingly, our results suggest that the lack of such refined maps does not compromise the ability to correctly perform activities related to spatial skills such as the localization of static sounds in space. Indeed, early blind participants performed as well as sighted individuals and late blind individuals in the localization of static sound sources. This result is in accordance with the idea that compensatory processes can be present in the absence of vision and that cortical reorganization may enhance some kind of auditory spatial performance in blind individuals (Collignon et al., 2009). It is also in accordance with other previous works (Ashmead et al., 1989; Lessard et al., 1998; Roder et al., 1999; Lewald, 2002, 2013; Voss et al., 2004; Kolarik et al., 2013) which suggest that blind individuals have normal or supra-normal sound localization abilities compared to sighted participants (Collignon et al., 2009; Voss et al., 2015) and that they show more accurate information with regard to distance (Zahorik, 2001; Kolarik et al., 2013, 2016).

### Navigation Responses

Secondly, we observed a different navigation pattern in early blind individuals: (i) they compress the shapes to be reproduced during navigation, (ii) they tend to reproduce more circular paths and, (iii) they tend to move faster. We think that all these aspects can be associated with an egocentric frame of reference adopted by blind individuals during navigation. Spatial navigation can be differentiated in route navigation, that refers to egocentric coordinates, and in inferential navigation that relies on allocentric coordinates (Loomis et al., 1993; Thinus-Blanc and Gaunet, 1997; Schmidt et al., 2013). Visual information in the first period of life may provide a contextual frame of reference to encode stimuli using allocentric (relative to external objects) rather than egocentric (relative to the observer) coordinates (Pasqualotto and Proulx, 2012). Some studies support this view, showing that blind individuals rely more on route navigation which is based on kinematic strategies relative to the experienced movement by using an egocentric reference (Bigelow, 1992). They encode stimuli using allocentric rather than egocentric, coordinates (Roder et al., 2007, 2004; Pasqualotto and Proulx, 2012). Impaired abilities in blind individuals associated with inferential navigation (Seemungal et al., 2007) and lower sensitivity to changes in perspective structure when walking without vision (Rieser et al., 1986) have been also reported. We think that spatial impairments delineated in this work can be associated with the use of a route navigation strategy by early blind individuals. This could indeed explain why they tend to be more likely to perform circular paths than angular paths. To perform a circular path they can rely on vestibular and proprioceptive information that is not impaired (Valko et al., 2012; Moser et al., 2015): they can internally set a specific rotation, speed and acceleration and maintain it during the entire path. Contrarily, in the square and triangular shapes, early blind participants need to set a metric between the geometrical components and to segment the movement with many stoprotate and go sub-tasks. These aspects might affect navigation and orientation, especially for the reproduction of triangular and square shapes, where the body representation in the space and body coordination is more important. This might require a more complex spatial representation for which visual experience is fundamental. The use of a route navigation strategy is also supported by the different walking speed

we observed in early blind individuals when compared with sighted and late blind individuals. Early blind individuals walk faster than the other two groups: in sighted individuals the slower walking speed could be associated with high task complexity, energy requirements and equilibrium demands in an unusual non-visual navigation task. Following the same line, the higher velocity in early blind individuals can be attributed to familiarity with non-visual tasks. On the other hand, the fact that late blind individuals also show slower walking speed compared to early blind participants, suggests that the walking pattern might be associated with early visual experience more than with confidence with the task. A possible explanation, which is in accordance with our previous observations, is that slower walking speed could facilitate spatial perception in an allocentric frame of reference where spatial-temporal information needs to be integrated in order to comprehend the global path produced. Contrarily, a faster walking speed, observed in sighted and late blind individuals, could reflect a more egocentric frame of reference in which global information is not integrated in space and time. Finally, we can speculate that the shape compression around the body region observed in early blind individuals can similarly reflect a more egocentric frame of reference that attracts the navigation toward the initial body position, resulting in it being compressed around the body. Further studies will be necessary to clarify this aspect.

### Visual Experience

From our results a difference between early and late blind individuals clearly emerges, supporting our second hypothesis. Early, but not late blind or sighted, participants had difficulties in performing the auditory and navigation tasks: the impairment was specific for complex audio paths, suggesting that simpler tasks such as static audio localization might require less subtle mechanisms. The performance of late blind and sighted individuals is similar in that they do not compress the area when reproducing the path and they correctly and precisely find the end point of the shape performed. Although only few late blind individuals were tested the result is stable and the variability among participant was minimal. This is in accordance with previous studies showing that the age of onset and duration of sight loss can affect auditory abilities (Voss et al., 2004; Gori et al., 2010; Wan et al., 2010) and the extent of cross-modal recruitment in dorsal brain regions in response to auditory spatial information (Dormal et al., 2012).

### CONCLUSION

To conclude, our results suggest that early blind individuals have a significant deficit in the interpretation of auditory geometric cues for navigation. The present study provides support for the cross-sensory calibration theory (Gori et al., 2008; Burr and Gori, 2011; Burr et al., 2011; Gori, 2015), suggesting that visual information is necessary in the first period of life for the normal development of auditory spatial representations. The auditoryimpairment observed in this study could be related to the lack of Euclidean representation which is typically mediated by the visual modality in sighted individuals. The navigation impairment observed in this study, on the other hand, may be related to the persistence of an egocentric frame of reference in early blind individuals. We can speculate that the visual cortex plays a key role in the transformation from egocentric to allocentric reference systems and that coordinate transformation could be mediated in sighted individuals by pathways involving the superior colliculus as previously proposed (King et al., 1988; Gori et al., 2014; King, 2014). We can also speculate that the lack of allocentric representation might hamper the processing of complex auditory geometric cues for spatial navigation investigated in this work. This could result from a different construction of the topographical representation of auditory space and suggests that the role of vision in this transformation process could occur in the first years of development, since the deficit is not present in late individuals.

Efficient walking is mediated by the integration of audiotactile signals with motor feedback. Understanding how this process occurs in early blind individuals might improve the development of scientific driven rehabilitation technologies for navigation. We hope that this work can provide inputs for further studies to better understand how non-visual navigation can be optimized in visually impaired individuals and what are the limits of cortical plasticity in case of sensory loss are.

### ETHICS STATEMENT

The study was approved by the ethics committee of the local health service (Comitato Etico, ASL3 Genovese, Italy). All the participants had a similar level of education (at least an Italian high school diploma, indicating 13 years of school). All the early blind participants were blind at birth. None of the participants had any history of hearing impairment. Blind participants were contacted from a list of participants that had taken part in our previous experiments. Sighted participants were contacted from a list of participants that had asked to take part in our studies. When the subjects arrived, we briefly outlined the experiment and they had to read and sign the ethical approval form. For the blind participants, the ethical documentation was read by the experimenter. All the participants provided written informed consent in accordance with the Declaration of Helsinki.

### AUTHOR CONTRIBUTIONS

MG, SF, and GC collected data. SF and MG analyzed the data. MG wrote the manuscript. SF, GC, and GB-B participated in protocol definition and paper discussion.

### ACKNOWLEDGMENTS

This research was supported by the EU Project ABBI (FP7-ICT 611452) and CODEFROR (FP7-PEOPLE-2013-IRSES).

## REFERENCES

fpsyg-08-00010 January 17, 2017 Time: 10:26 # 11


information processing in the auditory cortex. Neuron 81, 664–673. doi: 10. 1016/j.neuron.2013.11.023


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Gori, Cappagli, Baud-Bovy and Finocchietti. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

fpsyg-08-00010 January 17, 2017 Time: 10:26 # 12

# Corrigendum: Shape Perception and Navigation in Blind Adults

Monica Gori <sup>1</sup> \*, Giulia Cappagli <sup>1</sup> , Gabriel Baud-Bovy 2, 3 and Sara Finocchietti <sup>1</sup>

<sup>1</sup> Unit for Visually Impaired People, Istituto Italiano di Tecnologia, Genoa, Italy, <sup>2</sup> Robotics, Brain and Cognitive Science Department, Istituto Italiano di Tecnologia, Genoa, Italy, <sup>3</sup> The Unit of Experimental Psychology, Division of Neuroscience, IRCCS San Raffaele Scientific Institute, Vita-Salute San Raffaele University, Milan, Italy

Keywords: shape, perception, audio perception, blindness, motor, navigation

Edited and reviewed by:

Olivier Pascalis, Université Grenoble Alpes, France

> \*Correspondence: Monica Gori monica.gori@iit.it

#### Specialty section:

This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology

Received: 24 March 2017 Accepted: 15 May 2017 Published: 12 June 2017

#### Citation:

Gori M, Cappagli G, Baud-Bovy G and Finocchietti S (2017) Corrigendum: Shape Perception and Navigation in Blind Adults. Front. Psychol. 8:882. doi: 10.3389/fpsyg.2017.00882

#### **Shape Perception and Navigation in Blind Adults**

by Gori, M., Cappagli, G., Baud-Bovy, G., and Finocchietti, S. (2017). Front. Psychol. 8:10. doi: 10.3389/fpsyg.2017.00010

There is an error in the Funding statement. The correct Name for the Funder is **This research has been supported by the European ABBI project (FP7-ICT 611452)**. The authors apologize for this error and state that this does not change the scientific conclusions of the article in any way.

### ACKNOWLEDGMENTS

**A corrigendum on**

This research was supported by the European ABBI project (FP7-ICT 611452) and CODEFROR (FP7-PEOPLE-2013-IRSES).

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Gori, Cappagli, Baud-Bovy and Finocchietti. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# The Influence of Tactile Cognitive Maps on Auditory Space Perception in Sighted Persons

Alessia Tonelli1,2 \*, Monica Gori<sup>1</sup> and Luca Brayda<sup>2</sup>

<sup>1</sup> Unit for Visually Impaired People, Science and Technology for Children and Adults, Istituto Italiano di Tecnologia, Genova, Italy, <sup>2</sup> Robotics, Brain and Cognitive Sciences Department, Istituto Italiano di Tecnologia, Genova, Italy

We have recently shown that vision is important to improve spatial auditory cognition. In this study, we investigate whether touch is as effective as vision to create a cognitive map of a soundscape. In particular, we tested whether the creation of a mental representation of a room, obtained through tactile exploration of a 3D model, can influence the perception of a complex auditory task in sighted people. We tested two groups of blindfolded sighted people – one experimental and one control group – in an auditory space bisection task. In the first group, the bisection task was performed three times: specifically, the participants explored with their hands the 3D tactile model of the room and were led along the perimeter of the room between the first and the second execution of the space bisection. Then, they were allowed to remove the blindfold for a few minutes and look at the room between the second and third execution of the space bisection. Instead, the control group repeated for two consecutive times the space bisection task without performing any environmental exploration in between. Considering the first execution as a baseline, we found an improvement in the precision after the tactile exploration of the 3D model. Interestingly, no additional gain was obtained when room observation followed the tactile exploration, suggesting that no additional gain was obtained by vision cues after spatial tactile cues were internalized. No improvement was found between the first and the second execution of the space bisection without environmental exploration in the control group, suggesting that the improvement was not due to task learning. Our results show that tactile information modulates the precision of an ongoing space auditory task as well as visual information. This suggests that cognitive maps elicited by touch may participate in cross-modal calibration and supra-modal representations of space that increase implicit knowledge about sound propagation.

Keywords: cognitive maps, space perception, bisection, calibration, auditory perception, non-informative touch, multisensory

## INTRODUCTION

Several studies show that vision is essential in the domain of space perception influencing also other sensory modalities. It is well known that auditory space perception is modulated by visual inputs. When an auditory and visual stimuli are simultaneously presented although in two different space locations, the auditory stimulus is localized toward the location of the visual stimulus. This

#### Edited by:

Matthew William Geoffrey Dye, Rochester Institute of Technology, USA

#### Reviewed by:

Patrice Voss, McGill University, Canada Lotfi B. Merabet, Harvard Medical School, USA

> \*Correspondence: Alessia Tonelli alessia.tonelli@iit.it

#### Specialty section:

This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology

Received: 02 August 2016 Accepted: 13 October 2016 Published: 01 November 2016

#### Citation:

Tonelli A, Gori M and Brayda L (2016) The Influence of Tactile Cognitive Maps on Auditory Space Perception in Sighted Persons. Front. Psychol. 7:1683. doi: 10.3389/fpsyg.2016.01683

phenomenon is known as Ventriloquist effect (Bertelson and Radeau, 1981; Warren et al., 1981). Unlike the visual system, the auditory system cannot rely on a retinotopic organization of space in the inner ear. Specifically, the brain has to infer the direction of sound sources by taking into account the relative intensity of sound received at each ear as well as the time delay between arrival at the two ears in the superior olivary complex (Middlebrooks and Green, 1991). For this reason the auditory system is normally less accurate and reliable in spatial representation, compared with the visual system. Interestingly, vision can interact with audition even when a visual stimulus is not provided during an auditory task (Jackson, 1953; Shelton and Searle, 1980; Tabry et al., 2013). We recently demonstrated in sighted people that performance in auditory space bisection tasks is calibrated by short-term environmental observation only in a reverberant room, meaning that vision helps to construct complex auditory cognitive maps (Tonelli et al., 2015) thanks to a mental representation of the environment and not by direct visual information. Along with that, several studies have demonstrated, at a perceptual level, that auditory space perception can also be biased by tactile stimuli. Similarly to the audio-visual Ventriloquist effect, auditory localization seems biased toward the side of the concurrent tactile stimulus in bimodal tasks (Caclin et al., 2002; Bruns and Röder, 2010a,b; Bruns et al., 2011). Specifically, tactile stimulation influences the auditory cortical activity through higher areas assigned to multimodal association (Bruns and Röder, 2010a).

In the present study, we investigate whether it is possible to use touch as substitute of vision to modify, and possibly to improve, auditory spatial representations through the creation of a mental representation of the environment. We tested whether the construction of a cognitive map of a room through touch, can indirectly influence the perception of a complex auditory task (i.e., auditory spatial bisection task) in sighted people.

The hypothesis of this study is that haptic three-dimensional knowledge of an environment helps to build more precise auditory cognitive maps. This would match our previous results where vision calibrates the auditory modality (Tonelli et al., 2015). We supposed that spatial information obtained by exploring a 3D map would be poorer than that gained by visual observation. However, we wondered if, still, tactile information would be 'enough for space', meaning that essential information about the perimeter of the room, the kind of objects and their spatial relation would constitute sufficient knowledge to emulate the contribution of vision in auditory space perception (Pasqualotto et al., 2013).

To test this hypothesis, we tested a group of blindfolded sighted people in an auditory space bisection task and allowed them to explore with the hands a 3D tactile model of the room between the first and the second execution of the auditory task.

We recall that mental representation is an internal cognitive idea that represents external reality or else a mental process that makes use of such idea: "a formal system for making explicit certain entities or types of information, together with a specification of how the system does this" (Marr, 1985).

In our case we wanted to evaluate the mental spatial ability of the participants, through mental manipulation of objects in space. In addition, studies have demonstrated that the inter-personal variability in performing mental manipulations (Parsons et al., 2004; Guillot et al., 2007) is quite high. We therefore hypothesized that the ability in representing or manipulating an object could possibly predict auditory space bisection performance, when supported by additional haptic or visual knowledge of the room.

We administered to each participants two mental rotation questionnaire: the paper folding test (PFT) and the mental rotation test (MRT). The PFT requires participants to mentally perform complex spatial manipulations (Ekstrom et al., 1976) of a 2D item. Instead, the MRT evaluates the ability of mentally rotating a 3D object (Shepard and Metzler, 1971). The hypothesis was that PFT may predict an improvement obtained after the exploration of the tactile map – more similar to elicit mainly bi-dimensional representation, while the MRT would predict an improvement obtained after visual observation, which is more likely to elicit three-dimensional representations.

### MATERIALS AND METHODS

### Participants

Twenty sighted participants (13 females and 7 males, with an average age of 28.5, SD = 7) were recruited to participate in the experiment. All participants gave written informed consent before starting the test. The study was approved by the ethics committee of the local health service (Comitato Etico, ASL 3, Genova).

### Apparatus and Stimuli

Stimuli were delivered with scripts exploiting the Psychophysics 3.08 (Brainard, 1997) tool and Matlab (R2009a, The Mathworks, Natick, MA, USA). The acoustic stimuli for the auditory space bisection task were pink noise bursts, lasting 75 ms each. The sounds were produced by a linear array of 23 loudspeakers, 161 cm long and spanning ±25◦ of visual angle (see **Figure 1A**). The participants sat 180 cm from the center of the array. The auditory space bisection task consists in playing three consecutive sounds (duration of 75 ms) with an interval between each sounds of 500 ms. The first stimulus came always from the loudspeaker to the left (−25◦ ) and the third stimulus from the loudspeaker to the right (+25◦ ). The second stimulus came from an intermediate position, which was determined by QUEST (Watson and Pelli, 1983), an adaptive algorithm which, based on the current estimation of the participant, estimates the best stimulus value to be presented in the subsequent trial. The proportion of 'rightward' responses was calculated for each speaker distance. Gaussian functions by means of the Maximum Likelihood method were used to estimate both the accuracy, i.e., the bias in localize the center of the array, and the standard deviation. The standard deviation of the fit was taken as an estimate of the threshold, indicating the precision of the task, i.e. the reliability with which the task is performed.

The room size was 4.2 m × 3.0 m × 3.2 m (height) and the 3D reproduction of the room was made by bricks of Lego© on a scale 1:15 (see **Figure 1B**). Therefore, the space of the room

was represented by a 30 × 22 Lego dots matrix (excluding the walls, which were two Lego dot thick), i.e., a tactile map of 27 cm × 20.7 cm. The walls of the map were 10 Lego bricks high.

The bricks represented the perimeter of the room, the relevant openings (door, window) and the main objects located in the room (two tables, the chair hosting the subject), including a tactile representation of the loudspeaker array. A small model of a man, representing the subject, gave hint about his/her correct position and orientation inside the room and with respect to the objects. We respected the approximate relative proportions of all objects in the room.

Each participant was given two questionnaires evaluating mental manipulation ability: a PFT and a MRT. The PFT required participants to mentally perform complex spatial manipulations (Ekstrom et al., 1976). For each item on the PFT, the drawings depicted two or three folds in a square sheet of paper. The last drawing of folded paper showed a hole punched in it. Participants selected one of five drawings showing how the punched paper would look like when fully reopened. It was composed by 20 questions with scores ranging from 0 to 20. The MRT, instead, is composed by figures provided by Shepard and Metzler (1971), modified by Peters et al. (1995). The participants had to rotate the figures both around the horizontal and vertical axis in order to obtain the correct solution. The score was calculated by giving one and only one point for each correctly solved problem. A correct solution consists in identifying those

two stimuli from a group of four, which represent rotated versions of the target stimulus.

### Procedure

The sample of participants was randomly assigned in one of two groups (see **Figure 1C**): an experimental group and a control group.

Both groups performed an auditory space bisection task. The participants verbally reported whether the second sound was spatially closer to the first sound (produced by the first speaker on the left, number 1) than the last sound (produced by the last speaker on the right, number 23).

Both groups performed the task three times. All the participants were blindfolded before entering the room, so that during the first execution of the auditory task, they had neither knowledge of the room nor of the setup used to deliver the acoustic stimuli. The experimental group, blindfold on, explored with both hands the 3D tactile model of the room to understand the structure of the room, the disposition of the main objects inside the room, their own relative position with respect to the room and the objects when performing the auditory task. After that, each participant was led counterclockwise along the perimeter of the actual room. The participant had the chance to touch the walls and the acoustic stimulation setup. The participant of the experimental group then performed the auditory task a second time. Following that, the blindfold was removed for 1 min – allowing visual observation. Finally, the participant performed the auditory task a third time. Instead, the control group, after the first execution of the task, had a break of 5 min, keeping the blindfold on, then performed the auditory task a second time. As a last action, the control group followed the same procedure of the experimental group for the tactile exploration and navigation through the environment, then performed the task a third time. Each subject performed 80 trials of the auditory task per repetition, for a total of 240 trials.

At the end of the auditory space bisection task the participants of the experimental group were administered the PFT and the MRT, in random order. The PFT was administered in two parts of 10 questions each and they had 3 min to complete each part with a break of 1 minute between the first and the second parts. To complete the MRT the participants had 10 min.

### RESULTS

We ran a Lilliefors (Kolmogorov−Smirnov) test to check the normality of the sample. Results showed that both the experimental and control groups were not normally distributed for the precision in the first execution of the task (experimental group, D = 0. 279, p < 0.03; control group, D = 0. 277, p < 0.03; for more information, see Supplementary Materials). We used non-parametric statistical analysis. The failure in respecting criteria for normality is due to the presence of two outliers performances: participant 3 in the experimental group and participant 6 in the control group.

To see if the two samples were comparable we performed a Wilcoxon-test analysis (two-paired sample) between the first execution of the two groups. The results (**Figure 2**) revealed no significant difference between the first execution of the experimental group (black bars) and the control (red bars) for both precision (W = 65.5, p = 0.26) and bias (W = 41.5, p = 0.54), suggesting that the two groups are comparable, even if the control group is slightly more precise as compared to the experimental group.

We decided to normalize the results of the post-touch and post-vision, in the experimental group, and, second execution and post-touch, in the control group, by the performance of each participant in the first execution to avoid biases. For both precision and accuracy (bias), we computed a relative improvement: we subtracted to each performance that obtained in the first execution, then we divided it again for the first execution.

After that, we analyzed the precision in both the experimental and control groups, performing a one-sample Wilcoxon test for each condition of the experimental group, post-touch and post-vision conditions, and control group post-touch and second execution. In the post-touch condition, we had nine participants, instead of 10. As showed in **Figure 3**, for the experimental group, we found a significant improvement in precision for the experimental group (blue bars) in post-touch condition (filled blue bar − V = 1, p < 0.01), but not in the post-vision (lined blue bar: V = 9, p = 0.06), even if there is a trend. For the control group (green bars), we found a significant improvement for the post-touch condition (lined green bar: V = 3, p < 0.02) and not for the second execution (filled green bar: V = 16.5, p = 0.28).

On the contrary for the bias in performing the task, as showed in **Figure 4**, we did not found a significant improvement for accuracy in any condition for both control group (green bars − 2nd execution, V = 32.5, p = 0.65; post-touch V = 21, p = 0.91) and the experimental group (blue bars − post-touch, V = 27, p = 1; post-vision V = 39, p = 0.27).

Concerning the questionnaires, the average scores for the PFT was 62% of correct responses (SD = 16.5) and for the MRT was 51.7 % of correct responses (SD = 19). We computed a correlation between the percentage of correct responses in each questionnaire and the performance after tactile or visual information for both precision and accuracy. Thus, we computed a non-parametric Spearman correlation (RHO). After a Bonferroni correction for multiple comparisons, we found a negative and highly significant correlation only between the precision of post-touch condition (ρ(20) = −0.83, p < 0.01) and PFT. For the other results, see **Table 1**.

### DISCUSSION

Although previous studies (Bruns and Röder, 2010b; Gori et al., 2014b) demonstrated how direct tactile stimuli can influence auditory perception, this is the first study showing that the sense of touch, through active exploration of a surrounding environment and of its 3D map, can indirectly influence complex audio-spatial tasks that are known to benefit from previous environmental knowledge. This work contributes to argue that spatial representations are unlinked to specific sensory modalities

and that cross-modal calibration therefore contributes to build supra-modal mental representations.

Recent studies highlighted the importance of vision during development (Gori et al., 2012) showing that during childhood vision calibrates the other senses to process spatial information. When this calibration cannot take place, the non-visual modalities, especially audition, cannot properly encode some spatial information (Gori et al., 2014a; Finocchietti et al., 2015; Voss et al., 2015) that required a metric representation of space, while other auditory tasks are preserved (Lessard et al., 1998; Voss et al., 2004). In our previous study (Tonelli et al., 2015) we found that, in a reverberant room, the absence of knowledge of the environment leads to a decrease in precision of a complex auditory task, while no decrease occurs if the same task is performed in an anechoic chamber. This "impairment" is recovered after a brief observation of the room. The idea is that the person during the visual observation of the room has the chance to create a mental representation of the space.

TABLE 1 | Results of the correlation between two questionnaires about spatial abilities and precision/accuracy of the auditory space bisection tasks in two conditions (post-touch and post-vision).

bars represent the control group for the condition second execution (filled bar) and post-touch (lined bar).


Thanks to this representation the auditory system becomes able to compensate the noise produced by the reverberation. However, spatial knowledge does not come from vision only. Similar findings in which cognitive maps are developed from other modalities can shed light on the underpinnings of auditory spatial processing. For example, we have shown that touch helps to develop cognitive maps of surroundings in absence of vision (Campus et al., 2012) by eliciting the known phenomenon of sensory substitution (Bach-y-Rita and Kercel, 2003). However, when comparing persons with different degrees of visual disabilities, vision modulates the extent to which tactile information builds up abstract mental models (Brayda et al., 2013, 2015). On the other hand we have also found that brain regions deputed to tacto-spatial processing are similar to those elicited by audio-spatial processing (Leclerc et al., 2005; Campus et al., 2012).

In the present study, we investigated two points: (i) whether cognitive maps created by touch could influence space auditory perception with the same efficiency of maps generated by visual information, and (ii) whether the ability to mentally manipulate an object could predict auditory space auditory perception, when supported by additional haptic or visual knowledge of the room.

Contrarily to the study mentioned above, in the present study we allowed participants to construct a cognitive map by exploring with their hands a 3D model of the room and by being led along the perimeter of the real room between two executions of the space bisection task.

We found that tactile exploration significantly increases precision in a space bisection task. One could argue that the improvement might be due to a learning process and not to the tactile exploration. We have shown that this is not the case, because a control group, who performed the task twice and without any feedback on the structure of the room, did not show significant improvement in precision after the second task execution, but exhibited a significant improvement in the third execution, after tactile exploration of the 3D model of the room and by being led along the perimeter of the real room. The smaller magnitude of the accuracy improvement of the control group after touch, as compared to the experimental group, may be partially due to a learning effect. In fact, the sum of s the improvement of the control group after the second execution (−0.08) and the post-touch (−0.27) equals that of the experimental group after the tactile exploration (experimental group = −0.39; sum of the control group = −0.35).

Therefore we maintain that, in agreement with our previous study (Tonelli et al., 2015), the information obtained by touch, combined with vestibular feedback during navigation, are sufficient cues to create a mental representation of the space that helps to improve the understanding of room acoustics. Since observing the room a does not further increase auditory precision compared to touch a 3D model of the room, we assert that touch gives sufficient cues to create a mental representation of

space, even if vision is generally more suitable to address space perception and representation.

The results obtained in this study may appear not surprising, because previously studies demonstrated that passive tactile stimuli can directly influence auditory localization. For example, Gori et al. (2014b) demonstrated that a direct tactile feedback interacts with auditory spatial localization system improving the precision, if it is presented right after the auditory stimuli and in a congruent position. On the other hand, if it is present a spatial discrepancy between the tactile stimuli and the auditory stimuli, the auditory localization seems biased concurrent with the tactile stimuli in bimodal tasks (Caclin et al., 2002; Bruns and Röder, 2010a,b; Bruns et al., 2011). An explanation is that an incongruent condition may cause a cortical remapping of the auditory spatial representation, which tends to be more similar to the tactile spatial representation. What differs in this study is that the influence of tactile information on a complex auditory task is indirect and resides in the mental map create thanks to tactile information. This is visible in three main aspects. First, the tactile stimuli are not passively delivered on the human body, but are actively generated from spontaneous haptic exploration. The role of active exploration as compared passive stimulation is known to be important when building cognitive maps (Heller and Schiff, 1991). Second, comparing to previous studies, tactile feedback does not occur simultaneously with the auditory spatial task as in Gori et al. (2014b): here the spatio-tactile and audiotactile information are not linked to the same stimulus, but are just a mean to create a mental representation of the environment. Third, in our experiment tactile and audio feedbacks do not necessarily share the same frame of reference, since haptic exploration involved navigation and consequent stimulation of the vestibular system, while acoustic stimulation was a task to be achieved while seating. In fact, tactile stimuli initially have egocentric reference frame and then are remapped into external coordinates influencing the auditory space perception (Bruns and Röder, 2010a). The cognitive map obtained through touch is the additional piece of information that improves auditory precision.

The choice of space bisection deserves further explanation. Performing space bisection requires establishing a specific ordering relation between the three sound sources and take a decision based on these relations. This operation may require Euclidian representation of space (Gori et al., 2014a) and involves more spatial processing, possibly related to cues linked to the room structure.

The mental representation of space allows to interact with objects, to move into the environment and is based onto two frame of reference: allocentric and egocentric (Klatzky, 1998). The first is based on external salient landmarks in the environment; the second refers to coordinates anchored to the body. Starting from these two spatial coding modes we are able to create cognitive maps of space based on two different perspectives: survey and route. The 'survey' prospect provides a holistic view of the environment, preserving the information on the position of the objects and the Euclidean distances between them (Shelton and McNamara, 2004). One inevitable limitation of this study is that we could not counterbalanced across the participants of the experimental group the tactile and visual condition, because, otherwise, we would not have been able to assess the effect that the mental representation, built through tactile exploration, would have on the space bisection task.

Given the importance of the mental representation to perform the task and the nature of the space bisection, we decided to see whether there was a correlation between the results in the space bisection task and two mental rotation questionnaires: PFT and MRT. Mental rotation and mental folding have in common underlying cognitive process (Pellegrino et al., 1984; Wright et al., 2008). However, these two abilities differ, because mental folding is a non-rigid spatial transformation ability where the features of the manipulated object change. Instead, mental rotation involves a rigid manipulation, the object itself results unchanged, rather its spatial orientation differs. (Harris et al., 2013).

Our results show a significant negative correlation between the percentage of correct responses in the PFT and the precision of the space bisection task for post tactile exploration, meaning that the higher precision in the bisection task, the greater the ability to mentally manipulate a folded object. This correlation is much weaker with the MRT. One possible explanation could be that to solve a mental folding test, analytic strategy (Kyllonen et al., 1984) is needed, that helps to perform a non-rigid spatial transformation of the features of the manipulated object. This may in principle differ from MRT that requires, instead, a single rotation. The same strategy involved in PFT could be applied to perform the space bisection task. After having acquired spatial information through tactile or visual exploration, a common need to put in relationship the coordinates of the three sounds (or of the facets on the paper) may appear, which establishes a specific ordering relation between the map of sound sources or, alternatively, the map of the facets.

Our results show that a mental representation of the environment helps to perform complex spatial auditory tasks and that this representation can be create using both visual and tactile information. Moreover, we found that it is possible to correlate the precision in the space bisection task based on the results obtained in the PFT.

### AUTHOR CONTRIBUTIONS

AT, MG, and LB have contributed in designing the work. Data acquisition and analysis was provided by AT. AT, MG, and LB have contributed in drafting the work and gave final approval to the manuscript.

### FUNDING

This research is partly funded by the FP7 EU STREP project BLINDPAD (www.blindpad.eu), under grant 611621, by the FP7 EU STREP project ABBI (www.abbiproject.eu), under grant 611452, and by the Fondazione Istituto Italiano di Tecnologia (www.iit.it).

### ACKNOWLEDGMENT

fpsyg-07-01683 November 1, 2016 Time: 12:46 # 8

The authors would like to thank all the participants to this study and Claudio Campus for the tips about the statistical analysis.

### REFERENCES


### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fpsyg. 2016.01683/full#supplementary-material


Wright, R., Thompson, W. L., Ganis, G., Newcombe, N. S., and Kosslyn, S. M. (2008). Training generalized spatial skills. Psychon. Bull. Rev. 15, 763–771. doi: 10.3758/PBR.15.4.763

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Tonelli, Gori and Brayda. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Cerebral versus Ocular Visual Impairment: The Impact on Developmental Neuroplasticity

Maria B. C. Martín<sup>1</sup> \*, Alejandro Santos-Lozano1,2, Juan Martín-Hernández<sup>1</sup> , Alberto López-Miguel<sup>3</sup> , Miguel Maldonado<sup>3</sup> , Carlos Baladrón<sup>1</sup> , Corinna M. Bauer<sup>4</sup> and Lotfi B. Merabet<sup>4</sup>

<sup>1</sup> GIDFYS, European University Miguel de Cervantes, Valladolid, Spain, <sup>2</sup> Research Institute of Hospital 12 de Octubre (i+12), Madrid, Spain, <sup>3</sup> Refractive Surgery and Visual Rehabilitation, Ophthalmology, Instituto Universitario de Oftalmobiología Aplicada, Eye Institute, Valladolid, Spain, <sup>4</sup> Laboratory for Visual Neuroplasticity, Massachusetts Eye and Ear Infirmary, Harvard Medical School, Boston, MA, USA

Cortical/cerebral visual impairment (CVI) is clinically defined as significant visual dysfunction caused by injury to visual pathways and structures occurring during early perinatal development. Depending on the location and extent of damage, children with CVI often present with a myriad of visual deficits including decreased visual acuity and impaired visual field function. Most striking, however, are impairments in visual processing and attention which have a significant impact on learning, development, and independence. Within the educational arena, current evidence suggests that strategies designed for individuals with ocular visual impairment are not effective in the case of CVI. We propose that this variance may be related to differences in compensatory neuroplasticity related to the type of visual impairment, as well as underlying alterations in brain structural connectivity. We discuss the etiology and nature of visual impairments related to CVI, and how advanced neuroimaging techniques (i.e., diffusion-based imaging) may help uncover differences between ocular and cerebral causes of visual dysfunction. Revealing these differences may help in developing future strategies for the education and rehabilitation of individuals living with visual impairment.

Keywords: cortical, cerebral, visual impairment, ocular blindness, connectivity

### INTRODUCTION

### The Nature of Visual Impairment and the Case of CVI

Humans are highly dependent on their sense of vision in order to interact with the surrounding world. Not surprisingly, the loss of visual function associated with blindness and visual impairment has a dramatic impact on an individual's quality of life and independence (Dagnelie, 2013). Vision is a complex sensory function that requires the hierarchical participation of receptors (the photoreceptors of the eye), transmission (the optic nerves and optic radiations), and processing (the visual cortex) structures to transform captured visual information into meaningful percepts. Damage anywhere along the visual pathway typically results in some degree of visual impairment with characteristic clinical and functional manifestations. Traditionally, much of our understanding of visual impairment has been focused on the consequences of diseases and conditions affecting the eye and optic nerve (e.g., cataracts, macular degeneration, and glaucoma). Damage to these structures early, or later in life, can lead to ocular-related blindness or visual impairment (WHO definition of blindness: visual acuity of 20/200 or worse in the better seeing

#### Edited by:

Matthew William Geoffrey Dye, Rochester Institute of Technology, USA

#### Reviewed by:

Hong-Yan Bi, Institute of Psychology (CAS), China Zaira Cattaneo, University of Milano-Bicocca, Italy

> \*Correspondence: María B. C. Martín iobabego@gmail.com

#### Specialty section:

This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology

Received: 24 August 2016 Accepted: 30 November 2016 Published: 26 December 2016

#### Citation:

Martín MBC, Santos-Lozano A, Martín-Hernández J, López-Miguel A, Maldonado M, Baladrón C, Bauer CM and Merabet LB (2016) Cerebral versus Ocular Visual Impairment: The Impact on Developmental Neuroplasticity. Front. Psychol. 7:1958. doi: 10.3389/fpsyg.2016.01958

eye with corrective lenses, or visual field restriction to 20 degrees diameter or less in the better eye. WHO definition of visual impairment: visual acuity of 20/60 or worse in the better seeing eye<sup>1</sup> ). In the case of ocular blindness, the rest of the downstream processing structures within the brain appear to remain largely intact despite the loss of visual sensory input early in development. It is important to realize that blindness and visual impairment may also result from acquired damage (e.g., stroke or trauma) occurring at the level of transmission and/or processing structures lying outside of the eye itself. This includes at the level of the geniculate, optic radiations, and primary and associative areas of the visual cortex. Damage to these structures is generally referred to as neurological visual impairment (Good, 2007) and is associated with perceptual deficits that are typically more complex than those resulting from damage to the eye (Dutton, 2003; Hoyt, 2003).

In contrast to these two scenarios, there has been a more recent and dramatic rise in the incidence of children born with profound visual impairment not fitting with the typical profile of ocular blindness or visual impairment related to acquired brain injury. In this situation, the term cortical/cerebral visual impairment (CVI) was coined to describe damage to visual pathways and structures occurring during early perinatal development (Hoyt, 2003; Good, 2007, 2009). The term "cortical" visual impairment was originally proposed by Whiting et al. (1985) to describe visual dysfunction in pediatric populations of non-ocular cause, and its presumed association with damage to early visual cortical areas. However, as further characterization of this condition progressed, it became evident that CVI was often associated with damage to sites beyond early visual cortex including subcortical structures, white matter pathways, as well as higher-order associative processing areas of the cortex. Thus, the word "cortical" has been viewed as somewhat of a misnomer, and there has been the suggestion that the term "cerebral" would be a more encompassing and appropriate term (Good et al., 1994; Colenbrander, 2010). While the naming convention of CVI (i.e., cortical or cerebral) remains the subject of debate within the education and medical communities, it is important to acknowledge that employing accurate descriptive terms to characterize and localize the site of brain-based injury remains important issue. Two aspects merit consideration in this regard. First, it should be considered that brain injury in CVI may also affect areas beyond those ascribed to visual processing. Indeed, children with CVI often present with other neurological disorders such as cerebral palsy, seizures, or cognitive and developmental delays related to the location of brain damage (Huo et al., 1999; Philip and Dutton, 2014). With this taken into account, it is important to note that the terms cortical and cerebral fail to capture the possibility of more global neurological injury (Good, 2007). Second, injury to the visual system can result in a myriad of visual deficits. While these deficits can occur in isolation or in combination, the current anatomical definition is not sensitive to discriminate between CVI patients based on their apparent perceptual and cognitive dysfunctions. With these limitations in mind, the use of CVI as an umbrella term has emerged, encompassing both clinical and functional characteristics, stating that CVI is defined as a significant deficit in visual function associated with damage to retrochiasmatic visual pathways and cerebral structures in the absence of major ocular disease (or more accurately, the presence of visual deficits that cannot be explained by ocular abnormalities alone) (Dutton, 2003). Within the visual domain, these deficits include decreased visual acuity (ranging from mild to moderate impairment/low vision to profound blindness) as well as visual field impairments (typically in the lower hemifield) (Good et al., 2001; Kozeis, 2010). Most notably, however, are observed difficulties in higher-order visuospatial processing leading to substantial functional limitations that profoundly impact a child's learning, mobility, development, independence, and ultimately their quality of life (Fazzi et al., 2004; Boot et al., 2010). This broad spectrum of visual deficits makes the diagnosis of CVI not only more difficult to characterize and quantify, but also raises challenges in terms of developing appropriate and individualized rehabilitative strategies (McKillop and Dutton, 2008; Good, 2009).

There is now, more than ever, a greater need for improved accuracy in diagnosing, assessing, and developing effective education and rehabilitation programs for individuals with CVI. Furthermore, given that visual deficits in CVI are very diverse, their impact upon education and rehabilitative strategies remain much less well understood than in the case of ocular impairment (Baker-Nobles and Rutherford, 1995). This distinction is of utmost importance when considering that educational strategies designed to increase independence and functionality in children with ocular visual impairments are largely ineffective, and perhaps even detrimental, when applied to children with CVI (Groenveld et al., 1990; Farrenkopf et al., 1997). Neuroscience (particularly with regards to advanced neuroimaging techniques) may enable researchers to shed light on to these issues. In particular, it is crucial to uncover the neurophysiological differences between children with CVI and other forms of visual impairment and understand the association between potential risk factors and observed visual deficits. At the same time, this also represents a unique opportunity to rethink traditional interpretations related to what it means to be "visually impaired" (Swift et al., 2008) likewise, explore the developmental potential of the brain.

### Epidemiology and Etiology of CVI

From a public health perspective, CVI represents a pressing issue as it is now the leading cause of congenital visual impairment in children in developed countries including the United States (Good et al., 1994; Huo et al., 1999; Hoyt, 2007; Kong et al., 2012; Philip and Dutton, 2014). In the United Kingdom, CVI is the predominant disorder affecting up to 40–48% of the impaired children younger than 15 years old (Rahi et al., 2003; Rahi, 2007). It is worth noting that while visual impairment worldwide has decreased since early estimates in the 1990s, current evidence suggests that the incidence of CVI is continuing to rise in developed countries. This is due in large part to advancements in the delivery of neonatal intensive care resulting in greater infant survival from neurological damage and complications occurring

<sup>1</sup>http://www.who.int/mediacentre/factsheets/fs282/en/

during pregnancy and perinatal period (Good, 2001; Kozeis, 2010).

Individuals with CVI will also usually present with other coexisting disabilities and neurological disorders. In addition, secondary etiologies associated with CVI are also often present and include seizure, metabolic diseases, and underlying genetic syndromes. These same conditions can also further exacerbate complications and developmental delays accompanying CVI (Flodmark et al., 1990; Good et al., 1994). This makes assessment and evaluation difficult in the setting of concomitant cognitive (including attention), motor (such as cerebral palsy), and language deficits (Good et al., 1994). Finally, it is also crucial to realize that many individuals with CVI (when properly evaluated) are not truly blind per se based on their visual acuity alone, though they may ultimately function as a blind individual given inherent difficulties in the processing and interpretation of visual information (Swift et al., 2008).

In characterizing CVI, it is useful to distinguish between children born at term versus those born premature (WHO definition of prematurity is born alive before 37 weeks of pregnancy. Sub-categories of preterm birth based on gestational age include extremely preterm (<28 weeks) and very preterm (28 to <32 weeks)<sup>2</sup> . In infants born term, the most common cause of CVI is perinatal hypoxia-ischemia encephalopathy (HIE) (Huo et al., 1999; Fazzi et al., 2007; Khetpal and Donahue, 2007). The sequelae of HIE are dependent not only on the severity and duration of the hypoxic event, but also the gestational age (Hoyt, 2003). Specifically, regional differences in vascular perfusion (e.g., "watershed" zones; Baburamani et al., 2012) and the higher metabolic demand of the near term fetus (Bennet et al., 1999) alter the susceptibility of different brain locations to hypoxic-ischemic damage as the baby matures (Van den Broeck et al., 2008). In HIE, the areas that are most commonly damaged include deep gray matter, hippocampus, brainstem, and thalamic regions (Swarte et al., 2009). In contrast, premature infants will often present with periventricular leukomalacia (PVL) as the most common form of brain injury (Van den Broeck et al., 2008). This is associated with hemorrhagic necrosis in the periventricular white matter just dorsal and lateral to the external angle of the lateral ventricles (Volpe, 1998). The main factors commonly associated with PVL are an underdeveloped vasculature of the surrounding white matter, as well as impairment of the regulation of cerebral blood flow; both of which can predispose white matter to ischemic injury. Crucially, as the tracts of the optical radiations and of higher order visual functions travel within the periventricular white matter, PVL is often associated with impaired visual processing (Good, 2001; Dutton, 2003; Hoyt, 2007).

In both preterm and term infants, a common consequence is cell death (i.e., necrosis) of myelinated and pre-myelinated fibers obstructing the normal development of white matter pathways that communicate between sensory and motor areas of the brain. As a result, children with CVI will often exhibit motor and cognitive impairments associated with cerebral palsy (Lim et al., 2005).

While perinatal hypoxia remains the most common cause of CVI, other possibilities include traumatic brain injury (TBI; 10% of the cases in one reported study; (Khetpal and Donahue, 2007) as well as infectious etiologies (e.g., meningitis and encephalitis) leading to inflammatory-mediated white matter injury (Huo et al., 1999). Finally, seizure disorder is also a significant cause of CVI (estimated at 10% of cases) and also represents the most common associated neurological abnormality [reported in nearly 50% of CVI cases (Huo et al., 1999)].

### Compensatory Behaviors and Crossmodal Plasticity in the Setting of Visual Impairment

It is clear that in a world which heavily relies on sight, blind individuals have to make striking adjustments in order to remain functionally independent. Accumulating evidence suggests that individuals with ocular blindness (particularly, when blind from birth or very early in life) demonstrate comparable, and in some cases even superior, behavioral skills in the tactile and auditory domains as compared to their sighted counterparts (e.g., Lessard et al., 1998; Van Boven et al., 2000; Amedi et al., 2003; Gougoux et al., 2004; Wong et al., 2011; for review, see Merabet and Pascual-Leone, 2010). This has led to the suggestion that compensatory behaviors may be intimately related to underlying changes in the overall structural and functional organization of the brain resulting from profound vision loss (Voss et al., 2014). Interestingly, it has been shown that this reorganization implicates areas of the brain responsible for the processing of intact senses such as touch, hearing, and smell (e.g., Sterr et al., 1998; Hamilton et al., 2004; Rombaux et al., 2010) as well as the crossmodal reorganization of areas of the brain normally associated with the processing of visual information. Regarding the latter, numerous neuroimaging studies (predominately functional magnetic resonance imaging, or fMRI) have demonstrated that blind individuals show robust activation within occipital cortical areas while performing a variety of non-visual tasks [e.g., Braille reading (Sadato et al., 1996), sound localization (Gougoux et al., 2005; Voss et al., 2008; Collignon et al., 2011), and odor perception (Kupers and Ptito, 2014), as well as higher order cognitive tasks including language processing (Burton et al., 2002; Bedny et al., 2011; Striem-Amit et al., 2012) and verbal memory recall (Amedi et al., 2003; Raz et al., 2005). In summary, there appears to be mounting evidence supporting the view that the brain undergoes dramatic structural and functional changes in response to ocular blindness.

While the vast majority of scientific research investigating structural and functional brain changes resulting from profound visual deprivation has been carried out within the context of ocular causes of blindness, comparatively little research has been conducted investigating the developmental repercussions and neuroplastic compensatory mechanisms in CVI. As mentioned earlier, this is despite the high prevalence of this condition and its potential detrimental consequences on visual function and development. Indeed, often in the absence of ocular signs or pathology (combined with a lack of awareness of this condition by health care providers) many children

<sup>2</sup>http://www.who.int/mediacentre/factsheets/fs363/en/

with CVI are misdiagnosed. As such, often their visual difficulties are mistakenly attributed to a behavioral and psychological disorder (Swift et al., 2008). For those children that are accurately diagnosed, there remains the issue of the aforementioned observation that education and rehabilitation strategies developed for people with ocular blindness are not effective in the case of CVI (Groenveld et al., 1990; Farrenkopf et al., 1997). Put another way, how is it that two individuals with a comparable level of visual impairment (as characterized by measured visual acuity) could respond differently to the same training strategy? Could this be related to underlying structural and functional changes within the brain that differ between ocular compared to cerebral causes of visual impairment? In other words, does the brain develop and adapt differently in the setting of damage to visual cerebral structures compared to damage to the eye? Uncovering these differences would be of utmost importance not only in terms of developing appropriate education and rehabilitative strategies, but also to help better characterize the underlying physiology of CVI.

### CVI: A Disorder of Brain Connectivity?

As mentioned previously, children and adolescents with CVI often present with marked impairments in visual function including decreased visual acuity and visual field deficits impairments (Good, 2001; Kozeis, 2010). However, most striking are their deficits in the processing of higher-order and complex visual information (Fazzi et al., 2004; Boot et al., 2010). This includes cognitive and visuomotor processing difficulties related to object and spatial processing tasks (i.e., identifying common objects such as faces and locating them in space). Often, individuals with CVI will exhibit difficulties in locating a target object in a crowded or complex visual scene. For example, a child with CVI will often report not being able to identify a parent in a crowd or locate their favorite toy in a box filled with other toys (Dutton, 2003). They often will report troubles with finding their way around as well as perceiving complex moving scenes including interpreting biological motion (Braddick et al., 2003).

In describing visual processing, the concept of a two-stream hypothesis (i.e., dorsal/spatial processing and ventral/object processing) has often been purported in order to explain the division of labor as to how certain attributes are analyzed within a visual scene (Mishkin and Ungerleider, 1982). Given the nature of the visual dysfunctions observed in CVI, certain investigators have proposed that CVI may be a condition best characterized as a dorsal stream "dysfunction" or "vulnerability"; consistent with an impairment in the functioning of the dorsal/spatial visual processing pathway (i.e., connecting the occipital to parietal cortices and terminating in frontal areas) (Braddick et al., 2003; Dutton, 2009; Taylor et al., 2009). Certainly, this dichotomy represents a practical and useful conceptual framework to characterize observed deficits in CVI (Dutton, 2003). However, it is worth noting that neurophysiological support for this concept remains lacking despite strong psychophysical evidence. Furthermore, spatial processing deficits are not universal in CVI, nor do they occur in isolation from other deficits that can be characterized as non-spatial in nature. Indeed, many individuals with CVI also exhibit a broad spectrum of visual dysfunctions including object identification such as recognizing faces and shapes (Good et al., 1994; Porro et al., 1998; Fazzi et al., 2004, 2007, 2009). The nature of these perceptual impairments remains unclear in that is unknown whether these deficits represent true "agnosias" or rather are related to other cognitive issue such as imagery, language, and memory (see Fazzi et al., 2009, for further discussion). According to the two stream hypothesis, impairments in object identification (e.g., faces, toys, and other objects) would be suggestive of damage along the ventral visual processing pathway (i.e., connecting the occipital and temporal cortices) which must also be explained from a neuroanatomical and neurophysiological point of view. Lastly, concomitant oculomotor and attentional issues are also often present in individuals with CVI, which are not typically ascribed to the two stream visual processing model, but rather, may implicate different pathways and structures entirely. Therefore, while damage along key visual processing streams may be associated with observed perceptual deficits, the underlying maldevelopment of the brain in CVI appears to be more extensive and complex than previously assumed.

Indeed, clinical editorials have highlighted that the relationship between observed clinical manifestations and the extent of brain damage in CVI is complex and not yet fully understood (Guzzetta et al., 2010). Additionally, attention has been called to the value of advanced neuroimaging techniques in helping to better understand the relationship between brain maldevelopment and visual impairment in CVI (Good, 2001; Edmond and Foroozan, 2006; Murakami et al., 2008). With this in mind, early studies have attempted to associate visual impairments in CVI with alterations in brain structure using standard clinical neuroimaging modalities (predominantly magnetic resonance imaging, or MRI). For example, Serdaroglu et al. (2004) reported that the degree of gross cerebral morphological changes (and in particular, the severity of PVL) was correlated with neurodevelopmental outcomes. For example, children with low severity of PVL had minor motor problems or mild to normal functional outcomes, whereas the presence of cortical atrophy and thinning of the corpus callosum were associated with more developmental delays (Serdaroglu et al., 2004). While standard clinical neuroimaging techniques like structural MRI and computerized tomography (CT) can help characterize gross changes in cerebral structure, the underlying microarchitecture of white matter pathways cannot be ascertained. In the case of CVI, this is particularly relevant in terms of the need to characterize the relationship of specific types of visual impairments with key processing pathways such as the optic radiations and dorsal and ventral processing streams. There are advanced neuroimaging techniques that enable the examination of brain structure and anatomical pathways in a more detailed manner that may help further our understanding of the neuroanatomical basis of CVI. In particular, diffusion based imaging studies (such as diffusion tensor imaging, or DTI) combined with white matter tractography analysis can be used to reveal the organization of white matter pathways and thereby reveal how the brain is inter-connected. Briefly, by tracking the diffusion movement of water molecules in the brain, the overall organization of white matter connectivity can be inferred (Jones,

2008; Ffytche et al., 2010). Furthermore, it is possible to employ tractography techniques that allow for the "virtual dissection" of the brain so that pathways to be reconstructed and individually examined (Catani and Thiebaut de Schotten, 2008). For example, it is the neuroanatomical correlate of the dorsal/spatial stream has been identified as the superior longitudinal fasciculus (SLF) (Catani and Thiebaut de Schotten, 2008). In contrast, the inferior longitudinal fasciculus (ILF) represents the neuroanatomical correlate of the ventral visual processing stream (Catani and Thiebaut de Schotten, 2008). It is also important to identify a third pathway (though previous reports associate this pathway as part of the dorsal pathway; see Schmahmann et al., 2007; the inferior fronto-occipital fasciculus (IFOF) which appears to the be associated with visual attention and eye movements given its strong connections between occipital and frontal areas (Ffytche et al., 2010). Thus, using advanced diffusion based imaging techniques and white matter tract reconstruction, there is the unique opportunity to explore the association between visual perceptual deficits and the structural integrity of the visual pathways that support normal visual function and development.

In the specific case of CVI, previous studies using diffusion based MRI have identified marked alterations in white matter structure and further suggest that there is an association between the maldevelopment of key visual pathways and the visual dysfunctions observed in this condition. In one recent study by Lennartsson et al. (2014), diffusion weighted MRI was carried out in a group of individuals with white matter damage predominantly in the superior posterior periventricular white matter and with documented visual dysfunction. Specifically, it was found that early injury to the optic radiations was associated with characteristic patterns of visual field deficits. Interestingly, in a review study by Guzzetta et al. (2013), this group reported that many individuals diagnosed with CVI and with early periventricular damage to the optic radiations often showed normal development of visual field function. In their review, the authors suggested that the preservation of visual field function they reported may be the result of compensatory neuroplastic reorganization; an important observation that needs further and careful study as this may have important rehabilitative implications (Guzzetta et al., 2013).

A number of recent studies have investigated the individual pathways implicated in the processing of visual information between cortical areas of the brain with the aim of establishing a possible association between the structural integrity of these pathways and the visual dysfunction in CVI. In one study, the integrity of the SLF (measured by fractional anisotropy, or FA) was examined using DTI in association with impairments in object identification observed in a cohort of individuals with CVI. Specifically, it was shown that the structural integrity of the ILF was significantly decreased in CVI compared to normally developed controls (Ortibus et al., 2011). Given the role supported by the ILF, SLF, and IFOF in various aspects of visual processing (object, spatial, as well as visually guided attention and eye movement control, respectively), it seems plausible that the visual impairments observed in CVI could be associated with specific structural changes in white matter connectivity at the individual level. Indeed, in a recent report by two of the coauthors of this review (Bauer and Merabet), individuals with CVI were shown to have dramatic reductions in the volume and number of fibers of the ILF, SLF, and IFOF (Bauer et al., 2014b). To characterize potential differences in white matter connectivity and integrity, these investigators employed high angular resolution diffusion imaging (HARDI) rather than DTI as was previously done in aforementioned studies. While both DTI and HARDI techniques provide information regarding the degree of water diffusivity in the brain in order to derive local axonal fiber orientation, it is becoming increasingly established that HARDI is superior in its ability to delineate the organization of crossing fibers, and ultimately the overall microarchitecture of the brain (Tuch, 2004; Jones, 2008; Tournier et al., 2011). Interestingly, preliminary results from Bauer et al. (2014a) showed that these dramatic reductions in all three fasciculi were

not observed in ocular blind subjects as compared to normally sighted/developed controls (**Figure 1**). These early findings using diffusion based MRI techniques and whiter matter tractography reconstruction appear to support the notion that the brains of individuals with CVI show dramatic differences compared to normally developed brains as well as the case of ocular blind individuals. Finally, in a recent case study using a combined HARDI and fMRI approach, Merabet et al. (2016) demonstrated that in a CVI subject with a clinically documented inferior visual field deficit (assessed by formal perimetric testing), there was a structural-functional correspondence between the location and extent of the visual field deficit, damage to superior branches of the optic radiations (characterized by HARDI), and reduced retinotopic activation of early visual cortical areas associated with the representation of the inferior visual field (as indexed by fMRI). This correspondence is in accordance to the known anatomical and functional organization of visual pathways and geniculo-cortical representation of visual field space (Wandell, 1999) and demonstrates the advantage of a combining a clinical and multimodal neuroimaging approach to help characterize the underlying neurophysiology of visual deficits in CVI.

Taken together, these studies suggest that CVI may be associated with a more generalized vulnerability implicating numerous key pathways supporting the developing visual system. Furthermore, neuroplastic changes within the developing brain (such as the "re-wiring" of key geniculo-cortical or corticocortical connections) may support the sparing of visual function in certain individuals with CVI. It is important that future work be focused on correlating the degree of structural impairment of individual processing pathways with a broad range of measured outcomes of visual processing deficits at the individual level.

While the extent and integrity of individual visual processing pathways can be investigated using diffusion based imaging techniques, it is also possible to use the same data to explore whole brain networks of connectivity of the entire brain (Sporns et al., 2005). Specifically, comparing whole brain network connectivity between CVI and ocular blind individuals may further provide insight into developmental differences between these two groups. In a preliminary analysis, comparison of whole brain connectivity networks was carried out based on white matter connectivity derived from HARDI (**Figure 2**). In this analysis, it was found that whole brain connectivity was very similar when comparing ocular blind and normally sighted controls (Bauer et al., 2014a). As with the case of examining the dorsal and ventral pathways independently, robust network connectivity throughout the brain was evident in individuals with ocular blindness and may further be related to the compensatory behaviors observed in this group (Merabet and Pascual-Leone, 2010). In contrast, whole brain connectivity appeared to be markedly reduced in individuals with CVI as compared to ocular blind and normally sighted controls (Bauer et al., 2014a). While further confirmation of these results are ongoing, they do suggest that a global impairment in overall brain connectivity may be associated with observed cognitive visual dysfunctions as well as other associated sensorimotor and cognitive delays in CVI. This level of whole brain analysis may also provide hints as to why education and rehabilitative strategies designed for individuals with ocular blindness may not be effective in the case of CVI.

The use of advanced neuroimaging modalities like diffusion based imaging in CVI are still in the early stages. However, it is becoming clearer that much can be learned regarding the underlying neurophysiology of this condition beyond what can be ascertained by standard structural imaging alone. This may help uncover associated links between underlying brain connectivity and cognitive visual dysfunction in CVI and provide clues as to how to develop novel education and rehabilitation

strategies for individuals living with blindness and profound visual impairment. Equally evident is that the diagnosis of visual impairment/blindness based on singular criteria such as visual acuity fails to accurately characterize the true overall functioning, as well as potential, of an individual. In this regard, better characterization of visual deficits using a more even-handed and comprehensive testing battery will be important. Finally, it will also be important to associate, as well as disentangle, the contributory effects of potential risk factors associated with CVI and further establish their relationship to overall prognosis. For example, while there may be a suspected link between CVI and certain developmental disorders such as autism spectrum disorder (ASD), clear epidemiological evidence is still lacking (note that there is mounting evidence establishing low birth weight, prematurity, and neonatal encephalopathy as important risk factors for ASD and other cognitive delays; (Badawi et al., 2006; Schendel and Bhasin, 2008; Schieve et al., 2014). Indeed, the neurobiological impact of prematurity on the brain is likely to be extensive as well as variable. Thus the ability to parse out observed perceptual, cognitive, and motor deficits as a function of underlying developmental impairments remains crucial.

### CONCLUSION

Studies using advanced neuroimaging techniques have contributed greatly to our understanding of how the brain adapts to the loss of sight and have helped uncover the neuroplastic mechanisms that relate to compensatory behaviors. While considerable knowledge has been gained in the study

### REFERENCES


of individuals living with ocular visual impairment, a similar concerted effort is needed to gain important insight regarding the visual dysfunctions of children and adolescents with CVI. This may help to better understand the interrelationship between specific developmental deficits, underlying brain anatomy and function, and compensatory behavioral adaptations. In the end, understanding the conditions that promote neuroplasticity within the brain, both in the setting of ocular and cortical/cerebral blindness, will be crucial to help to maximize the learning, development, and well-being of these individuals.

### AUTHOR CONTRIBUTIONS

Study concept and design: MBM and LM. Study of CVI vs. Visual Impairment: MBM, AL-M, and MM. Epidemiology of CVI: AS-L, JM-H, and CB. Etiology of CVI: MBM, AS-L, and JM-H. Relationship between Brain Damage and CVI: CMB and LM. Study of neuroimaging techniques: CMB, LM, and CB. Critical revision of the manuscript: all authors.

### ACKNOWLEDGMENTS

This work was supported by grants from the NIH/NEI (RO1 EY019924) and the Low Vision Research Award from Research to Prevent Blindness (RPB) and the Lions Clubs International Foundation (LCIF) to LM.

fetal sheep to acute umbilical cord occlusion. J. Physiol. 517(Pt 1), 247–257. doi: 10.1111/j.1469-7793.1999.0247z.x




**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Martín, Santos-Lozano, Martín-Hernández, López-Miguel, Maldonado, Baladrón, Bauer and Merabet. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Visible Social Interactions Do Not Support the Development of False Belief Understanding in the Absence of Linguistic Input: Evidence from Deaf Adult Homesigners

#### Deanna L. Gagne<sup>1</sup> \* and Marie Coppola1,2

<sup>1</sup> Department of Psychological Sciences, University of Connecticut, Storrs, CT, United States, <sup>2</sup> Department of Linguistics, University of Connecticut, Storrs, CT, United States

#### Edited by:

Matthew William Geoffrey Dye, Rochester Institute of Technology, United States

#### Reviewed by:

Gary Morgan, City University of London, United Kingdom Stephen Butterfill, University of Warwick, United Kingdom

\*Correspondence: Deanna L. Gagne deanna.gagne@uconn.edu

#### Specialty section:

This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology

Received: 18 September 2016 Accepted: 08 May 2017 Published: 02 June 2017

#### Citation:

Gagne DL and Coppola M (2017) Visible Social Interactions Do Not Support the Development of False Belief Understanding in the Absence of Linguistic Input: Evidence from Deaf Adult Homesigners. Front. Psychol. 8:837. doi: 10.3389/fpsyg.2017.00837 Congenitally deaf individuals exhibit enhanced visuospatial abilities relative to normally hearing individuals. An early example is the increased sensitivity of deaf signers to stimuli in the visual periphery (Neville and Lawson, 1987a). While these enhancements are robust and extend across a number of visual and spatial skills, they seem not to extend to other domains which could potentially build on these enhancements. For example, congenitally deaf children, in the absence of adequate language exposure and acquisition, do not develop typical social cognition skills as measured by traditional Theory of Mind tasks. These delays/deficits occur despite their presumed lifetime use of visuo-perceptual abilities to infer the intentions and behaviors of others (e.g., Pyers and Senghas, 2009; O'Reilly et al., 2014). In a series of studies, we explore the limits on the plasticity of visually based socio-cognitive abilities, from perspective taking to Theory of Mind/False Belief, in rarely studied individuals: deaf adults who have not acquired a conventional language (Homesigners). We compared Homesigners' performance to that of two other understudied groups in the same culture: Deaf signers of an emerging language (Cohort 1 of Nicaraguan Sign Language), and hearing speakers of Spanish with minimal schooling. We found that homesigners performed equivalently to both comparison groups with respect to several visual socio-cognitive abilities: Perspective Taking (Levels 1 and 2), adapted from Masangkay et al. (1974), and the False Photograph task, adapted from Leslie and Thaiss (1992). However, a lifetime of visuo-perceptual experiences (observing the behavior and interactions of others) did not support success on False Belief tasks, even when linguistic demands were minimized. Participants in the comparison groups outperformed the Homesigners, but did not universally pass the False Belief tasks. Our results suggest that while some of the social development achievements of young typically developing children may be dissociable from their linguistic experiences, language and/or educational experiences

clearly scaffolds the transition into False Belief understanding. The lack of experience using a shared language cannot be overcome, even with the benefit of many years of observing others' behaviors and the potential neural reorganization and visuospatial enhancements resulting from deafness.

Keywords: theory of mind (ToM), visual perspective taking, false photograph, deafness, homesign, false belief task, Nicaraguan Sign Language

### INTRODUCTION

Congenitally deaf individuals can exhibit enhanced visual perception and visuospatial abilities that reflect neural reorganization in response to an altered sensory landscape and/or experience using a natural sign language in the visual modality, such as American Sign Language (ASL). Examples of skills demonstrating such enhancement in deaf individuals include attention to motion in the periphery (e.g., Neville and Lawson, 1987b), mental rotation (Emmorey et al., 1998), and face processing (McCullough and Emmorey, 1997). An extensive literature has documented this phenomenon and has also begun to discern the relative contributions of deafness and language experience to the locus and nature of the subsequent neural reorganization (e.g., Neville and Lawson, 1987a; Emmorey et al., 1993, 1998; Bavelier et al., 2001; Codina et al., 2011). Recent work has uncovered more details regarding the associations between experiential factors such as type and timing of language exposure, and the corresponding reorganization of specific neural areas. For example, Cardin et al. (2013) showed that changes in regions of the left superior temporal cortex (STC) in deaf individuals can be attributed to exposure to sign language, while plasticity in the right STC results from their altered sensory landscape (namely, a lack of auditory input). In this special issue, Codina et al. (2017) report differences in peripheral visual sensitivity due to differential sensory experiences (deafness vs. hearing) and the timing of acquisition of sign language (early vs. later).

However, these enhancements due to a lack of auditory input or use of a visual language do not necessarily extend to other cognitive domains. For example, although deaf children and adults spend their entire lives observing the behaviors of others and using this visual information to navigate social interactions, the ability to predict others' beliefs, desires, and subsequent behaviors, commonly known as Theory of Mind (ToM) (e.g., Premack and Woodruff, 1978; Baron-Cohen, 2001; Call and Tomasello, 2008, among many others), is more strongly associated with the quality and amount of language input, and the age of exposure to such input (e.g., de Villiers and Pyers, 2002; Schick et al., 2007; Howard et al., 2008; Meristo et al., 2012).

Wellman and Liu (2004) and Peterson et al. (2005) present a scaling of precursor mental (belief, desire, and emotion) abilities to ToM abilities, that is, an ordering of tasks assessing knowledge and understanding of mental states that are necessary for children to succeed on false belief tasks. Here we focus on the contribution of visual experiences to the development of these social cognitive skills, and report a similar scaling of visuospatial abilities that appear to be prerequisites for social cognitive abilities. This investigation is particularly motivated by our interest in the development of socio-cognitive abilities among rarely studied individuals who do not have access to language but who nevertheless have rich visuo-social experiences, and is informed by arguments such as those proposed by Dijksterhuis and Bargh (2001), who present a theoretical link between gains in perception and subsequent gains in social cognition.

The series of studies we report here is designed to discern the contribution of visuospatial perceptual experiences to the development of a sequence of skills pertinent to social cognition (**Figure 1**). All of these abilities have been identified as possible precursors to False Belief understanding, the hallmark measure of a mature ToM in typically developing children (e.g., Flavell et al., 1981; Zaitchik, 1990; Sodian et al., 2007; Moll and Meltzoff, 2011). Of course, other abilities have also been identified as precursors to ToM (e.g., joint attention, specific syntactic constructions, etc.). In the present study we focus on those precursor abilities that (a) are visually based and (b) can be assessed using tasks that require minimal linguistic demands in terms of instructions and responses.

In the majority of previous studies, the perceptual experiences, language experiences, educational experiences, and biological maturation of participants are all tightly intertwined and presumably synergistic. One way to attempt to disentangle these factors is to look to populations whose experiences vary systematically in particular ways. One such population is deaf children: approximately 5–10% of deaf children are born to deaf signing parents (Mitchell and Karchmer, 2004), the remaining 90–95% have hearing parents who do not know sign language. Only a small percentage of these hearing parents become fluent users of sign language (Meadow-Orlans et al., 2003), and the resulting variability in deaf children's language experiences offers a window into the contribution of early language exposure to these precursor abilities. However, prior work examining these specific skills has tested deaf children in the context of an educational or early intervention setting, whether it be focused on spoken language, sign language, or both modalities. Thus, most deaf children in prior studies have been exposed to at least some language, intervention, and education (e.g., Peterson and Siegal, 1998; Meristo et al., 2007; Shield et al., 2016).

Our goal in the present paper is to determine which, if any, of these precursor abilities are robust in the face of a lack of access to language and education, especially given arguments that exposure to an established visual language whose structure capitalizes on spatial perspective taking may scaffold sociocognitive abilities (Courtin, 2000). The rarely studied groups in the current work offer a unique opportunity to identify the relative contributions of visual experiences alone. When we say "alone" here, we refer to the unusual situation faced by

homesigners (described in more detail below). That is, studying homesigners offers a way to assess the limits of visual experiences in driving socio-cognitive skills in the absence of a language model, participation in a shared linguistic community, and rich educational experiences. To that end, we asked participants to (a) indicate what (image) another person sees (Perspective Taking Level 1); (b) indicate how another person sees [a 3-D object (Perspective Taking Level 2; both tasks adapted from Masangkay et al. (1974)]; and (c) recall a visual reality that does not match the current visual reality [False Photograph, (e.g., Zaitchik, 1990)].

These specific tasks were chosen for several reasons. First, they are well-known precursors to False Belief success along the social cognition continuum. That is, typically developing children can often succeed on these tasks before or at about the same time they are able to succeed on False Belief tasks (**Figure 1**). Second, their focus on visual information (**Table 1**) is well suited to our question of how far the use of visual information can take an individual on the developmental path to ToM. Third, they minimize linguistic tasks demands relative to tasks that rely on storytelling (e.g., the Sally-Anne Task, Baron-Cohen et al., 1985), use particular linguistic structures (e.g., de Villiers and de Villiers, 2000) or specific mental-state vocabulary (e.g., Howard et al., 2008). Two facts motivated our examination of these abilities in adult homesigners: (1) these tasks have not been done with deaf children (regardless of language delay); and (2) these tasks have never been done with homesigners. Here we will see whether reducing the linguistic load in the task might reveal explicit ToM understanding with this population.

### Participant Groups

For our purposes, Homesigners are a group with little to no exposure to conventional linguistic input, little to no educational experience (**Table 2**), but lifelong experience using a visual communicative system. However, homesigners' experiences differ from those of the "atypical" child and adult participants typically studied in the domain of ToM in at least two additional ways. Homesigners do not participate in a linguistic community, and they have scant experience with formal education. Therefore, we included two comparison groups to better discern the contributions of each of these experiences, and help focus our attention on the role played by visual experiences: Deaf signers of the first cohort of Nicaraguan Sign Language (NSL) and hearing Nicaraguans who have not had formal education. We now describe each of these participant groups, and explain how their particular characteristics serve our goal of narrowing down the potential explanations for performance on tasks measuring various abilities in the domain of ToM.

#### Homesigners

Homesigners are deaf individuals who have not acquired either a spoken or sign language; their deafness precludes adequate access to the spoken language around them (e.g., Goldin-Meadow, 2003). In Nicaragua, homesigners' family and community environments do not include opportunities to access sign language (Polich, 2005). They are raised by hearing families who do not sign, and their geographical and/or economic circumstances preclude access to special education and/or the signing Deaf community. Despite this lack of access to linguistic input, they develop their own gesture systems called homesign (Coppola, 2002; Coppola and Newport, 2005). In developing countries like Nicaragua, few resources exist for identification, intervention, and education of children with disabilities. An extremely small proportion of the deaf population participates in the recently emerged Deaf community who uses NSL (Labato, 2017). Thus, it is not uncommon for many deaf individuals to reach adulthood without benefiting from the crucial language exposure provided by participating in a community of deaf signing individuals.

Despite their lack of conventional linguistic input, homesigners in Nicaragua continue to use their gesture systems as their primary means of communication into adulthood. They show no signs of social impairment or inhibition, readily engaging socially with familiar and unfamiliar individuals, and exhibit none of the social impairments/difficulties found in individuals with autism. Thus, they enjoy relatively typical social interactions with their hearing families, friends, and neighbors, with the caveat that these hearing communication partners do not fully share the homesign system with the deaf individual in their family (Carrigan and Coppola, 2017). While homesign is not a fully developed language, mature homesign systems exhibit a

#### TABLE 1 | Summary of the requirements of each task in the current study.


Note that Mental Rotation [as tested by Shield et al. (2016) was not tested here, but is provided for the sake of completeness]. PTL1 = perspective taking level 1, PTL2 = perspective taking level 2.

#### TABLE 2 | Summary of participant groups and characteristics.


<sup>∗</sup>One Unschooled Spanish Speaker did not participate in the False Photograph task.

range of linguistic properties found in fully developed languages, such as morphophonological regularities (Brentari et al., 2012), morphosyntactic structure (e.g., Coppola et al., 2013), and the grammatical relation of subject (Coppola and Newport, 2005).

Finally, homesigners provide us with an attractive population to explore these areas of interest particularly because they are biologically more mature than the deaf children in most previous studies of this nature. In the absence of language, and the consequent reduced opportunities to take others' perspectives (visual or otherwise), we might expect a protracted trajectory of ToM development. By exploring how ToM abilities do or do not emerge on the path to adulthood given these unusual circumstances, we hope to contribute to the current understanding of ToM development.

#### Nicaraguan Sign Language: An Emerging Language

Nicaraguan Sign Language is an indigenous sign language that emerged from a newly expanded center for special education in Managua during the late l970s to early 1980s (Polich, 2005). The first group of signers is known as "Cohort 1"; these signers initially formed the Deaf community and began creating the language through their interactions with each other at the center for special education. Like Homesigners, Cohort 1 signers of NSL did not have access to linguistic input transmitted from any pre-existing language model. However, Cohort 1 signers did engage in language genesis with their peers (e.g., Senghas, 2003; Senghas et al., 2005). NSL signers of all cohorts (Cohort 1 and the subsequent children who entered the school later, representing Cohorts 2, 3, and so on) interact with many other users who use the system as a primary language, i.e., members of the Deaf community in Managua, and are thus part of a linguistic community. Homesigners, by comparison, use their gesture systems with hearing people their entire lives – hearing people who only use these gestures with the homesigner and never with each other.

Another significant difference between Homesigners and Cohort 1 signers is the fact that Cohort 1's (and subsequent cohorts') introduction to the linguistic community is situated within an educational or vocational context (Polich, 2005; Senghas et al., 2005). As it is for most deaf children born to parents who do not already know a sign language, it is the schools that provide both educational and primary linguistic experiences through peer interactions. This will be explained further in the context of the next group, the Unschooled hearing Spanish Speakers.

Thus, the main motivations for including Cohort 1 signers as a comparison group are: (1) to investigate two deaf populations in the same cultural context whose language experiences differ minimally; and (2) to establish an anchor point using a previously studied group whose false belief abilities had previously been studied using this methodology.

#### Unschooled Nicaraguan Spanish Speakers

As mentioned, like most deaf children born to hearing parents who do not know a sign language, the vast majority of NSL signers in past and present studies gained access to their linguistic community via educational settings (Polich, 2005). We therefore cannot separate having a linguistic community from education in either NSL signers, who have both, or Homesigners, who have neither. Unschooled hearing Spanish Speakers have had a complementary set of experiences to Cohort 1 signers: they have full access to an established language and a language community from birth, but have little to no education (**Table 2**).

The reasons that the Unschooled Spanish Speakers did not go to school are straightforward and are unlikely to reflect an uncontrolled selection bias: five of the eight unschooled hearing participants were full-time agricultural workers; the other three worked for their family businesses, making and selling food products in their local communities. Their lack of education primarily resulted from economic restrictions and distance to the nearest school.

### STUDY 1: PERSPECTIVE TAKING LEVEL 1

We began by assessing all participants on a simple baseline visual perspective task. Based on results from their "mountain

task," Piaget and Inhelder (1956) theorized that children could not understand the visual perspective of others until well into childhood, after about the age of 9. Subsequent research has shown that young children do understand others' perspectives, though this ability is acquired incrementally, with the ability to understand what someone else sees (Perspective Taking Level 1) available earlier in development than the understanding of how that person sees it (Perspective Taking Level 2) (e.g., Masangkay et al., 1974; Flavell et al., 1981).

By the age of 3 typically developing children can easily answer the what question, as measured by verbal tasks [e.g., "what does the experimenter see?" (e.g., Masangkay et al., 1974)] or by the second year of life as measured by looking (Luo and Baillargeon, 2007; Sodian et al., 2007) or assisting (Moll and Tomasello, 2006) behavior.

No published studies have reported on Perspective Taking Level 1 abilities in deaf children<sup>1</sup> . Given previous studies with typically developing children, which confound maturation, language exposure, and possibly educational experiences, we cannot make a clear prediction for the homesigners' performance on Perspective Taking Level 1 tasks.

### Method

All procedures for all studies reported were approved under University of Connecticut IRB #H10-306. All participants provided written informed consent to participate in this study, and those identified in the images of the manuscript provided written informed consent for their publication.

### Participants

The 4 Nicaraguan Homesigners, the 6 Cohort 1 NSL users, and the 8 Unschooled Spanish Speakers described in **Table 1** participated in this study.

### Materials

Level 1 perspective taking abilities were tested using twodimensional stimuli, namely eight images of common objects, animals, and humans presented on 81/2" × 11" laminated sheets (see **Figure 2** for examples of Cat/Cap). All the images chosen were familiar items in Nicaragua, varying in category from humans to animals to inanimate tools, and have been used successfully with all of these participant groups (Richie and Yang, 2013).

### Procedure

Each round used two images from the previously mentioned list, arranged into four pairs: Cat/Cap, Wheelbarrow/Fishing Rod, Cow/Girl and Pitcher/Chicken. First, both images were shown to the participants, and participants were asked to identify the objects. This was done for two reasons – first, it familiarized the participants with the images, and second, it created common referential expressions between the participant and the experimenter (particularly important because homesigners each have their own idiosyncratic gesture systems).

The participant and experimenter sat across from each other at a table (**Figure 2A**). For each of the aforementioned sets, after familiarization and naming of the images, the two images were placed back to back, with one image facing the participant and the other facing the experimenter. The experimenter then asked, using the appropriate communication system (gesture/homesign, NSL, or Spanish): "What do you see?" and "What do I [the experimenter] see?" (**Table 3**). Both perspective questions were asked for each set, and each set of images was flipped so that the participant had an opportunity to see every image. Feedback was provided as needed during the first pair of images, Cat/Cap, to clarify instructions.

### Results and Discussion

Participants in all groups answered the control questions correctly ("what do you see?") and correctly named the image that the experimenter saw during test trials; that is, all participants performed at ceiling. Keep in mind that the homesigners in this study have had extremely limited experience with a conventional sign language; however, this lack of linguistic input and community did not have a negative impact on their performance. Therefore, if language is required in order to succeed on this task, their idiosyncratic visual and gestural experiences must exceed the threshold. The fact that the groups differ dramatically in their language experiences, but not in their performance, seems to suggest that simple maturation and/or life experience can support success.

### STUDY 2: PERSPECTIVE TAKING LEVEL 2

Our next question investigates whether the homesigners understand how someone else sees an object – that is, do they understand that when they and another person are looking at the same object from different angles, that the other person's perspective is different from their own, and what that other perspective on that object would look like? We address this in the next study, with a task investigating Perspective Taking Level 2 (Masangkay et al., 1974; Flavell et al., 1981). Previous studies have shown that Level 2 perspective taking abilities are available later in development than Perspective Taking Level 1 (e.g., Masangkay et al., 1974; Flavell et al., 1981), though Perspective Taking Level 2 may be available to children as young as 36 months of age (Moll and Meltzoff, 2011).

As it was for Perspective taking Level 1, few studies have directly measured Level 2 Perspective Taking in deaf populations. In a study of natively signing children with and without Autism Spectrum Disorders (ASD), Shield et al. (2016) found differences in results for Level 2 perspective taking: native-signing children with ASD did not perform

<sup>1</sup> Given that the literature contains only one unpublished study [Remmel's (2003) dissertation] of Perspective Taking Level 1 abilities in deaf children, insufficient basis exists for making a prediction regarding Homesigners' performance.

FIGURE 2 | (A) The experimenter (right) and a participant (left) engaged in the Perspective Taking Level 1 task. He sees one image, (B, "cat" or C, "cap"), while the experimenter could see the other. All images were photographs of real objects and not line drawings.

#### TABLE 3 | Sample interaction for Cat/Cap.

Crucial perspective questions are presented in bold text.

as well on Level 2 tasks as did the typically developing native-signing Deaf children. However, the children with ASD did not show this difficulty with mental rotation tasks, which consider different perspectives of the same object, but without considering another person's perspective of the object (**Table 1**).

There has also been some debate in the literature as to whether exposure to and use of sign language and the spatial perspective taking inherent in using a spatial language boosts perspective taking abilities, which may in turn boost ToM development (Courtin, 2000). However, more recent studies suggest that using a visual language confers no benefit to signing children or to subsequent ToM development (Courtin and Melot, 2005; Shield et al., 2016). Therefore, we hypothesized that there would be no differences between the groups' success on Perspective Taking Level 2 tasks.

### Method

The participants for this study were the 4 Nicaraguan Homesigners, the 6 Cohort 1 NSL users, and the 8 Unschooled Spanish Speakers shown in **Table 2**. 2

### Materials

Level 2 perspective taking abilities were tested using a procedure adapted from Reed and Peterson (1990) using minimal language/gesture with the homesigners and three-dimensional objects (e.g., Shield et al., 2016). Three familiar objects were included in our task: a toy duck, a mug with identifiable sides

<sup>2</sup>One Unschooled Spanish Speaker who was tested outside at the farm where he worked did not complete the second half of Level 2 (Left/Right) because the onset of a rainstorm necessitated packing up all of the equipment. We therefore based his overall score on just the first half of the task (Front/Back). Across participants, Left/Right (M = 0.76, SD = 0.21) and Front/Back (M = 0.71, SD = 0.3) scores did not differ [t(16) = 0.84, p = 0.413].

(a hand design on one side and a handle), and a toy truck (**Figure 3**).

### Procedure

Each object was presented to the participant on a turntable so that it could be rotated easily either by the experimenter or the participant. Testing was done with the turntable on a surface between the participant and the experimenter. Front/back perspectives were tested first with all objects, then side (left/right) views were tested. As in Level 1, participants were asked, using appropriate communication methods, "What do you see?" and "What do I [the experimenter] see?" (**Table 4**). Participants were given an 8" × 11" laminated sheet displaying the four possible perspectives of the object being tested (**Figure 4**). Participants could respond either by selecting the correct image or by describing the correct perspective (e.g., "You see the back of the duck" or "You see the duck's feet."), however, the experimenter encouraged participants to indicate the correct image whenever possible for clarity in coding and for consistency across participant groups, particularly for the homesigners, who often solely relied on selecting images rather than describing the experimenter's perspective. The experimenter repeated the question for insufficiently descriptive responses such as "You see the duck." The duck was used for familiarization with the task, and feedback was provided during duck trials. By the end of the familiarization/practice trials, all participants save for two (one homesigner and one hearing Spanish speaker) correctly responded at 100% to the experimenter's view for all four perspectives on the duck, as measured by the last time each of these four perspectives was tested (front, back, left side, and right side). These two participants are discussed further in the "Results and Discussion."

### Results

Level 2 Perspective Taking (three-dimensional objects) was coded for accuracy across test trials (Mug and Truck). Only the first response to questions asking about the experimenter's perspective are reported (i.e., the "What do I see" questions for the two objects mug and truck, totaling four trials). Because no significant differences were found between Left/Right and Front/Back scores across participants via a paired-samples t-test, (Left/right M = 0.76, SD = 0.21; Front/Back M = 0.71, SD = 0.3), t(16) = 0.84, p = 0.413, we report a single overall score per participant (**Figure 5**).

All group means were above chance (0.25); indeed, all but two participants scored at or above chance. Groups did not differ in their performance [Kruskal–Wallis non-parametric test, H(2) = 0.44, p = 0.802] with a mean of 8.8 for homesigners, 10.7 for NSL Cohort 1, and 9 for the Unschooled Spanish Speakers<sup>3</sup> .

### Discussion

We asked whether homesigners, whose language exposure and educational experiences are very limited, can correctly predict another person's visual perspective of the same object from a different angle. When compared to individuals who have language exposure from birth (the Unschooled Spanish Speakers), or other deaf individuals who have had educational experiences (Cohort 1 NSL signers), we found no differences in overall performance.

The one published study that has investigated Perspective Taking Level 2 in deaf individuals found differences between deaf individuals with and without ASD (Shield et al., 2016). The current findings contribute to that small body of work, affirming that Shield et al.'s (2016) finding was likely due to a social deficit in the deaf individuals with ASD, rather than to a languagespecific deficit. Both of the current analyses contribute to this interpretation: (1) the fact that homesigners, despite their lack of exposure to linguistic input, performed above chance on the Perspective Taking Level 2 task; and (2) the lack of difference among the groups, who differed markedly in their language experiences, on the Perspective Taking Level 2 task.

Although we found no differences among groups, we did find interesting results in two individuals – one Homesigner and one Unschooled Spanish speaker<sup>4</sup> , indicated by gray circles in **Figure 5**. While the presence of their scores did not affect the overall results, we wish to address two points about these participants. First, their scores on the test items parallel those of their practice items; that is, they were the only two participants to not achieve ceiling on the experimenter's perspective by the end of the practice items. However, they did reach ceiling for their own perspective ("What do you see?"), showing that their difficulty with the task was likely not due to any inability to interpret the three-dimensional object to two-dimensional images in the answer array<sup>5</sup> .

Second, their incorrect answers for the experimenter's perspective in the test trials seem to indicate egocentric perspectives – 80% of the Homesigner's responses and 60% of the Unschooled Spanish Speaker's incorrect responses were the image of their own perspective, rather than the other two potentially incorrect choices in the 4-picture array. If they had chosen the other potentially incorrect choices, one could argue that they understood that the experimenter could not possibly see the same perspective as they, but they just could not mentalize or translate that perspective to the image(s). Because their responses were so strongly egocentric, we must conclude that they likely do struggle with understanding other people's perspectives.

We also observed an interesting strategy employed by some Unschooled Spanish Speakers that was not observed in the other two groups. Namely, they used language to talk their way to the correct answer out loud by verbalizing their own perspective (e.g., "I see the driver's side. . ."), and then coming to conclusions

<sup>3</sup>This analysis includes the two participants who did not score at ceiling by the end of the practice trials. Removing these participants does not change the nonsignificant result [H(2) = 0.12, p = 0.941], with means of 9.2 for homesigners, 8.7 for NSL Cohort 1, and 8.1 for the Unschooled Spanish speakers.

<sup>4</sup> Interestingly, these two participants are a mother/son pair, which raises the question of a familial/genetic contribution. Nothing in their everyday behavior suggests that they would not succeed on this relatively straightforward task.

<sup>5</sup>This observation has also been made about 4-year-olds by Gzesh and Surber (1985).

### TABLE 4 | Sample interaction for Front/Back trials of object: duck.


Crucial perspective questions are presented in bold text.

about the experimenter's perspective (e.g., ". . . so you must see the passenger side."). Employing this strategy did not elevate this group's scores as compared to the other two groups.

Although the homesigners superficially scored similarly to the other two groups in this visual perspective taking task, it is possible that the quality or depth of their understanding of others' visual states does not translate to an understanding of mental states. We hypothesize that the underlying quality of their visual social interactions with others are different, given their different linguistic experiences (namely, that they are severely restricted, such as playing hide-and-seek without the benefit of phrases like "where are you? I can't see/find you!"). Another main difference is that while they interact daily with their families and friends, these communicative interactions do not rely on a shared linguistic system. We propose that this lack of a shared linguistic system is the likely cause for their difficulties with more advanced ToM tasks, even when those tasks do not strongly rely on language in their procedure or instructions.

### STUDY 3: FALSE PHOTOGRAPH

In order to test participants' ability to maintain a previous reality in the face of changes to the present reality, we conducted a "false photograph" task. Pioneered by Zaitchik (1990), the False

FIGURE 4 | Sample answer array showing the four perspectives tested in Perspective Taking Level 2 (A), and the experimenter observing a participant choosing a perspective from the array (B).

Photograph task closely imitates False Belief tasks. However, instead of requiring participants to consider another's belief, it requires them to consider a previous reality that does not represent the current state of affairs.

Zaitchik's (1990) original study and others since then (e.g., de Villiers and Pyers, 2001) suggested that typically developing children struggle with false photograph tasks just as much as they do with false belief tasks. This pattern suggests that the difficulty with false beliefs for typically developing children may not be in the belief as much as in the conflict between the prior knowledge and current state of reality. Some groups of children show a different pattern, calling this interpretation into question. Children with autism, and deaf children with hearing, non-signing parents (who do not have the early benefit of accessible language), reliably pass false photograph tasks while struggling with false belief tasks (e.g., Leslie and Thaiss, 1992; Peterson and Siegal, 1998; de Villiers and Pyers, 2001). This difference in patterning may point to a benefit of language – the children with autism were minimally able to communicate (in order to participate in these tasks), and the deaf children had access (albeit late) to either spoken or signed language. However, the children with autism or hearing loss were, at minimum, 2 years older than the typically developing children studied in False Photograph tasks. This age difference suggests that given enough developmental time, even without the benefit of language, false photograph understanding may develop in any individual. Working with homesigners – individuals who have never had the benefit of learning from linguistic input, or engaging in a shared language environment, can help us disentangle these factors.

### Participants

The participants for this study were the 4 Nicaraguan Homesigners, the 6 Cohort 1 NSL users, and 7 of the 8 Unschooled Spanish Speakers represented in **Table 2**. 6

<sup>6</sup>Due to time constraints, one Unschooled Spanish Speaker did not participate in the False Photograph task.

### Procedure

The procedure for the current False Photograph task was modeled after the Identity-Change Photograph<sup>7</sup> task in Leslie and Thaiss (1992), also used with deaf children in de Villiers and Pyers (2001). Participants were first presented with a Fujifilm Instax 210 camera (similar to a classic Polaroid camera whose prints develop within minutes), and three objects: a doll, a duck, and a truck. All participants were first asked to name the three objects. This was mainly so that the experimenter would be familiar with the specific gesture used by each homesigner to name each object, since these gestures could vary across homesigners.

First, as a control question, the doll was placed on a chair and a picture was taken. The developing photograph was then placed face-down on a table and the participants were asked what the photograph would show. All participants correctly stated that the image would be that of the doll. Then the doll on the chair was replaced by the duck and another photograph was taken. The new developing image was placed, again, facedown on a nearby table, within sight of the participant. Before the participant was asked about the new photograph, however, the duck was then replaced by a toy truck and the participant was finally asked what image would appear on the face-down photo.

### Results and Discussion

All participants in all groups were able to correctly state that the face-down polaroid image would be of the duck that previously occupied the chair, and not the toy truck that was currently in that location.

Typically, False Photograph tasks are reported along with results from False Belief tasks, which we will address in the general discussion. It is interesting to note, however, that the age of success for typically developing children on False Photograph tasks is similar to the age of success for False Belief tasks – again, an age at which their language has developed quite a bit. This would typically lead one to believe that False Photograph success may also be associated with a certain amount of language exposure. However, the homesigners, who have not acquired any established language, spoken or signed, succeeded on this task; this finding supports and strengthens previous findings (e.g., Peterson and Siegal, 1998) showing that success on False Photograph tasks may be dissociable from language experience. Interestingly, given the degree and duration of the homesigners' lack of exposure to language and education, we show that this ability may develop from life experience alone.

### STUDY 4: EXPERIENTIAL FALSE BELIEF

Thus far we have explored various visually based precursor abilities that have previously been related to False Belief success, and have shown that homesigners succeed on all of these tasks: Study 1 (Perspective Taking Level 1), Study 2 (Perspective Taking Level 2), and Study 3 (False Photograph) (see **Figure 1**). Homesigners' success on Perspective Taking Level 2 shows their ability to understand and consider another person's visual perspective of an object (e.g., Masangkay et al., 1974; Flavell et al., 1981). With respect to the relative timing of these precursor abilities, typically developing hearing children pass False Photograph and False Belief tasks at around the same point in development (Zaitchik, 1990). On the basis of this set of findings, one might predict that adult homesigners would also succeed on False Belief.

A number of previous studies have shown poorer performance on FB tasks among deaf individuals than their normally hearing peers. The consensus of these studies is that the relatively poor performance of deaf individuals does not result from the experience of being deaf itself, but from the consequent delay in language exposure (Peterson and Siegal, 1999; Rhys-Jones and Ellis, 2000; Woolfe et al., 2002; Courtin and Melot, 2005; Moeller and Schick, 2006; Morgan and Kegl, 2006; Schick et al., 2007; Meristo et al., 2012, among others). What these studies have shown is that ToM abilities are delayed commensurate with the degree of delay of exposure to sign language. But what if the child is never exposed to an established language? Does ToM never progress past a certain point? What if that child creates a system of communication themselves? Is that system enough to scaffold FB success? Most previous tests of false belief (e.g., Peterson et al., 2005; Peterson and Slaughter, 2006; Schick et al., 2007) relied on linguistically conveying a sequence of events and then explicitly asking the child a critical question such as "Where will Sally look for the marble?" [from the classic Sally-Anne task in Baron-Cohen et al. (1985)]. Less linguistic means of conveying the story events, including, for example, thought bubbles (Morgan and Kegl, 2006) or sequenced pictures (Pyers and Senghas, 2009) still rely on participants' experience with literacy conventions that is typically only gained in formal education settings. Due to their relative lack of formal education, homesigners have extremely



No participant passed the Unexpected Contents condition without also passing the Appearance/Reality condition.

<sup>7</sup>This particular version of the False Photograph task was chosen in anticipation of our False Belief tasks, which do not include a Change-of-Location question (as several False Photograph questions do) but do include Unexpected Contents and Appearance-Reality questions, which more closely relate to an identity question.

limited experience with books, stories presented as a sequence of pictures, or other sophisticated literacy conventions. Thus the current minimally linguistic, experiential approach gives us a means of probing homesigners' FB understanding that avoids these issues.

In line with previous studies showing a link between language experience and FB performance, Pyers and Senghas (2009) also found an effect of using a still-emerging language on FB success, comparing the FB performance of successive cohorts of signers of NSL. Both Cohort 1 and Cohort 2 signers in their study entered the signing community before the age of 6 years; in this way they are similar to the deaf children in the studies described above whose access to language is delayed. However, it is worth noting two differences: the NSL signers were tested as adults, and each cohort of signers entered a very different situation with respect to the type of language available in their environment, as we elaborate below.

Cohort 1 signers were the first creators of what is now NSL, and are from the same cohort of NSL signers as those in the current studies; Cohort 2 signers are the individuals who entered the Deaf signing community after Cohort 1, and who learned NSL from them. Pyers and Senghas (2009) found that Cohort 1 signers, who used fewer mental verbs when describing videos depicting belief or desire events, succeeded less frequently on False Belief tasks than the Cohort 2 signers, who used more mental verbs in their descriptions of those events<sup>8</sup> . Note that the Cohort 2 signers are the first in the Nicaraguan Deaf community to benefit from a language model, namely, the signing of the Cohort 1 signers who preceded them in the center for special education (vertical input)<sup>9</sup> . While they did not have language models, Cohort 1 signers benefited from peer-to-peer interactions (horizontal input) in the context of a linguistic community. The individual homesigners who are the focus of the current studies lack such a linguistic community. Thus, comparing the homesigners and Cohort 1 signers will reveal the contribution of Cohort 1's shared emerging language to FB success.

Despite scant access to language input and formal education, homesigners show remarkable abilities to access and express information. As noted earlier, they create relatively complex gesture systems that display structure at various levels of linguistic analysis (e.g., Coppola and Newport, 2005). They also spend their entire lives observing the behavior of the people around them, relying solely on the visual information accessible to them. Therefore, we ask: can life experiences apart from language, that is, visuo-social experiences, provide enough information about others' thoughts, beliefs and desires to scaffold ToM development in the absence of linguistic input?

### Participants

The participants were those described in **Table 2**: 4 Homesigners, 6 NSL Cohort 1 signers, and 8 Unschooled Spanish Speaking adults.

### Procedure

In order to minimize effects of having a shared language and/or educational experiences (such as those found when using picturecompletion tasks), we employed an experiential False Belief task developed by Pyers (2005). Instead of conveying a narrative using language, or relying on literacy conventions, this methodology uses the participant's own personal experiences in the course of the task to establish the false belief situation (Experience Phase). Then, in the Prediction Phase, the participant is asked to make predictions about another person's behavior (choices). Both phases impose minimal productive and receptive communication demands on the participant. We now describe the procedure in detail.

As described above, each participant was given first-hand experiences with Appearance-Reality (A/R) and Unexpected Contents (UC) false belief situations. They then participated in a prediction game in which they earned an incentive for making correct predictions. The procedure is described in great detail because the incremental, implicit understanding of the task instructions, and how participants should respond, are essential to our commitment to a minimally verbal procedure that fairly assesses the ToM abilities of homesigners in particular. **Figure 6** summarizes the 14 trials that each participant saw, first in the Experience phase, and then again in the Prediction phase.

### Phase 1: Participant as Experiencer

#### **Part 1: Control/training: Stickers**

All participants first engaged with six sticker trials of two types: "obvious choice" or "individual preference" (3 of each, totaling 6 trials). As an experiencer this phase familiarized participants with the process of choosing items from an array, and (nonverbally) demonstrated that a choice on a particular trial may be obvious (e.g., three stickers bearing identical images in which one is pristine, and the other two are crumpled or ripped), or that a choice might be based on one's preference (e.g., two differentcolored smiley face stickers) (**Figure 6**). The sticker trials also ensured that participants understood that they had to choose only one element of the array on each trial.

#### **Part 2: Appearance/Reality False Belief**

In the A/R experiencer phase, the participant saw three plates holding one, two, and four cookies. Unbeknownst to the participant, the four "cookies" were very convincing ceramic composite replicas. The experimenter encouraged each participant to indicate the "best" plate. For the homesigners, this was done by pointing at the participant, indicating the three plates of cookies, followed by a thumbs-up, thumbs-down gesture combined with a questioning look. All participants in all groups chose the plate with four cookies during their experience phase. After selecting this plate, they were then encouraged to try a cookie from that plate, at which point they discovered that the cookies were not real. The cookies were then returned

<sup>8</sup>This finding accords with studies in the United States showing that frequency of mental-state language in children's input (e.g., Howard et al., 2008) and ability to use mental-state language in complement clause structures (e.g., Hale and Tager-Flusberg, 2003; de Villiers, 2007) predict False Belief success.

<sup>9</sup> See Senghas et al.'s (2005) excellent description and analysis of the community structure and interaction patterns that catalyzed the emergence of Nicaraguan Sign Language, including the distinction between vertical (more mature/proficient model-to-learner) and horizontal (peer-to-peer) language input.

to their original locations and the question or gestures were repeated, this time to check the participant's knowledge that the plate only appeared to contain four "cookies," and that the plate containing two cookies should be considered the "best." At this point, all participants in all groups selected the plate with two cookies, demonstrating that they successfully internalized their false belief. That is, they understood that their previous belief that the four cookies on the plate were real was in fact, a false belief.

#### **Part 3: Unexpected contents false belief**

In the UC experiencer condition, the participant was shown one of the following arrays of four real objects: (1) a sheet of paper, a glass, a small padlock, and a candle; or (2) a notebook, a mug, a lockbox, and a box of cigarettes (**Figure 6**). The participant was then presented with a series of tools and was asked to indicate which object in the array each tool is used with. First, the participant was presented with a pen, and was asked to match it with an object. The correct choices were the paper (first array described above) or notebook (second array described above); the order of arrays was counterbalanced across participants. Upon choosing the paper/notebook, the participant was asked to make a mark on the paper/notebook. This was done for two reasons: first to show that the pen was functional, and second to establish a routine of "using" the tool presented with the object it was used with. Second, the participant was presented with a pitcher of water and asked to match it to its object (the correct choice

FIGURE 7 | A participant engaged in the Unexpected Contents portion of the Experiencer Phase. He has just opened the matchbox after having matched it with the cigarettes, and has discovered that the matchbox actually contains a key, not matches.

being the glass or mug). Again, upon choosing, the participant was asked to pour water into the vessel (**Figure 7**). Third, the participant was presented with a matchbox (which, unbeknownst to the participant, contained a key, but no matches), and again was asked to match it to one of the objects. Note that at this point there are two objects in the array that have not been matched to a

tool: one that could be lit with a match (candle or cigarettes) and one that was seemingly unrelated (a small padlock or a lockbox). As an experiencer, the first correct response should be based on the external appearance of the matchbox, and thus the correct match would be the object that could be lit: either the candle or the cigarettes. Upon choosing, the participant was encouraged to light the candle or a cigarette, and subsequently discovered that the matchbox contained a key, not matches (**Figure 7**).

The objects in the array were then switched out for their functional equivalents [i.e., array (1) to (2), that is, paper to notebook, etc.]. However, the three tools (pen, pitcher, and matchbox) were left in view of the participant, so he or she could see that no one, including the experimenter, touched them during this changeover. The entirety of the UC condition was repeated, now with the participant knowing what was in the matchbox, matching tools to their objects. At this point, the key trial is the matchbox trial, during which the participant should demonstrate his or her knowledge that the matchbox contained a key (instead of matches), and should therefore be matched with the padlock/lockbox, and not the candle/cigarettes.

### Phase 2: Participant as Predictor (Confederate as Experiencer)

To reiterate, after experiencing all 14 trials of the task, and more importantly, after directly experiencing the two false beliefs (the four objects on the plate only appear to be cookies, and the matchbox contains a key and not matches), each participant participated in all of the trials again, but this time as a predictor of another person's choices (**Figure 8**). The confederate whose choices were predicted by the participant was a member of the research team who had not previously participated in any aspect of this task with the current participant and who had remained out of sight of the participant and experimenter for the duration of the task up until this point. Moreover, the experimenter invited the participant to collude with the experimenter by emphasizing that the other person who would be brought in (the confederate) had "not seen" the game before. Finally, for each trial, the participant indicated the item he or she thought the confederate would choose before the confederate actually made a selection, by marking a set of laminated sheets depicting each array of objects.

### Part 1: Control: Sticker Trials

In the prediction phase, sticker trials served to implicitly instruct the participants that: (1) sometimes it is easy to predict someone else's behavior (obvious choice trials), (2) sometimes it is harder (individual preference trials), and (3) correct predictions earn a small monetary reward [5 Córdobas per correct prediction (max. 70 Córdobas or US\$2.75)] across all trials (a healthy incentive given typical local incomes).

Sticker trials also gave participants the opportunity to show their understanding that individual preferences might vary and/or encouraged participants to consider the other person's preferences in cases where the participant predicted that the confederate would choose the same sticker that they themselves had chosen in the experiencer phase. All participants experienced one or both of two possible outcomes: (1) making an incorrect choice when predicting the confederate's choice of sticker. In this case their subsequent failure to earn the incentive emphasized that they should consider the other person's preferences. In outcome (2), participants predicted sticker preferences for the confederate that differed from their own, showing that they indeed were considering the other person's preferences and not just going with what they (the participant) themselves had chosen previously as an experiencer.

Finally, the sticker trials served to train participants how to indicate their predictions of the confederate's choices by marking the laminated sheet displaying the array of objects associated with each trial. This was particularly important for the two groups who did not have educational experiences (the Homesigners and Unschooled Spanish Speakers, many of whom are not literate), and who may not have many opportunities to interact with images in this way or to use writing tools (dry erase marker). No participant displayed any difficulty in using these items during the task – they could indicate their choices by circling, making dots, or marking lines on the images they chose, and all participants showed their understanding of the task procedure during the sticker trials.

Note that while each Experiencer and Predictor phase began with sticker trials, the order of A/R and UC False Belief trials was counterbalanced across participants, as well as across the Experiencer and Predictor phases of the experiment for each participant.

### Part 2: Appearance/Reality False Belief

In the A/R prediction phase, the participant observed the experimenter ask the confederate "which is [the] best [plate of cookies]?" Before the confederate made his choice, however, the participant was asked to indicate which image on the laminated sheet (one, two, or four cookies) the confederate would choose (out of sight of the confederate, of course). Then the confederate made his choice of four cookies as the "best." Recall that the confederate at this point is acting as a naïve participant who should not know that the four cookies are fake – he should have a false belief given the appearance of the cookies – the same false belief the participant just experienced themselves. The participant is then told whether he/she was correct in his/her prediction and rewarded (or not). Just as in the participant's version of the experiencer phase, the confederate is then told to take a cookie from the plate he chose, at which point he "realizes" that the four cookies are fake. The fake cookie is returned to the plate and the knowledge question is posed to the confederate, with the participant predicting the confederate's current knowledge of the fake cookies, and earning the monetary incentive for the correct knowledge prediction. All participants in all groups correctly predicted that the confederate would choose the plate with two cookies at this point, regardless of whether their initial prediction was correct.

### Part 3: Unexpected Contents False Belief

In the prediction phase of UC, the participants were again asked to predict how the confederate would match the tools to the objects in the array before them by indicating their prediction on the laminated sheet for each trial, with correct

predictions earning incentive payments. Recall that the third tool (a matchbox containing a key) is the crucial match – the confederate continued the naïve act and first matched the matchbox based on the appearance of the matchbox – that it ought to have matches and thus be used with the item that could be lit (the candle or cigarettes). The participant was then told whether their prediction for the confederate's choice was correct, earning the appropriate reward for correct predictions. Importantly, the participant ought to have recalled their own false belief and predicted the candle or cigarettes. Regardless of the participant's correct/incorrect prediction, the confederate was then asked to proceed and "use" the matches to light the candle or cigarettes, at which point the confederate "realizes" that the matchbox contains a key and no matches. The array is switched out, as it was for the participant's experiencer phase, and the confederate is then asked the knowledge question(s), going through all three tools, with the participant making predictions now "knowing" that the confederate has realized what is in the matchbox, and earning the appropriate incentives through the remaining trials.

### Results

In this task, participants were asked to correctly predict a confederate's choices across a variety of preference trials and tool-object matching trials. The crucial questions were an A/R condition (one, two, or four cookies, with the plate of four cookies containing fake cookies) and an UC condition in which a box of matches actually contained a key. For the A/R condition, a participant passed if they initially (that is, before the confederate realized that the four cookies were fake) predicted that the confederate would choose the plate containing four "cookies." Participants failed this task if they initially predicted that the confederate would choose the plate containing two (real) cookies. For the UC condition, a participant passed if they initially predicted that the confederate would match the matchbox to the item to be lit (i.e., the cigarettes or the candle), and failed if they predicted that the confederate would match the matchbox to the item that needed a key (i.e., the padlock or the lockbox). None of the Homesigners, who lack a linguistic community, passed; however, immersion in a linguistic community did not guarantee passing for NSL signers and Unschooled Spanish Speakers. In sum, for the A/R condition, no homesigner passed, 3 of the 6 NSL signers passed (50%), and 4 of the 8 Unschooled Spanish Speakers passed (50%). For the UC condition, no homesigner passed, 1 of the 6 NSL signers passed (17%), and 3 of the 8 Unschooled Spanish Speakers passed (37%) (**Figure 9** and **Table 5**).

### Discussion

This experiential false belief study is the first to be conducted with a rarely studied population: homesigners, individuals free of congenital social impairments whose linguistic input is extremely limited. Despite generating gesture systems featuring a surprising degree of linguistic complexity, many years of communicative engagement with family members and friends, and relatively typical social and vocational experiences, none of the four adult homesigners passed either of the experiential false belief tasks. We now address (and reject) alternative explanations and interpretations of this poor performance.

## Alternative Explanations and Interpretations

### **Task issues**

One potential objection is that homesigners performed poorly because the task design was too difficult for anyone to succeed. The fact that 1/6 NSL signers and 3/8 Unschooled Spanish Speakers scored 100% (that is, succeeded on both the A/R and UC questions) demonstrates that the task is passable. A second possibility is that the homesigners did not understand the task. We are confident that the incremental nature of the task design, and the gradual escalation of task demands, ensured that homesigners understood what was expected of them. Below we argue that the homesigners did not respond randomly, that they in fact systematically performed worse than chance, and that their patterns of responding are interpretable given our theoretical framework.

Two patterns in the data indicate that the homesigners (and participants in other groups who we report as "not succeeding"

at the A/R or UC false belief questions) did not choose answers at random. First, these participants always chose the answer that they themselves knew to be the true state of the world. For example, in the A/R cookie trials, no one predicted that the confederate would prefer the plate with one cookie; everyone who responded in error predicted the plate with two cookie (the plate with the largest number of "real" cookies). Similarly, no participant (who answered incorrectly) predicted that the confederate would pair the matchbox with anything but the lock or lockbox, an even more striking "error" because superficially, matchboxes do not pair with things that lock. They then gave these same responses when asked the true belief question after seeing the confederate experience a false belief.

Participants' overall better performance on A/R trials supports the validity of the task. Recall that the order of false belief trial types was counterbalanced such that half of the participants received the A/R trial first, and half received the UC trials first. Those participants who made an error on their first false belief trial had the opportunity to learn from their failure to obtain the financial incentive. However, the pattern of results shows that participants in all groups failed to learn from their errors on the false belief trials, suggesting that the task indeed measured their false belief abilities rather than their ability to learn.

#### **Memory issues**

Memory problems, and not an inability to represent and manipulate the confederate's mental representation of the world, might account for homesigners' poor performance. Maybe homesigners forgot their previous experiences with false beliefs in the Experiencer phase; in that case, they would have responded in accord with the "face value" of the cookies on the plates and the matchbox and thus should have answered correctly that the plate containing four cookies was best, and that the matchbox matched the candle/cigarettes. Because they consistently chose the "correct wrong answer" – meaning the plate with two cookies (not the plate with one) and the padlock/lockbox (rather than the paper or the mug), they demonstrated that their memory of their previous experiences is intact and is in fact driving their choices.

### **Other methodological considerations**

Our extensive experience with the homesigners who participated in this study indicates robust comprehension and production of pointing and communicative intent (Coppola and So, 2005; D'Entremont and Seamans, 2007; Tomasello et al., 2007). Indeed we take advantage of this for all tasks in this study. Thus, the minimal-communication demands of the task are unlikely to be the source of the homesigners' difficulties. One drawback shared by many extant false belief studies with deaf and hard of hearing participants (including the present study) is that they were run by hearing experimenters. Deaf and hard of hearing children routinely experience that hearing people have information they do not have access to (e.g., Jenny is running down the hall); these cumulative experiences may lead them to generalize/over attribute knowledge to hearing experimenters in these types of tasks (Gagne, 2015).

Given previous studies showing delays in False Belief abilities in deaf children and adults with compromised access to language (e.g., Peterson and Siegal, 1999; de Villiers and Pyers, 2002; Morgan and Kegl, 2006; Schick et al., 2007; Pyers and Senghas, 2009), it is not surprising that the homesigners performed poorly. However, these results add to our understanding that ToM abilities do not emerge in adulthood on their own, as a result of many years of life experiences and social interactions, in the absence of linguistic input and participation in a shared linguistic community. Indeed, these findings highlight the contribution of language experience to the later ToM success of latelanguage acquiring deaf adults in studies such as Peterson and Wellman (2009) and O'Reilly et al. (2014). These studies all support the following conclusion: experience with language later in development (i.e., after early childhood) helps ToM skills, but it does not universally lead to success for all individuals. Importantly, the current study underscores the finding that a lack of exposure to a linguistic community uniformly results

in striking impairments in ToM understanding, even in mature individuals.

### Malleability of Theory of Mind Abilities in Adulthood

Our results also replicate the results of Pyers (2005), who employed (and innovated) the minimal-language, minimalcommunication Experiential False Belief task with signers of NSL from Cohorts 1 and 2. In fact, 5 of the 6 Cohort 1 signers in the current study were the same individuals tested by Pyers (2005). Strikingly, the performance of these 5 individuals did not improve, despite 10 additional years of life experience and social interactions. These results appear to conflict with studies showing that ToM abilities can improve later in life. How can we resolve this? One explanation might be that false belief performance relates not only to the age at which an individual is exposed to language, but also to the type of language they are exposed to (i.e., established or emerging) (Pyers and Senghas, 2009). The O'Reilly et al. (2014) participants were exposed to Australian Sign Language, an established sign language, which already had abundant mental-state verbs in its lexicon. Thus, a main difference between the late-exposed adults in O'Reilly et al. (2014) and the Cohort 1 signers in the current study (who were all exposed to sign by age 5) is their relatively greater access to mental-state verbs such as "believe" and "know." This lexical richness may have helped to develop the Australian participants' ToM understanding and growth into adulthood. Pyers and Senghas (2009) argue that the reduced availability of such mental-state verbs in Cohort 1 signing was the limiting factor for development of their false belief abilities. They further argue that increased exposure to those words over a 2-year period improved Cohort 1's performance, though not to ceiling, and not on all types of false belief tasks.

### Unschooled Spanish Speakers' Apparent Difficulties with the Task

Our most surprising result is that the Unschooled Spanish Speakers, who did have exposure to an established language from birth, did not universally succeed: only 4 of the 8 participants succeeded on the A/R task, and only 3 of 8 succeeded on UC. One noteworthy pattern is that a greater proportion of the Unschooled Spanish Speakers passed both the UC and A/R tasks, compared to the NSL signers, leading to higher overall scores. Given our focus on the role of language in the development of ToM abilities, we were initially a bit surprised that the unschooled hearing Spanish speakers did not succeed on all false belief tasks given their full access to an established language. However, recall that while the Spanish speakers, like Cohort 1, do engage others using a shared language (Spanish), they are also like the homesigners in that they do not have educational experiences. Several studies have demonstrated the positive influence of a shared community language on the development of ToM (e.g., Morgan and Kegl, 2006; Pyers and Senghas, 2009), as well as the positive impact of mental-state language (e.g., Howard et al., 2008) or even specific linguistic structures (e.g., de Villiers and de Villiers, 2000). Previous studies have also found that even adults with full access to language and education in westernized cultures do not necessarily perform at ceiling in behavioral ToM tasks requiring explicit responses (e.g., O'Reilly et al., 2014), or even implicit responses (e.g., Senju et al., 2009).

Indeed, studies of children in preliterate cultures show they do not always perform at ceiling (e.g., Avis and Harris, 1991; Vinden, 1996, 1999; Chasiotis et al., 2006) but do show developmental gains between 3 to 8 years of age (e.g., Callaghan et al., 2005). These children also show better performance on A/R tasks than on UC tasks (Vinden, 1996). These results parallel our current findings with the two non-Homesigning groups: participants (including the Unschooled Spanish Speakers) were more likely to pass the A/R task than the UC task, though neither group performed at ceiling.

We also understand the findings of the Unschooled Spanish Speakers in the following context. First, the hearing Spanish speakers in this study spend the majority of their time around familiar people and in familiar contexts. This is, in part, a consequence of how we selected them: we did not want to recruit participants who chose (or whose families chose for them) not go to school because of an endogenous factor (such as mild intellectual disability). Thus the participants in our study came from families where formal schooling was too remote, or deemed not necessary in order to sustain the family. Consequently, participants were from relatively self-sufficient families, and do not regularly interact with a wide variety of others. In this way they are well matched to the social interaction profiles of the homesigners.

Second, and related to the first reason, for both the homesigners and the Spanish speakers, other people tend to fill in gaps in social cognition, and thus these skills are not challenged to develop further. Third and finally, it is difficult to compare the false belief performance of Spanish speakers with no formal schooling to prior research with unschooled adults, because none exists that we are aware of (though we would be very happy to learn of it). Previous work has focused exclusively on the abilities of unschooled children (e.g., Avis and Harris, 1991; Vinden, 1996, 1999; Callaghan et al., 2005; Chasiotis et al., 2006, among others). It is important to keep in mind, too, that research with welleducated adults in developed countries also reveals surprising limits on their propensity (but not necessarily capacity) to use ToM skills in appropriate communicative contexts (see, for example, Keysar et al., 2003; Apperly et al., 2010).

Having addressed issues related to the use of the experiential false belief task with these understudied populations and their results and interpretations, we turn now to considering the overall pattern of results from this series of four studies, and their implications for our understanding of the relationship between mental and visual representations of others' experiences without the contributions of a shared language.

### OVERALL DISCUSSION

In a series of four developmentally sequenced studies, we investigated the possibility that a lifetime of socio-visual experiences scaffolds socio-cognitive development, particularly in the realm of ToM. We worked with three understudied populations in Nicaragua: (1) Homesigners, deaf individuals

who have extremely limited language input and educational experiences, (2) Cohort 1 NSL signers, the first group of deaf individuals to contribute to the creation of a new sign language in Nicaragua, who benefit from both a language community and educational experiences, and (3) Unschooled Spanish Speakers, hearing individuals in Nicaragua who have been exposed to spoken Spanish from birth, but who, like the Homesigners, have very little to no formal educational experience.

Across three tasks previously suggested to be precursor abilities to False Belief success [the gold standard task for measuring mature ToM abilities (e.g., Wimmer and Perner, 1983; Baron-Cohen et al., 1985)], we found that Homesigners either performed at ceiling (Perspective Taking Level 1 and False Photograph), or did not significantly differ from the two comparison groups (Perspective Taking Level 2). However, when presented with an innovative, minimally linguistic, experiential False Belief task (Pyers, 2005) which tested False Beliefs that can be conveyed primarily and effectively in the visual modality (A/R and UC), Homesigners as a group did not succeed (**Table 6**).

Our results show that Homesigners do not have difficulty in understanding that others may have perspectives of the world that differ from their own (Perspective Taking Levels 1 and 2), and that the current state of the world does not necessarily reflect a previous state (False Photograph). Their understanding of these things is not limited to an understanding of identity, but extends to differing perspectives of the same object (Perspective Taking Level 2). Homesigners' difficulties in the Experiential False Belief task is not due to problems with memory (they could have passed by "forgetting" their previous experiences and responding solely based on the external appearance of the items in question), or difficulties with understanding others' desires (they can successfully proceed through the control/sticker trials of the Experiential False Belief task, which asks about desire without presenting a False Belief about the stickers). They also made no incorrect predictions about the "true belief " questions – those that asked about the confederate's choice after the confederate realized the true state of the world. Thus, we can be confident that the homesigners understand that seeing is knowing, yet may still struggle to understand that not seeing is not knowing (as shown by their difficulty with false belief questions). Taken together, these results demonstrate that visual information, plus potential contributions of maturation, can get one pretty far along the ToM developmental trajectory (**Figure 1**). Importantly, the abilities in question are still limited to the visual realm – this accumulated experience gathering and processing visual information throughout a lifetime does not support the ability to predict another person's behavior in a false belief context.

### Probing Homesigners' Lack of False Belief Success

We suggest that the homesigners' failure to predict others' behavior is also not due to an inability to apply a metarepresentation, at least visually (Smith et al., 2013). However, the limitation may be one of linguistic meta-representation, which is not testable with homesigners, given the communicative structures available to them. Although the precursor abilities we investigated required consideration of another's mind in that they asked about another person's perspective (a visual representation), they did not necessarily ask about the content of another person's belief (a mental representation) (**Table 6**). It may be the case that the relatively greater lexical resources available to the Cohort 1 signers [i.e., mental-state verbs (Pyers and Senghas, 2009)], and to the Unschooled Spanish Speakers contributed to the successes of some participants in those groups. We also acknowledge that success may not be solely based on the availability of mental verbs, but the frequency of their use (e.g., Howard et al., 2008). We should note that not all Cohort 1 and Unschooled Spanish Speaker participants passed the false belief tasks, which begs the question of possible contributions of education and other language-use contexts that may employ mental-state language frequently. These individuals in Nicaragua (deaf or not) may not engage as often in the types of play that may use the kind of language that contributes to ToM development as we know it in the United States (e.g., "Hide and Seek").

Perspective Taking is likely a precursor to False Belief success. The current findings suggest that the complete lack of language input (for Homesigners) and the sparse mental language available (to Cohort 1) limits ToM development without hindering the development of Perspective Taking abilities.

### Implicit vs. Explicit Measures

While the Homesigners did not succeed on the Experiential False Belief task we conducted, we cannot necessarily conclude that they are incapable of understanding others' beliefs. We raise this point given previous studies with typically developing infants who, at the time of testing, also had not had much language or educational experience (e.g., Onishi and Baillargeon, 2005). Perhaps implicit measures of False Belief would be able to detect abilities in the homesigners that are not detectable using the Experiential False Belief task we employed here [e.g., the anticipatory looking task used by Senju et al. (2009)]. Though the current experiential false belief task avoids many of the linguistic and literacy convention demands imposed by the majority of false belief tasks, it does still require the participants to produce an explicit response (marking a prediction on the laminated sheet depicting the array of possible objects). Two previous studies of implicit FB in deaf infants and children show that difficulties in FB may start as early as 17 months (Meristo et al., 2012), and may persist into middle childhood (Meristo et al., 2016). What is left to explore, however, is whether the delay in implicit False Belief prediction found in these deaf infants persists into adulthood. Senju et al. (2009) suggest that for individuals with ASD, difficulties in implicit abilities persist into adulthood, though these same individuals are able to – using their increasing language abilities – overcome difficulties in explicit tasks. The homesigners may prove to be a population with the inverse pattern: language barriers that persist into adulthood but no congenital cognitive deficit to bar the development of implicit ToM abilities.


TABLE 6 | Summary of tasks and the aspects of representational conflict and self/other addressed by each, and performance of each of the groups in the current studies.

PTL1, Perspective Taking Level 1; PTL2, Perspective Taking Level 2; A/R, Appearance/Reality; UC, Unexpected Contents.

Positive findings with homesigners using an implicit measure could lend credence to two-system accounts of social cognition (e.g., Apperly and Butterfill, 2009). In these accounts, implicit abilities are available early in life, and possibly shared with non-human animals, but explicit responses are developed later by humans, and may depend on language. This suggestion would align with studies of the neural basis of False Belief and Perspective Taking abilities that show differential activation in the temporo-parietal junction (TPJ)/posterior superior temporal sulcus (pSTS) and the medial prefrontal cortex (MPFC). Aichhorn et al. (2006) suggest that the pSTS/TPJ may be responsible for making "cold" or factual judgments about others' minds (such as in Perspective Taking, where no behavioral prediction is needed), whereas the MPFC may be employed for behavior prediction. Given the results of our studies, we offer that the development of MPFC calculations of predicted behavior may be language-dependent, whereas the development of pSTS/TPJ computation may be divorced from language (either maturational, or developed from visual or language-independent social interaction). However, this is an empirical question beyond the scope of the current paper.

### CONCLUSION

In our view, the contribution of the current work is demonstrating that success in visual perspective taking does not automatically or inevitably lead to understanding others' unseen mental states, despite extensive observations of human interactions. The findings from the four tasks reported here, taken together, fill gaps in the existing literature regarding the relationship between visual socio-cognitive abilities and later (visually based) False Belief success. First, we contribute data on ToM abilities in one population that has not been studied previously (Homesigners), as well as from two highly understudied populations: signers of an emerging language (Cohort 1 NSL signers) and hearing Spanish speakers with little to no educational experiences. We found that all three groups are equally able to succeed on Visual Perspective Taking (Levels 1 and 2) as well as False Photograph, but that they vary in their abilities to succeed at False Belief. Additionally, we showed that while False Belief success may likely be language dependent (even when the task minimizes the need for communication and maximizes firsthand visual experiences), success on Visual Perspective Taking (Levels 1 and 2) and False Photograph tasks is likely independent from language experience. Further studies are needed, most likely involving implicit measures of false belief, to investigate ToM abilities in those who do not succeed at the current False Belief tasks, as well as possible neural correlates for such differential abilities.

### ETHICS STATEMENT

University of Connecticut Institutional Review Board Protocol # H10-306. All participants were consented according to an approved process by the University of Connecticut IRB protocol H10-306. Both authors are native American Sign Language signers, are fluent in spoken Spanish and have also had previous extensive contact with the homesigners in this study as well as with the Cohort 1 signers. All signed/gestured consent interactions were videotaped and a simplified consent information sheet was created to accommodate basic reading abilities for Cohort 1 signers and Unschooled Spanish Speakers. Homesigners were consented with a hearing, Spanish speaking relative present who was familiar with homesign to confirm understanding. Cohort 1 participants were usually consented in groups of 2–3, to allow for group questions and clarifications as needed. Because the majority (if not all) of the Unschooled Spanish

Speakers had minimal reading abilities, the consent form(s) and information sheet(s) were read aloud to the Spanish speakers with ample opportunity for questions. All participants indicated their agreement to the consent process by printing or signing their name, or making their mark.

### AUTHOR CONTRIBUTIONS

DG and MC contributed to the stimuli creation, DG collected the data, DG and MC analyzed the data and wrote the manuscript.

### REFERENCES


### FUNDING

This work was supported by NSF GRF # DGE-1247393, El Instituto: Institute of Latina/o, Caribbean, and Latin American Studies at the University of Connecticut, and IGERT/NBL International Experience grants to DLG, and NSF #1227908 to MC and Diane Brentari. We gratefully acknowledge all of the Nicaraguan participants and the indispensable support of Dr. Claudia Molina and Leybi Tinoco. We thank Russell Richie, Kurt Gagne, Cassandra Svelnys, Julia Adell, Karianna Montalvo, Jennie Pyers, Ann Senghas for their myriad contributions to this work, as well as the reviewers for their observations and suggestions, which greatly improved the manuscript.



of a visual language. Brain Res. 405, 284–294. doi: 10.1016/0006-8993(87) 90297-6



**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Gagne and Coppola. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# American Sign Language Syntax and Analogical Reasoning Skills Are Influenced by Early Acquisition and Age of Entry to Signing Schools for the Deaf

, Rama Novogrodsky3,4 and

#### Edited by:

Jon Henner<sup>1</sup>

Robert Hoffmeister<sup>4</sup>

Matthew William Geoffrey Dye, Rochester Institute of Technology, USA

#### Reviewed by:

Gary Lupyan, University of Wisconsin–Madison, USA Diane Lillo-Martin, University of Connecticut, USA Ronnie Bring Wilbur, Purdue University, USA Patrick Boudreault, Gallaudet University, USA

> \*Correspondence: Jon Henner J\_henner@uncg.edu

#### Specialty section:

This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology

Received: 09 September 2016 Accepted: 05 December 2016 Published: 26 December 2016

#### Citation:

Henner J, Caldwell-Harris CL, Novogrodsky R and Hoffmeister R (2016) American Sign Language Syntax and Analogical Reasoning Skills Are Influenced by Early Acquisition and Age of Entry to Signing Schools for the Deaf. Front. Psychol. 7:1982. doi: 10.3389/fpsyg.2016.01982 <sup>1</sup> Professions in Deafness, Department of Specialized Education Services, University of North Carolina at Greensboro, Greensboro, NC, USA, <sup>2</sup> Department of Psychological and Brain Sciences, Boston University, Boston, MA, USA, <sup>3</sup> Department of Communication Sciences and Disorders, University of Haifa, Haifa, Israel, <sup>4</sup> Programs in Deaf Studies, Center for the Study of Communication and the Deaf, Boston University, Boston, MA, USA

\*, Catherine L. Caldwell-Harris<sup>2</sup>

Failing to acquire language in early childhood because of language deprivation is a rare and exceptional event, except in one population. Deaf children who grow up without access to indirect language through listening, speech-reading, or sign language experience language deprivation. Studies of Deaf adults have revealed that late acquisition of sign language is associated with lasting deficits. However, much remains unknown about language deprivation in Deaf children, allowing myths and misunderstandings regarding sign language to flourish. To fill this gap, we examined signing ability in a large naturalistic sample of Deaf children attending schools for the Deaf where American Sign Language (ASL) is used by peers and teachers. Ability in ASL was measured using a syntactic judgment test and language-based analogical reasoning test, which are two sub-tests of the ASL Assessment Inventory. The influence of two age-related variables were examined: whether or not ASL was acquired from birth in the home from one or more Deaf parents, and the age of entry to the school for the Deaf. Note that for non-native signers, this latter variable is often the age of first systematic exposure to ASL. Both of these types of age-dependent language experiences influenced subsequent signing ability. Scores on the two tasks declined with increasing age of school entry. The influence of age of starting school was not linear. Test scores were generally lower for Deaf children who entered the school of assessment after the age of 12. The positive influence of signing from birth was found for students at all ages tested (7;6–18;5 years old) and for children of all age-of-entry groupings. Our results reflect a continuum of outcomes which show that experience with language is a continuous variable that is sensitive to maturational age.

Keywords: ASL acquisition, deaf children, analogies, syntax, age of acquisition

## INTRODUCTION

fpsyg-07-01982 December 22, 2016 Time: 15:54 # 2

Studying language deprivation in Deaf children has led to important findings on brain plasticity and sensitive periods in human development (Corina and Singleton, 2009). The temporal lobes, important for processing and understanding auditory language, are activated by sign language in congenitally Deaf subjects, indicating striking neural plasticity (Nishimura et al., 1999). In the realm of language acquisition, the topic of this paper, studying Deaf individuals with late acquisition of a first language has helped quantify the notion of sensitive periods for language learning (e.g., Mayberry and Eichen, 1991; Mayberry, 1993). Mayberry and colleagues have shown across two decades of research that outcomes for learning a first language, and any subsequent languages, are progressively worse for individuals with later age-of-acquisition (Mayberry and Lock, 2003; Boudreault and Mayberry, 2006; Mayberry, 2010).

The notion of "windows for language and cognitive development" that are briefly opened and then closed continues to be debated by scientists, educators, and the public. The keen interest in this topic has also engendered myths and misunderstandings, alongside genuine unknowns (e.g., Giraud and Lee, 2007). To set the stage for presenting our data on a large naturalistic sample of over 600 school-aged Deaf children, we discuss two claims about the timing of acquisition of sign language that are based on little or no data but used to buttress advice to parents of Deaf children.

### (1) Time Windows for ASL Acquisition Close Too Soon

Knoors and Marschark (2012) claimed that the hearing parents of Deaf children may not be able to learn ASL well enough, or quickly enough, for their children to benefit from sign language used in the home. They cite data on sensitive periods for language acquisition, including the studies we cited above, of delayed language acquisition of ASL by Mayberry and Lock (2003) and Mayberry (2010). They further asserted that children of hearing parents are exposed to insufficient sign language at home during early and middle childhood, the time when the brain has maximal plasticity for learning a first language (Mayberry and Lock, 2003). Given this advice, hearing parents may hesitate to enroll their Deaf children in an academic signing environment, either as part of an early intervention program, or a preschool. They may infer that little is to be gained by learning sign language themselves, if they cannot become fluent quickly enough, or attain a high enough level soon enough, to facilitate their children's language and cognitive development.

### (2) Can Sign Language Be the Back-Up Communication Method If Speech Training Fails?

Many medical, speech, and language therapists, audiological, and early intervention professionals recommend to parents that they not teach their Deaf children sign language lest it interfere with the acquisition of spoken language (although there is considerable diversity of option among practitioners, see Mellon et al., 2015). But, parents are sometimes told they can consider sign language as a back-up language in case speech therapies do not work (see Mauldin, 2016 for more information). Underlying these recommendations is the findings from neuroimaging studies that visual processing can "take over" neural regions that mediate hearing and spoken language in children (Nishimura et al., 1999). However, research indicates sign language acquisition does not interfere with spoken language acquisition (Hassanzadeh, 2012; Davidson et al., 2014). Indeed, there is emerging evidence that sign language may actually facilitate spoken language development and that it may counter the cognitive effects of language deprivation (Davidson et al., 2014; Amraei et al., 2017). The aspect of this debate addressed by our data concerns the advisability of using sign language as a back-up system.

It is noteworthy that these two claims give opposite advice to hearing parents of Deaf children. The first one tells hearing parents that sign language cannot be learned past a critical period, so do not bother placing your child in a school that uses ASL. The second claim advises parents that they can always switch to sign-language later in childhood, thus implying that maturational constraints on learning sign-language are minimal. Critical to this discussion is the fact that knowing a language fluently is a major factor in learning about the world and crucial for success in an academic environment. Additionally, for Deaf children, research supports the notion that fluency in a first language (e.g., ASL) supports learning a second language (Novogrodsky et al., 2014a). In fact, several studies have found that ASL knowledge support the development of English literacy skills (Lange et al., 2013; Andrew et al., 2014; Ausbrooks et al., 2014; Hrastinski and Wilbur, 2016). Hrastinski and Wilbur (2016) examined the relationship between ASL abilities and performance on several different standardized assessments, including the Stanford Achievement Test 10th edition, and the Measures of Academic Progress and their associated subtests. ASL proficiency was the most significant predictor of performance on the different assessments; more important than home language, whether or not the child was implanted, whether or not they had a speech or language impairment, or how old they were during assessment.

Important to both these claims are the actual ASL skills of Deaf children who vary in their entry to an academic signing environment. Our data addresses this, together with the comparison between signing from birth with Deaf parents vs. absent or unsystematic signing with hearing parents. As further background for our study, we review below what is known about language development in Deaf children.

### LANGUAGE DEVELOPMENT IN DEAF CHILDREN

Language deprivation occurs when Deaf children are not exposed to sign language from birth and gain minimal information from spoken language. Delays in language milestones are typically observed in non-native signers and late-signers, broadly conceptualized as systematic exposure after age 5–7. The negative

effects of late linguistic exposure are present in either the signed or spoken language modalities (Mayberry, 2010). Late exposure has far-reaching consequences. The most important are cognitive and social impairments which then compound difficulties adjusting to the mainstream classroom and larger society (e.g., Branson and Miller, 1993; Ramsey, 1997; Nunes et al., 2001). Late exposure often leads to behavioral problems, mental illness, and substance abuse (Black and Glickman, 2006; Glickman, 2007; Anderson et al., 2015).

In contrast to Deaf children with language deprivation, children acquiring sign language as a first language (L1) from birth pass through the same language acquisition stages and achieve the same cognitive milestones as do children acquiring spoken language as their first language (Newport and Meier, 1985; Petitto, 1987; Goldin-Meadow and Mylander, 1998; Mayberry and Squires, 2006; Corina and Singleton, 2009; Novogrodsky et al., 2014b). The primary barrier to acquiring sign language as a first language is that 95% Deaf children are born to hearing, non-signing parents, who most frequently use only spoken language (Mitchell and Karchmer, 2005). According to survey research from 2009 to 2010, 5.8% of hearing parents in the US reported using ASL (Gallaudet Reseach Institute, 2011).

Today, with the focus on early intervention and cochlear implants, many Deaf children have increased access to spoken language. But implants do not give sufficient support for spoken language acquisition to be successful for all Deaf children (Mellon et al., 2015). While attrition rates for implant users vary and appear to be low (Watson and Gregory, 2005; Archibold et al., 2009), not all children can use implants well enough to acquire spoken language to levels characteristic of their hearing peers (Niparko et al., 2010; Geers and Sedey, 2011; Geers et al., 2015). Implanted children also have poorer executive function abilities than typically developing children (Figueras et al., 2008; Kronenberger et al., 2013). They do not seem to acquire the same kind of language-based reasoning skills as hearing children (Edwards et al., 2011). We cite these findings not to disparage cochlear implants, but to remind readers that cochlear implants do not fully remediate language deprivation in Deaf children.

There is also now considerable evidence that learning sign language does not interfere with learning spoken language. Deaf children can be bi-modal bilinguals, as shown in these two research studies:


Children in these studies were able to acquire both sign language and spoken language without conflict. They demonstrated language performance within the normal agerange in both modalities (Davidson et al., 2014). These bimodal children preformed at monolingual English age-targets on standardized language tests, including the Preschool Language Scale test (Zimmerman et al., 2002) and the Expressive Vocabulary Test (Williams, 2007). Receiving natural language input via the visual modality apparently minimized the negative effects of early auditory deprivation (Davidson et al., 2014). This allowed spoken language acquisition to be acquired in a time frame that is later than standard first language acquisition. This indicates the positive effects that are possible following early intervention.

Language development in Deaf children is tied to the development of language based analogical reasoning skills. Edwards et al. (2011) argued that historically low performance of Deaf children on analogical reasoning assessments, including language based analogical reasoning, may be related to language deprivation. Bandurski and Galkowski (2004), studied this in a sample of Deaf children who were native signers of Polishsign language. When given an analogical reasoning assessment using polish-sign language, native signers performed on par with typically developing hearing children who were given an equivalent assessment in written Polish. Henner (2016), in his dissertation, demonstrated that the best predictor of performance on an ASL language based analogical reasoning assessment was ASL vocabulary ability, thereby building on the work of Bandurski and Galkowski (2004).

### THE EFFECT OF AGE OF ACQUISITION ON SPOKEN AND SIGN LANGUAGE DEVELOPMENT OF DEAF CHILDREN

As mentioned briefly above, the most compelling evidence regarding maturational constraints on first language learning has relied on late first language learners who are usually Deaf (e.g., Newport, 1990; Pénicaud et al., 2013). Newport (1990) discussed the ASL abilities of 30 adults, aged 35–70 years, who had a minimum of 30 years' daily exposure to ASL, but differed in the age of first exposure to ASL. All participants were near ceiling in their understanding of basic ASL word order. Late learners (those exposed to sign language after age 12) had difficulties with morphology. They produced frozen signs, omitted obligatory morphemes, and were inconsistent on test items requiring the same morpheme. Early learners (who entered the school for the Deaf between ages 4–6) also had lower scores on tests of morphology than those who were exposed to ASL from birth from Deaf parents, indicating the importance of early and consistent ASL exposure at home.

Age of acquisition effects on language development have been more extensively studied by Mayberry and colleagues (Mayberry and Fischer, 1989; Mayberry and Lock, 2003; Boudreault and Mayberry, 2006; Mayberry and Squires, 2006; Mayberry, 2007). Different aspects of sign language have been studied: syntactic acquisition (Boudreault and Mayberry, 2006), narrative comprehension (Mayberry and Fischer, 1989), sentence memory (Mayberry and Fischer, 1989), sentence interpretation (Mayberry and Lock, 2003), and on-line grammatical processing (Mayberry and Eichen, 1991). These studies revealed that age of acquisition has long-lasting effects that are observable even when learners

were adults who were tested after years of sign language experience (see also Mayberry, 1992).

The general public often learns about late first language acquisition through the exceptional case study of Genie (Curtiss, 1977), which involved extreme neglect and physical abuse. It remains unappreciated by the public at large that language deprivation among Deaf students continues to be common, even the norm. Language deprivation can occur in stable, loving families who work to provide their children with language.

It may also not be well known among parents that early intervention programs exist to support Deaf children's acquisition of language, both signed and spoken. Hearing parents can expose their Deaf children to sign language by enrolling them into schools for the Deaf, where both peers and teachers use ASL. First exposure to ASL for hearing children is thus frequently the age of entrance to a school for the Deaf. Age of entry to a school for the Deaf is widely used in psycholinguistics to index of age of acquisition (Newport, 1990; Mayberry and Eichen, 1991; Mayberry, 1993; Mayberry et al., 2002, 2011; Henner et al., 2015; Novogrodsky et al., under review).

Age-of-entry to a school for Deaf varies greatly; there are several reasons why this is the case. The current policy in special education is for Deaf children to attend their local public school (mainstreaming) (e.g., Ramsey, 1997). Thus, when students transfer into a school for the Deaf, they have often transferred from either a non-signing school program or a non-ASL program. An example of a non-ASL program is one that uses an artificial signing system, like Signed Exact English (SEE). Such transfers often occur when students have failed academically in the mainstream education environment, typically due to problems caused by language deprivation. The majority of Deaf children of hearing parents appear to transfer into schools for the Deaf after the age of 6 (Henner et al., 2015).

In the current study we examine age effects in acquiring the different domains of syntax and vocabulary-based analogical reasoning. Prior work on age-effects in syntactic acquisition found difficulties when ASL was acquired between age 5–7, and even stronger difficulties when age of acquisition was between 8 and 13 (Boudreault and Mayberry, 2006). Boudreault and Mayberry studied adults; we attempt to replicate and extend those findings by testing school age Deaf children. Vocabulary-based analogical reasoning is also highly dependent on language skills and would also be affected by late age-of-acquisition (Richland and Burchinal, 2013). It is noteworthy that previous research (Sharpe, 1985) argued that auditory stimulation is necessary to develop language-based analogical reasoning skills. We examine these issues via a large sample of Deaf children growing up as visual learners, with ASL as the primary language.

### OVERVIEW OF METHOD

As reviewed above, substantial literature exists on the importance of early exposure to signed language<sup>1</sup> for later language development. We wanted to extend these findings to a large cohort of school-aged Deaf children (ages 7;6–18;5), and to ask more detailed questions about long-term outcomes of early signed language exposure, and, in the more specific case, ASL. Is experience with ASL from parents at birth more important for syntactic acquisition compared to acquisition of vocabulary? Deaf children frequently enter a school for the Deaf, where ASL is used in an academic context, at different ages. Does the age of entry to a school for the Deaf influence later ASL ability? Which of these factors, early experience with the language at home with parents, vs. systematic exposure to ASL in a school setting, is more important?

Our questions required participants with different histories of exposure to ASL. This required a larger sample than has been common in prior research with Deaf students. We were able to test the largest sample of Deaf children in the United States by collaborating with Deaf schools. This was also a necessity because the current project was part of our team's larger goal of developing the American Sign Language Assessment Instrument (ASLAI), a computerized inventory of sign language assessments (Hoffmeister et al., unpublished). As part of a funded project by the US Institute of Education Sciences running from 2010 to 2015, and with agreements with the schools for the Deaf where we tested, we were able to secure "blanket consent" through the Boston University Institutional Review Board. Blanket consent meant that parents needed to opt their children out of assessment rather than opt in. In exchange for blanket consent, schools were provided with detailed reports about their students' ASL abilities for use in individualized education program (IEP) planning. Blanket consent allowed us access to large and varied numbers of Deaf children.

The schools we targeted were residential schools for the Deaf with at least 100 or more students. These schools typically have relatively high numbers of Deaf teachers and staff, including native ASL fluent adults who provide high-quality visual language input. Schools for the Deaf tend to be favored among Deaf parents. These schools use ASL as the medium of academic instruction and thus have high levels of ASL classroom use in the classroom. Residential schools also have a large ASL-using peer population. Students thus learn ASL naturalistically via immersion, during ASL-mediated activities after school or during free time with both peers and adults. These schools have strong early intervention programs, meaning ASL exposure can begin in infancy. Given these characteristics, the environment of the residential schools facilitates both natural language acquisition and provides exposure to academic language in the classroom. One of our goals is to determine how much students can benefit from these sign-rich environments even if age of entry to the school for the Deaf occurs later than early childhood, during the elementary school years (age 6–12).

To make testing manageable at diverse schools around the country, we focused on collecting information about the students from the schools, and did not additionally survey parents. Schools for the Deaf record how many Deaf parents are at home, the year students entered the school, and other background information. We relied on these records and thus do not have information about the amount of signing at home, nor do we have key information about socio-economic status. We also do not know about any sign language exposure prior to entry into the school of assessment.

Henner et al. ASL Early Acquisition

Relying on information provided by schools has the benefit that we have generally the same information for all students. It obviates the drawback of missing and subjective data that is customary with reliance on parental surveys. School records on whether students had Deaf parents allowed us to categorize those students as being native signers, meaning exposed to sign from birth. We categorized students with hearing parents as nonnative signers, meaning that exposure to ASL before entry into the school of assessment was likely either absent or erratic in nature (Mitchell and Karchmer, 2005). Our primary variables were thus: (1) Native/non-native signing, and (2) Age of entry to the current school of assessment. Although these are imperfect measures of our underlying theoretical constructs (see below), they proved to be powerful predictors of students' language outcomes, with effects that were measurable well into the teen years.

Our variables, operationalizations, and predictions were as follows:

### Native vs. Non-native Signing Status

Operationalized by having at least one Deaf vs. hearing parent, this variable captures the theoretical construct of early/systematic exposure to language vs. late/uneven exposure. We predicted better ASL for native signers, and predicted that this advantage would hold across all ages tested (i.e., from the onset of schooling until age 18;5, the age of ending high school).

### Age of Entry to School of Assessment

This was operationalized as the year that students entered the school where they were tested. This is frequently students' first exposure to a consistent signing school environment that includes peers with signing abilities of equal or better skills. For a substantial majority of non-native students, date of entry to the current school for the Deaf represents the first systematic exposure to a signed language, and in some cases their first systematic exposure to any accessible language. It is thus related to the theoretical construct of age of first language acquisition. Note that the classic studies of the influence of late exposure to sign language used date of entry to a residential school for the Deaf as the onset of acquisition of ASL (see Henner et al., 2015 for review). However, when interpreting the variable of date of entry to the school for the Deaf, it is important to bear in mind that our population includes students with Deaf parents (native signers). For native signers, date of entry to the current school signals the onset of classroom ASL and the challenges and rigors that academic language presents compared to conversational language (Cummins, 1982).

Given that age of entry to school is related to first systematic exposure to ASL for non-native signers, we predicted that younger age of entry to school would correspond to higher scores on the two ASLAI tasks. This reflects the widespread idea in the language acquisition literature of "the younger the better" (see Singleton and Ryan, 2004). However, there are two caveats which make age of entry to school an exploratory variable. First, some students will have attended a signing school before the current school. For this reason, age of entry to the current school may overestimate that real average age of first exposure to signed language. It also reduces the opportunity to find that a late date of school entry is associated with poorer language outcomes. The second problem goes in the reverse direction. Late entry to a school of the Deaf is often a response to low academic achievement in a mainstream school setting. These students thus enroll at their first signing school not simply with advanced age, but with a history of poor language and academic failure. This will increase the association between late age and poor ASL ability.

### MATERIALS AND METHODS

### Participants

Data for this study came from 688 Deaf students between the ages of 7;6 and 18;5 years from schools for the Deaf in the US who scored above chance (25%) on the tasks used to collect data (**Table 1**). The students attended the large schools mentioned above, which are considered "signing schools." That is, they use sign language as the primary language of instruction. The students were administered the American Sign Language Assessment Instrument (ASLAI; Hoffmeister et al., unpublished). Two of the ASLAI tasks were analyzed for this study: a language based analogical reasoning task that represents vocabulary and grammatical knowledge and requires metalinguistic skills (Henner, 2016), and a syntactic judgment task representing ASL syntactic knowledge (Novogrodsky et al., under review). While all 688 students were tested on the analogy task, only 455 took the more recently developed syntactic judgment task. Parental hearing status was used as an approximate indicator of exposure to signed language from birth. The label "native" was given to participants who had at least one Deaf parent. The rationale for this categorization is that 92% of families with two Deaf parents use ASL at home and 84% of families with one Deaf parent use ASL at home (Mitchell and Karchmer, 2005). We thus considered 244 students to be native signers. As noted earlier, many of the remaining 444 non-native signers may not have been exposed to ASL until they entered the school for the Deaf where they were tested. Although the native signing group was smaller than the non-native signing group, they composed 35% of the sample population of this study. This is a significant representative sample considering it is estimated that only 5–10% of Deaf children are born to Deaf parents (Mitchell and Karchmer, 2005).

An important variable for our analyses was the age at which students entered the school of assessment, as a proxy for the likely onset of systematic ASL exposure. For native signers, age of school entry indicates the onset of exposure to classroom ASL (i.e., academic ASL), which is significant given that schooling extends and enriches language abilities even for native speakers (e.g., Cummins, 1982). Three groupings were made based on the age of entry to the school of assessment. These groupings were 0–6 (early intervention programs to kindergarten/1st grade), 7–12 (elementary school), and 13–18 (post-elementary school). **Figure 1** shows the proportion of the frequency count for the different ages at which native and non-native students entered the schools for the Deaf.

A higher percentage of native signers entered a signing school for the Deaf by age six compared to non-native signers (63% vs.


TABLE 1 | Number of students by task, age, and signing status.

52%). For date of entry between age 7–12, the percentages were 27 and 32%, respectively, and for entry after age 12, 9 and 16%. The sample of non-native signers in this study, compared to the US population of non-native Deaf signers, likely includes a higher percentage who entered a school for the Deaf before the age of six. Nevertheless, one can see that age of entry into a signing school is generally earlier for students with Deaf parents than those with hearing parents. It is worth noting that our sample also contained a large number of native signers who entered schools for the Deaf after the age of 6. Because we do not have information about students prior to entering the school of assessment, we cannot describe the exact reasons for this placement. Some of these students may have attended a different Deaf school prior to the school of assessment. Just like hearing parents, Deaf parents may have tried mainstreaming, oralism, or other educational options before deciding to place their child in a traditional school for the Deaf.

# Materials

Novogrodsky et al., under review). (a) The Syntax Task

The measures used in this study were two subtasks of the ASL Assessment Instrument. The ASLAI is a comprehensive displaying both pictures and videos. For most tasks, a stimulus is displayed, followed by four videos which are presented sequentially. Participants can interact with the videos to pause or replay them. Responses are selected by clicking a button above a corresponding video (e.g., A–D). While the test order was randomized, the question order was fixed for each participant. It is comparable in scope and content to spoken language tests administered to school-aged children. For assessing ASL syntactic knowledge, the Syntax Difficult subtest (for comparison, there is a Syntax Simple subtest) was used; for assessing vocabulary and related metalinguistic skills, the Analogy subtest was used. Here we briefly review the specific design of each task (see also

battery of 11 multiple choice receptive language tasks presented via computer. Video presentation of stimuli and responses vary according to the task. The ASLAI platform is capable of

This syntactic judgment task includes 27 test items designed to tap knowledge of nine syntactic ASL structures (roughly following Boudreault and Mayberry, 2006): (a) Topicalization,

(b) Subject-Verb-Object (SVO), (c) Complements, (d) Relative Clauses, (e) Verb agreement, (f) Negation, (g) Conditionals, (h) Wh-Q, and (i) Rhetorical Questions. The Syntax test in the ASLAI was modeled after Boudreault and Mayberry's assessment, but it differs in crucial ways. First, the assessment developed by Boudreault and Mayberry had 168 sentences that functioned as individual questions (stimuli). These questions were either grammatical or not grammatical. In the ASLAI, a question was composed of four different videos: a stimulus item containing a grammatically correct sentence and three foils containing different syntactic violations. Glosses of a sample complement structure question are presented in 1a–d below. 1a is a gloss of the correct response, 1b and 1c are foils with word order violations, and 1d is a foil with incorrect co-indexing between FRIEND<sup>j</sup> and HE<sup>k</sup> representing a grammatical violation. Items varied in types of foils, such as word order, or incorrect non-manual markers.


Second, Boudreault and Mayberry (2006)'s assessment was designed for adults and their sample was composed of 30 Deaf adults. The Syntax test in the ASLAI was designed for children.

#### (b) The Analogy Task

It is an ASL analogical reasoning assessment. Items were presented using the classical sentence presentation, A : B :: C : ?. Students viewed a signed sentence in which the A, B, and C parts were signed, followed by the ASL sign for WHAT (five handshape, both hands). They then were presented with four signed responses, of which one was the intended target. The task was composed of 24 different questions that corresponded to one of six different types of analogies: (a) causality, (b) antonym (opposites), (c) part-whole, (d) ASL phonology, (e) purpose, and (f) noun-verb pairs (a derivational process common in ASL that changes a verb to a noun through reduplication).

### Testing Procedures

Participants completed the Analogies and Syntax tasks in groups of up to 20 students, with typical group sizes ranging from 5 to 20. The ASLAI computer platform presented the two tasks in four different phases using only ASL via video windows. We describe the three phases of each of the two tasks here.

#### Instructions

Students viewed instructions for each of the two tasks, presented in ASL in the actual task, but translated into English for ease of reading in this paper. The instructions for the syntax task were as followed: "Now you will take a different test. This test is a syntax test. You will see four different video windows presenting signed sentences. The format will be the same as the previous tests you have taken, with four separate movie windows on your screen. Each movie window will play one sentence. You need to carefully watch each sentence and decide which one of the four sentences is correct. Three of the sentences are wrong – the signing is wrong/incorrect and the way the signs fit together do not make sense. One sentence is right – it is produced correctly, the sign order is correct, facial expressions are correct, and everything fits together in the right way. You need to pick that one correct sentence."

The instructions for the analogy task are as follows: "Now you will take an analogies test. What's an analogies test? First you'll see three signs. Two, A and B, have a relationship. The third sign C has the same relationship with D as A and B. Where is D? You have to find it from the four signs shown. How will you decide which one is the right one? Well, the first two signs A : B have a relationship and you will use that relationship to find the correct sign for C :."

#### Practice

Before each task, students viewed practice items and were provided feedback on whether or not their selection was correct. Students were instructed to select the response that best reflected a correct response in ASL.

#### Task Procedures

In the Analogies task, for each of the 24 items, the testing platform presented a signed stimulus item that included an item that was missing the response in the second part of the analogy equation, this was followed with four videos containing one target sign and three foil signs. In the syntax task for each of the 27 items, the testing platform presented an instructional sentence for the stimulus (CHOOSE CORRECT ASL SENTENCE), and four video stimuli, which contained one target sentence and three foil sentences. Upon completion of each task a review screen appeared allowing students to go over their choices and make corrections before the final choice was made. In the review screen, the target item was displayed along with the freeze frame of the student's selected response. Students could also return to a selected item and confirm their selection or select a different response if they choose.

### RESULTS

### Hypothesis Testing With Mixed Linear Effects Models

Our goal was to determine the effects of early language experience with ASL on subsequent ASL skills by measuring outcomes during the school years. Effects in two domains were investigated: ASL syntactic judgment and language-based analogical reasoning. We were interested in two ways to measure early language experience:


We were further interested in which one of these types of early language experience was more important for subsequent ASL outcomes.

Our analysis was accomplished using mixed linear effects models, with model fit measured by maximum likelihood estimations (MLL; Baayen et al., 2008). Statistical significance of the predictors was calculated by using chi-square to compare a new MLL value to the prior MLL value without that predictor in the model.

### Age of Entry Into the School of Assessment

Age was included as a fixed effect to control for age-based abilities. We first ran the analysis with only age as a fixed effect, and test takers (students) as a random effect. We then added age at date of entry (as a fixed effect) and assessed whether this resulted in a significant change in the maximum likelihood estimation. Adding age at date of entry did improve the model fit for both syntactic judgment [χ 2 (5) = 33.95, p < 0.001], and language based analogies [χ 2 (5) = 30.65, p < 0.001]. In both models, the students accounted for roughly 2% of the variance (see summary in **Table 2**).

The data in **Table 2** provide verification that increasing age-of-entry to school is associated with lower scores on both the syntactic and analogy tasks. The beta values shown in **Table 2** are the same for the two tasks, with the – 0.01 value indicating that age-of-entry is a weak predictor. We suspected this was because the relationship between ageof-entry and the dependent variables was not linear. In the next analysis, age-of-entry to school was grouped into three categories, as described in the methods section. This allowed us to examine the decline in syntactic judgment and language based analogical reasoning skills with increasing age of school entry for specific age-of-entry periods. We could thus determine more precisely, where in childhood the maximum decline occurred and how strong it was. In the next analysis, we also added in native vs. non-native signing status.

#### Native vs. Non-native Signing Status

Because we changed age at date of entry into a grouping variable with three levels, we first calculated their predictive strength, without the presence of native vs. non-native signing status. We then added native vs. non-native as a predictor in the model (see statistical summary in **Table 3**).

Age at date of entry was a significant predictor for both syntactic judgment [χ 2 (7) = 81.08, p < 0.001] and analogical reasoning [χ 2 (7) = 117.59, p < 0.001]. The age-at-date-of-entry groups were significant for syntactic judgment [χ 2 (7) = 23.14, p < 0.001] and analogical reasoning [χ 2 (7) = 23.52, p < 0.001]. Native vs. non-native status was also significant for syntactic judgment abilities [χ 2 (7) = 45.18, p < 0.001] and analogical reasoning skills [χ 2 (7) = 56.45, p < 0.001]. Again, students themselves accounted for 2% of the variance.

The decrease in test scores for those entering the school for the Deaf after age 12 was especially drastic, as can be seen by the relatively large beta weights of −0.13. Non-native signers on average scored 12 and 13 points lower on the syntactic judgment task and analogical reasoning tasks compared to native signers. The results highlight the importance of early language experience on syntactic sensitivity and language based analogical reasoning skills.

### Visualizing the Influence of Signing Status on ASL Outcomes

To graphically depict the influence of early age of exposure on subsequent ASL syntactic and vocabulary ability, we used density plots to describe the distributions of scores, with native and non-native signers plotted in side-by-side panels (as in **Figures 2** and **3**).

The reason for using density plots is that the diversity of abilities of students in our sample meant that dividing our students into consecutive age groups results in non-normal distributions. For example, when students have only a few years in the current school of assessment (and thus lower likelihood of intensive exposure to ASL), many students cluster in the bottom of the distribution of syntactic total scores. With older ages, the distribution of ASL ability scores shifts so that an increasing proportion of students are in the middle or higher of the distribution. Density plots are a powerful method of visualizing both central tendencies and overall distribution shape. The bar in the middle of each plot is the median of the sample. The long vertical box encompassing it shows the interquartile range, similar to conventional boxplots. This box indicates the "middle fifty" in a data set. That is, when scores of a distribution are divided into fourths, the two quartiles in the middle show the middle half of the distribution. The outline of the distribution encompasses all points.

We connected each median across the three age grouping with a line, as is conventional in line graphs, to aid visualizing trends across age groupings. Medians were employed in statistical analyses because mean scores were compromised by the variability in the samples.

#### Density Plots of Syntactic Judgment Scores

As shown in **Figure 2**, for native signers, the median score on the syntactic judgment task was lowest for students with the oldest age of school entry. The distribution also changed across the three age groupings. For students who entered school between the years of 0–6, the largest bulge in the plot is around a score of 70% correct. In contrast, for students who entered school at ages 7–12, most frequent scores lie at the bottom of the distribution.

For non-native signers, median scores are almost unchanging across the age groupings. However, like the distribution for native signers, the distribution grows increasingly bottom-heavy, with half of students who entered the current school after age 12 having scores below 50% correct, indicating poor ability to select the syntactically correct sentence from the set of four options.

One question is why few students had tests scores above 80% correct (see **Figure 2**). Syntactic judgment tasks are known to be difficult. In Boudreault and Mayberry (2006), even native signers only had a mean score of 85% – and they were adults. We therefore considered 85% to the ceiling, signifying adult competence. Based on this, our best performing group, the native signers, are on an expected developmental trajectory.

#### Density Plots of Analogy Task Scores

For native signers, who entered their current school for the Deaf by age 6, analogical reasoning ability was relatively high (75% correct). Indeed, the density plots for these native signers, shown in **Figure 3**, have a bulge at 80% correct, indicating good understanding of analogies. However, scores show a different distribution for students who entered their current school after age 12. Here, 50% of the scores were lower than 60% correct.

A bleaker picture is portrayed by the distributions for nonnative signers. Age of entry to the current school did not influence median scores on the analogies task. Interestingly, the age-ofschool-entry group with the largest bulk of low scores was

#### TABLE 2 | Mixed effects multiple regression with age-of-entry as a predictor.


TABLE 3 | Mixed effects multiple regression with predictors age-of-entry and signing status.


<sup>1</sup>By signed language, we mean a fully operationalized language, such as ASL rather than a sign system invented to model a spoken language through signs, such as Signed Exact English (SEE).

the group with the earliest age of entry. This is presumably because non-native signers with age of entry 0–6 includes young children, many of whom are just beginning to learn ASL. There is surprisingly little change from the 7–12 group to the 13–18 group. Like native signers, the non-native signers in the 13–18 group show uniform density spanning the range from 40 to 80% correct. This indicates that students with late age of school entry are a diverse group, spanning the spectrum from having poor to good analogical reasoning abilities.

### DISCUSSION

### Overview of Findings

We investigated two variables related to age of exposure of sign language. The first variable was whether or not parents fluently signed to their children starting at birth, measured using parental hearing status (native ns. non-native). The second variable was children's age at date of entry into an academic signing environment, which was the school for the Deaf where students were assessed. Both of these types of age-dependent language experiences influenced subsequent signing ability:

• Age-of-entry into an academic signing environment after 6 years of age was associated with poorer performance on both the syntactic judgment task and the language-based analogical reasoning task. The influence of age-of-entry was not linear. Test scores were markedly lower for Deaf children who entered the school of assessment after the age of 12.

• The positive influence of signing from birth-on was found for learners of all ages and for all age-of-entry groupings. Being a non-native signer had negative effects on later ASL skills that were similar in magnitude to entering the school for the Deaf after age 12 (i.e., see beta weights in **Table 3**).

We comment on these after describing age effects in our dataset.

### Age Effects in Syntactic Judgment and Analogical Reasoning Abilities

Age effects for the acquisition of ASL syntactic judgment abilities, documented in Novogrodsky et al. (under review), were surprisingly modest in magnitude. Effect sizes for the contribution of age to syntactic judgment scores were weaker than the effect of having Deaf parents (i.e., early exposure to language), and weaker than the cumulative number of years enrolled in a Deaf school. In addition, interpreting age effects is difficult in a naturalistic sample such as ours, because test scores can be worse for older children than for younger children. The reason is that test scores of older children include students who transferred to the current school for the Deaf after experiencing academic failure at a non-signing school. An example of this is that older children in our database included subsamples who

were 10–13 years old at their first systematic exposure to ASL. For students who have been in school for the Deaf continuously, test scores are linearly correlated with age. But averaging together students who entered early with those who entered late resulted in a flattened slope for test scores as a function of age during ages 10–14.

Age at test is important because the ability to judge grammaticality is a metalinguistic ability requiring cognitive maturity. At ages 7–8, when Deaf children are able to produce syntactically correct signs, their ability on the syntactic judgment task remained poor (Novogrodsky et al., under review). At this age, non-native scorers as a group were near chance (average percent correct 27%, range 20–49%). Native signers did better, but the judgment task was still challenging, with an average percent correct of 40% (range 20–62%).

Age effects are also evident in analogical reasoning tasks that use words rather than shapes and figures (Alexander et al., 1989; Goswami, 1991; Gentner et al., 2009). Rattermann and Gentner (1998) argue that relational reasoning abilities reflect the cumulative effect of knowledge. As children age, they acquire more vocabulary and concept knowledge which helps them better detect, analyze, and apply relationships between words or concepts. The results in **Tables 2** and **3** show clear improvement in performance on the Analogies and Syntax tasks based on age. However, while age effects were present in our study, they were more limited than the effects of early exposure to language, and the cumulative effects of academic sign language in schools.

### The Advantage of Signing from Birth: Extending Prior Research

The advantages of early exposure to sign language are well documented (Mayberry and Lock, 2003; Mayberry, 2010). Our data extend these results to a large naturalistic sample of schoolaged children between the ages of 7;5 to 18;6. Participants had better ASL syntax and analogical reasoning skills if they signed from birth. Native signers scored on average 13 points higher than non-native signers on an ASL syntactic judgment task, and 12 points higher on the ASL language-based analogies task. Our study is the first study to demonstrate the advantage of signing from birth on children's ability to solve language-based analogical reasoning problems in ASL. We found high analogy test scores for Deaf students who grew up as primarily visual learners, with ASL as the native language. This refutes Sharpe's (1985) claim that auditory stimulation is necessary to develop language-based analogical reasoning skills.

### Signing in School Facilitates ASL Syntactic Judgment and Language-Based Analogical Reasoning Abilities

Because only a small percentage of Deaf children have fluent signing experiences in the home, entry into a signing classroom represents, for most Deaf children, both the first systematic exposure to sign language and also the first exposure to academic sign language. We therefore looked at age at date of entry as another variable in the acquisition process for ASL based syntactic judgment skills and for ASL based analogical reasoning skills.

Non-native signers who entered the school of the Deaf before age 6 had better ASL abilities than did those who entered after age 6. Deaf children who entered between ages 6 and 12 had better ASL skills than those who entered after age 12 but never equaled or caught up to the native signers. These patterns held for both syntactic judgment and analogical reasoning abilities.

Native signers who entered school after the age of 12 also showed poorer performance on both syntactic judgment and analogical reasoning tasks compared to those who entered school before the age of 12. There are likely to be two reasons, which future work can investigate: as noted earlier, Deaf parents may have emphasized oral training or a different signing system at home, such as cued speech or even a different natural sign language. This will mean low levels of ASL. Another contributing factor is that sign language in the classroom is academic language. Enrollment at Deaf school may be important even for students who have ASL at home. Academic language prepares students for the metalinguistic abilities required by syntactic judgment and analogical reasoning tasks.

### Implications

In the introduction we noted two pieces of advice to hearing parents, based on quite different views of the influence of early language experiences on subsequent language development. One claim assumed that ASL acquisition is under such tight maturational constraints that hearing parents stand little chance of being able to learn it in time for children to benefit. Parents are similarly told that enrolling Deaf children in schools for the Deaf where ASL is used after early intervention periods would also be too late (Knoors and Marschark, 2012). In contrast, others have told parents to exclude sign from the home to give speech training the best chance of taking hold, with the idea that sign language could be learned later as a back-up language if oral methods failed (Mauldin, 2016). We wanted to see if the windows for learning ASL close in early childhood, or can ASL be learned well anytime in childhood provided the correct environment?

Our results demonstrate a continuum of outcomes that reflect experience with language as a continuous variable that is sensitive to maturational age. The best sign-language outcomes were for exposure from birth from parents, next best was ASL exposure before age 6, next was academic ASL exposure before 12, and the worst was academic ASL exposure after age 12. Advocates for a "windows closing" view could see their perspective supported by our findings that non-native signers as a group were delayed, at all ages tested, relative to native signers. Later school enrollment was also associated with poorer sign language than earlier entry to a signing school.

On the other hand, advocates for "sign language as a backup" could focus on the considerable overlap between native and nonnative signers. As we noted, half of Deaf children with hearing parents had ASL scores which were as good as the scores achieved by three-quarters of the Deaf children with Deaf parents. This can be seen by comparing medians and quartiles across the two

density plots in each of **Figures 2** and **3**. Comparing native and non-native signers showed the robust advantage of being a native signer, but the density charts also demonstrate that many Deaf children with hearing parents performed as well as did their native-signing peers. Finally, some non-native signers who enrolled after age 12 also were able to score on our tasks at similar levels to native signers. These children may have had some signing experience before transferring into the school of assessment. Ultimately, the results show that time spent in a good signing environment (e.g., ASL based programs for the Deaf) leads to across the board improvements in language and language related abilities in Deaf children.

We suggest that the best advice for parents is to avoid allor-none thinking, and recognize the impact of the continuum of age-related declines revealed by our data. The best signing outcome is when signing occurs from birth, but considerable plasticity exists and most students may be able to take advantage of the fully accessible exposure to ASL when presented consistently by peers and adults. Our advice is that parents should, when possible, choose ASL as either a primary or a supplementary means of communicating with their Deaf children.

### CONCLUSION

Our findings confirm and extend to a large naturalistic sample the well-known advantage of early, systematic exposure to ASL. Native signers had an advantage in ASL syntactic skills and vocabulary-based analogical reasoning that held irrespective of age-of-entry to an academic signing environment. While native signers will remain a small percent of the Deaf population, this shows that hearing parents who learn to sign can and do influence their children's language skills and the development of their higher level language abilities. We additionally documented that non-native signers have the best chance of developing their ASL abilities when they are exposed to an academic signing environment before the age of 12. Parents who place their children in good signing programs for the Deaf by age 6 (for the

### REFERENCES


best result) can expect that their children will approximate the language skills of native-signing, Deaf children.

### NOTES

The data used in this analysis has also been used in other analyses by the same team. However, we affirm that the analysis and discussion here is novel.

### ETHICS STATEMENT

IRB approval was provided by the Boston University Charles River Campus Institutional Review Board. Consent for data collection was provided via Blanket Consent procedures. Parents were required to opt their children out of assessment. Information about the assessment was provided in both print and via ASL videos. Adults over the age of 18 who were included in assessment were also provided text or video consent documents.

### AUTHOR CONTRIBUTIONS

JH was the lead writer for this article. He did the analyses and guided the team during the writing process. CC-H was vital during the editing process. She helped make the article appropriate for publication in Frontiers. RN helped JH build the initial few drafts and developed the theoretical background. RH developed the application that collected the data, built the research team, and helped edit the draft.

### FUNDING

Funding for this research was provided by IES: National Center for Special Education Research, Grant Award Number R234A100176.




**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Henner, Caldwell-Harris, Novogrodsky and Hoffmeister. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Reduced Working Memory Mediates the Link between Early Institutional Rearing and Symptoms of ADHD at 12 Years

Florin Tibu<sup>1</sup> \*, Margaret A. Sheridan<sup>2</sup> , Katie A. McLaughlin<sup>3</sup> , Charles A. Nelson<sup>4</sup> , Nathan A. Fox<sup>5</sup> and Charles H. Zeanah<sup>6</sup>

1 Institute of Child Development, Bucharest, Romania, <sup>2</sup> Department of Psychology and Neuroscience, University of North Carolina, Chapel Hill, NC, USA, <sup>3</sup> Department of Psychology, University of Washington, Seattle, WA, USA, <sup>4</sup> Harvard Medical School – Boston Children's Hospital – Harvard Center on the Developing Child – Harvard Graduate School of Education, Boston, MA, USA, <sup>5</sup> Department of Human Development and Quantitative Methodology, University of Maryland, College Park, MD, USA, <sup>6</sup> Department of Psychiatry and Behavioral Sciences, Tulane University School of Medicine, New Orleans, LA, USA

Children who are raised in institutions show severe delays across multiple domains of development and high levels of psychopathology, including attention deficit/hyperactivity disorder (ADHD). Low performance in executive functions (EFs) are also common in institutionally reared children and often do not remediate following improvements in the caregiving environment. ADHD symptomatology also remains elevated even after children are removed from institutional care and placed in families. We investigate whether poor EF is a mechanism explaining elevated rates of ADHD in children reared in institutional settings in the Bucharest Early Intervention Project (BEIP). In the current study, we examine the potentially mediating role of poor EF in the association between institutionalization and symptoms of ADHD at age 12 years. A total of 107 children were assessed with the Cambridge Neuropsychological Test Automated Battery (CANTAB) on working memory, set-shifting and planning. We also obtained concurrent teacher reports on their levels of ADHD symptoms (inattention and impulsivity separately). Institutionalization strongly predicted elevations in symptoms of inattention and impulsivity at age 12 years (ps < 0.01). Indices of working memory and planning were also associated with ADHD after controlling for potential confounders (ps < 0.03). Mediation analyses revealed that poor working memory performance mediated the link between exposure to early institutionalization and higher scores of both inattention and impulsivity. These results replicate and extend the findings that we reported in the BEIP sample at age 8 years. Together, they suggest that compromised working memory is a key mechanism that continues to explain the strikingly high levels of ADHD in late childhood among children institutionalized in early life. Interventions targeting working memory may help to prevent ADHD among children exposed to institutional care.

Keywords: children, institutionalization, executive functioning, working memory, ADHD

## INTRODUCTION

Considerable evidence indicates that early adverse environments can render children vulnerable to various psychiatric disorders that develop later in life (Green et al., 2010; McLaughlin et al., 2012). The link between adverse environments and the onset of psychopathology and other developmental problems is mediated, in part, by disruptions in brain structure and function

#### Edited by:

Matthew William Geoffrey Dye, Rochester Institute of Technology, USA

#### Reviewed by:

Iris Wiegand, Max Planck Institute for Human Development, Germany Vincent Joseph Samar, Rochester Institute of Technology, USA

> \*Correspondence: Florin Tibu florin.tibu@idc.ro

#### Specialty section:

This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology

Received: 12 August 2016 Accepted: 08 November 2016 Published: 24 November 2016

#### Citation:

Tibu F, Sheridan MA, McLaughlin KA, Nelson CA, Fox NA and Zeanah CH (2016) Reduced Working Memory Mediates the Link between Early Institutional Rearing and Symptoms of ADHD at 12 Years. Front. Psychol. 7:1850. doi: 10.3389/fpsyg.2016.01850

(McLaughlin et al., 2014a; Sheridan and McLaughlin, 2014; Teicher and Samson, 2016). A typical neural development is particularly likely when exposure to adversity occurs during infancy and early childhood, a period of heightened neural sensitivity to environmental inputs of numerous kinds when the trajectory of brain development is tuned based on the environment the child experiences (Nelson et al., 2009). Early adaptations to an adverse environment can produce lasting changes in cognition and enduring deficits.

A particularly egregious form of early adversity is institutional rearing, which currently impacts approximately eight million children around the world (Save the Children UK, 2009). Institutionalization represents an extreme form of psychosocial and sensory deprivation with a profound impact on multiple aspects of development, including IQ (Nelson et al., 2007), attention (Pollak et al., 2010; Loman et al., 2013), and executive functions (EFs). With regard to EF, there is accumulating evidence to suggest that working memory (Bauer et al., 2009; Bos et al., 2009; Pollak et al., 2010; Hanson et al., 2013; Loman et al., 2013), inhibitory control (Colvert et al., 2008; Bruce et al., 2009; McDermott et al., 2013), error monitoring (Troller-Renfree et al., 2016), and set-shifting (Hanson et al., 2013) are all negatively impacted by early institutionalization. For other EFs, however, such as planning, the findings have been somewhat mixed, with most prior studies reporting negative findings (e.g., Bos et al., 2009; Pollak et al., 2010; Bick et al., in press).

These disruptions in EF might represent a developmental pathway linking early institutionalization to the onset of some forms of psychopathology. The prevalence of mental health problems is unusually high in children reared in institutions, especially attention deficit/hyperactivity disorder (ADHD; Stevens et al., 2008; Zeanah et al., 2009; Humphreys et al., 2015). Current evidence suggests that for young children reared in institutions, adoption or foster care placement does not lead to attenuation in signs of ADHD (Zeanah et al., 2009; Rutter et al., 2010; Humphreys et al., 2015) or EF performance (Colvert et al., 2008; Bos et al., 2009; McDermott et al., 2013; Bick et al., in press), unless the children were removed before 6 months of age (Rutter et al., 2010). Moreover, recent evidence emerging from the English Romanian Adoptees (ERA) study with children adopted in the UK from Romanian orphanages shows persistent ADHD symptomatology through early adulthood (Kennedy et al., 2016), with new cases being diagnosed with ADHD beyond age 20 years. These findings suggest a persistent course of ADHD among children raised in deprived early environments, even after intensive psychosocial intervention, and highlight the need for identifying mechanisms that lead to the very high prevalence of this phenotypic variant of ADHD. Here, we examine the role of disruptions in EF as a mechanism explaining persistent ADHD symptoms among children reared in deprived institutions in early life.

Mechanisms linking institutionalization to ADHD have included multiple manifestations of atypical brain structure and function (McLaughlin et al., 2010, 2014b). However, it is likely that these disruptions in neural development correspond to patterns of atypical cognitive development that might also contribute to the onset of ADHD among children reared in deprived environments. Neuropsychological links between institutional rearing and ADHD have also been demonstrated, including poor inhibitory control (Colvert et al., 2008) and, most recently, we found that working memory and response inhibition mediated the link between institutionalization and two dimensions of ADHD, inattention and impulsivity, among children aged 8 years (Tibu et al., 2016). To the best of our knowledge, no other studies to date have tested the role of EF as a mechanism linking institutional rearing to ADHD symptomatology.

In the current study, we aimed to replicate and extend our previous findings at age 8 by exploring the potentially mediating role of EFs in the association between institutionalization and symptoms of ADHD at age 12. Given ongoing development of EFs during early adolescence (Best and Miller, 2010), and given that ADHD symptoms often decrease in severity during the transition to adolescence (Willoughby, 2003), we were interested in whether EFs would continue to explain the link between early institutionalization and ADHD in this developmental period. Moreover, a recent review on EF findings from studies conducted with previously institutionalized children indicates some contradictory results during late childhood and the need to gather more evidence as children enter adolescence (Merz et al., 2016). Based on BEIP findings at earlier assessments, we expected that children with histories of institutionalization would have increased ADHD symptoms and decreased performance on the EF tasks (Bos et al., 2009; Humphreys et al., 2015; Bick et al., in press). We also expected to find links between executive EF abilities and ADHD symptoms. Furthermore, we investigated whether elevations in ADHD symptoms in the institutionalized children were mediated by differences in EF abilities. To test our hypotheses we included behavioral measurements of multiple EFs (i.e., working memory, setshifting, and planning) and used teacher reports of ADHD symptomatology.

## MATERIALS AND METHODS

### Participants

The study participants were children from our longitudinal investigation (Bucharest Early Intervention Project, BEIP) who were recruited in infancy from institutions in Bucharest, Romania, and randomized to either a care-as-usual-group (CAUG) who continued to live in the institutions, or a foster care group (FCG) whom we placed in foster families that we recruited and offered support until children were 42 months old. A third community control group of never institutionalized (NIG) children was recruited at the same time with the institutionalized children through General Practitioner offices. At 12 years, 107 participants in the BEIP were assessed with the Cambridge Neuropsychological Test Automated Battery (CANTAB) on working memory, set-shifting, and planning and we also obtained teacher reports on their levels of ADHD symptoms (with separate measurements for inattention and impulsivity).

Demographic characteristics for the participants in the two groups (i.e., with and without exposure to institutionalization) are presented in **Table 1** and show significant differences in birth weight. Mean birth weight in the institutionalized children was significantly lower than in the community controls (mean difference = −494 grams; t(95) = −6.58; p < 0.001). Therefore we included birth weight as a covariate in all subsequent analyses.

### Measures ADHD

We obtained teacher reports on children's symptoms of inattention and hyperactivity using the Health and Behavior Questionnaire (HBQ; Boyce et al., 2002; Essex et al., 2002), a questionnaire with high reliability and validity that has been used extensively with school-aged children (Essex et al., 2002; Lemery-Chalfant et al., 2007). The teacher version of the HBQ was used in the BEIP previously at age 8 years (Tibu et al., 2016), as well as in other studies with formerly institutionalized children (Wiik et al., 2011; Pitula et al., 2014). Symptoms are rated on a 3-point Likert scale: 0 ("never or not true"), 1 ("sometimes true") or 2 ("often or very true"), with higher scores indicating more severe symptomatology. There are 15 items in the HBQ for assessing ADHD (six items for inattention and nine items for impulsivity).

### Executive Functioning

The CANTAB was used to measure working memory, planning, and set-shifting. The CANTAB contains behavioral subtests that have been widely used with typically developing children, atrisk children, children with ADHD, and adults (Nigg, 2001; Fried et al., 2015). The CANTAB has been validated extensively on samples of school-age children and has been found to discriminate well between clinical and non-clinical populations (Luciana and Nelson, 2002). Five CANTAB subtests were administered on a desktop computer with a touch-sensitive computer screen to assess working memory, planning, and setshifting skills.

Delayed Matching to Sample (DMS) is a short term memory task in which the child is presented with a stimulus pattern and then needs to select a matching pattern from a series of four patterns shown below the stimulus. Trials are either simultaneous (both the stimulus and the four choices are shown on the screen at the same time), with a zero-second delay (the stimulus disappears just before the choices are presented), or with a delays of 4000 or 12000 ms. The main outcome variables are the percentage of correct trials and latency of response for each type of trial.

Paired Associates Learning (PAL) subtest assesses spatial working memory and new learning. Six to eight boxes are presented sequentially on the screen, with some or all containing a different pattern. The patterns are then shown again in the middle of the screen, one at a time and in random order, and the child has to touch the box that contained the pattern. The difficulty increases with the number of patterns contained in the boxes. Outcome variables include stages completed at first trial, total stages completed, and memory score (i.e., number of patterns correctly located after the first trial summed across the stages completed).

Spatial Working Memory (SWM) tests the ability to retain spatial information across a delay and to manipulate remembered items in working memory. A number of 3–8 colored boxes are shown on the screen and the subject is invited, by process of elimination, to find a blue token in each of these boxes and use it to fill up an empty column on the right of the screen. Variables of interest are total errors (i.e., total number of errors made by the subject during all trials), between errors (i.e., number of times across trials in which the subject revisits a box in which a token has previously been found), within errors (i.e., number of times within a search in which the subject revisits an empty box), and strategy (i.e., presence/absence of organized patterns of search).

Stockings of Cambridge (SOC) is a planning task derived from the Tower of London test (Shallice, 1982). The child views a set of three hanging stockings on the top of the screen that contain colored circles in a given order and another set of stockings on the lower half of the screen containing the same circles but positioned differently across the stockings. The child is instructed to move the circles in the lower display to copy the upper model using as few moves as possible. The difficulty of the trials increases gradually from 2-move problems to more complex models that require five moves to solve. Key outcomes are number of trials solved in minimum moves, mean number of moves for each level of difficulty, and initial and subsequent thinking times.

Intra-Extra Dimensional Set Shift (IED) assesses rule acquisition and reversal and attention flexibility. Two stimuli (one correct and one incorrect) are displayed on the computer


Abbreviations: EIG, ever institutionalized group; NIG, never institutionalized group.

screen. Stimuli initially represent only one category (i.e., shape), then two categories (i.e., line and shape). Participants learn to select the correct categories based on instantaneous feedback from the computer, but the stimuli and/or rules are changed after a specific number of correct responses. The initial shifts in correct stimuli are intra-dimensional (e.g., within the shape dimension) and later become extra-dimensional, requiring a category shift (e.g., from the shape dimension to the line dimension). Performance is assessed based on, among other indicators, number of stages completed and number of errors made.

### Procedure

Ethics approval was obtained from the University of Bucharest's Ethics Committee, as well as the Institutional Review Boards (IRBs) of the US universities to which the project's Principal Investigators were affiliated. As the children approached 12 years of age, the families were invited to come for assessments to our laboratory in Bucharest. Consent forms were signed by the legal guardians, and verbal and written assent was also obtained from the children. The CANTAB was administered during one of the lab sessions. The HBQs were distributed to the teachers. Both the families and the teachers were compensated commensurate to the amount of time involved and deemed appropriate in consultation with local staff.

### Data Analysis Plan

To test our hypothesis that EF mediated the link between exposure to institutionalization and ADHD, we first sought to establish links between the predictor and each of the two dependent variables (i.e., ADHD inattention and impulsivity scale scores), as well as associations between our proposed mediators (i.e., indices of EF) and both the predictor and the dependent variables. As a final step, we conducted two sets of mediation analyses (one for inattention and one for impulsivity) and calculated mediation effects (i.e., indirect effects) only for the EF variables that were significantly linked to the predictor and the dependent variables following recommendations from Preacher and Hayes (2008). A bootstrapping procedure was used to test the significance of the indirect effects with 5,000 bootstrap samples and 95% confidence intervals to yield more valid estimates of the indirect effects (Preacher and Hayes, 2008). To support mediation, bootstrapped confidence intervals for the indirect effects cannot contain the value of zero. We included the participants' birth weight and gender as covariates in all these analyses, and statistical significance was evaluated at the 0.05-level, using two-sided tests. Data were analyzed using the application IBM SPSS Statistics version 23.

### RESULTS

### Foster Care Intervention Effects on ADHD and EF

In line with earlier findings in our BEIP sample at age 8 years, we did not see an intervention effect on children's ADHD scores or their performance on any of the CANTAB tasks at the age of 12 years. Specifically, children assigned to the FCG did not differ from those in the CAUG on their levels of ADHD symptomatology [t(79) = 0.54, p = 0.59 for inattention; t(79) = 0.50, p = 0.62 for impulsivity, which is congruent with parent reports of children's symptomatology in this sample (Humphreys et al., 2015). Similarly, the two groups did not differ on measurements of working memory, planning, and set-shifting (detailed results not reported here; see Bick et al., in press). Therefore, for the purposes of this paper we combine the FCG and CAUG and examine only two groups, the ever institutionalized group (EIG, N = 73) and the never institutionalized group (NIG, N = 34). The same approach has been employed in several other BEIP studies that have examined mechanisms linking institutionalization to elevations in ADHD (McLaughlin et al., 2010, 2014b; Slopen et al., 2012; Tibu et al., 2016).

### Exposure to Institutionalization and Child Outcomes

History of institutional rearing predicted scores on both ADHD scales. As shown in **Table 2**, children with history of institutionalization had markedly higher levels of inattention, F(1,105) = 19.93, p < 0.001, and impulsivity, F(1,104) = 8.80, p = 0.004, compared to children in the community group.

Children exposed to institutional rearing had worse performance than those who had never been institutionalized on multiple EF indices in each of the five CANTAB tasks, including all three tests of working memory, planning, and set-shifting (detailed results not reported here; see Bick et al., in press).

Given the disproportionately high number of Rroma children among the EIG compared to the NIG, we tested the potentially confounding role of ethnicity in the link between exposure to institutionalization and ADHD and EF scores. All these associations were unchanged when the Rroma children were removed from the sample.

### Links between ADHD Symptoms and EF Performance

**Table 3** shows the associations of inattention and impulsivity with each of the EF variables that were associated with institutionalization. Performance on the EF tasks was consistently associated with inattention and impulsivity, such that worse performance was associated with higher levels of symptoms. The

#### TABLE 2 | Institutionalization status and attention deficit/hyperactivity disorder (ADHD) scores.


<sup>∗</sup>Analyses control for birth weight and gender.

Abbreviations: EIG, ever institutionalized group; NIG, never institutionalized group.


one exception was the set-shifting task, in which performance was not associated with inattention and impulsivity.

After controlling for birth weight and gender, several working memory and planning indices continued to predict ADHD levels. Specifically, associations remained significant between inattention and DMS percent accuracy 12000 ms (β = −0.22; p = 0.024), PAL first trial memory score (β = −0.23; p = 0.016), SWM strategy score (β = 0.30; p = 0.002), SWM total errors (β = 0.42; p < 0.001), and SOC problems solved in minimum moves (β = −0.29; p = 0.003), and also between impulsivity and PAL first trial memory score (β = −0.22; p = 0.023), SWM strategy score (β = 0.32; p = 0.001), SWM total errors (β = 0.38; p < 0.001), and SOC problems solved in minimum moves (β = −0.26; p = 0.007).

### Mediation Analyses

In our final sets of analyses we tested our main hypotheses; namely that poor EF performance explained the link between exposure to institutionalization and elevated ADHD symptoms. To do that, we tested two mediation models, one for each of the two dependent variables, in which we included the institutionalization status as the predictor, EF indices as mediators and birth weight and gender as covariates. For each model, we tested separately whether the indirect effect of institutionalization on ADHD symptoms through working memory and planning was statistically significant. These findings are presented jointly in **Figure 1**.

In predicting inattention, we first included as mediators the four working memory variables jointly (i.e., DMS percent correct at the 12,000 ms trials, PAL first trial memory score, SWM strategy score, and SWM total errors), and then the planning index (i.e., SOC problems solved in minimum moves) as sole mediator. The indirect effect of exposure to institutionalization on inattention through the working memory indices was significant (95% CI: −0.24, −0.04). The total effect of institutionalization in predicting inattention was attenuated by 48.6% to non-significance level (B = −0.13; p = 0.067) when these working memory variables were included as mediators. By contrast, when the planning index was included as a mediator, the indirect effect of institutionalization on inattention was not statistically significant (95% CI: −0.07, 0.00). These findings suggest that the link between exposure to institutionalization and inattention is mediated by working memory, and not explained by planning.

Likewise, in predicting impulsivity, we first included as mediators three working memory variables jointly (i.e., PAL first trial memory score, SWM strategy score, and SWM total errors), then the planning index (i.e., SOC problems solved in minimum moves) as sole mediator. The indirect effect of exposure to institutionalization on impulsivity through the working memory indices was significant (95% CI: −0.23, −0.04). The total effect of exposure to institutionalization in predicting impulsivity was attenuated by 66.5% to non-significance level (B = −0.05; p = 0.43) with the inclusion of the working memory indices as joint mediators. In contrast, the indirect effect of institutionalization

fpsyg-07-01850 November 22, 2016 Time: 13:44 # 5

TABLE 3 | Correlations

 between ADHD and

EF variables.

in the prediction of impulsivity when the SOC variable was entered as mediator was statistically non-significant (95% CI: −0.07, 0.00). These results suggest that the link between exposure to institutionalization and impulsivity is, as with inattention, explained by performance in working memory and cannot be attributed to the children's planning ability.

### DISCUSSION

Children with histories of institutional rearing continue to display elevated symptoms of ADHD and poor EF performance in the pre-pubertal period. In the current study, we show that EF mediates the link between institutionalization and symptoms of both inattention and impulsivity, which provides support for the hypothesis that low cognitive abilities represent a distinct pathway through which early institutional care exerts a persistent impact on mental health problems. As we expected, children reared in institutions had higher ADHD symptomatology and worse performance across EF tasks compared to a control group (for detailed results on the EF, see Bick et al., in press). We found that poor working memory specifically acted as a mediator in the associations of institutional rearing with both inattention and impulsivity, highlighting the key role played by this higher-order cognitive function in explaining how institutionalization can contribute to persistent ADHD. Working memory appears to be one factor explaining the link between institutional rearing and ADHD, although clearly other mechanisms are likely to be involved.

The results in the current study replicate and extend findings from other studies that have tested samples of previously institutionalized and adopted children of comparable ages to the children in our study. Specifically, our finding that previously institutionalized children exhibit markedly elevated rates of inattention and impulsivity as reported by their teachers is consistent with caregiver reports obtained through a structured interview in the BEIP (Humphreys et al., 2015) and with other studies showing high ADHD symptoms among children who were institutionalized in early life and later adopted into families in Western Europe (Stevens et al., 2008; Vorria et al., 2014; Kennedy et al., 2016) and North America (Gunnar et al., 2007; Wiik et al., 2011). Our current results extend findings from this sample at earlier ages, which also observed high levels of ADHD symptoms (Zeanah et al., 2009; McLaughlin et al., 2014b; Tibu et al., 2016), by documenting persistently elevated levels of ADHD throughout childhood and into early adolescence following early institutionalization. It is possible that the trajectories of ADHD symptoms in children who grew up in institutions follow a different trajectory compared to ADHD seen in family-reared populations in that the levels of symptoms remain high, sometimes even becoming higher as children enter young adulthood (Kennedy et al., 2016).

Additionally, we replicate and extend our previous findings on the mediating role of EF in the link between early institutional care and ADHD symptomatology in middle childhood (Tibu et al., 2016). Here, we found again that compromised working memory ability is a key mechanism that explains elevated ADHD symptoms in children who had been exposed to institutionalization. These findings reveal a cognitive mechanism that explains the link between institutionalization and ADHD, emphasizing the pervasive influence of early institutionalization on cognitive development. Interestingly, experiences of institutionalization early in life appear to impact cognitive function even in domains associated with areas of the brain known to exhibit a protracted developmental trajectory into adolescence. Working memory is supported by prefrontal and superior parietal cortex function and shows a protracted developmental trajectory into late adolescence (Thomason et al., 2009; Finn et al., 2010; Peverill et al., 2016). Disruptions in spatial

working memory have previously been observed among children with ADHD (Westerberg et al., 2004), as well as atypical neural structure and function in fronto-parietal networks (Giedd et al., 2001; Castellanos et al., 2002).

How exactly reduced working memory exerts influence on the development of ADHD is difficult to answer given that multiple pathways resulting from interplays between individual predispositions and environmental adversities are likely to be involved (Sonuga-Barke and Halperin, 2010) and institution-related ADHD appears to differ substantially from ADHD in typical populations (Kennedy et al., 2016). It is possible that institutional care in the first years of life disrupts profoundly neural structures (e.g., the prefrontal cortex; PFC) that are responsible for the on-going development of memory and attention (Nelson et al., 2011), which in turn, can result in abnormally high levels of inattention and impulsivity. The PFC is an area of the brain that has been shown in numerous studies to be influenced by deprived early environments (McLaughlin et al., 2014a,b; Sheridan and McLaughlin, 2014). Specifically, children raised in deprived environments exhibit reductions in cortical thickness in the PFC as well as the parietal cortex (McLaughlin et al., 2014b; Mackey et al., 2015). This pattern is consistent with conceptual models of deprivation, which argue that accelerated synaptic pruning occurs in the PFC and other areas of association cortex when the environment is lacking in cognitive and social stimulation (McLaughlin et al., 2014a; Sheridan and McLaughlin, 2014). This accelerated cortical thinning may contribute to the development of poor working memory and, ultimately, ADHD. Indeed, the association between institutional rearing and ADHD has been shown to be explained, in part, by exaggerated cortical thinning and a pattern of blunted cortical activity, including in the PFC and parietal cortex (McLaughlin et al., 2010, 2014b). It is likely that early deprivation associated with institutionalization alters the development of fronto-parietal networks that underlie working memory, and that this altered neural circuitry ultimately contributes to behavioral problems in the form of elevated ADHD symptoms. Clearly, more research is needed into identifying specific cortical and subcortical abnormalities impacting EF and ADHD in the institutionalized children.

Planning ability, although impacted by institutionalization and linked to ADHD, was not a construct that explained ADHD symptoms in our current study, which is consistent with our BEIP findings at age 8 years. It is possible that planning, which is thought to be a more sophisticated problem-solving cognitive skill, is not affected by institutionalization in the same manner as other more fundamental EFs like working memory. There is evidence suggesting that working memory and inhibition are more central processes that help toward the development of most of the other EFs (Senn et al., 2004), and it may be that disruptions that occur during early childhood impact on the more basic functions, which may in turn lead to signs of ADHD in the first years of life.

There are several noteworthy clinical and theoretical implications regarding the findings in the current study. First, caregivers and specialists should be informed that many of the children who had been reared in institutions will exhibit high levels of ADHD symptoms that are likely to persist throughout childhood and beyond. Foster and adoptive parents need to become aware of the enduring neurocognitive deficits and impulsive/inattentive manifestations that many of these children will show across settings. Second, there is potential for interventions to be designed that may decrease these symptoms by improving children's cognitive abilities, at least their working memory. Indeed, there are programs that have proved their efficacy in addressing ADHD problems through working memory training (Klingberg et al., 2005; Gray et al., 2012; van der Donk et al., 2015), although there is no evidence yet on how effective such interventions might be with institutionalized children. Lastly, developmental models of ADHD linked to institutionalization should not omit testing working memory and possibly other EFs too, among its contributing factors.

The strengths of the current study include a longitudinal sample from an RCT with a rarely studied population that had been exposed to institutional rearing, some from as early as 6 months of age, a robust mediation model that allows testing a causal understanding on ADHD while controlling for confounders, and good retention rates in a sample of highly vulnerable adolescents recruited more than a decade ago.

Our study should also be discussed in terms of its limitations. First, we included concurrent measurements of EF and ADHD, which may pose some limitations in inferring causality. Second, we made use of questionnaires as opposed to interviews for measuring ADHD symptoms. Nevertheless, teacher reports are reliable instruments in measuring levels of symptomatology, particularly of externalizing nature (Cai and Kaiser, 2004) and have been used before in studies with institutionalized children and adolescents. Finally, we were not able to account for the possible influence of other factors (e.g., prenatal, genetic risks, or medical illnesses during the gestational or early infancy periods, etc.) that might have played a direct or interactive role in the link between exposure to institutionalization and ADHD and which should be examined in future studies. Interestingly though, evidence from the ERA study indicates that children removed from institutions before age 6 months had comparable ADHD scores with control participants (Rutter et al., 2010), suggesting that the high rates of ADHD in postinstitutionalized children do not simply reflect prenatal and genetic factors but also the impact of being raised in a deprived environment. Certainly, prenatal and genetic factors play an important role, and we cannot adjust for their influence in the current study.

This study identified poor working memory as a mediator of the association between institutional rearing and ADHD symptoms of inattention and impulsivity at 12 years of age in children who, during their early childhood, grew up in institutions. Mental health problems, ADHD in particular, are a public health concern at global level and our current findings have the potential to inform specialists on some of the neuropsychological mechanisms to psychopathology during early adolescence so that interventions can be effectively implemented to reduce maladaptation and increase chances of academic success and optimal social and occupational functioning.

### AUTHOR CONTRIBUTIONS

fpsyg-07-01850 November 22, 2016 Time: 13:44 # 8

CN, NF, and CZ designed research; FT performed research; FT, MS, and KM analyzed data; FT, MS, KM, CN, NF, and CZ wrote the paper.

### REFERENCES


### FUNDING

This work was supported by the National Institute of Mental Health Grant MH091363 (to CN).

### ACKNOWLEDGMENT

We are grateful to all the children and caregivers who participated in the Bucharest Early Intervention Project and our dedicated lab staff in Romania.

comorbid ADHD: a randomized controlled trial. J. Child Psychol. Psychiatry 53, 1277–1284. doi: 10.1111/j.1469-7610.2012.02592.x


and symptoms of attention-deficit/hyperactivity disorder. Biol. Psychiatry 68, 329–336. doi: 10.1016/j.biopsych.2010.04.005


early intervention? J. Child Psychol. Psychiatry 51, 368–389. doi: 10.1111/j.1469- 7610.2009.02195.x


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The reviewer VS and the handling Editor declared their shared affiliation, and the handling Editor states that the process nevertheless met the standards of a fair and objective review.

Copyright © 2016 Tibu, Sheridan, McLaughlin, Nelson, Fox and Zeanah. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

fpsyg-07-01850 November 22, 2016 Time: 13:44 # 9