Abstract
The use of digitally modified stimuli with enhanced diagnostic information to improve verbal communication in children with sensory or central handicaps was pioneered by Tallal and colleagues in 1996, who targeted speech comprehension in language-learning impaired children. Today, researchers are aware that successful communication cannot be reduced to linguistic information—it depends strongly on the quality of communication, including non-verbal socio-emotional communication. In children with cochlear implants (CIs), quality of life (QoL) is affected, but this can be related to the ability to recognize emotions in a voice rather than speech comprehension alone. In this manuscript, we describe a family of new methods, termed parameter-specific facial and vocal morphing. We propose that these provide novel perspectives for assessing sensory determinants of human communication, but also for enhancing socio-emotional communication and QoL in the context of sensory handicaps, via training with digitally enhanced, caricatured stimuli. Based on promising initial results with various target groups including people with age-related macular degeneration, people with low abilities to recognize faces, older people, and adult CI users, we discuss chances and challenges for perceptual training interventions for young CI users based on enhanced auditory stimuli, as well as perspectives for CI sound processing technology.
Introduction
In 1996, Paula Tallal, Michael Merzenich and their colleagues published two seminal companion papers in Science which focused on how processing deficits in language-learning impaired (LLI) children could be ameliorated by training with acoustically modified, exaggerated speech stimuli. One paper () demonstrated that, with only 8–16 h of child-appropriate adaptive training over 20 days, LLI children improved substantially their temporal processing abilities to recognize brief sequences of both non-speech and speech stimuli. The other manuscript () used speech modifications to create salient, exaggerated speech stimuli to train speech comprehension. They assessed effects of daily training over 4 weeks with temporally modified speech in which the fast (mostly consonant) parts were exaggerated relative to the slower (mostly vowel) parts. Compared with training with unmodified speech, training with temporally exaggerated speech caused far larger posttraining benefits. Importantly, benefits corresponded to about 2 years of developmental age, were achieved with only 4 weeks of training, generalized to unmodified natural speech comprehension, and were maintained at follow-up 6 weeks after training completion.
We do not know how these interventions affected quality of life (QoL) in these LLI children. But in children with hearing loss and with cochlear implants (CIs), speech recognition also has generally been treated as a benchmark for success of intervention.
At the same time, non-verbal communication skills1 (e.g., ) have been comparatively disregarded. This seems unfortunate, because seminal work has shown only a relatively weak relationship between perceived QoL and speech recognition in CI users (), and because consistent positive correlations between QoL and abilities to perceive vocal emotions were reported more recently (; ; ). These are sometimes larger than those between QoL and speech comprehension, such that the role of vocal emotions in communication can hardly be overstated (). In children with CIs, studies with large samples suggest that psychosocial wellbeing is tightly related to communication skills (). Moreover, in hearing children from low-income families, the quality of communication is a more important predictor for expressive language development in the first 3 years of life than the quantity of caregivers’ words during interaction ().
The voice—like the face—not only conveys a rich set of paralinguistic cues about a speaker’s arousal and emotions, but also more time-stable speaker characteristics including speaker identity, gender, or age (). Beyond emotions, there is now tentative evidence from adult CI users that QoL can be positively related to abilities to perceive speaker age or gender (), which could emphasize the importance of social information in terms of being aware who one is talking to. Nonetheless, more research is needed to understand the role of these abilities for communication success and QoL with a CI, especially in children. This can be seen in line with findings that children with a CI are particularly disadvantaged in communication and social participation, including in the school context, even when performing well on linguistic tests (). QoL in adults is typically regarded to depend on four pillars of physical health, psychological factors, social relationships, and environmental factors (). Similarly, QoL in children is adapted to four relevant domains of life relating to physical, emotional, social and school functioning (). We define socio-emotional skills to include abilities for emotion recognition and expression, perspective taking, theory of mind, empathy, prosocial behavior, and conflict resolution (; ; ). It is worth remembering that there may be bidirectional interactions between psychosocial problems and hearing performance (), and also that socio-emotional skills appear systematically affected in children with hearing impairment more generally (for a systematic review, cf. ). Accordingly, many of the arguments made in this manuscript may apply to this target group as well.
Simple morphing and parameter-specific morphing in basic science
Soon after image morphing was invented to manipulate faces () this triggered a real revolution to social psychophysics: Morphing suddenly allowed researchers to perform objective, quantitative, subtle and photorealistic manipulations of a social signal—regardless of whether this was identity, emotion, age, or another domain of facial variation; morphing is a general purpose technique. More than a decade later, STRAIGHT software was introduced as an analogous technology for researchers in audition (). With morphing, researchers can interpolate two voices (or faces), and can also create digital averages across many speakers. One key finding here is that averages are consistently perceived as more attractive than would be expected from the individual contributing faces (), or voices (). Moreover, attractiveness is negatively correlated with distinctiveness for both faces and voices (Zäske et al., 2020), and attractive faces are less memorable than unattractive ones (Wiese et al., 2014). More crucially for present purposes, a suitable average face (or voice) can serve as a reference for caricaturing: Interpolating between an average and an individual speaker can then be used to create an anti-caricature of an individual, whereas extrapolation beyond the individual can be used to create a digital caricature, in which all idiosyncratic features of speaker which deviate from the average are accentuated. Figure 1 illustrates this for faces, and shows parameter-specific caricatures of a face which were performed separately for shape and texture (note that both parameters can be combined in a full caricature).
FIGURE 1
In audition, morphing has supported investigations into neurocognitive mechanisms of voice perception with controlled stimuli, focusing on different signals such as gender (; ), emotions (, ; ; ), identity (; ), or age (Zäske and Schweinberger, 2011). Here, one consistent finding has been that the perception of the same voice (e.g., one at an intermediate position on an angry-to-fearful continuum) can be influenced by previous adaptation to salient (e.g., uniformly angry) voices. Such contrastive adaptation effects were generally interpreted as evidence for norm-based coding of social signals in voices, complementing earlier analogous findings in faces. At a technical level, it should be noted that both sound and image morphing operate on computationally independent parameters which, at least in principle, can be manipulated separately (e.g., Figure 1). In image morphing, two independent parameters are shape (the 2D or 3D shape and metric relationships between facial features) and texture (skin reflectance characteristics in terms of coloration and pigmentation). Analogously, in sound morphing with TANDEM-STRAIGHT () five parameters can be manipulated separately (for technical detail, cf. ; ): fundamental frequency (F0), formant frequencies (FF), spectrum level information (SL), aperiodicities (AP, including shimmer and jitter), and time (T). Note though that parameter-specific morphing (PSM) is rare—most published studies use stimuli in which all morph parameters are simply manipulated conjointly.
In the section “Clinical relevance of morphing and caricaturing: Voices” below, we propose that morphing, and PSM in particular, provides researchers with a powerful toolbox which allows, firstly, to better understand the role of auditory information for the perception of different social signals (emotion, identity, etc.) in the voice. Secondly, PSM allows to better understand and quantify individual differences in high and low performers, including in clinical conditions with sensory impairments to hearing or with central impairments (e.g., autism and phonagnosia). Finally, diagnostic information obtained with PSM can be used to develop and test tailor-made perceptual trainings, and, as a perspective, to contribute to developing tailor-made sound processing technology in CI devices. In the next section “Improving the recognition of social signals by digital caricaturing: History and perspectives”, we provide directly relevant context for this perspective.
Improving the recognition of social signals by digital caricaturing: History and perspectives
Historically, the first computerized facial caricatures by preceded the advent of image morphing. Brennan’s work focused on computerized line drawings reminiscent of traditional hand-drawn caricatures. Thus, the first digital caricatures were entirely lacking texture (grayscale luminance or coloration) information and worked exclusively by enhancing distinctive aspects of shape (spatial positioning of salient landmarks) information. As a comment, this work may have primed researchers to assume that shape information is crucial for face recognition (e.g., )—although it is now clear that the recognition of familiar faces from shape information alone is extremely poor ().
A substantial proportion of studies using caricaturing to improve face recognition used shape caricatures only, leaving texture unchanged. On one hand, this seems unfortunate, because texture caricatures are more efficient than shape caricatures to improve recognition of experimentally familiarized faces ()—a finding which generalizes to faces of different “races” relative to the observer (Zhou et al., 2021). On the other hand, the good news could be that already promising published results may actually underestimate the full potential of caricaturing.
Two recent studies used caricatured faces in a face recognition training program adapted for older adults (), or for young adults with poor face recognition skills (). Both employed 6–12 laboratory-based training sessions (ca. 1 h each) between pre- and post-training sessions, and quantified both training-induced cortical plasticity in the EEG/ERP and generalizations of training effects to other face perception tests. Training induced consistently enhanced face-sensitive ERP responses in both studies, suggesting training-induced cortical plasticity. Although behavioral benefits of this short training on face perception tests were small in magnitude, showed PSM-specific training-induced improvements of training either with shape or with texture caricatures: Whereas shape-caricature training improved unfamiliar face matching, texture training elicited more marked improvements in face learning and memory. Together, this indicates that PSM caricature training is promising to improve performance in people with low face recognition skills.
Clinical relevance of morphing and caricaturing: Faces
In vision, caricaturing has been successfully used as a general method to improve face recognition, and may work best under poor visibility conditions, in older adults (), or in people with lower-than-average face recognition skills (). Importantly, when initially learned as caricatures, newly learned faces can be recognized better from veridical images at test, even compared to when learned in veridical versions also (). The Canberra group around Elinor McKone uses caricaturing to improve face recognition in people with sensory impairment due to age-related macular degeneration (AMD), and reports convincing evidence that caricaturing consistently benefits face recognition in affected individuals (). In parallel, the group reports evidence showing the impact of impaired face perception in AMD patients on both social interaction and QoL (), and identifies impaired face perception as an important contributor to lower QoL in AMD.
By implication, effective interventions to improve face perception should have potential to enhance QoL in these individuals. Note that the above-mentioned effects of caricaturing were obtained with static faces; for better transfer into everyday life, real-time facial caricaturing technology would be desirable. Although unavailable today, real-time caricaturing may approach feasibility in the foreseeable future, and may eventually be combined with bionic eyes in patients with prosthetic vision ().
Clinical relevance of morphing and caricaturing: Voices
To illustrate how PSM can improve assessment of voice perception, consider a recent study into the ability of adult CI users to perceive speaker gender in morphed voices in three conditions. In these conditions, acoustic information about speaker gender was preserved (a) in all STRAIGHT parameters (“full morphs”), (b) only in the F0 contour (“F0 morphs”; with all other parameters set to a non-informative intermediate level), or (c) only in vocal timbre (“timbre morphs,” which reflect a combination of FF, SL, and AP information). Although F0 and timbre both contribute to speaker gender perception in normal-hearing listeners, this study revealed that adult CI users exclusively used F0 cues to perceive speaker gender, and did not make efficient use of timbre in this task (). This might be thought to reflect inefficient transmission of timbre cues by a CI, but another recent study with analogous methods and participant samples seems to exclude this simple interpretation: In a task to perceive vocal emotions, adult CI users were far more efficient to use timbre, compared to F0 information (). In addition, both studies revealed that these qualitative differences are seen despite large interindividual differences between CI users. This suggests that the processing of acoustic information with a CI should be considered in the context of the target social signal.
Together, PSM can help to understand sensory determinants of successful recognition of communicative signals and their impairments. But PSM might also help to devise tailor-made training programs with acoustically enhanced stimuli, for instance by exaggerating aspects of the signal that can still be processed relatively efficiently by an individual child or adult with a CI. Arguably, the prospects of such an approach depends on the potential of caricatures to improve the recognition of socio-emotional signals. Thus, we discuss relevant evidence in the next section “Discussion.”
Importantly, although there is still a relative lack of research on caricatures of voices, a recent study has established that morphing of vocal emotions causes linear effects to the perception of emotional intensity and arousal, with caricatured emotions obtaining the highest ratings (Whiting et al., 2020). Together with our own groundwork, these results have encouraged us to develop an online training program which utilizes vocal caricatures of emotions to enhance vocal emotion recognition in CI users. Figure 2 illustrates initial results from this currently ongoing study, which tentatively suggest to us that caricature training indeed might be promising to improve vocal emotion recognition in CI users. We note that these findings clearly will need to be substantiated in a full study which also controls for procedural training effects; until then, they should be seen as preliminary, and interpreted with due caution.
FIGURE 2
Discussion
Our current training program involves three (pre-training, post-training, and follow-up) lab-based sessions to optimize standardization and data quality for evaluation purposes. At the same time, accessibility and user-friendliness of the training itself should be given high priority. From a feasibility perspective, lab-based training is time-consuming, requires substantial personnel assistance, and often implicates few but long training sessions (i.e., massed rather than distributed training). On-line (or mobile) training versions, in which participants exercise at home in more frequent but shorter training sessions can facilitate transfer of acquired skills into daily life. More and shorter training session also may better exploit well-known benefits of distributed over massed learning, which seem operational for children at least from primary school ages (e.g., ). Trainings at home are convenient, cost-effective and accessible, particularly for children in families with financial or geographical difficulties for accessing center-based services. There is increasing proof-of-efficiency for such home-based trainings in the domain of social skills (e.g., ).
A key issue for contributions to this Special Research Topic is how QoL should be best assessed in children with a CI, who often receive their implant early in life. Good arguments to prefer self-reports over parent (or caregiver) reports include that there tends to be only limited agreement between these, and that young children from 5 years of age can give increasingly detailed and reliable self-ratings of QoL, provided that child-centered and age-appropriate instruments are used (), such as the KINDL-R (e.g., Zaidman-Zait et al., 2017) or the PedsQL () and its derivates. We hypothesize that efficient trainings of vocal emotion recognition will have positive effects on several domains of pediatric QoL, particularly on emotional, social, and school functioning.
Provided the training benefits with adult CI users (Figure 2) are confirmed upon study completion, this would indicate that the potential of PSM-based caricature trainings calls for in-depth exploration. We believe that such training development and evaluation should proceed in parallel with basic research on PSM methods in voice perception. As one of the next steps, we anticipate the development and evaluation of a child-friendly version of the training targeted at children with a CI. Within the format restrictions of this article, we cannot discuss the potential of audiovisual (voice with congruent dynamic facial information) versions of a training, but given the multimodal nature of emotional communication (Young et al., 2020), audiovisual trainings may be both promising and timely: Recent research has revealed adaptive benefits from visual facial information, thus modifying earlier beliefs that effectively discouraged the use of visual stimulation during rehabilitation with a CI (; ; ). Of relevance, there is enormous current progress in digital speech synthesis technology (Yamagishi et al., 2012; ). Our understanding from cross-field personal communications at the first interdisciplinary conference on voice identity (VoiceID; see ) is that technological progress may soon enable real-time caricaturing of voices. Accordingly, this approach could inform intelligent and adaptive CI sound processors to enhance interaction quality. From a user’s viewpoint, it seems central to evaluate any evidence-based future training programs against the degree to which they cannot only improve socio-emotional communication, but also QoL in young CI recipients.
Conclusion
Socio-emotional skills are of key importance for QoL in young CI recipients. Unfortunately, skills to perceive emotions and other non-verbal communicative signals from the voice have been neglected by previous research which, we argue, overfocused on speech recognition as the main benchmark for CI success. We show perspectives for how parameter-specific morphing and caricaturing can provide a methodological toolbox for better individual assessment and intervention in the domain of voice perception. Vision impairment and face perception are examples from a related domain for which caricaturing was already shown to improve communication and QoL; we present arguments and first results that advocate this approach for CI users. In summary, this manuscript provides perspectives both for more efficient perceptual training programs and for enhanced sound processing technologies that may benefit socio-emotional communication and QoL with a CI.
Statements
Data availability statement
The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.
Ethics statement
Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.
Author contributions
SRS: conceptualization, writing—initial draft, and review and editing. CIvE: conceptualization, writing—review and editing. Both authors contributed to the article and approved the submitted version.
Funding
CIvE was funded by a fellowship from the Studienstiftung des deutschen Volkes. The authors’ research in this topic area is now funded by a grant from the Deutsche Forschungsgemeinschaft (DFG), grant reference SCHW 511/25-1.
Acknowledgments
We gratefully acknowledge the support of Lukas Erchinger, M.Sc., and Jenny M. Ruttloff, B.Sc., in preparing and conducting the ongoing online caricature training study.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Footnotes
1.^Please note that we do not refer to sign languages in this manuscript, but to paralinguistic social-communicative vocal and facial signals (e.g., a speaker’s emotions).
References
1
AmbridgeB.TheakstonA. L.LievenE. V. M.TomaselloM. (2006). The distributed learning effect for children’s acquisition of an abstract syntactic construction.Cogn. Devel.21174–193. 10.1016/j.cogdev.2005.09.003
2
AndersonC. A.WigginsI. M.KitterickP. T.HartleyD. E. H. (2017). Adaptive benefit of cross-modal plasticity following cochlear implantation in deaf adults.Proc. Natl. Acad. Sci. U.S.A.11410256–10261. 10.1073/pnas.1704785114
3
AndicsA.McQueenJ. M.PeterssonK. M.GalV.RudasG.VidnyanszkyZ. (2010). Neural mechanisms for voice recognition.NeuroImage521528–1540.
4
AsadiH.DellwoV.LavanN.PellegrinoE.RoswandowitzC. (2022). 1st Interdisciplinary Conference on Voice Identity (VoiceID): Perception, Production, and Computational Approaches.Zurich62022.
5
BeaumontR.WalkerH.WeissJ.SofronoffK. (2021). Randomized Controlled Trial of a Video Gaming-Based Social Skills Program for Children on the Autism Spectrum.J. Autism Devel. Disor.513637–3650. 10.1007/s10803-020-04801-z
6
BensonP. J.PerrettD. I. (1991). Perception and recognition of photographic quality facial caricatures: Implications for the recognition of natural images.Europ. J. Cogn. Psychol.3105–135.
7
BestelmeyerP. E. G.MaurageP.RougerJ.LatinusM.BelinP. (2014). Adaptation to Vocal Expressions Reveals Multistep Perception of Auditory Emotion.J. Neurosci.348098–8105. 10.1523/jneurosci.4820-13.2014
8
BestelmeyerP. E. G.RougerJ.DeBruineL. M.BelinP. (2010). Auditory adaptation in vocal affect perception.Cognition117217–223.
9
BrennanS. E. (1985). Caricature generator: the dynamic exaggeration of faces by computer.Leonardo18170–178. 10.2307/1578048
10
BruckertL.BestelmeyerP.LatinusM.RougerJ.CharestI.RousseletG. A.et al (2010). Vocal Attractiveness Increases by Averaging.Curr. Biol.20116–120.
11
BurtonA. M.SchweinbergerS. R.JenkinsR.KaufmannJ. M. (2015). Arguments Against a Configural Processing Account of Familiar Face Recognition.Perspect. Psychol. Sci.10482–496. 10.1177/1745691615583129
12
DammeyerJ. (2010). Psychosocial Development in a Danish Population of Children With Cochlear Implants and Deaf and Hard-of-Hearing Children.J. Deaf Stud. Deaf Educ.1550–58. 10.1093/deafed/enp024
13
DawelA.WongT. Y.McMorrowJ.IvanoviciC.HeX.BarnesN.et al (2019). Caricaturing as a General Method to Improve Poor Face Recognition: Evidence From Low-Resolution Images, Other-Race Faces, and Older Adults.J. Exp. Psychol. Appl.25256–279. 10.1037/xap0000180
14
FrühholzS.SchweinbergerS. R. (2021). Nonverbal auditory communication - Evidence for integrated neural systems for voice signal production and perception.Progr. Neurobiol.199:101948. 10.1016/j.pneurobio.2020.101948
15
Hirsh-PasekK.AdamsonL. B.BakemanR.OwenM. T.GolinkoffR. M.PaceA.et al (2015). The Contribution of Early Communication Quality to Low-Income Children’s Language Success.Psychol. Sci.261071–1083. 10.1177/0956797615581493
16
HuberM. (2005). Health-related quality of life of Austrian children and adolescents with cochlear implants.Int. J. Pediatr. Otorhinolaryngol.691089–1101. 10.1016/j.ijporl.2005.02.018
17
HuberM.HavasC. (2019). Restricted Speech Recognition in Noise and Quality of Life of Hearing-Impaired Children and Adolescents With Cochlear Implants - Need for Studies Addressing This Topic With Valid Pediatric Quality of Life Instruments.Front. Psychol.10:2085. 10.3389/fpsyg.2019.02085
18
ItzM. L.SchweinbergerS. R.KaufmannJ. M. (2017). Caricature generalization benefits for faces learned with enhanced idiosyncratic shape or texture.Cogn. Affect. Behav. Neurosci.17185–197. 10.3758/s13415-016-0471-y
19
ItzM. L.SchweinbergerS. R.SchulzC.KaufmannJ. M. (2014). Neural correlates of facilitations in face learning by selective caricaturing of facial shape or reflectance.NeuroImage102736–747. 10.1016/j.neuroimage.2014.08.042
20
JiamN. T.CaldwellM.DerocheM. L.ChatterjeeM.LimbC. J. (2017). Voice emotion perception and production in cochlear implant users.Hear. Res.35230–39. 10.1016/j.heares.2017.01.006
21
KaufmannJ. M.SchulzC.SchweinbergerS. R. (2013). High and low performers differ in the use of shape information for face recognition.Neuropsychologia511310–1319.
22
KawaharaH.MatsuiH. (2003). Auditory morphing based on an elastic perceptual distance metric in an interference-free time-frequency representation.IEEE Proc. ICASSP1256–259.
23
KawaharaH.MoriseM.TakahashiT.NisimuraR.IrinoT.BannoH. (2008). Tandem-STRAIGHT: A temporally stable power spectral representation for periodic signals and applications to interference-free spectrum F0, and aperiodicity estimation.Proc. ICASSP20083933–3936.
24
KawaharaH.SkukV. G. (2019). “Voice Morphing,” in The Oxford Handbook of Voice Processing, edsFrühholzS.BelinP. (Oxford: Oxford University Press), 685–706.
25
KlimeckiO. M. (2019). The Role of Empathy and Compassion in Conflict Resolution.Emot. Rev.11310–325. 10.1177/1754073919838609
26
LaneJ.RohanE. M. F.SabetiF.EssexR. W.MaddessT.BarnesN.et al (2018a). Improving face identity perception in age-related macular degeneration via caricaturing.Scientif. Rep.8:15205. 10.1038/s41598-018-33543-3
27
LaneJ.RohanE. M. F.SabetiF.EssexR. W.MaddessT.DawelA.et al (2018b). Impacts of impaired face perception on social interactions and quality of life in age-related macular degeneration: A qualitative study and new community resources.PLoS One13:e0209218. 10.1371/journal.pone.0209218
28
LangloisJ. H.RoggmanL. A. (1990). Attractive faces are only average.Psychol. Sci.1115–121.
29
LatinusM.BelinP. (2012). Perceptual Auditory Aftereffects on Voice Identity Using Brief Vowel Stimuli.PLoS One7:e41384. 10.1371/journal.pone.0041384
30
LimbachK.ItzM. L.SchweinbergerS. R.JentschA. D.RomanovaL.KaufmannJ. M. (2022). Neurocognitive effects of a training program for poor face recognizers using shape and texture caricatures: A pilot investigation.Neuropsychologia165:108133. 10.1016/j.neuropsychologia.2021.108133
31
LimbachK.KaufmannJ. M.WieseH.WitteO. W.SchweinbergerS. R. (2018). Enhancement of face-sensitive ERPs in older adults induced by face recognition training.Neuropsychologia119197–213. 10.1016/j.neuropsychologia.2018.08.010
32
LuoX.KernA.PullingK. R. (2018). Vocal emotion recognition performance predicts the quality of life in adult cochlear implant users.J. Acoust. Soc. Am.144EL429–EL435. 10.1121/1.5079575
33
LynessC. R.WollB.CampbellR.CardinV. (2013). How does visual language affect crossmodal plasticity and cochlear implant success?Neurosci. Biobehav. Rev.372621–2630. 10.1016/j.neubiorev.2013.08.011
34
McKoneE.RobbinsR. A.HeX.BarnesN. (2018). Caricaturing faces to improve identity recognition in low vision simulations: How effective is current-generation automatic assignment of landmark points?PLoS One13:10. 10.1371/journal.pone.0204361
35
MerzenichM. M.JenkinsW. M.JohnstonP.SchreinerC.MillerS. L.TallalP. (1996). Temporal processing deficits of language-learning impaired children ameliorated by training.Science27177–81. 10.1126/science.271.5245.77
36
MushtaqF.WigginsI. M.KitterickP. T.AndersonC. A.HartleyD. E. H. (2020). The Benefit of Cross-Modal Reorganization on Speech Perception in Pediatric Cochlear Implant Recipients Revealed Using Functional Near-Infrared Spectroscopy.Front. Hum. Neurosci.14:308. 10.3389/fnhum.2020.00308
37
NussbaumC.von EiffC. I.SkukV. G.SchweinbergerS. R. (2022). Vocal emotion adaptation aftereffects within and across speaker genders: Roles of timbre and fundamental frequency.Cognition219:104967. 10.1016/j.cognition.2021.104967
38
RichlerJ. J.MackM. L.GauthierI.PalmeriT. J. (2009). Holistic processing of faces happens at a glance.Vis. Res.492856–2861. 10.1016/j.visres.2009.08.025
39
RijkeW. J.VermeulenA. M.WendrichK.MylanusE.LangereisM. C.van der WiltG. J. (2021). Capability of deaf children with a cochlear implant.Disabil. Rehabil.431989–1994. 10.1080/09638288.2019.1689580
40
SchorrE. A.RothF. P.FoxN. A. (2009). Quality of Life for Children With Cochlear Implants: Perceived Benefits and Problems and the Perception of Single Words and Emotional Sounds.J. Speech Lang. Hear. Res.52141–152. 10.1044/1092-4388(2008/07-0213)
41
SchurzM.RaduaJ.TholenM. G.MaliskeL.MarguliesD. S.MarsR. B.et al (2021). Toward a Hierarchical Model of Social Cognition: A Neuroimaging Meta-Analysis and Integrative Review of Empathy and Theory of Mind.Psychol. Bull.147293–327. 10.1037/bul0000303
42
SchweinbergerS. R.CasperC.HauthalN.KaufmannJ. M.KawaharaH.KlothN.et al (2008). Auditory adaptation in voice perception.Curr. Biol.18684–688.
43
SchweinbergerS. R.KawaharaH.SimpsonA. P.SkukV. G.ZäskeR. (2014). Speaker perception.Wiley Interdisc. Rev.-Cogn. Sci.515–25.
44
ShinM.-S.SongJ.-J.HanK.-H.LeeH.-J.DoR.-M.KimB. J.et al (2015). The effect of psychosocial factors on outcomes of cochlear implantation.Acta Oto Laryngol.135572–577. 10.3109/00016489.2015.1006336
45
SismanB.YamagishiJ.KingS.LiH. Z. (2021). An overview of voice conversion and its challenges: from statistical modeling to deep learning.IEEE ACM Transac. Aud. Speech Lang. Proc.29132–157. 10.1109/taslp.2020.3038524
46
SkevingtonS. M.LotfyM.O’ConnellK. A. (2004). The World Health Organization’s WHOQOL-BREF quality of life assessment: Psychometric properties and results of the international field trial - A report from the WHOQOL group.Qual. Life Res.13299–310. 10.1023/b:qure.0000018486.91360.00
47
SkukV. G.KirchenL.OberhoffnerT.Guntinas-LichiusO.DobelC.SchweinbergerS. R. (2020). Parameter-Specific Morphing Reveals Contributions of Timbre and Fundamental Frequency Cues to the Perception of Voice Gender and Age in Cochlear Implant Users.J. Speech Lang. Hear. Res.633155–3175. 10.1044/2020_jslhr-20-00026
48
SkukV. G.SchweinbergerS. R. (2013). Adaptation aftereffects in vocal emotion perception elicited by expressive faces and voices.PLoS One8:e81691. 10.1371/journal.pone.0081691
49
SkukV. G.SchweinbergerS. R. (2014). Influences of Fundamental Frequency, Formant Frequencies, Aperiodicity and Spectrum Level on the Perception of Voice Gender.J. Speech Lang. Hear. Res.57285–296. 10.1044/1092-4388(2013/12-0314)
50
StevensonJ.KreppnerJ.PimpertonH.WorsfoldS.KennedyC. (2015). Emotional and behavioural difficulties in children and adolescents with hearing impairment: a systematic review and meta-analysis.Europ. Child Adolesc. Psych.24477–496. 10.1007/s00787-015-0697-1
51
TallalP.MillerS. L.BediG.BymaG.WangX. Q.NagarajanS. S.et al (1996). Language comprehension in language-learning impaired children improved with acoustically modified speech.Science27181–84. 10.1126/science.271.5245.81
52
VarniJ. W.SeidM.KurtinP. S. (2001). PedsQL (TM) 4.0: Reliability and validity of the pediatric quality of life Inventory (TM) Version 4.0 generic core scales in healthy and patient populations.Med. Care39800–812. 10.1097/00005650-200108000-00006
53
von EiffC. I.SkukV. G.ZaskeR.NussbaumC.FruhholzS.FeuerU.et al (2022). Parameter-Specific Morphing Reveals Contributions of Timbre to the Perception of Vocal Emotions in Cochlear Implant Users.Ear Hear.431178–1188. 10.1097/aud.0000000000001181
54
WhitingC. M.KotzS. A.GrossJ.GiordanoB. L.BelinP. (2020). The perception of caricatured emotion in voice.Cognition200:104249. 10.1016/j.cognition.2020.104249
55
WieseH.AltmannC. S.SchweinbergerS. R. (2014). Effects of attractiveness on face memory separated from distinctiveness: Evidence from event-related brain potentials.Neuropsychologia5626–36. 10.1016/j.neuropsychologia.2013.12.023
56
YamagishiJ.VeauxC.KingS.RenalsS. (2012). Speech synthesis technologies for individuals with vocal disabilities: Voice banking and reconstruction.Acous. Sci. Technol.331–5.
57
YoungA. W.FrühholzS.SchweinbergerS. R. (2020). Face and voice perception: Understanding commonalities and differences.Trends Cogn. Sci.24398–410. 10.1016/j.tics.2020.02.001
58
Zaidman-ZaitA.CurleD.JamiesonJ. R.ChiaR.KozakF. K. (2017). Health-Related Quality of Life Among Young Children With Cochlear Implants and Developmental Disabilities.Ear Hear38399–408. 10.1097/aud.0000000000000410
59
ZäskeR.SchweinbergerS. R. (2011). You are only as old as you sound: Auditory aftereffects in vocal age perception.Hear. Res.282283–288. 10.1016/j.heares.2011.06.008
60
ZäskeR.SkukV. G.SchweinbergerS. R. (2020). Attractiveness and distinctiveness between speakers’ voices in naturalistic speech and their faces are uncorrelated.Roy. Soc. Open Sci.7:12. 10.1098/rsos.201244
61
ZhouX.ItzM. L.VogtS.KaufmannJ. M.SchweinbergerS. R.MondlochC. J. (2021). Similar use of shape and texture cues for own- and other-race faces during face learning and recognition.Vis. Res.18832–41. 10.1016/j.visres.2021.06.014
Summary
Keywords
quality of life, children, cochlear implant, voice morphing, training
Citation
Schweinberger SR and von Eiff CI (2022) Enhancing socio-emotional communication and quality of life in young cochlear implant recipients: Perspectives from parameter-specific morphing and caricaturing. Front. Neurosci. 16:956917. doi: 10.3389/fnins.2022.956917
Received
30 May 2022
Accepted
26 July 2022
Published
25 August 2022
Volume
16 - 2022
Edited by
Maria Huber, Paracelsus Medical University, Austria
Reviewed by
Mickael L. D. Deroche, Concordia University, Canada
Updates
Copyright
© 2022 Schweinberger and von Eiff.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Stefan R. Schweinberger, stefan.schweinberger@uni-jena.de
This article was submitted to Auditory Cognitive Neuroscience, a section of the journal Frontiers in Neuroscience
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.