Abstract
In face-to-face communication, humans are faced with multiple layers of discontinuous multimodal signals, such as head, face, hand gestures, speech and non-speech sounds, which need to be interpreted as coherent and unified communicative actions. This implies a fundamental computational challenge: optimally binding only signals belonging to the same communicative action while segregating signals that are not connected by the communicative content. How do we achieve such an extraordinary feat, reliably, and efficiently? To address this question, we need to further move the study of human communication beyond speech-centred perspectives and promote a multimodal approach combined with interdisciplinary cooperation. Accordingly, we seek to reconcile two explanatory frameworks recently proposed in psycholinguistics and sensory neuroscience into a neurocognitive model of multimodal face-to-face communication. First, we introduce a psycholinguistic framework that characterises face-to-face communication at three parallel processing levels: multiplex signals, multimodal gestalts and multilevel predictions. Second, we consider the recent proposal of a lateral neural visual pathway specifically dedicated to the dynamic aspects of social perception and reconceive it from a multimodal perspective (“lateral processing pathway”). Third, we reconcile the two frameworks into a neurocognitive model that proposes how multiplex signals, multimodal gestalts, and multilevel predictions may be implemented along the lateral processing pathway. Finally, we advocate a multimodal and multidisciplinary research approach, combining state-of-the-art imaging techniques, computational modelling and artificial intelligence for future empirical testing of our model.
Introduction
In face-to-face communication, we encounter multiple layers of discontinuous multimodal signals: head, face, mouth movements, hand gestures, speech and non-speech sounds. This implies a fundamental computational challenge: optimally binding only signals belonging to the same communicative action while segregating unrelated signals (Noppeney, 2021). Within this challenge, the temporal misalignment of fast-changing signals across different sensory channels raises a central binding problem (). Finally, each conversational partner is taxed by fast turn-taking dynamics (Levinson, 2016). Despite these critical constraints, we process multimodal communicative signals faster than speech alone (; ). Crucially, we use non-verbal communicative signals to facilitate semantic understanding (Özyürek, 2014) and pragmatic inference (). How do we achieve such an extraordinary feat?
To address this question, we need to move beyond the prominent speech-centred research perspective on the neurocognitive mechanisms of human communication. Building on previous calls for the need to study language in its multimodal manifestation and ecological context (Levinson and Holler, 2014; Vigliocco et al., 2014; ; Perniss, 2018), the view we put forward here seeks to reconcile two explanatory frameworks recently proposed in psycholinguistics and sensory neuroscience. Specifically, we first highlight that verbal and non-verbal communicative signals are integrated to represent socially relevant acts (Levinson and Holler, 2014) through domain-general mechanisms of multimodal integration and prediction (). Accordingly, we then reconceive the neuroscientific evidence of a third visual pathway, specialised for dynamic aspects of social perception (Pitcher and Ungerleider, 2021), from a multimodal perspective. Finally, we propose that the resulting brain network implements the sensory processing gateway necessary toward successful multimodal processing and interpretation of face-to-face communicative signals.
Multimodal processing in face-to-face interactions: A possible computational framework
recently outlined the key computational principles that support fast and efficient multimodal processing in face-to-face communication, with the ultimate goal of interpreting communicative social actions (Figure 1A). First, domain-general mechanisms of multimodal integration (Stein, 2012; Noppeney, 2021) are hypothesised to be co-opted for detecting communicative signals. For example, faster processing of multimodal relative to unimodal communicative inputs mirrors multimodal facilitation outside the domain of communication in humans (Murray et al., 2001; Senkowski, 2005; ) and animals (). proposed that multimodal interactions resting on statistical regularities among sensory inputs allow chunking the stream of concurrent dynamic inputs into multiplex signals at a perceptual, pre-semantic level. Further, the statistical regularities between multiplex signals and communicative meanings generate multimodal gestalts that bear semantic and pragmatic value, thus signalling a specific social action. For example, eyebrow frowns often accompany a raising voice pitch to signal the intention to ask a question (Nota et al., 2021). Mechanisms of Gestalt perception (Wagemans et al., 2012), social affordance (), and relevance (Sperber and Wilson, 1995) may jointly contribute to the recognition of multimodal communicative gestalts (Trujillo and Holler, 2023). Finally, the recognition of a specific social action may trigger top-down multilevel predictions about how the message will unfold in time. For example, frowning and pointing at an object typically anticipates a question about that object, triggering top-down hierarchical predictions at multiple sensory levels (e.g., vocal sounds, bodily movements) and linguistic levels (e.g., words, sentential units). Multiplex signals, multimodal gestalts, and multilevel predictions are thought to interact in a continuous, dialectic process, leading to incremental unification while the message unfolds (, ). Specifically, this supports a parallel processing framework whereby the beginning of the message simultaneously activates multiple potential interpretations (i.e., multimodal gestalts). As the message unfolds, concurrent bottom-up sensory processing and multilevel predictions iteratively refine each other toward a final gestalt solution (Trujillo and Holler, 2023). Such a parallel account accommodates evidence that processing of communicative social actions starts early (Redcay and Carlson, 2015), perhaps in parallel to semantic comprehension (Tomasello et al., 2022).
FIGURE 1
Supporting this framework, there is substantial psycholinguistic evidence for systematic associations between facial-bodily signals and social actions (; Nota et al., 2021). Moreover, the early emergence of these perceptual associations in infants (), as well as parallels in non-human primates (Rossano and Liebal, 2014), suggest they might be deeply rooted in the human onto- and phylogenesis.
Multimodal processing in face-to-face interactions: A possible neural framework
Accumulating evidence (Pitcher et al., 2014; Walbrin and Koldewyn, 2019; Landsiedel et al., 2022) suggests that dynamic visual aspects of social perception (e.g., face, hand and body movements across the visual field) cannot be easily accommodated within the classic dual-stream model for visual perception (Ungerleider and Mishkin, 1982). Accordingly, resting on both anatomical and functional evidence in humans and non-human primates, Pitcher and Ungerleider (2021) proposed the existence of a third visual processing pathway (Figure 1B) that projects on the lateral cortical surface from the early visual cortex into the mid-posterior superior temporal sulcus (pSTS) via motion-selective occipito-temporal areas (V5/hMT). Consistent evidence shows that pSTS preferentially responds to multiple types of dynamic social bodily inputs including eye, mouth, hands, and body movements (; ; ). Importantly, both anterior hMT (; Huk et al., 2002) and pSTS (; Pitcher et al., 2020; ) respond to dynamic signals across both visual hemifields in human and non-human primates, in opposition to the contralateral field bias that characterises the ventral pathway (). Together, these functional properties are thought to support social interaction, which is an inherently dynamic process requiring the integration of sensory information across the entire visual field (Pitcher and Ungerleider, 2021).
Relevantly, Pitcher and Ungerleider (2021) note that the “proximity (to pSTS, a.n.) of brain areas computing multisensory information relevant to social interactions further dissociates the third pathway from the established role of the ventral and dorsal pathways.” We further elaborate on this by reconceiving the third visual pathway as a fundamental part of a larger multimodal neural system that implements fast analysis of multisensory communicative signals during face-to-face interactions. This pathway projects from early visual and auditory regions along the lateral brain surface and into the pSTS (lateral processing pathway; LPP). From this perspective, regions in the mid-posterior and lateral superior temporal gyrus, which are sensitive to auditory motion, animacy, sounds of moving bodies and dynamic aspects of human vocalisation (i.e., prosodic intonation), become candidate nodes of the auditory bank of LPP.
Analogously to the third visual pathway, evidence supporting the existence of a third lateral auditory cortical pathway, independent of dorsal/ventral pathways (Rauschecker, 1998; Rauschecker and Tian, 2000) and projecting via motion-sensitive regions into the posterior STS, comes from both tracer studies in macaques and in vivo white matter tractography in humans (see Table 1, connectivity profiles). These mid-posterior lateral areas showing anatomical connectivity with the pSTS also show motion-sensitive and voice-sensitive responses, suggesting functional selectivity for dynamic biologically-relevant information along this lateral auditory pathway (see Table 1, functional properties). Relevantly, functional interactions and direct anatomical connections have also been observed between auditory and visual motion-sensitive regions (see Table 1), suggesting a structural scaffolding for early convergence of multimodal information () within temporo-occipital regions of the LPP that might share the same computational goal: fast and reliable analysis of multimodal information relevant to social interactions.
TABLE 1
| Auditory area | Functional/Connectivity profile | References |
| (a) In non-human primates | ||
| Mid-posterior parabelt | Auditory motion processing | Poirier et al., 2017 |
| Mid-lateral parabelt | Processing of conspecific vocalization | Petkov et al., 2008; Perrodin et al., 2011 |
| Mid-posterior parabelt | Connection to the mid-posterior STS | ; ; ; ; Smiley et al., 2007 |
| Motion-sensitive areas | Monosynaptic connection to visual MT | Ungerleider and Desimone, 1986; ; Palmer and Rosa, 2006 |
| (b) In human primates | ||
| Bilateral hPT | Preferential processing of moving sounds | Krumbholz et al., 2005; |
| Right lateral hPT | Responses to ipsilateral auditory field | Krumbholz et al., 2005 |
| Bilateral anterior hPT | Encoding of living and human-action sounds categories | |
| Right anterior hPT and area adjacent to TVA | Responses to socially meaningful prosody | ; Sammler et al., 2015; |
| Bilateral lateral hPT | White matter connections to mid- and posterior upper bank of STS | |
| Bilateral mid-lateral STG | White matter connections to mid-upper bank of STS | |
| Bil. motion-selective portions of hPT | White matter connections to motion-selective hMT | |
Functional properties and structural connectivity profile of mid-posterior and lateral auditory areas in the superior temporal gyrus as described in (a) non-human and (b) human primates.
STS, superior temporal sulcus; MT, middle temporal visual area; hPT, human planum temporale; TVA, temporal voice area; STG, superior temporal gyrus; Bil., Bilateral.
Toward a neurocognitive model of face-to-face communication
In the following section, we attempt to reconcile the psycholinguistic () and sensory neuroscience (Pitcher and Ungerleider, 2021) frameworks, reviewed so far, toward a coherent neurocognitive model of multimodal face-to-face communication. Accordingly, we propose how key computational principles underlying the perception of multimodal social actions (multiplex signals, multimodal gestalts, and multilevel predictions) might be implemented along the LPP (Figure 1C).
Detecting multimodal co-occurrences: Multiplex signals via upstream sensory regions
Traditionally, it was thought that multimodal integration takes place in higher-order polysensory areas such as parietal or prefrontal cortices, after unimodal processing in early sensory regions (Ungerleider and Mishkin, 1982; Rauschecker and Tian, 2000); however, accumulating evidence over the past two decades shows clear cross-modal interactions between early sensory areas (; ; Kayser and Logothetis, 2007; ). In fact, several studies with humans (, ; Schürmann et al., 2006; Martuzzi et al., 2007; ; Lewis and Noppeney, 2010) and primates (Schroeder et al., 2001; ; Kayser et al., 2005, 2008; Lakatos et al., 2007) have proved driving or modulatory effects of cross-modal stimuli at the bottom of the sensory processing hierarchy. Beyond identifying multimodal interactions, such evidence also revealed their ubiquity across the (sub)cortical hierarchy and called for the need to further characterise the computational principles, neural properties and behavioural relevance of these interactions. One possibility is that they differ at different processing stages (i.e., multistage integration) along the (sub)cortical hierarchy (; Noppeney et al., 2018; Noppeney, 2021).
Since visual bodily signals typically precede speech during natural face-to-face interactions (Nota et al., 2021), they may modulate the sound-induced activity in the auditory cortex by resetting the phase of ongoing oscillations (; Mégevand et al., 2020; Pouw et al., 2021). In support of a temporally-sensitive mechanism, neurophysiological (Kayser et al., 2010; ), and fMRI studies (Lewis and Noppeney, 2010; Werner and Noppeney, 2011) have shown that audiovisual interactions in early auditory cortex and hPT depended on audiovisual temporal coincidence or coherence over time. Sensitivity to temporal co-occurrences is crucial to multiplex signals, which rest on temporal statistical regularities across sensory channels at a perceptual, pre-semantic level (). Therefore, it seems plausible that upstream sensory regions (e.g., visual and auditory cortices) interact in a temporally-sensitive fashion at corresponding processing stages (i.e., via multistage integration) to implement multiplex signals [see also ]. Specifically, it may be that primary visual and auditory cortices concur to support the automatic, salience-driven detection of multimodal co-occurrences, while secondary visual and auditory cortices along the LPP (hMT/EBA and hPT/TVA) concur to represent dynamic aspects of audiovisual bodily signals, mirroring results outside the realm of face-to-face communication (Lewis and Noppeney, 2010).
Recognizing communicative meanings: Multimodal gestalts via pSTS
As reviewed above, upstream visual and auditory sensory regions are structurally and functionally interconnected with pSTS. Crucially, this region represents a site of multimodal integration of social and non-social sensory information, as shown in neuroimaging and neurophysiological studies with humans (; ; Werner and Noppeney, 2010a,b; ; Noah et al., 2020) and non-human primates (; ). While these studies employed non-linguistic but meaningful world categories such as animals, manipulable objects, and human actions, pSTS is also involved in the processing of communicative and meaningful audiovisual stimuli such as lip-speech (MacSweeney et al., 2000; Wright, 2003; Macaluso et al., 2004; van Atteveldt et al., 2004; Stevenson and James, 2009; Price, 2012; Venezia et al., 2017) and gesture-speech (, ; Hubbard et al., 2009; Willems et al., 2016). Consistently, multimodal integration in pSTS may allow the creation of meaningful neural representations (; Noppeney et al., 2018), including those bearing semantic and pragmatic values for social communication (i.e., multimodal gestalts; ). In particular, we propose that pSTS might concur toward such (multimodal) neural representations based on Bayesian Causal Inference principles (Körding et al., 2007; Shams and Beierholm, 2010; Noppeney, 2021), mirroring effects found along the dorsal audiovisual pathways for spatial localisation (Rohe and Noppeney, 2015, 2016; ; ).
Intriguingly, pSTS is positioned at the intersection of three brain systems respectively responsible for social perception, action observation, and theory of mind (Yang et al., 2015). As noticed by Pitcher and Ungerleider (2021), perceptual analysis of goal-directed actions in the pSTS likely influences activity in parietal and frontal systems that are responsible for action and intention recognition. As such, after receiving converging inputs from upstream sensory regions of the LPP, pSTS may represent the sensory processing gateway that feeds to higher-order networks for social action recognition during face-to-face communication. As a result, multiplex signals may be processed at the semantic and pragmatic levels, enabling the recognition of multimodal gestalts ().
Predicting how the conversation unfolds: Multilevel predictions along the cortical hierarchy
Increasing evidence shows that humans, among other species, build on their past experiences to construct predictive models of themselves and their sensory environment (). Accordingly, the brain can be conceived as a “prediction machine” () that attempts to match bottom-up sensory inputs with top-down expectations. Following hierarchical predictive coding (Rao and Ballard, 1999; , ), any mismatch between expectation and actual input is signalled as a prediction error that propagates up the processing hierarchy to higher-level areas; vice versa, expected inputs are “explained away,” resulting in “expectation suppression” (Summerfield et al., 2008; ; Richter et al., 2018; Walsh et al., 2020). Importantly, expectation suppression reflects the neural tuning properties along a given processing hierarchy. For example, predictions about visual object and face identity are associated with expectation suppression respectively in object-selective regions (Meyer and Olson, 2011; Kaposvari et al., 2018; Richter et al., 2018; ; ) and face-selective regions (Summerfield et al., 2008; ; Schwiedrzik and Freiwald, 2017) along the ventral visual stream [for corresponding effects in the auditory domain, see e.g., Jaramillo and Zador (2011), Todorovic et al. (2011), , ].
Similarly, multilevel predictions during face-to-face interactions () may be implemented via mechanisms of hierarchical predictive processing in neural pathways that are responsible for coding the relevant sensory information (e.g., vocal sounds, bodily movements) and linguistic information (e.g., words, sentential units, social actions). Increasing evidence shows signatures of hierarchical predictive processing during language comprehension in left-lateralized fronto-temporal regions of the language network (; Sohoglu and Davis, 2016; Willems et al., 2016; Schmitt et al., 2021; ). Accordingly, predictive processing mechanisms may implement multimodal sensory predictions relevant to face-to-face interactions along the cortical hierarchy of the LPP. Initial evidence shows that hMT and pSTS activity is reduced in response to expected than unexpected visual actions (Koster-Hale and Saxe, 2013), such as human movements violating biomechanical predictions (; Saygin et al., 2012). Further, pSTS activity is reduced in response to actions that fit rather than violate the spatiotemporal structure of the environment (Koster-Hale and Saxe, 2013), such as shifting head and gaze toward rather than away an abrupt warning signal (Pelphrey et al., 2003). Interestingly, there is evidence of a functional dissociation between hMT and pSTS, with only the latter being sensitive to violations of action intentions (Pelphrey et al., 2004). Such dissociation is suggestive of a hierarchy of computations from sensory processing of dynamic inputs in hMT (at the level of multiplex signals) to semantic and pragmatic analysis in pSTS (at the level of multimodal gestalts), which may then be reflected in the respective expectation suppression profiles. Yet, it remains an open question whether and how multimodal (e.g., audiovisual) predictions arising from face-to-face interactions generate neural signatures of hierarchical predictive processing along the entire LPP, down to upstream sensory regions [for complementary evidence, see Lee and Noppeney (2014)]. Further, it is unknown whether and how higher-order expectations from language, action recognition and theory of mind networks may feed-back to pSTS (Yang et al., 2015) and thus travel down the LPP.
Discussion and conclusion
The current proposal leaves many aspects of the model un- or under-specified, including issues of hemispheric lateralization (Pitcher and Ungerleider, 2021) and the exact relationship between LPP and brain networks responsible for language (, ; ; ), action recognition (Lingnau and Downing, 2015; Wurm and Caramazza, 2022), and theory of mind (; Mar, 2011; Schaafsma et al., 2015). Future research must provide direct empirical evidence to support our framework, as well as refine and enrich it at the algorithmic and neural levels. To start, neuroimaging and neurostimulation techniques may characterise the functional and representational properties of the LPP as proposed here, as well as its degree of lateralization and interconnection with other brain networks (Thiebaut de Schotten and Forkel, 2022). Further, it will be crucial to combine these techniques with methodological approaches that enable human motion-tracking and near-to-optimal preservation of naturalistic, ecological contexts of face-to-face social interactions, such as virtual reality (Peeters, 2019). Complementarily, hyperscanning (Redcay and Schilbach, 2019; ) and multibrain stimulation techniques (Novembre and Iannetti, 2021) will be necessary to probe the functional relevance of the LPP during multimodal face-to-face processing across interacting brains. In parallel, the use of computational models (e.g., Bayesian Causal Inference) and neuroscientific-inspired artificial intelligence (i.e., convolutional or deep neural networks) could formalise the empirical evidence and test its role (e.g., necessity, sufficiency) for human behaviour () during face-to-face interactions. Last, but not least, it will be crucial to further embrace an interdisciplinary perspective in which psycholinguistics and neuroscientific frameworks would be reciprocally validated.
We conclude that the time is mature to accept the challenge we, among others before, advocated in this perspective and move beyond the speech-centred perspective dominating research on the neurocognitive mechanisms of human communication and language. We offer an original perspective bridging two recent propositions in psycholinguistics () and sensory neuroscience (Pitcher and Ungerleider, 2021) into a neurocognitive model of multimodal face-to-face communication. Testing this framework represents a novel and promising endeavour for future research.
Statements
Author contributions
SB and AF contributed equally to the original conception of the perspective and wrote the first draft of the manuscript. FP contributed to further developing the preliminary conception. All authors contributed to manuscript revision, read, and approved the submitted version.
Funding
SB was supported by a “Starting Grant DM 737/21” from the University of Trento (R06). SB and FP were supported by a “Progetto di Rilevante Interesse Nazionale (PRIN)” from the Italian Ministry for Education, University and Research (MIUR-PRIN 2017 n.20177894ZH).
Acknowledgments
We would like to express our gratitude to Eugenio Parise for providing insightful comments on the preliminary version of this perspective.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. The reviewer JT declared a shared parent affiliation with the author AF, and the handling editor declared a past collaboration with the author SB, at the time of review.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
1
AlinkA.SchwiedrzikC. M.KohlerA.SingerW.MuckliL. (2010). Stimulus predictability reduces responses in primary visual cortex.J. Neurosci.302960–2966. 10.1523/JNEUROSCI.3730-10.2010
2
AllerM.NoppeneyU. (2019). To integrate or not to integrate: Temporal dynamics of hierarchical Bayesian causal inference.PLoS Biol.17:e3000210. 10.1371/journal.pbio.3000210
3
AllisonT.PuceA.McCarthyG. (2000). Social perception from visual cues: Role of the STS region.Trends Cogn. Sci.4267–278. 10.1016/S1364-6613(00)01501-1
4
AmadoC.HermannP.KovácsP.GrotheerM.VidnyánszkyZ.KovácsG. (2016). The contribution of surprise to the prediction based modulation of fMRI responses.Neuropsychologia84105–112. 10.1016/j.neuropsychologia.2016.02.003
5
AtilganH.TownS. M.WoodK. C.JonesG. P.MaddoxR. K.LeeA. K. C.et al (2018). Integration of visual information in auditory cortex promotes auditory scene analysis through multisensory binding.Neuron97640–655.e4. 10.1016/j.neuron.2017.12.034
6
BarascudN.PearceM. T.GriffithsT. D.FristonK. J.ChaitM. (2016). Brain responses in humans reveal ideal observer-like sensitivity to complex acoustic patterns.Proc. Natl. Acad. Sci. U.S.A.113:E616–E625. 10.1073/pnas.1508523113
7
BattalC.RezkM.MattioniS.VadlamudiJ.CollignonO. (2019). Representation of auditory motion directions and sound source locations in the human planum temporale.J. Neurosci.392208–2220. 10.1523/JNEUROSCI.2289-18.2018
8
BeauchampM. S. (2005). See me, hear me, touch me: Multisensory integration in lateral occipital-temporal cortex.Curr. Opin. Neurobiol.15145–153. 10.1016/j.conb.2005.03.011
9
BeauchampM. S.ArgallB. D.BodurkaJ.DuynJ. H.MartinA. (2004). Unraveling multisensory integration: Patchy organization within human STS multisensory cortex.Nat. Neurosci.71190–1192. 10.1038/nn1333
10
BeauchampM. S.YasarN. E.FryeR. E.RoT. (2008). Touch, sound and vision in human superior temporal sulcus.Neuroimage411011–1020. 10.1016/j.neuroimage.2008.03.015
11
BeerA. L.PlankT.MeyerG.GreenleeM. W. (2013). Combined diffusion-weighted and functional magnetic resonance imaging reveals a temporal-occipital network involved in auditory-visual object processing.Front. Integr. Neurosci.7:5. 10.3389/fnint.2013.00005
12
BelykM.BrownS. (2014). Perception of affective and linguistic prosody: An ALE meta-analysis of neuroimaging studies.Soc. Cogn. Affect. Neurosci.91395–1403. 10.1093/scan/nst124
13
BenettiS.CollignonO. (2022). Cross-modal integration and plasticity in the superior temporal cortex.Handb. Clin. Neurol.187127–143. 10.1016/B978-0-12-823493-8.00026-2
14
BesleJ.FischerC.Bidet-CauletA.LecaignardF.BertrandO.GiardM.-H. (2008). Visual activation and audiovisual interactions in the auditory cortex during speech perception: Intracranial recordings in humans.J. Neurosci.2814301–14310. 10.1523/JNEUROSCI.2875-08.2008
15
BiauE.TorralbaM.FuentemillaL.de Diego BalaguerR.Soto-FaracoS. (2015). Speaker’s hand gestures modulate speech perception through phase resetting of ongoing neural oscillations.Cortex6876–85. 10.1016/j.cortex.2014.11.018
16
BizleyJ. K.MaddoxR. K.LeeA. K. C. (2016). Defining auditory-visual objects: Behavioral tests and physiological mechanisms.Trends Neurosci.3974–85. 10.1016/j.tins.2015.12.007
17
BlankH.DavisM. H. (2016). Prediction Errors but Not Sharpened Signals Simulate Multivoxel fMRI Patterns during Speech Perception.PLoS Biol.14:e1002577. 10.1371/journal.pbio.1002577
18
BoussaoudD.UngerleiderL. G.DesimoneR. (1990). Pathways for motion analysis: Cortical connections of the medial superior temporal and fundus of the superior temporal visual areas in the macaque.J. Comp. Neurol.296462–495. 10.1002/CNE.902960311
19
BruceC.DesimoneR.GrossC. G. (1981). Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque.J. Neurophysiol.46369–384. 10.1152/jn.1981.46.2.369
20
CalvertG. A.ThesenT. (2004). Multisensory integration: Methodological approaches and emerging principles in the human brain.J. Physiol. Paris98191–205. 10.1016/j.jphysparis.2004.03.018
21
Cameron-FaulknerT.TheakstonA.LievenE.TomaselloM. (2015). The relationship between infant holdout and gives, and pointing.Infancy20576–586. 10.1111/infa.12085
22
ChenL.VroomenJ. (2013). Intersensory binding across space and time: A tutorial review.Atten. Percept. Psychophys.75790–811. 10.3758/s13414-013-0475-4
23
ClarkA. (2013). Whatever next? predictive brains, situated agents, and the future of cognitive science.Behav. Brain Sci.36181–204. 10.1017/S0140525X12000477
24
CostantiniM.GalatiG.FerrettiA.CauloM.TartaroA.RomaniG. L.et al (2005). Neural systems underlying observation of humanly impossible movements: An FMRI study.Cereb. Cortex151761–1767. 10.1093/cercor/bhi053
25
de la MotheL. A.BlumellS.KajikawaY.HackettT. A. (2006). Cortical connections of the auditory cortex in marmoset monkeys: Core and medial belt regions.J. Comp. Neurol.49627–71. 10.1002/cne.20923
26
de LangeF. P.HeilbronM.KokP. (2018). How do expectations shape perception?Trends Cogn. Sci.22764–779. 10.1016/j.tics.2018.06.002
27
DeenB.SaxeR.KanwisherN. (2020). Processing communicative facial and vocal cues in the superior temporal sulcus.Neuroimage221:117191. 10.1016/J.NEUROIMAGE.2020.117191
28
DesimoneR.UngerleiderL. G. (1986). Multiple visual areas in the caudal superior temporal sulcus of the macaque.J. Comp. Neurol.248164–189. 10.1002/cne.902480203
29
DiederichN. J.FénelonG.StebbinsG.GoetzC. G. (2009). Hallucinations in Parkinson disease.Nat. Rev. Neurol.5331–342.
30
DrijversL.HollerJ. (2022). The multimodal facilitation effect in human communication.Psychon. Bull. Rev.[Epub ahead of print]. 10.3758/S13423-022-02178-X
31
DriverJ.NoesseltT. (2008). Multisensory interplay reveals crossmodal influences on “sensory-specific” brain regions, neural responses, and judgments.Neuron5711–23. 10.1016/j.neuron.2007.12.013
32
FerrariA.NoppeneyU. (2021). Attention controls multisensory perception via two distinct mechanisms at different levels of the cortical hierarchy.PLoS Biol.19:e3001465. 10.1371/journal.pbio.3001465
33
FerrariA.RichterD.de LangeF. P. (2022). Updating contextual sensory expectations for adaptive behaviour.J. Neurosci.428855–8869. 10.1523/JNEUROSCI.1107-22.2022
34
FinziD.GomezJ.NordtM.RezaiA. A.PoltoratskiS.Grill-SpectorK. (2021). Differential spatial computations in ventral and lateral face-selective regions are scaffolded by structural connections.Nat. Commun.12:2278. 10.1038/s41467-021-22524-2
35
FoxeJ. J.MoroczI. A.MurrayM. M.HigginsB. A.JavittD. C.SchroederC. E. (2000). Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping.Cogn. Brain Res.1077–83. 10.1016/S0926-6410(00)00024-0
36
FoxeJ. J.SchroederC. E. (2005). The case for feedforward multisensory convergence during early cortical processing.Neuroreport16419–423. 10.1097/00001756-200504040-00001
37
FoxeJ. J.WylieG. R.MartinezA.SchroederC. E.JavittD. C.GuilfoyleD.et al (2002). Auditory-somatosensory multisensory processing in auditory association cortex: An fMRI study.J. Neurophysiol.88540–543. 10.1152/jn.00694.2001
38
FriedericiA. D. (2012). The cortical language circuit: From auditory perception to sentence comprehension.Trends Cogn. Sci.16262–268. 10.1016/j.tics.2012.04.001
39
FristonK. (2005). A theory of cortical responses.Philos. Trans. R. Soc. B360815–836. 10.1098/rstb.2005.1622
40
FristonK. (2010). The free-energy principle: A unified brain theory?Nat. Rev. Neurosci.11127–138. 10.1038/nrn2787
41
FrithC. D.FrithU. (2006). The neural basis of mentalizing.Neuron50531–534. 10.1016/j.neuron.2006.05.001
42
FroeselM.GacoinM.ClavagnierS.HauserM.GoudardQ.HamedS. (2021). Neural correlates of audio-visual integration of socially meaningful information in macaque monkeys.bioRxiv [Preprint]. 10.1101/2021.05.02.442333
43
FuK. M. G.JohnstonT. A.ShahA. S.ArnoldL.SmileyJ.HackettT. A.et al (2003). Auditory cortical neurons respond to somatosensory stimulation.J. Neurosci.237510–7515.
44
GalaburdaA. M.PandyaD. N. (1983). The intrinsic architectonic and connectional organization of the superior temporal region of the rhesus monkey.J. Comp. Neurol.221169–184. 10.1002/CNE.902210206
45
GallagherS.(ed.) (2020). “Direct Social Perception,” in Action and Interaction, (Oxford: Oxford University Press), 121–154. 10.1093/oso/9780198846345.003.0007
46
GhazanfarA.SchroederC. (2006). Is neocortex essentially multisensory?Trends Cogn. Sci.10278–285. 10.1016/j.tics.2006.04.008
47
GhazanfarA. A.ChandrasekaranC.LogothetisN. K. (2008). Interactions between the superior temporal sulcus and auditory cortex mediate dynamic face/voice integration in rhesus monkeys.J. Neurosci.284457–4469. 10.1523/JNEUROSCI.0541-08.2008
48
GingrasG.RowlandB. A.SteinB. E. (2009). The differing impact of multisensory and unisensory integration on behavior.J. Neurosci.294897–4902. 10.1523/JNEUROSCI.4120-08.2009
49
GiordanoB. L.McAdamsS.ZatorreR. J.KriegeskorteN.BelinP. (2013). Abstract encoding of auditory objects in cortical activity patterns.Cereb. Cortex232025–2037. 10.1093/CERCOR/BHS162
50
Gurtubay-AntolinA.BattalC.MaffeiC.RezkM.MattioniS.JovicichJ.et al (2021). Direct structural connections between auditory and visual motion-selective regions in humans.J. Neurosci.412393–2405. 10.1523/jneurosci.1552-20.2021
51
HackettT. A.SmileyJ. F.UlbertI.KarmosG.LakatosP.De La MotheL. A.et al (2007). Sources of somatosensory input to the caudal belt areas of auditory cortex.Perception361419–1430. 10.1068/p5841
52
HackettT. A.StepniewskaI.KaasJ. H. (1998). Subdivisions of auditory cortex and ipsilateral cortical connections of the parabelt auditory cortex in macaque monkeys. J. Comp. Neurol. 394, 475-495. 10.1002/(SICI)1096-9861(19980518)394:4
53
HagoortP. (2005). On broca, brain, and binding: A new framework.Trends Cogn. Sci.9416–423. 10.1016/j.tics.2005.07.004
54
HagoortP. (2019). The neurobiology of language beyond single-word processing.Science36655–58. 10.1126/science.aax0289
55
HamiltonA. F. C. (2021). Hyperscanning: Beyond the Hype.Neuron109404–407. 10.1016/j.neuron.2020.11.008
56
HassabisD.KumaranD.SummerfieldC.BotvinickM. (2017). Neuroscience-Inspired Artificial Intelligence.Neuron95245–258. 10.1016/j.neuron.2017.06.011
57
HassonU.EgidiG.MarelliM.WillemsR. M. (2018). Grounding the neurobiology of language in first principles: The necessity of non-language-centric explanations for language comprehension.Cognition180135–157. 10.1016/J.COGNITION.2018.06.018
58
HeT.RichterD.WangZ.de LangeF. P. (2022). Spatial and temporal context jointly modulate the sensory response within the ventral visual stream.J. Cogn. Neurosci.34332–347. 10.1162/jocn_a_01792
59
HeilbronM.ArmeniK.SchoffelenJ. M.HagoortP.de LangeF. P. (2022). A hierarchy of linguistic predictions during natural language comprehension.Proc. Natl. Acad. Sci. U.S.A.119:e2201968119. 10.1073/pnas.2201968119
60
HeilbronM.ChaitM. (2018). Great expectations: Is there evidence for predictive coding in auditory cortex?Neuroscience38954–73. 10.1016/j.neuroscience.2017.07.061
61
HeinG.KnightR. T. (2008). Superior temporal sulcus - it’s my area: Or is it?J. Cogn. Neurosci.202125–2136. 10.1162/jocn.2008.20148
62
HellberndN.SammlerD. (2018). Neural bases of social communicative intentions in speech.Soc. Cogn. Affect. Neurosci.13604–615. 10.1093/SCAN/NSY034
63
HickokG.PoeppelD. (2000). Towards a functional neuroanatomy of speech perception.Trends Cogn. Sci.4131–138. 10.1016/S1364-6613(00)01463-7
64
HickokG.PoeppelD. (2007). The cortical organization of speech processing.Nat. Rev. Neurosci.8393–402. 10.1038/nrn2113
65
HirschJ.NoahJ. A.ZhangX.DravidaS.OnoY. (2018). A cross-brain neural mechanism for human-to-human verbal communication.Soc. Cogn. Affect. Neurosci.13907–920. 10.1093/scan/nsy070
66
HolleH.GunterT. C.RüschemeyerS. A.HennenlotterA.IacoboniM. (2008). Neural correlates of the processing of co-speech gestures.Neuroimage392010–2024. 10.1016/j.neuroimage.2007.10.055
67
HolleH.ObleserJ.RueschemeyerS. A.GunterT. C. (2010). Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions.Neuroimage49875–884. 10.1016/j.neuroimage.2009.08.058
68
HollerJ. (2022). Visual bodily signals as core devices for coordinating minds in interaction.Philos. Trans. R. Soc. B377:20210094. 10.1098/rstb.2021.0094
69
HollerJ.KendrickK. H.LevinsonS. C. (2018). Processing language in face-to-face conversation: Questions with gestures get faster responses.Psychon. Bull. Rev.251900–1908. 10.3758/s13423-017-1363-z
70
HollerJ.LevinsonS. C. (2019). Multimodal Language Processing in Human Communication.Trends Cogn. Sci.23639–652. 10.1016/j.tics.2019.05.006
71
HubbardA. L.WilsonS. M.CallanD. E.DaprettoM. (2009). Giving speech a hand: Gesture modulates activity in auditory cortex during speech perception.Hum. Brain Mapp.301028–1037. 10.1002/hbm.20565
72
HukA. C.DoughertyR. F.HeegerD. J. (2002). Retinotopy and functional subdivision of human areas MT and MST.J. Neurosci.227195–7205. 10.1523/jneurosci.22-16-07195.2002
73
JaramilloS.ZadorA. M. (2011). The auditory cortex mediates the perceptual effects of acoustic temporal expectation. Nat. Neurosci. 14, 246–253. 10.1038/nn.2688
74
KaposvariP.KumarS.VogelsR. (2018). Statistical learning signals in macaque inferior temporal cortex.Cereb. Cortex28250–266. 10.1093/cercor/bhw374
75
KayserC.LogothetisN. K. (2007). Do early sensory cortices integrate cross-modal information?Brain Struct. Funct.212121–132. 10.1007/s00429-007-0154-0
76
KayserC.LogothetisN. K.PanzeriS. (2010). Visual enhancement of the information representation in auditory cortex.Curr. Biol.2019–24. 10.1016/j.cub.2009.10.068
77
KayserC.PetkovC. I.AugathM.LogothetisN. K. (2005). Integration of touch and sound in auditory cortex.Neuron48373–384. 10.1016/j.neuron.2005.09.018
78
KayserC.PetkovC. I.LogothetisN. K. (2008). Visual modulation of neurons in auditory cortex.Cereb. Cortex181560–1574. 10.1093/cercor/bhm187
79
KördingK. P.BeierholmU.MaW. J.QuartzS.TenenbaumJ. B.ShamsL. (2007). Causal inference in multisensory perception.PLoS One2:e943. 10.1371/journal.pone.0000943
80
Koster-HaleJ.SaxeR. (2013). Theory of mind: A neural prediction problem.Neuron79836–848. 10.1016/j.neuron.2013.08.020
81
KrumbholzK.SchönwiesnerM.Von CramonD. Y.RübsamenR.ShahN. J.ZillesK.et al (2005). Representation of interaural temporal information from left and right auditory space in the human planum temporale and inferior parietal lobe.Cereb. Cortex15317–324. 10.1093/CERCOR/BHH133
82
LakatosP.ChenC. M.O’ConnellM. N.MillsA.SchroederC. E. (2007). Neuronal oscillations and multisensory interaction in primary auditory cortex.Neuron53279–292. 10.1016/j.neuron.2006.12.011
83
LandsiedelJ.DaughtersK.DowningP. E.KoldewynK. (2022). The role of motion in the neural representation of social interactions in the posterior temporal cortex.Neuroimage262119533. 10.1016/j.neuroimage.2022.119533
84
LeeH.NoppeneyU. (2014). Temporal prediction errors in visual and auditory cortices.Curr. Biol.24:R309–R310. 10.1016/j.cub.2014.02.007
85
LevinsonS. C. (2016). Turn-taking in human communication – origins and implications for language processing.Trends Cogn. Sci.206–14. 10.1016/J.TICS.2015.10.010
86
LevinsonS. C.HollerJ. (2014). The origin of human multi-modal communication.Philos. Trans. R. Soc. B369:20130302. 10.1098/rstb.2013.0302
87
LewisR.NoppeneyU. (2010). Audiovisual synchrony improves motion discrimination via enhanced connectivity between early visual and auditory areas.J. Neurosci.3012329–12339. 10.1523/JNEUROSCI.5745-09.2010
88
LingnauA.DowningP. E. (2015). The lateral occipitotemporal cortex in action.Trends Cogn. Sci.19268–277. 10.1016/j.tics.2015.03.006
89
MacalusoE.GeorgeN.DolanR.SpenceC.DriverJ. (2004). Spatial and temporal factors during processing of audiovisual speech: A PET study.Neuroimage21725–732. 10.1016/j.neuroimage.2003.09.049
90
MacSweeneyM.AmaroE.CalvertG. A.CampbellR.DavidA. S.McGuireP.et al (2000). Silent speechreading in the absence of scanner noise: An event-related fMRI study.Neuroreport11:1729. 10.1097/00001756-200006050-00026
91
MarR. A. (2011). The neural bases of social cognition and story comprehension.Annu. Rev. Psychol.62103–134. 10.1146/annurev-psych-120709-145406
92
MartuzziR.MurrayM. M.MichelC. M.ThiranJ. P.MaederP. P.ClarkeS.et al (2007). Multisensory interactions within human primary cortices revealed by BOLD dynamics.Cereb. Cortex171672–1679. 10.1093/cercor/bhl077
93
MégevandP.MercierM. R.GroppeD. M.GolumbicE. Z.MesgaraniN.BeauchampM. S.et al (2020). Crossmodal phase reset and evoked responses provide complementary mechanisms for the influence of visual speech in auditory cortex.J. Neurosci.408530–8542. 10.1523/JNEUROSCI.0555-20.2020
94
MeyerT.OlsonC. R. (2011). Statistical learning of visual transitions in monkey inferotemporal cortex.Proc. Natl. Acad. Sci. U.S.A.10819401–19406. 10.1073/pnas.1112895108
95
MurrayM. M.FoxeJ. J.HigginsB. A.JavittD. C.SchroederC. E. (2001). Visuo-spatial neural response interactions in early cortical processing during a simple reaction time task: A high-density electrical mapping study.Neuropsychologia39828–844. 10.1016/S0028-3932(01)00004-5
96
NoahJ. A.ZhangX.DravidaS.OnoY.NaplesA.McPartlandJ. C.et al (2020). Real-time eye-to-eye contact is associated with cross-brain neural coupling in angular gyrus.Front. Hum. Neurosci.14:19. 10.3389/fnhum.2020.00019
97
NoppeneyU. (2021). Perceptual inference, learning, and attention in a multisensory world.Annu. Rev. Neurosci.44449–473. 10.1146/annurev-neuro-100120-085519
98
NoppeneyU.JonesS. A.RoheT.FerrariA. (2018). See what you hear-How the brain forms representations across the senses.Neuroforum24237–246. 10.1515/nf-2017-A066
99
NotaN.TrujilloJ. P.HollerJ. (2021). Facial signals and social actions in multimodal face-to-face interaction.Brain Sci.11:1017. 10.3390/brainsci11081017
100
NovembreG.IannettiG. D. (2021). Hyperscanning alone cannot prove causality. Multibrain stimulation can.Trends Cogn. Sci.2596–99. 10.1016/j.tics.2020.11.003
101
ÖzyürekA. (2014). Hearing and seeing meaning in speech and gesture: Insights from brain and behaviour.Philos. Trans. R. Soc. B369:20130296. 10.1098/rstb.2013.0296
102
PalmerS. M.RosaM. G. P. (2006). Quantitative analysis of the corticocortical projections to the middle temporal area in the marmoset monkey: Evolutionary and functional implications.Cereb. Cortex161361–1375. 10.1093/cercor/bhj078
103
PeetersD. (2019). Virtual reality: A game-changing method for the language sciences. Psychon. Bull. Rev. 26, 894–900. 10.3758/s13423-019-01571-3
104
PelphreyK. A.MitchellT. V.McKeownM. J.GoldsteinJ.AllisonT.McCarthyG. (2003). Brain activity evoked by the perception of human walking: Controlling for meaningful coherent motion.J. Neurosci.236819–6825. 10.1523/JNEUROSCI.23-17-06819.2003
105
PelphreyK. A.MorrisJ. P.McCarthyG. (2004). Grasping the intentions of others: The perceived intentionality of an action influences activity in the superior temporal sulcus during social perception. J. Cogn. Neurosci. 16, 1706–1716. 10.1162/0898929042947900
106
PernissP. (2018). Why we should study multimodal language.Front. Psychol.9:1109. 10.3389/FPSYG.2018.01109/BIBTEX
107
PerrodinC.KayserC.LogothetisN. K.PetkovC. I. (2011). Voice cells in the primate temporal lobe.Curr. Biol.211408–1415. 10.1016/j.cub.2011.07.028
108
PetkovC. I.KayserC.SteudelT.WhittingstallK.AugathM.LogothetisN. K. (2008). A voice region in the monkey brain.Nat. Neurosci.11367–374.
109
PitcherD.DuchaineB.WalshV. (2014). Combined TMS and fMRI reveal dissociable cortical pathways for dynamic and static face perception.Curr. Biol.242066–2070. 10.1016/j.cub.2014.07.060
110
PitcherD.PilkingtonA.RauthL.BakerC.KravitzD. J.UngerleiderL. G. (2020). The human posterior superior temporal sulcus samples visual space differently from other face-selective regions.Cereb. Cortex30778–785. 10.1093/cercor/bhz125
111
PitcherD.UngerleiderL. G. (2021). Evidence for a third visual pathway specialized for social perception.Trends Cogn. Sci.25100–110. 10.1016/j.tics.2020.11.006
112
PoirierC.BaumannS.DheerendraP.JolyO.HunterD.BalezeauF.et al (2017). Auditory motion-specific mechanisms in the primate brain.PLoS Biol.15:e2001379. 10.1371/JOURNAL.PBIO.2001379
113
PouwW.ProkschS.DrijversL.GambaM.HollerJ.KelloC.et al (2021). Multilevel rhythms in multimodal communication.Philos. Trans. R. Soc. B376: 20200334. 10.31219/OSF.IO/PSMHN
114
PriceC. J. (2012). A review and synthesis of the first 20years of PET and fMRI studies of heard speech, spoken language and reading.Neuroimage62816–847. 10.1016/j.neuroimage.2012.04.062
115
RaoR. P. N.BallardD. H. (1999). Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects.Nat. Neurosci.279–87. 10.1038/4580
116
RauscheckerJ. P. (1998). Cortical processing of complex sounds.Curr. Opin. Neurobiol.8516–521. 10.1016/S0959-4388(98)80040-8
117
RauscheckerJ. P.TianB. (2000). Mechanisms and streams for processing of “what” and “where” in auditory cortex.Proc. Natl. Acad. Sci. U.S.A.9711800–11806. 10.1073/pnas.97.22.11800
118
RedcayE.CarlsonT. A. (2015). Rapid neural discrimination of communicative gestures.Soc. Cogn. Affect. Neurosci.10545–551. 10.1093/scan/nsu089
119
RedcayE.SchilbachL. (2019). Using second-person neuroscience to elucidate the mechanisms of social interaction.Nat. Rev. Neurosci.20495–505. 10.1038/s41583-019-0179-4
120
RichterD.EkmanM.de LangeF. P. (2018). Suppressed sensory response to predictable object stimuli throughout the ventral visual stream.J. Neurosci.387452–7461. 10.1523/JNEUROSCI.3421-17.2018
121
RoheT.NoppeneyU. (2015). Cortical hierarchies perform bayesian causal inference in multisensory perception.PLoS Biol.13:e1002073. 10.1371/journal.pbio.1002073
122
RoheT.NoppeneyU. (2016). Distinct computational principles govern multisensory integration in primary sensory and association cortices.Curr. Biol.26509–514. 10.1016/j.cub.2015.12.056
123
RossanoF.LiebalK. (2014). “Requests and offers,” in orangutans and human infants,” in Requesting in social interaction, ed.DrewP. (Amsterdam: Benjamins), 335–364. 10.1075/slsi.26.13ros
124
SammlerD.GrosbrasM. H.AnwanderA.BestelmeyerP. E. G.BelinP. (2015). Dorsal and ventral pathways for prosody.Curr. Biol.253079–3085. 10.1016/J.CUB.2015.10.009
125
SayginA. P.ChaminadeT.IshiguroH.DriverJ.FrithC. (2012). The thing that should not be: Predictive coding and the uncanny valley in perceiving human and humanoid robot actions.Soc. Cogn. Affect. Neurosci.7413–422. 10.1093/scan/nsr025
126
SchaafsmaS. M.PfaffD. W.SpuntR. P.AdolphsR. (2015). Deconstructing and reconstructing theory of mind.Trends Cogn. Sci.1965–72. 10.1016/j.tics.2014.11.007
127
SchmittL. M.ErbJ.TuneS.RysopA. U.HartwigsenG.ObleserJ. (2021). Predicting speech from a cortical hierarchy of event-based time scales.Sci. Adv.7:eabi6070. 10.1126/sciadv.abi6070
128
SchroederC. E.LindsleyR. W.SpechtC.MarcoviciA.SmileyJ. F.JavittD. C. (2001). Somatosensory input to auditory association cortex in the macaque monkey.J. Neurophysiol.851322–1327.
129
SchürmannM.CaetanoG.HlushchukY.JousmäkiV.HariR. (2006). Touch activates human auditory cortex.Neuroimage301325–1331. 10.1016/j.neuroimage.2005.11.020
130
SchwiedrzikC. M.FreiwaldW. A. (2017). High-level prediction signals in a low-level area of the macaque face-processing hierarchy.Neuron9689–97.e4. 10.1016/j.neuron.2017.09.007
131
SenkowskiD. (2005). Oscillatory beta activity predicts response speed during a multisensory audiovisual reaction time task: A high-density electrical mapping study.Cereb. Cortex161556–1565. 10.1093/cercor/bhj091
132
ShamsL.BeierholmU. R. (2010). Causal inference in perception.Trends Cogn. Sci.14425–432. 10.1016/j.tics.2010.07.001
133
SmileyJ. F.HackettT. A.UlbertI.KarmasG.LakatosP.JavittD. C.et al (2007). Multisensory convergence in auditory cortex, I. Cortical connections of the caudal superior temporal plane in macaque monkeys. J. Comp. Neurol. 502, 894–923. 10.1002/CNE.21325
134
SohogluE.DavisM. H. (2016). Perceptual learning of degraded speech by minimizing prediction error.Proc. Natl. Acad. Sci. U.S.A.113:E1747–E1756. 10.1073/pnas.1523266113
135
SperberD.WilsonD. (1995). Relevance: Communication and cognition, 2nd Edn. Hoboken, NJ: Blackwell Publishing.
136
SteinB. E. (2012). The new handbook of multisensory processing. Cambridge, MA: The MIT Press. 10.7551/mitpress/8466.003.0001
137
StevensonR. A.JamesT. W. (2009). Audiovisual integration in human superior temporal sulcus: Inverse effectiveness and the neural processing of speech and object recognition.Neuroimage441210–1223. 10.1016/j.neuroimage.2008.09.034
138
SummerfieldC.TrittschuhE. H.MontiJ. M.MesulamM. M.EgnerT. (2008). Neural repetition suppression reflects fulfilled perceptual expectations.Nat. Neurosci.111004–1006. 10.1038/nn.2163
139
Thiebaut de SchottenM.ForkelS. J. (2022). The emergent properties of the connected brain.Science378505–510. 10.1126/science.abq2591
140
TodorovicA.van EdeF.MarisE.de LangeF. P. (2011). Prior expectation mediates neural adaptation to repeated sounds in the auditory cortex: An MEG study.J. Neurosci.319118–9123. 10.1523/JNEUROSCI.1425-11.2011
141
TomaselloR.GrisoniL.BouxI.SammlerD.PulvermüllerF. (2022). Instantaneous neural processing of communicative functions conveyed by speech prosody.Cereb. Cortex324885–4901. 10.1093/cercor/bhab522
142
TrujilloJ. P.HollerJ. (2023). Interactionally embedded gestalt principles of multimodal human communication.Perspect. Psychol. Sci.10.1177/17456916221141422
143
UngerleiderL. G.DesimoneR. (1986). Cortical connections of visual area MT in the macaque.J. Comp. Neurol.248190–222. 10.1002/CNE.902480204
144
UngerleiderL. G.MishkinM. (1982). “Two cortical visual systems,” in Analysis of visual behavior, edsIngleD.GoodaleM.MansfieldR. (Cambridge: MIT Press).
145
van AtteveldtN.FormisanoE.GoebelR.BlomertL. (2004). Integration of letters and speech sounds in the human brain.Neuron43271–282. 10.1016/j.neuron.2004.06.025
146
VeneziaJ. H.VadenK. I.RongF.MaddoxD.SaberiK.HickokG. (2017). Auditory, visual and audiovisual speech processing streams in superior temporal sulcus.Front. Hum. Neurosci.11:174. 10.3389/fnhum.2017.00174
147
ViglioccoG.PernissP.VinsonD. (2014). Language as a multimodal phenomenon: Implications for language learning, processing and evolution.Philos. Trans. R. Soc. B369:20130292. 10.1098/RSTB.2013.0292
148
WagemansJ.ElderJ. H.KubovyM.PalmerS. E.PetersonM. A.SinghM.et al (2012). A century of Gestalt psychology in visual perception: I. Perceptual grouping and figure-ground organization.Psychol. Bull.1381172–1217. 10.1037/a0029333
149
WalbrinJ.KoldewynK. (2019). Dyadic interaction processing in the posterior temporal cortex.Neuroimage198296–302. 10.1016/j.neuroimage.2019.05.027
150
WalshK. S.McGovernD. P.ClarkA.O’ConnellR. G. (2020). Evaluating the neurophysiological evidence for predictive processing as a model of perception.Ann. N. Y. Acad. Sci.1464242–268. 10.1111/nyas.14321
151
WernerS.NoppeneyU. (2010b). Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization.Cereb. Cortex201829–1842. 10.1093/cercor/bhp248
152
WernerS.NoppeneyU. (2010a). Distinct functional contributions of primary sensory and association areas to audiovisual integration in object categorization.J. Neurosci.302662–2675. 10.1523/JNEUROSCI.5091-09.2010
153
WernerS.NoppeneyU. (2011). The contributions of transient and sustained response codes to audiovisual integration.Cereb. Cortex21920–931. 10.1093/cercor/bhq161
154
WillemsR. M.FrankS. L.NijhofA. D.HagoortP.van den BoschA. (2016). Prediction during natural language comprehension.Cereb. Cortex262506–2516. 10.1093/cercor/bhv075
155
WrightT. M. (2003). Polysensory interactions along lateral temporal regions evoked by audiovisual speech.Cereb. Cortex131034–1043. 10.1093/cercor/13.10.1034
156
WurmM. F.CaramazzaA. (2022). Two ‘what’ pathways for action and object recognition.Trends Cogn. Sci.26103–116. 10.1016/j.tics.2021.10.003
157
YangD. Y. J.RosenblauG.KeiferC.PelphreyK. A. (2015). An integrative neural model of social perception, action observation, and theory of mind.Neurosci. Biobehav. Rev.51263–275. 10.1016/j.neubiorev.2015.01.020
Summary
Keywords
multimodal communication, face-to-face interactions, social actions, lateral cortical processing pathway, psycholinguistics, sensory neuroscience
Citation
Benetti S, Ferrari A and Pavani F (2023) Multimodal processing in face-to-face interactions: A bridging link between psycholinguistics and sensory neuroscience. Front. Hum. Neurosci. 17:1108354. doi: 10.3389/fnhum.2023.1108354
Received
25 November 2022
Accepted
11 January 2023
Published
02 February 2023
Volume
17 - 2023
Edited by
Elena Nava, University of Milano-Bicocca, Italy
Reviewed by
James P. Trujillo, Radboud University, Netherlands; Huriye Atilgan, University of Oxford, United Kingdom; Jun-ichiro Watanabe, Hitachi, Japan
Updates
Copyright
© 2023 Benetti, Ferrari and Pavani.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Stefania Benetti, stefania.benetti@unitn.it
†These authors have contributed equally to this work and share first authorship
This article was submitted to Sensory Neuroscience, a section of the journal Frontiers in Human Neuroscience
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.