Abstract
The reach-to-grasp movement is ordinarily performed in everyday living activities and it represents a key behavior that allows humans to interact with their environment. Remarkably, it serves as an experimental test case for probing the multisensory architecture of goal-oriented actions. This review focuses on experimental evidence that enhances or modifies how we might conceptualize the “multisensory” substrates of prehension. We will review evidence suggesting that how reach-to-grasp movements are planned and executed is influenced by information coming from different sensory modalities such as vision, proprioception, audition, taste, and olfaction. The review closes with some considerations about the predominant role of the multisensory constituents in shaping prehensile behavior and how this might be important for future research developments, especially in the rehabilitative domain.
Introduction
The notion that senses are better conceptualized as interrelated modalities rather than independent channels is supported by several studies, providing evidence for common neural and psychological mechanisms for the processing of multisensory information (e.g., Graziano and Gross, ; Driver and Spence, ; Spence et al., 2000; Doyle and Walker, ). The creation of a unitary percept of objects is one of the classic roles attributed to multisensory integration (Newell, ). Indeed, we are able to recognize a mug not only by looking at it but also, for example, by touching it when it is out of view. Similarly, we can recognize a robin redbreast relying on visual information about its size, shape, and colors, but also by hearing its song.
In humans, most of the research conducted on crossmodal integration has typically focused on perceptual integration, studying this phenomenon by using arbitrary responses (e.g., reaction times, saccadic eye movements). Less clear are the effects of multisensory coding during more natural tasks, such as upper limb tasks, where actions are performed in three-dimensional space. Indeed, different sensory modalities are used in concert not only to perceive objects but also to represent actions (Fogassi and Gallese, ). In this respect, a fundamental role of multisensory integration is to help in the planning and execution of actions. In fact, most of the actions we perform daily rely on sensory information and, to act appropriately, we often have to process information coming from more than one sensory modality in parallel. The act of kicking a ball, for example, requires the integration of visual, proprioceptive, and tactile modalities. Writing is another example of an action that, to be accomplished accurately, requires the integration of visual, proprioceptive, and tactile information. In addition, recognizing and understanding what other individuals are doing depends on multimodal information (Fogassi and Gallese, ). As an example, by hearing the sound made by the flowing water into a glass, we are able to reasonably recognize the act of pouring even without seeing the acting individual. Thus, information arriving through different and multiple sensory modalities can greatly facilitate the retrieval of the representation of a given object, acting individual, or a given action in our brain.
To date, crossmodal links between vision, audition, sense of touch, and proprioception have been extensively documented for grasping an object with hands (Johansson and Westling, ; Klatzky et al., , ; Klatzky and Lederman, ; Goodwin et al., ; Jenmalm et al., ; van Beers et al., 2002; Patchay et al., , ; Sober and Sabes, 2003; Aziz-Zadeh et al., ; Gazzola et al., ; Zahariev and MacKenzie, 2007; Etzel et al., ; Castiello et al., ). Here we present a series of studies that demonstrate crossmodal links between vision and other modalities (audition, olfaction and proprioception) during grasping. Prehensile actions are one of the most frequent behavior we perform and represent a remarkable experimental test case to probe the multisensory nature of our behavior. Specifically, in virtue of the well-characterized kinematics profile of reach-to-grasp movements (Castiello and Ansuini, ; Jeannerod, ) and of the detailed and multifaced information that it can provide, we mainly focus on the contributions coming from the study of grasp kinematics to reveal its multisensory nature. The results indicate a strong multisensory effect on the posture assumed by the hand during a visually guided reach-to-grasp movement, which is also evident at the level of action planning.
The Role of Tactile Information
How tactile information influences grasping kinematics has been first explored by presenting targets of different dimensions either in visual or haptic modality (Chieffi and Gentilucci, ), and asking participants to judge their size and position. Results show that the sensory modality did not affect size estimation for the large target, whereas small stimuli tended to be underestimated when judged relying on tactile information. Adopting a similar approach, Camponogara and Volcic (,, ) recently showed how the role of the sensory modality changes over time when the right hand grasps an object which is perceived through vision, or haptically, with the left hand. When only haptic information was available, wider grip aperture and earlier initiation characterized hand preshaping, whereas the final phase of the action was slower. Conversely, vision appeared to be more relevant for the final phases of the movement, where the hand approaches the object and on-line visual feedback becomes more crucial. Visuo-haptic information made the action more efficient and precise, with the grip aperture becoming less variable and the movement execution faster.
The effect of tactile information on grasping kinematics has been documented also in terms of competing information: Gentilucci et al. () asked participants to reach and grasp visually presented objects presented in different sizes with one hand, while holding another unseen object (i.e., distractor) of different sizes (smaller or greater than the target) in the other hand. The main finding was that the size of the distractor did affect the kinematics of the grasping executed with the other hand: in particular, maximal hand aperture decreased and increased when the distractor was smaller and larger than the visual target-object, respectively. However, the effect of tactile information was observed only when the visual target-object was small and manipulation was performed using the right hand. This rendered unclear what caused the effect. These results have been confirmed and extended with a similar paradigm (Patchay et al., , ), showing that maximum hand aperture for the visual target was proportional to the dimension of the distractor, which was manipulated proprioceptively with the other hand (Figure 1A). Analogous patterns were observed when the distractor was manipulated with either the left or the right hand. Noticeably, the effect of tactile information occurred only when the distractor was actively grasped; the effect was absent when the non-reaching hand received passive tactile and proprioceptive stimulation.
Figure 1
Overall these findings show that exploring large or small object activates the movement parameterization which corresponds to the size of that object, i.e., big and small hand aperture, respectively, indicating that the selection of the appropriate “grasp” motor plan for interacting with an object can be based on tactile cues. Therefore, the mechanism underlying the guidance of actions is not only sensitive to the information conveyed via vision but also via the sense of touch. Subsequent studies suggested that the contribution of tactile information better emerges when the inputs coming from the sensory channels disagree: while in conditions of congruency between perceptual inputs (e.g., vision and touch) the benefit of adding tactile information over vision only is almost negligible, in case of a mismatch between sensory inputs a larger variability in performance is observed (Pettypiece et al.,
However, the interaction between vision and touch extends beyond grasp kinematics: for example, studies on haptic “memory” demonstrate that the weight of a previously manipulated target affects forces employed to grasp subsequent objects (Johansson and Westling,
The Role of Auditory Information
When using hands to manipulate objects and to interact with surfaces we generate contact sounds, providing important information concerned with the interaction between the moving effectors and the acted upon object. In particular, the contact sound signals both functional consequences and completion of manipulative actions. For instance, the “crash” sound associated with our hands breaking a walnut can be considered as a contact sound. Upon hearing the “crash,” we become aware of having broken the shell, thus, we stop the walnut handling, and we bring the husk to our mouth.
Zahariev and MacKenzie (2007) have focused on the role played by contact sounds in hand grasping by asking participants to perform reach-to-grasp actions toward a visual target (i.e., a wooden cube) either in the absence or in the presence of a “virtual” contact sound, delivered when digits entered the space immediately surrounding the object. Participants were informed in advance whether a contact sound was delivered or not, and the main finding was shorter movement time for the contact sound compared to the no contact sound trials. This result was taken as an evidence of the effect that auditory information might exert on the organization of hand grasping movements. However, the specific reason of why the presence of a contact sound reduced reach duration leaving unchanged hand kinematics was unclear. Furthermore, the delivered contact sound corresponded to the sound of a cork popping out of bottle, a sound which is not normally generated when touching a wooden block. Therefore, the nature of the effect caused by the contact sound remained unexplained.
A subsequent investigation addressed this issue by adopting a similar procedure (Castiello et al.,
The Role of Chemosensory Information
An aspect which has been largely neglected in terms of the multisensory processes underlying hand grasping movements concerns chemosensory information. To date, only few studies considered reach-to-grasp movements performed toward a visual target-object in the presence of olfactory cues (Castiello et al.,
Figure 2

(A) The experimental set up and examples of visual targets used in Castiello's et al. (
Altogether, the results of these studies indicate that the “size” of an odor influenced the kinematic profile of a reach-to-grasp movement. Crucially, the motor plan evoked by the odor is surprisingly fine grained and when elicited can modulate kinematics patterns both in terms of individual fingers movements and synergic movement amongst digits (Santello and Soechting,
Conclusion
This review presents the hand as a theoretical vehicle for understanding the multisensory nature of prehensile actions. The hand, an organ through which we explore our social and objective world, is integral to test the multisensory architecture of action. Unveiling how multisensory integration does shape our action not only has important implications for a full understanding of action planning and on-line control, but can also advance knowledge and applications in the fields of motor learning (Sigrist et al., 2013; Luan et al.,
Statements
Author contributions
SB, UC, and CB identified the topic, performed the literature overview, and wrote the manuscript. All authors contributed to the article and approved the submitted version.
Funding
This study was supported by a grant from MIUR (Dipartimenti di Eccellenza DM 11/05/2017 no. 262) to the Department of General Psychology.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
References
1
AlaertsK.SwinnenS. P.WenderothN. (2009). Interaction of sound and sight during action perception: evidence for shared modality-dependent action representations. Neuropsychologia47, 2593–2599. 10.1016/j.neuropsychologia.2009.05.006
2
Aziz-ZadehL.IacoboniM.ZaidelE.WilsonS.MazziottaJ. (2004). Left hemisphere motor facilitation in response to manual action sounds. Eur. J. Neurosci.19, 2609–2612. 10.1111/j.0953-816X.2004.03348.x
3
CamponogaraI.VolcicR. (2019a). Grasping adjustments to haptic, visual, and visuo-haptic object perturbations are contingent on the sensory modality. J. Neurophysiol.122, 2614–2620. 10.1152/jn.00452.2019
4
CamponogaraI.VolcicR. (2019b). Grasping movements toward seen and handheld objects. Sci. Rep.9:3665. 10.1038/s41598-018-38277-w
5
CamponogaraI.VolcicR. (2020). Integration of haptics and vision in human multisensory grasping. Cortex. 135, 173–185. 10.1101/2020.05.12.090647
6
CastielloU. (1999). Mechanisms of selection for the control of hand action. Trends Cogn. Sci.3, 264–271. 10.1016/S1364-6613(99)01346-7
7
CastielloU.AnsuiniC. (2009). Kinematic assessment of grasping, in Sensorimotor Control of Grasping: Physiology and Pathophysiology, eds. NowakD. A.HermsdörferJ. (Cambridge: Cambridge University Press), 20–32. 10.1017/CBO9780511581267.003
8
CastielloU.GiordanoB. L.BegliominiC.AnsuiniC.GrassiM. (2010). When ears drive hands: the influence of contact sound on reaching to grasp. PLoS ONE5:e12240. 10.1371/journal.pone.0012240
9
CastielloU.ZuccoG. M.ParmaV.AnsuiniC, Tirindelli, R. (2006). Cross-modal interactions between olfaction and vision when grasping. Chem. Senses31, 665–671. 10.1093/chemse/bjl007
10
ChieffiS.GentilucciM. (1993). Coordination between the transport and the grasp components during prehension movements. Exp. Brain Res.94, 471–477. 10.1007/BF00230205
11
di PellegrinoG.FadigaL.FogassiL.GalleseV.RizzolattiG. (1992). Understanding motor events: a neurophysiological study. Exp. Brain Res.91, 176–180. 10.1007/BF00230027
12
DoyleM. C.WalkerR. (2002). Multisensory interactions in saccade target selection: Curved saccade trajectories. Exp. Brain Res.142, 116–130. 10.1007/s00221-001-0919-2
13
DriverJ.SpenceC. (1998). Crossmodal attention. Curr. Opin. Neurobiol.8, 245–253. 10.1016/S0959-4388(98)80147-5
14
EtzelJ. A.GazzolaV.KeysersC. (2008). Testing simulation theory with cross-modal multivariate classification of fMRI data. PLoS ONE3:e3690. 10.1371/journal.pone.0003690
15
FadigaL.CraigheroL.OlivierE. (2005). Human motor cortex excitability during the perception of others' action. Curr. Opin. Neurobiol.15, 213–218. 10.1016/j.conb.2005.03.013
16
FogassiL.GalleseV. (2004). Action as a Binding Key to Multisensory Integration, in The Handbook of Multisensory Processes, eds. CalvertG. A.SpenceC.SteinB. E. (Cambridge, MA, US: MIT Press), 425–441.
17
GazzolaV.Aziz-ZadehL.KeysersC. (2006). Empathy and the somatotopic auditory mirror system in humans. Curr. Biol.16, 1824–1829. 10.1016/j.cub.2006.07.072
18
GentilucciM.DapratiE.GangitanoM. (1998). Haptic information differentially interferes with visual analysis in reaching-grasping control and in perceptual processes. Neuro Report9, 887–891. 10.1097/00001756-199803300-00023
19
GoodwinA. W.JenmalmP.JohanssonR. S. (1998). Control of grip force when tilting objects: effect of curvature of grasped surfaces and applied tangential torque. J. Neurosci.18, 10724–10734. 10.1523/JNEUROSCI.18-24-10724.1998
20
GrazianoM. S. A.GrossC. G. (1993). A bimodal map of space: somatosensory receptive fields in the macaque putamen with corresponding visual receptive fields. Exp. Brain Res.97, 96–109. 10.1007/BF00228820
21
JeannerodM. (2009). The study of hand movements during grasping. A historical perspective, in Sensorimotor Control of Grasping: Physiology and Pathophysiology, eds. NowakD. A.HermsdörferJ. (Cambridge: Cambridge University Press), 127–140. 10.1017/CBO9780511581267.011
22
JenmalmP.DahlstedtS.JohanssonR. S. (2000). Visual and tactile information about object-curvature control fingertip forces and grasp kinematics in human dexterous manipulation. J. Neurophysiol.84, 2984–2997. 10.1152/jn.2000.84.6.2984
23
JohanssonR. S.WestlingG. (1984). Roles of glabrous skin receptors and sensorimotor memory in automatic control of precision grip when lifting rougher or more slippery objects. Exp. Brain Res.56, 550–564. 10.1007/BF00237997
24
JohanssonR. S.WestlingG. (1988). Coordinated isometric muscle commands adequately and erroneously programmed for the weight during lifting task with precision grip. Exp. Brain Res.71, 59–71. 10.1007/BF00247522
25
KlatzkyR. L.LedermanS. J. (1988). The Intelligent Hand, in Psychology of Learning and Motivation, ed. BowerG. H. (San Diego: Academic Press), 121–151. 10.1016/S0079-7421(08)60027-4
26
KlatzkyR. L.LedermanS. J.ReedC. (1987). There's more to touch than meets the eye: the salience of object attributes for haptics with and without vision. J. Exp. Psychol. Gen.116, 356–369. 10.1037/0096-3445.116.4.356
27
KlatzkyR. L.PaiD. K.KrotkovE. P. (2000). Perception of material from contact sounds. Presence Teleoperators Virtual Environ.9, 399–410. 10.1162/105474600566907
28
LuanM.MaurerH.MirifarA.BeckmannJ.EhrlenspielF. (2020). Multisensory action effects facilitate the performance of motor sequences. Atten. Percept. Psychophys.10.3758/s13414-020-02179-9. [Epub ahead of print].
29
MaielloG.PaulunV. C.KleinL. K.FlemingR. W. (2018). The sequential-weight illusion. Percept9:2041669518790275. 10.1177/2041669518790275
30
NewellF. N. (2004). Cross-modal object recognition, in The Handbook of Multisensory Processes, eds. CalvertG. A.SpenceC.SteinB. E. (Cambridge, MA, US: MIT Press), 123–139.
31
ParmaV.GhirardelloD.TirindelliR.CastielloU. (2011a). Grasping a fruit. Hands do what flavour says. Appetite56, 249–254. 10.1016/j.appet.2010.12.013
32
ParmaV.RoveratoR.GhirardelloD.BulgheroniM.TirindelliR.CastielloU. (2011b). When flavor guides motor control: an effector independence study. Exp. Brain Res.212, 339–346. 10.1007/s00221-011-2733-9
33
PatchayS.CastielloU.HaggardP. (2003). A cross-modal interference effect in grasping objects. Psychon. Bull. Rev.10, 924–931. 10.3758/BF03196553
34
PatchayS.HaggardP.CastielloU. (2006). An object-centred reference frame for control of grasping: effects of grasping a distractor object on visuomotor control. Exp. Brain Res.170, 532–542. 10.1007/s00221-005-0240-6
35
PettypieceC. E.GoodaleM. A.CulhamJ. C. (2010). Integration of haptic and visual size cues in perception and action revealed through cross-modal conflict. Exp. Brain Res.201, 863–873. 10.1007/s00221-009-2101-1
36
RossiS.CapuaA. D.PasqualettiP.UlivelliM.FadigaL.FalzaranoV.et al. (2008). Distinct olfactory cross-modal effects on the human motor system. PLoS ONE3:e1702. 10.1371/journal.pone.0001702
37
SantelloM.SoechtingJ. F. (1998). Gradual molding of the hand to object contours. J. Neurophysiol.79, 1307–1320. 10.1152/jn.1998.79.3.1307
38
SeddaA.MonacoS.BottiniG.GoodaleM. A. (2011). Integration of visual and auditory information for hand actions: preliminary evidence for the contribution of natural sounds to grasping. Exp. Brain Res.209, 365–374. 10.1007/s00221-011-2559-5
39
ShullP. B.DamianD. D. (2015). Haptic wearables as sensory replacement, sensory augmentation and trainer – a review. J. Neuro Eng. Rehabil.12:59. 10.1186/s12984-015-0055-z
40
SigristR.RauterG.RienerR.WolfP. (2013). Augmented visual, auditory, haptic, and multimodal feedback in motor learning: a review. Psychon. Bull. Rev.20, 21–53. 10.3758/s13423-012-0333-8
41
SoberS. J.SabesP. N. (2003). Multisensory integration during motor planning. J. Neurosci.23, 6982–6992. 10.1523/JNEUROSCI.23-18-06982.2003
42
SpenceC.PavaniF.DriverJ. (2000). Crossmodal links between vision and touch in covert endogenous spatial attention. J. Exp. Psychol. Hum. Percept. Perform.26, 1298–1319. 10.1037/0096-1523.26.4.1298
43
TubaldiF.AnsuiniC.DemattèM. L.TirindelliR.CastielloU. (2008a). Effects of olfactory stimuli on arm-reaching duration. Chem. Senses33, 433–440. 10.1093/chemse/bjn010
44
TubaldiF.AnsuiniC.TirindelliR.CastielloU. (2008b). Odours grab his hand but not hers. Perception37, 1886–1889. 10.1068/p6286
45
van BeersR. J.BaraducP.WolpertD. M. (2002). Role of uncertainty in sensorimotor control. Philos. Trans. R. Soc. Lond. B. Biol. Sci.357, 1137–1145. 10.1098/rstb.2002.1101
46
van PolanenV.DavareM. (2015). Sensorimotor memory biases weight perception during object lifting. Front. Hum. Neurosci.9:700. 10.3389/fnhum.2015.00700
47
van PolanenV.TiboldR.NurukiA.DavareM. (2019). Visual delay affects force scaling and weight perception during object lifting in virtual reality. J. Neurophysiol.121, 1398–1409. 10.1152/jn.00396.2018
48
ZaharievM. A.MacKenzieC. L. (2007). Grasping at ‘thin air’: multimodal contact cues for reaching and grasping. Exp. Brain Res.180, 69–84. 10.1007/s00221-006-0845-4
Summary
Keywords
multisensory integration, kinematics, grasping, sensory perception, reach-to-grasp
Citation
Betti S, Castiello U and Begliomini C (2021) Reach-to-Grasp: A Multisensory Experience. Front. Psychol. 12:614471. doi: 10.3389/fpsyg.2021.614471
Received
06 October 2020
Accepted
18 January 2021
Published
09 February 2021
Volume
12 - 2021
Edited by
Ivan Camponogara, New York University Abu Dhabi, United Arab Emirates
Reviewed by
Guido Maiello, Justus Liebig University Giessen, Germany; Vonne Van Polanen, KU Leuven, Belgium
Updates

Check for updates
Copyright
© 2021 Betti, Castiello and Begliomini.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Chiara Begliomini chiara.begliomini@unipd.it
This article was submitted to Cognition, a section of the journal Frontiers in Psychology
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.