# EMBODYING TOOL USE: FROM COGNITION TO NEUROREHABILITATION

EDITED BY : Mariella Pazzaglia, Giulia Galli, Yusuf Ozgur Cakmak and Jan Babic PUBLISHED IN : Frontiers in Human Neuroscience, Frontiers in Neurorobotics, Frontiers in Neurology, and Frontiers in Neuroscience

#### Frontiers eBook Copyright Statement

The copyright in the text of individual articles in this eBook is the property of their respective authors or their respective institutions or funders. The copyright in graphics and images within each article may be subject to copyright of other parties. In both cases this is subject to a license granted to Frontiers. The compilation of articles constituting this eBook is the property of Frontiers.

Each article within this eBook, and the eBook itself, are published under the most recent version of the Creative Commons CC-BY licence. The version current at the date of publication of this eBook is CC-BY 4.0. If the CC-BY licence is updated, the licence granted by Frontiers is automatically updated to the new version.

When exercising any right under the CC-BY licence, Frontiers must be attributed as the original publisher of the article or eBook, as applicable.

Authors have the responsibility of ensuring that any graphics or other materials which are the property of others may be included in the CC-BY licence, but this should be checked before relying on the CC-BY licence to reproduce those materials. Any copyright notices relating to those materials must be complied with.

Copyright and source acknowledgement notices may not be removed and must be displayed in any copy, derivative work or partial copy which includes the elements in question.

All copyright, and all rights therein, are protected by national and international copyright laws. The above represents a summary only. For further information please read Frontiers' Conditions for Website Use and Copyright Statement, and the applicable CC-BY licence.

ISSN 1664-8714 ISBN 978-2-88966-263-0 DOI 10.3389/978-2-88966-263-0

#### About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

#### Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

### Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

#### What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office: researchtopics@frontiersin.org

# EMBODYING TOOL USE: FROM COGNITION TO NEUROREHABILITATION

Topic Editors: Mariella Pazzaglia, Sapienza University of Rome, Italy Giulia Galli, Santa Lucia Foundation (IRCCS), Italy Yusuf Ozgur Cakmak, University of Otago, New Zealand Jan Babic, Institut Jožef Stefan (IJS), Slovenia

Citation: Pazzaglia, M., Galli, G., Cakmak, Y. O., Babic, J., eds. (2020). Embodying Tool Use: from Cognition to Neurorehabilitation. Lausanne: Frontiers Media SA. doi: 10.3389/978-2-88966-263-0

# Table of Contents


Giorgio Scivoletto, Giulia Galli, Monica Torre, Marco Molinari and Mariella Pazzaglia

*40 Touching! An Augmented Reality System for Unveiling Face Topography in Very Young Children*

Michiko Miyazaki, Tomohisa Asai and Ryoko Mugitani


Xiaofen Xing, Zhenqi Li, Tianyuan Xu, Lin Shu, Bin Hu and Xiangmin Xu

*66 Subthreshold Vibrotactile Noise Stimulation Immediately Improves Manual Dexterity in a Child With Developmental Coordination Disorder: A Single-Case Study*

Satoshi Nobusako, Michihiro Osumi, Atsushi Matsuo, Emi Furukawa, Takaki Maeda, Sotaro Shimada, Akio Nakai and Shu Morioka

*76 Recent Technology-Aided Programs to Support Adaptive Responses, Functional Activities, and Leisure and Communication in People With Significant Disabilities*

Giulio E. Lancioni, Marta Olivetti Belardinelli, Nirbhay N. Singh, Mark F. O'Reilly, Jeff Sigafoos and Gloria Alberti

*90 Learning of Artificial Sensation Through Long-Term Home Use of a Sensory-Enabled Prosthesis*

Ivana Cuberovic, Anisha Gill, Linda J. Resnik, Dustin J. Tyler and Emily L. Graczyk


Josselin Baumard and François Osiurak

*132 How Tool-Use Shapes Body Metric Representation: Evidence From Motor Training With and Without Robotic Assistance* Valentina Bruno, Ilaria Carpinella, Marco Rabuffetti, Lorenzo De Giuli,

Corrado Sinigaglia, Francesca Garbarini and Maurizio Ferrarin

*141 Embodiment and Presence in Virtual Reality After Stroke. A Comparative Study With Healthy Subjects*

Adrián Borrego, Jorge Latorre, Mariano Alcañiz and Roberto Llorens


Yu Sun and Rixin Tang


Nadine Spychala, Stefan Debener, Edith Bongartz, Helge H. O. Müller, Jeremy D. Thorne, Alexandra Philipsen and Niclas Braun

*240 Factors Influencing Manipulation of a Familiar Object in Patients With Limb Apraxia After Stroke*

Gloria Pizzamiglio, Zuo Zhang, Mihaela Duta and Elisabeth Rounis

*253 Effect of External Force on Agency in Physical Human-Machine Interaction*

Satoshi Endo, Jakob Fröhner, Selma Musić, Sandra Hirche and Philipp Beckerle

# Editorial: Embodying Tool Use: From Cognition to Neurorehabilitation

Giulia Galli <sup>1</sup> , Yusuf Ozgur Cakmak 2,3,4,5, Jan Babicˇ <sup>6</sup> and Mariella Pazzaglia1,7 \*

1 Istituto di Ricovero e Cura a Carattere Scientifico (IRCCS) Fondazione Santa Lucia, Rome, Italy, <sup>2</sup> Department of Anatomy, University of Otago, Dunedin, New Zealand, <sup>3</sup> Brain Health Research Centre, Dunedin, New Zealand, <sup>4</sup> Medical Technologies Centre of Research Excellence, Auckland, New Zealand, <sup>5</sup> Centre for Health Systems and Technology, Dunedin, New Zealand, <sup>6</sup> Laboratory for Neuromechanics and Biorobotics, Jožef Stefan Institute, Ljubljana, Slovenia, <sup>7</sup> Department of Psychology, University of Rome "La Sapienza", Rome, Italy

Keywords: embodiment, tool use, body representation, assistive devices, bodily illusions, robot, apraxia

**Editorial on the Research Topic**

### **Embodying Tool Use: From Cognition to Neurorehabilitation**

Tool use is a ubiquitous, fundamental human characteristic that supports our ability to extend actions and thoughts beyond simple biological boundaries (Maravita and Iriki, 2004; Pazzaglia and Zantedeschi, 2016). The process of embodiment of tools may be defined as the sense that these non-corporeal objects extend bodily representations (Botvinick and Cohen, 1998) or become a "part of us" (Schettler et al.) In this framework, tool use consists of the mutual modification of the body, device, and world (i.e., other organisms, objects, the environment, or the self) (Pazzaglia and Molinari, 2016). Ownership of body parts or tools can be studied in patients after brain and spinal cord damage (Pazzaglia et al., 2018, 2020) as well as during experimental manipulations by using a virtual body and multisensory conflict (Botvinick and Cohen, 1998; Blanke et al., 2015). The scientific community has analyzed the sense of ownership (de Vignemont, 2011), the sense of agency (Pazzaglia and Galli, 2014), and the sense of self-location (Kilteni et al., 2012) to investigate the embodiment of corporeal objects and tools and their inherent bodily experience.

A substantial body of research on this Research Topic has focused on the developments and progress made in the embodiment of tools that extend the functionality of the body to support rehabilitation and to improve the capability for acting in the environment of patients with nervous system damage. Twenty-one novel publications have been collected that include 14 original research articles, 1 case report, 1 clinical trial, 3 reviews, 1 method, and 1 opinion.

Below is a brief overview of multiple original research contributions that focused on novel techniques, recognizing challenges, exploring new paradigms and methodologies, and integrating and expanding their theoretical reach. Taken together, these findings demonstrate the increasing attention and ongoing understanding with regard to how the body and the tool influence cognition.

Regarding the role of the body in clinical practice and especially in neurorehabilitation, Baumard and Osiurak have discussed the mechanisms underlying bodily experience as prerequisites for developing studies on tool incorporation. However, Weser and Proffitt noted that the subjective experience of the embodiment of a tool is quite different from that of experiencing body ownership. They argued that the body schema and the body image must work in harmony for one to experience a coherent sense of control over one's physical form as well as for identification with the physical form. However, it seems crucial to consider the different contributions of proprioception, sensory input from multiple modalities, body appearance, identity, position, tool function, proper grip, tool expertise, goals, and agency (Sun and Tang; Bruno et al.), as these factors cause different gradients of change in body representations.

Edited and reviewed by: Mark T. Wallace, Vanderbilt University, United States

> \*Correspondence: Mariella Pazzaglia mariella.pazzaglia@uniroma1.it

#### Specialty section:

This article was submitted to Sensory Neuroscience, a section of the journal Frontiers in Human Neuroscience

Received: 21 July 2020 Accepted: 27 August 2020 Published: 20 October 2020

#### Citation:

Galli G, Cakmak YO, Babic J and ˇ Pazzaglia M (2020) Editorial: Embodying Tool Use: From Cognition to Neurorehabilitation. Front. Hum. Neurosci. 14:585670. doi: 10.3389/fnhum.2020.585670

**5**

Novel technologies are expected to open up new neurorehabilitation opportunities for the body. One of the reviews examined experimental and clinical studies that have explored the manipulation of an embodied virtual body in immersive virtual reality for experimental as well as clinical pain relief, suggesting the potential of using an embodied virtual body in immersive virtual reality (Matamala-Gomez et al.). The paper from Lancioni et al. presents an overview of recent technology-aided programs designed to help people with significant disabilities to engage in adaptive responses, functional activities, leisure, and communication and thus help them interact with their physical and social environment and improve their performance/achievement. Two contributions are innovative research approaches and neurorehabilitation techniques during childhood. These approaches involve the use of an augmented reality system for unveiling face topography in very young children (Miyazaki et al.) and the application of subthreshold vibrotactile noise stimulation to improve manual dexterity in a child with developmental coordination disorder (Nobusako et al.). The only study approaching a psychiatric condition, applied the rubber hand illusion in individuals with obsessive-compulsive disorder (OCD). This study suggests heightened malleability of body image in OCD and paves the way for a tolerable technique for the treatment of OCD (Jalal et al.).

Some of the studies have considered the applicability of virtual reality and robotic assistance in clinical populations with damage to the central nervous system. Borrego et al. determined and compared the sense of embodiment and presence elicited by a virtual environment under different perspectives and levels of immersion in individuals with stroke. Gandolfi et al. evaluated the effects of robot-assisted training on upper limb spasticity, function, muscle strength, and electromyographic muscle activity in chronic stroke patients treated with Botulinum toxin.

However, among the original research articles, a large group of contributions focused on promising interventions for more conventional rehabilitation following motor impairment after brain lesions (Borrego et al.; Pizzamiglio, Zhang, Kolasinski et al.; Gandolfi et al.; Spichala et al.; Pizzamiglio, Zhang, Duta et al.). Spichala and colleagues explored self-paced embodiable neurofeedback for post-stroke motor rehabilitation (Spichala et al.). Pizzamiglio et al. showed that patients with apraxia have difficulties in selecting elements of object-directed actions pertaining to habitual as well as goal-directed factors. Additionally, a few studies have further examined the neural underpinning of the action observation network (Pazzaglia and Galli, 2019) in relation to object use and effector independence through the application of voxel-based lesion-symptom mapping (Pizzamiglio et al.) and transcranial magnetic stimulation (Betti et al.).

Patients with brain lesions were not the only clinical population considered in this Research Topic. One of the contributions focused on the use of outcome measures to assess various areas of interest pertaining to lesions of the spinal cord (Scivoletto et al.). Surprisingly, this study revealed that the majority of the respondents did not evaluate the use of assistive technology. This result emphasizes the need for more thorough knowledge and use of outcome scales for improvement in the quality of assistive device evaluation (Galli and Pazzaglia, 2015). The theme of assistive device evaluation has also been considered in patients with transradial amputation (Cuberovic et al.). This study examined how passive learning of a home-used, neural-connected, and sensoryenabled prosthetic hand influenced the perception of artificial sensory feedback in the user. Participant interviews indicated that the naturalness of the experience and engagement with the prosthesis progressively increased, suggesting that artificial somatosensation may decrease prosthesis abandonment. From a clinical perspective, these results support the use of new rehabilitation perspectives that are necessary for successful integration of rapidly evolving technology into more effective rehabilitation practices aimed at restoring a sense of integrity in a damaged body.

The final theme that we identified in the contribution centers around innovative methodology, pioneering paradigms, and novel techniques. Using a multi-channel electroencephalogram (EEG) for automatic emotion recognition can help braininspired robots in reading people's interactive intentions and states (Xing et al.). This novel framework consists of a linear EEG mixing model and an emotion timing model. Building the linear EEG mixing model and decomposing the EEG source signals from the collected EEG signals improves the temporal classification of the EEG feature sequences, elected as the emotion classifier. Endo et al. used an adapted intentional binding effect paradigm to investigate whether the sense of agency can be used to measure the quality and experience of physical humanmachine interaction schemes that allow the human operator and the collaborative machine to act as a "single entity." One of the studies used functional near-infrared spectroscopy to measure the effects of moderate-intensity and high-intensity short-term aerobic exercise on prefrontal activity related to food images and recorded the subjective feelings of appetite in methamphetaminedependent users (Wang et al.).

In conclusion, this Research Topic collects an impressive body of literature on "Embodying Tool Use." Overall, the contributions extend and enrich the previous multidisciplinary approach and translational applications. However, despite the significant progress made in our understanding and the real-world relevance, there are boundless directions, endless possibilities, and exciting challenges yet to be explored in future research.

### AUTHOR CONTRIBUTIONS

MP planned the topic. MP and GG edited the most of papers included in the topic. YC and JB edited some papers included in the topic. GG drafted the editorial which all authors reviewed and approved. All authors contributed to the article and approved the submitted version.

### FUNDING

This work was supported by the Italian Ministry of Health (RF-2018-12365682).

## REFERENCES


**Conflict of Interest:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2020 Galli, Cakmak, Babiˇc and Pazzaglia. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Tool Embodiment: The Tool's Output Must Match the User's Input

Veronica Weser\* and Dennis R. Proffitt

Department of Psychology, University of Virginia, Charlottesville, VA, United States

The embodiment of tools and rubber hands is believed to involve the modification of two separate body representations: the body schema and the body image, respectively. It is thought that tools extend the capabilities of the body's action schema, whereas prosthetics like rubber hands are incorporated into the body image itself. Contrary to this dichotomy, recent research demonstrated that chopsticks can be embodied perceptually during a modified version of the rubber hand illusion (RHI) in which tools are held by the rubber hand and by the participant. In the present research, two experiments examined tool morpho-functional (tool output affordance, e.g., precision grasping) and sensorimotor (tool input, e.g., precision grip) match as a mechanism for this tool-use dependent change to the body image. Proprioceptive drift in the RHI occurred when the tool's output and the user's input matched, but not when this match was absent. This suggests that this factor may be necessary for tools to interact with the body image in the RHI.

Keywords: tools, rubber hand illusion, embodiment, body representation, expertise

#### Edited by:

Giulia Galli, Fondazione Santa Lucia (IRCCS), Italy

#### Reviewed by:

Luke Edward Miller, INSERM U1028 Centre de Recherche en Neurosciences de Lyon, France Elisa Canzoneri, École Polytechnique Fédérale de Lausanne, Switzerland

> \*Correspondence: Veronica Weser vuw3nb@virginia.edu

Received: 07 October 2018 Accepted: 21 December 2018 Published: 11 January 2019

#### Citation:

Weser V and Proffitt DR (2019) Tool Embodiment: The Tool's Output Must Match the User's Input. Front. Hum. Neurosci. 12:537. doi: 10.3389/fnhum.2018.00537

## INTRODUCTION

Two bodies of literature have run in parallel for nearly two decades: tool use and body representation. Both fields employ overlapping terminology and examine the ways in which noncorporeal tools or prosthetics are incorporated into and extend bodily representations. Though there has been some effort to compare the two areas of research from a speculative standpoint, only minimal headway has been made to bridge the literatures experimentally. On one hand, the investigation of human tool use demonstrates that tools are incorporated into at least some form of representation of the user's body. However, researchers who use multisensory bodily illusions like the rubber hand illusion (RHI) (Botvinick and Cohen, 1998) to examine bodily representations have repeatedly shown that the feeling of body-ownership can be extended only to objects that resemble human body parts. Thus research on tools and research using rubber hands stand in direct opposition, with many arguing that rubber hands are incorporated into the body, while tools merely extend the body. The research presented herein investigates whether consistency between the affordance of a tool and the grip used to wield it might facilitate the incorporation of a tool into the body representation in a manner akin to the rubber hand in the RHI.

In an effort to experimentally reconcile the division in the tool and body-ownership illusion literatures, Weser et al. (2017) used a novel RHI paradigm in which both the participant and the rubber hand were equipped with tools. In the classic RHI, simultaneous visuo-tactile stimulation of a rubber hand and the participant's hidden hand induces feelings of ownership of the rubber hand (Botvinick and Cohen, 1998; Tsakiris and Haggard, 2005; Tsakiris, 2016). Importantly, the illusion also results in a change in the felt position of the hand undergoing stimulation known as

proprioceptive drift: stronger subjective ownership of the rubber hand coincides with the feeling that the participant's real hand is located closer to the rubber hand. In Weser et al. (2017), it was not the rubber hand and the participant's hand that received simultaneous tactile stimulation, but rather the tools held by both. The unseen stimulation of the held tool is readily detected by the participant, and indeed research has shown that tactile stimulation of a tool is subjectively felt at the tip of the tool (Yamamoto and Kitazawa, 2001; Yamamoto et al., 2005), even though the mechanoreceptors that process the tactile information are located in the hand. Moreover, Miller et al. provided strong evidence that tools function as sensory extensions of the body in a manner akin to animal whiskers by showing that humans can accurately identify the location where a held rod contacts external object (2018). Of particular relevance to the present work, participants were able to identify the location of the tactile contact with the held object even when the stimulation was delivered by the experimenter (Miller et al., 2018). This indicates that the visuo-tactile stimulation delivered to the tip of the tool in the tool-version of the RHI was perceived accurately in Weser et al. (2017) and in the present work. Previously, the illusion was successfully induced when the tool in question was a pair of chopsticks, but not when it was a teacup. Moreover, the proprioceptive drift was greater for participants who practiced using chopsticks immediately prior to experiencing the illusion than for those who did not. Remarkably, the proprioceptive drift also increased as a function of chopstick skill, such that those who were highly skilled with chopsticks tended to perceive their hand as even closer to the location of the rubber hand than those who were less skilled.

The facilitatory effect of tool use prior to the illusion induction is in keeping with the literature that examines action-specific body representations. However, this finding is also at odds with the majority of the literature demonstrating a strict separation between body image and schema, representations for perception and action.

The success of the traditional RHI is contingent on the visual similarity, postural congruency, body part identity and laterality of the seen object and the body part receiving tactile stimulation (e.g., Costantini and Haggard, 2007; Haans et al., 2008). Perceived ownership of the rubber hand arises due to the modification of the body image, an abstract body representation that persists through time and contains a reference description of the visual, anatomical, and postural properties of the body (De Preester and Tsakiris, 2009). This body representation would appear to be the polar opposite of the ever-changing representation of the body's position in space that is easily modified to include a handheld tool—the body schema. To reiterate, the body schema is what allows a tool wielder to account for the changes in his or her capacity for action during tool use, such as the longer reach afforded by a mechanical grabber. In contrast, the body image is what allows a person to recognize and identify with his or her own hand when for example, it is entwined with the hand of another. These two body representations are at the heart of each of the literatures on tool use and multisensory illusions of body ownership (e.g., RHI), respectively.

Comparing and contrasting RHI illusion and tool use work provides further support for a division between body representations for action and perception. Even though participants report feeling as if the rubber hand has become a part of their body, the reaching actions of participants who experience proprioceptive drift following the RHI remain accurate (Kammers et al., 2009). In other words, even though they report feeling as though their hand is located closer to the rubber hand, they can still accurately reach and grasp an object with the hand that was supposedly replaced by the rubber hand during the illusion. This suggests that the RHI is only modifying the perceptual representation of the body, as movements executed by the replaced hand are still accurate. This finding can be directly contrasted with work on tool use paradigms that demonstrate using tools will alter the kinematics of reach to grasp movements (Cardinali et al., 2011, 2012; Baccarini et al., 2014).

Moreover, tools have a similar null effect on the perceptual body image representation. Cardinali et al. (2011) demonstrated that the use of a reach-extending tool increases participants' indirect length estimates of their forearms, but only when the body schema was accessed to provide the estimates. In this study, participants used a 40 cm mechanical grabbing device to reach for, grasp, lift up, and replace an object. Participants then localized one of three positions on their arm (the tip of the index finger, the wrist or the elbow) by naming the position on a scale that represented the length of the arm in response to a cue from an experimenter that was either delivered verbally (by naming either finger, wrist, or elbow) or through direct tactile stimulation of the body part. The tool-using arm was kept out of the participant's sight behind a barrier throughout the experiment. Cardinali found that after tool use, participants overestimated the distance between their wrist and elbow if the body part was touched but not named. In contrast, localizing named body parts was not affected by tool use, suggesting that using a tool may change the body representation for action; the body schema, but not necessarily a more abstract understanding of the relative location of body parts contained in the body image (Cardinali et al., 2011). Since the work by Weser et al. (2017) represents a first attempt at using the RHI paradigm to examine whether or not the multisensory stimulation of held tools is sufficient to alter the body image as assessed by measuring proprioceptive drift, it is imperative to discover the necessary conditions and constraining factors of the tool version of the RHI. One possible mechanism is that presence or absence of morpho-functional and sensorimotor match for each tool.

In Weser et al. (2017) chopsticks and teacup were selected as comparison tools because they have a different form of morphofunctional tool output and an identical sensorimotor grip input (Cardinali et al., 2016). Chopsticks, but not teacups, have a morpho-functional match: they afford precision grasping and they are wielded with a precision grip. Teacups are typically held with a precision grip, but their morphology and function are not related to precision actions. Morpho-functional refers to the output of the tool: its shape and the action it affords. Sensorimotor refers to the input provided to the tool: the motor actions of the wielder. The match (or lack thereof) between

tool morphology (output) and arm movement/grasp mechanics (input) is thought to play a deterministic role in how and whether or not the use of a tool will cause a modulation of the wielder's body representation (Miller et al., 2014; Cardinali et al., 2016).

Broadly speaking, tools that extend one's reach (such as mechanical grabbers) influence the wielder's representation of the length of his or her arm, but not the size of his or her hand (Cardinali et al., 2009a,b; Miller et al., 2014). In contrast, tools that expand the grasp of the hand but not the length of the wielder's reach specifically alter the implicit representation of the size of the hand, but not the length of the arm (Miller et al., 2014). In addition to tool type, tool dimensions and the particular movements used by a wielder also determine how the representation of the body is affected: Sposito et al. (2012) found a change when a 60 cm grabber was used, but not when a 20 cm tool was used instead. In addition, Romano et al. (2018) demonstrated that using the same tool with either predominantly shoulder movements or predominantly wrist movements determined whether participants estimated the midpoint of their forearm to be closer to their shoulder or their wrist, respectively. This finding demonstrates that tools specifically alter the representation of the body part that they are functionally augmenting, that there are particular tool properties that determine whether or not this alteration takes place, and that the way a given tool is used in a task directly affects how the body representation is updated. These studies examine long reach extending tools, or gross wholehand grasping tools; they do not speak to whether or not small precision tools also alter the representation of the hand in a manner specific to the grip used to wield the tool (i.e., precision vs. power grips). While the difference between reach-lengthening and grip-widening tools and how they affect body representations may seem obvious, additional finer-grained comparisons as in Romano et al. (2018) are still needed to assess whether it is the shape of the tool or the grip and movements used to wield the tool that has the greater impact on the body representation.

In other words, the morpho-functional output of a tool has a clear impact on the effect that tool has on one's body representation, but the sensorimotor input of the grip used to wield the tool and whether or not that input matches the type of action or output the tool affords must be investigated by comparing small handheld tools wielded with varying grips. Cardinali et al. (2016) found that sticks attached to the thumb and index finger and pliers both cause an increase in the represented length of the wielder's fingers. However, the pliers caused a global increase in finger length while the two sticks specifically lengthened the representation of the wielder's thumb and index finger. Even though both tools offered a precision output, the sensorimotor input to the tools differed. The power grip input for pliers caused the representation of the hand to shift to one where only the fingers as a unit moving in opposition to the thumb (similar to the two prongs of the pliers) were relevant. However, the precision grip input for using the sticks kept the middle, ring and pinkie finger separable from the index finger.

Cardinali et al. (2016) controlled for the morpho-functional characteristics of the tools (both had a precision output) and determined that the difference in sensorimotor input for wielding the tool affected how the hand came to be represented following use. Weser et al. (2017) controlled for the sensorimotor aspect of the tools (both had a precision grip input) and revealed that the tool without a morpho-functional and sensorimotor match (precision grip input and precision output) did not affect the body representation of the wielder. Thus, experiments that compare tools by controlling for either morpho-functional tool output or sensorimotor tool input provide a promising avenue for investigating the conditions necessary for the extension or incorporation of tools into body representations.

Experiments 1 and 2 presented herein expand on this premise by using the tool-version of the RHI to compare two tools that differ in their morpho-functional characteristics and are identical in their sensorimotor traits, as in Cardinali et al. (2016). Needlenose pliers and tweezers both have a precision output, but the inputs differ. Needle-nose pliers are used with a whole-hand power grip while tweezers are wielded with only the thumb and index finger in a precision grip. If tool morpho-functional and sensorimotor match is a constraining factor on whether or not the tool-version of the RHI succeeds, then there should be a difference between these two tools. The tweezers (Experiment 2) should result in high proprioceptive drift following the illusion while the proprioceptive drift in the case of the pliers (Experiment 1) will not significantly differ from control conditions. This finding would bring a new level of nuance to the literature on the effects of tools on body representation, as it would indicate an advantage for morpho-functional and sensorimotor match when it comes to altering the proprioceptive information about the location of a tool and the hand wielding it. It would suggest that though a match is not necessary for a modification of the hand representation to occur (see Cardinali et al., 2016), it is required for an update to be made to the model of the body's location in space following simultaneous multisensory stimulation. This would indicate that, as in the classic RHI, match allows for more than just the extension of the body to include the tool, but also the incorporation of the tool into the body.

## EXPERIMENT 1 PLIERS: MORPHO-FUNCTIONAL AND SENSORIMOTOR MISMATCH

Chopsticks and teacups were used as comparison tools in Weser et al. (2017) because the two tools have different morphofunctional outputs and identical and sensorimotor inputs. Chopsticks have a morpho-functional/sensorimotor match and teacups do not. In Experiment 1, needle-nose pliers lack this match, as they are wielded with a full-hand power grip and act on the environment in a precision-grip manner. Therefore, it follows that a pliers-version of a RHI should not be as successful as a tool-version where there is both a morpho-functional match and a sensorimotor match, such as chopsticks (Weser et al., 2017) or tweezers (Experiment 2).

## Methods

### Participants

A total of 71 right-handed individuals (18 males; mean age: 19.0; SD = 1.0) participated in exchange for credit in an

introductory psychology course at the University of Virginia. The data from 5 participants was lost due to experimenter error (2) and participants' failure to follow instruction (3), leaving 66 participants. Thirty-one participants completed the tool-skill task prior to experiencing the illusion, while the remaining 35 completed the tool-skill task at the end of the study. All participants had normal or corrected to normal vision and provided written informed consent.

### Materials

### Pliers Rubber Hand

A life cast of author VW's hand holding a pair of needle nose pliers was made from flesh-tinted plastic resin (see **Figure 1A**). An identical pair of pliers was provided for the participant to hold throughout the study and use during the tool-skill task. Together, the hand and tool measured approximately 9 cm × 20 cm × 5 cm, with the tips of the pliers resting about 2 cm above the surface of the table. The handle of the pliers contained a small spring that caused the jaws of the pliers to open whenever the user relaxed his or her grip.

### Wooden Block

The other viewed item was the wooden block (**Figure 1B**) used in Weser et al. (2017). The piece of wood was a 9 cm × 23 cm × 2 cm block, pale and beige in color, with the outline of a hand drawn on the surface in black ink. This wooden stimulus was comparable in overall size to the rubber hand holding the tool, and is comparable to the control (non-corporeal) items used in classic RHI studies (i.e., Haans et al., 2008; Longo et al., 2008).

### Tool-Skill Task

The same task used in Weser et al. (2017) was used to measure participant pliers skill. Two-hundred seventy plastic beads of various colors that measured 0.8 cm in diameter were presented to participants in a tray. Participants used their pliers to transfer each bead to a container with 6 color-labeled compartments. There were 30 beads of each color to be sorted, and 90 "distractor beads." Participants were required to move all beads of one color to the container before starting on the next color. Participants were allotted 5 min to transfer as many beads as possible. The number of beads transferred was recorded and used as a proxy value for participant plier-skill.

measured 13 cm in length, with a 10 cm handle and jaws 3 cm in length. (B) The wooden block used as the control viewed object in both experiments presented here and in Weser et al. (2017).

### Rubber Hand Illusion Questionnaire

Twenty-five questions from Longo et al. (2008) were adapted to measure the subjective experience of the tool-version of the RHI (see **Supplementary Material** for the pliers version of the questionnaire). In particular, the adapted questions referred to five different components of the experience of the illusion: embodiment of the rubber hand (10 statements), loss of the real hand (5 statements), movement of the real or rubber hand (3 statements), deafference of the real hand (3 statements), and affect (3 statements). All questions were modified to refer to the tool held by the rubber hand, rather than to the rubber hand itself.

### Experimental Design

A 2 × 2 × 2 mixed design was employed. The viewed object (pliers rubber hand vs. wooden block) and timing of visuo-tactile stimulation (synchronous vs. asynchronous) were within subjects factors, and the group (tool-skill task prior to the illusion vs. following the illusion) was a between-subjects factor. The number of beads transferred with pliers was included as a covariate, and a random effect of participant was added to account for individual differences in pliers-skill and illusion susceptibility. The 4 within-subjects conditions, completed in a random order, were: (i) pliers rubber hand synchronous (ii) pliers rubber hand asynchronous; (iii) wooden block synchronous; (iv) wooden block asynchronous. Participants completed a RHI questionnaire following the completion of each condition. Participants held pliers during all four conditions and were encouraged to stretch and rest their hand while completing the questionnaire between conditions.

In the synchronous visuo-tactile stimulation conditions, the experimenter used 2 paintbrushes to manually stroke the tip of the participant's held pliers and the pliers held by the viewed object at the same time. In the asynchronous visuo-tactile conditions, the experimenter stroked the participant's pliers first, while the pliers held by the viewed object was stroked with a latency of 500–1000 ms. Each stimulation period lasted 180 s and was timed using a stopwatch. Experimenters were instructed to apply enough pressure to the pliers that the contact would be felt. The paintbrush used measured 22 cm in length, with a 2 cm × 1 cm bristle.

### Procedure

Participants were greeted and informed that they would be using pliers and making self-perception estimates throughout the duration of the experiment. Upon arrival, participants were randomly assigned to either first complete the tool-skill task or to undergo the RHI procedure prior to using the pliers to transfer beads. During the RHI procedure, participants were seated across from the experimenter with their right, pliers-holding hand placed inside a specially constructed box, measuring 100 cm in width, 40 cm in height, and 20 cm in depth. The box was divided into three compartments of equal size, and the viewed object rested inside the central compartment in front of the participant's midline. The viewed object and the participant's hand were aligned such that both rested at the same distance in front of the participant's chest. The lateral distance between the tip of the participant's pliers

and the tip of the pliers held by the rubber hand was kept constant at 25.5 cm. The top of the box was covered by a one-way mirror. The portion of the one-way mirror above the compartment containing the participant's hand was obstructed such that the interior of the compartment could not be seen by the participant at any time during the experiment, and the surface always appeared to be a regular, two-way mirror (**Figure 2A**).

The proprioceptive judgment phase was conducted before and after each visuo-tactile stimulation phase, allowing the perceived position of the participant's held tool to be used as an implicit, quantitative proxy for measuring the strength of the illusion. A ruler with the numbers printed in reverse was supported between two poles 45 cm above the box. When illuminated from above, the mirrored surface of the box reflected the ruler numbers in their proper orientation at the same gaze depth as the floor of the box containing the rubber hand.

Before and after the visuo-tactile stimulation phase, participants verbally reported the number on the ruler that was in line with the jaws of their held pliers by projecting a parasagittal line from the tip of the tool to the ruler. During the visuo-tactile stimulation phase, the ruler was always shifted to a different random position such that the numbers the participant viewed during the judgment phases were always different. This ensured that participants did not memorize previously stated numbers and that the participant estimated the proprioceptively perceived position of their hand independently during each condition.

The central compartment of the box was illuminated from below during the visuo-tactile stimulation phase (**Figure 2B**), making the one-way mirror transparent such that the participant could view the stroking of the object inside the box. Throughout this procedure, participants were instructed to apply light pressure to the pliers' handle and keep the jaws slightly closed. This allowed the experiment to stroke both jaws of the pliers simultaneously with the paint brush. During the wooden block condition, the front corner of the block (on the participant's right) was stroked with the paint brush.

Upon completion of all four RHI conditions and the tool-skill task, participants provided a written response to a few questions about their age, sex, and a 5-point Likert question regarding their

previous experience using pliers. The Likert responses ranged from: 1—I never use pliers; 2—I very rarely use pliers (e.g., I've used them in the last year); 3—I occasionally use pliers (e.g., I've used them a few times in the last 6 months); 4—I frequently use pliers (e.g., I often use them in crafts or projects); 5—I use pliers regularly (e.g., I use them once a week or more).

### Results and Discussion Proprioceptive Drift

As predicted, participants did not experience a significant difference in proprioceptive drift during the synchronous stroking of their held pliers and the pliers held by the rubber hand as compared to the control conditions with asynchronous stroking and the wooden block. For this analysis, assumptions of normal distribution, independence of residuals, and sphericity were met. Using R (R Core Team, 2017) and the lmer() function in the lme4 library (Bates et al., 2014), a model was fitted to the data that predicted drift from the interaction of timing of visual-tactile stimulation (synchronous or asynchronous), viewed object (pliers rubber hand vs. wooden block), recency of tool-use (before or after the illusion phase) as betweensubjects fixed effects. The amount of beads transferred during the tool skill task was included as a covariate and a random effect of participant was used to account for the repeated measures nature of the design. The main effect of viewed object was significant: Wald Chi-Square (1) = 5.46, p = 0.019, with the pliers rubber hand (M = 0.87, SE = 0.24) yielding significantly higher proprioceptive drift than the wooden block (M = 0.14, SE = 0.23). Most RHI studies focus on the interaction between visual-tactile stimulation and the viewed object, and as **Figure 3** illustrates, this interaction was not significant (p = 0.33).

There was also a significant interaction of timing of visuotactile stimulation (synchronous vs. asynchronous) and the number of beads transferred during the tool-skill task: Wald Chi-Square (1) = 13.92, p < 0.001. This interaction is plotted in **Figure 4**. There were no other main effects or interactions that

stimulation and viewed object. The significant main effect of viewed object is apparent, as drift was larger when participants viewed a rubber hand holding pliers than when a wooden block was viewed. Error bars represent ±1 SEM.

reached significance. Clearly, this interaction was not predicted given the opposite findings with chopsticks in Weser et al. (2017); however, the pliers and chopsticks conditions differed in a number of respects. Unlike previous studies in which many participants reported that they used chopsticks daily, no participants in Experiment 1 reported frequently using pliers. Indeed, the majority of participants (n = 43) said they "very rarely" used pliers. Moreover, when grouping participants by their response to the pliers-use question, the 4 participants who said they "frequently use pliers" transferred the fewest beads during the took-skill task of any group (M = 103, SD = 9.2; Occasionally: n = 4, M = 161, SD = 11.7; Very Rarely: n = 43, M = 155, SD = 20.9; Never: n = 15, M = 115, SD = 26.5).

This suggests that the bead transfer task may not have been an ecologically valid assessment of tool skill, as it was for chopsticks. It also indicates that those performing better (e.g., transferring more beads) were not necessarily those participants with more skill and experience at using pliers. This may offer an explanation both as to why there was no effect of group (tool-skill task prior to the illusion vs. after the illusion) on the illusion, and importantly why the interaction of number of beads transferred and timing of visuo-tactile stimulation was the opposite direction as previously seen in the chopsticks study in Weser et al. (2017). In that experiment, the tool-skill task successfully quantified the toolusers' skill with chopsticks, as participants who reported more frequent chopstick use far outperformed those who reported never or infrequently using the tool. It is therefore a possibility that transferring beads with pliers was not so much a measure of skill with pliers, but rather of overall hand dexterity.

Though as of yet there is no definitive experiment that demonstrates a decrease in RHI strength for those with greater hand dexterity or awareness (dancers or pianists, for example), it has long been speculated that such individuals would have reduced susceptibility to the illusion (Botvinick and Cohen, 1998; Tsakiris and Haggard, 2005; Tsakiris, 2010). Indeed, those with lower interoceptive awareness (as measured with an established heart-rate monitoring task) were far more susceptible to the RHI than were those with high interoceptive abilities (Tsakiris et al., 2011; but see David et al., 2014). Therefore, the strong negative relationship seen in this study between the number of beads transferred with pliers and the amount of proprioceptive drift experienced during synchronous illusion conditions may actually index the decreased illusion susceptibility of more dexterous, bodily aware participants who are able to use an unfamiliar tool more easily than participants with less bodily awareness.

#### Rubber Hand Illusion Questionnaire

Following Longo et al. (2008), the mean ratings for the five components of the rubber hand illusion questionnaire (Embodiment, Loss of one's hand, Movement, Affect, and Deafference) were submitted to a mixed ANOVA with the 4 illusion conditions as within-subjects factors and a between subject factor of group (tool-skill task prior to the illusion vs. after). The analysis revealed a significant main effect of questionnaire component [F(4,62) = 80.47, p < 0.001] and of condition [F(4,88) = 5.92, p < 0.001]. No other main effects of interactions reached significance (all F's > 1.0). Follow-up analyses examining each questionnaire component individually revealed no significant differences between illusion conditions, suggesting that the subjective experience of the pliers-version of the rubber hand illusion was not greatly affected by the appearance of the viewed object or by the timing of the visuotactile stimulation. It seems likely that participants' lack of familiarity with pliers made it just as difficult for them to embody a rubber hand holding pliers as it would be to embody a wooden block. As a result, they failed to endorse questions about the rubber hand and the wooden block at equal rates. The nonsignificant interaction between illusion condition and component of the RHI Questionnaire is plotted in **Figure 5**.

### EXPERIMENT 2 TWEEZERS: MORPHO-FUNCTIONAL AND SENSORIMOTOR MATCH

fnhum-12-00537 January 7, 2019 Time: 19:45 # 7

Like the chopsticks and teacups used in Weser et al. (2017), tweezers and pliers similarly differ on their level of morphofunctional match and sensorimotor match: Chopsticks and tweezers match, while teacups and pliers do not. Weser et al. (2017) examined tools with an identical sensorimotor match (both tools were held with a precision grip) while the studies presented here examine tools with an identical morphofunctional match (both tools act on the environment in a precision fashion). If the match between tool morphology and grasp mechanics determines whether or not the use of a tool will cause a modulation of the wielder's body representation (e.g., Miller et al., 2014; Cardinali et al., 2016; Weser et al., 2017), then participants in Experiment 2 should experience a RHI when viewing a rubber hand holding tweezers.

## Methods

### Participants

Data was collected from 76 right-handed participants (24 males; mean age: 18.7; SD = 1.0). All participants had normal or corrected to normal vision, participated in exchange for credit in an introductory psychology course at the University of Virginia, and provided written informed consent prior to commencing the study. Data from 4 female participants was lost due to experimenter error (1) and the failure of 3 participants to follow directions. This brought the total sample size down to 72, with 37 completing the tool-skill task prior to engaging in the illusion, and the final 35 completing the tool-skill task after the completion of the illusion procedure.

### Materials

### Tweezers Rubber Hand

A life cast of author VW's hand holding a pair of tweezers was made from flesh-tinted plastic resin (see **Figure 6**). An identical pair of tweezers was provided for the participant to hold throughout the study and use during the tool-skill task. Together, the hand and tool measured approximately 9 cm × 22 cm × 6 cm, with the tips of the tweezers resting about 2 cm above the surface of the table.

### Wooden Block

The other viewed item was the wooden block (**Figure 1B**) described in Experiment 1.

### Tool-Skill Task

The bead-transfer task described previously was altered so that it would be more appropriate for tweezers. The beads were replaced with "seed beads," tiny plastic beads that measured 1.8 mm in diameter. As before, participants were required to use tweezers to pick up 1 bead at a time and move it from 1 container to another, sorting by color. There were 40 beads of each of 8 colors (320 beads total), and participants were allotted 5 min to sort as many beads as possible.

### Rubber Hand Illusion Questionnaire

The same 25 questions from Longo et al. (2008) were used. The questions were altered so as to reference the rubber hand holding tweezers, rather than the rubber hand alone or the rubber hand holding pliers.

FIGURE 6 | The life cast of a hand holding tweezers. The tweezers measured

### Tweezers Use Question

9 cm in length.

Given our hypothesis that tool experience (or lack the thereof) played a large role in the findings presented in Experiment 1 in which most participants reported infrequent pliers-use, we decided to closely examine tweezers use in Experiment 2. As in Experiment 1, we used a brief post-experiment questionnaire assessing tool familiarity. The Likert responses ranged from: 1—I never use tweezers; 2—I very rarely use tweezers (e.g., I've used them to remove a splinter here and there); 3—I occasionally use tweezers (e.g., I used them here and there for splinters, projects, or personal grooming); 4—I frequently use tweezers (e.g., I use them monthly for splinters, projects, or personal grooming); 5—I use tweezers regularly (e.g., I use them once a week or more).

### Experimental Design

A 2 × 2 × 2 × 2 mixed design was employed. The viewed object (tweezers rubber hand vs. wooden block) and timing of visuo-tactile stimulation (synchronous vs. asynchronous) were within-subjects factors, the group (tool-skill task prior to the illusion vs. following the illusion), and frequency of tweezers use were between-subjects factors. The number of beads transferred with tweezers was included as a covariate, and a random effect of participant was added to account for individual differences in tweezers-skill and illusion susceptibility. The four withinsubjects conditions, completed in a random order, were: (i) tweezers rubber hand synchronous (ii) tweezers rubber hand asynchronous; (iii) wooden block synchronous; (iv) wooden block asynchronous. The participant held tweezers during all four conditions of the illusion.

### Procedure

Upon arrival, participants were randomly assigned to either first complete the tool-skill task or to undergo the RHI procedure prior to using the tweezers to transfer seed beads. During the illusion-induction procedure, participants were instructed to apply light pressure to the tweezers handle and keep the prongs slightly closed. This allowed the experimenter to stroke both prongs of the tweezers simultaneously with a paint brush. All other aspects of the procedure were identical to Experiment 1.

## Results and Discussion

### Proprioceptive Drift

We found that participants who reported frequent tweezers use experienced proprioceptive drift when their tweezers were stroked in synchrony with the tweezers held by the rubber hand. Participants who did not report frequent tweezers use did not experience proprioceptive drift that differed significantly between the synchronous tweezers condition of interest and the control conditions. Assumptions of normal distribution, independence of residuals, and sphericity were met. A linear mixed-effects model that included parameters for viewed object (tweezers rubber hand vs. wooden block), visuo-tactile stimulation (synchronous vs. asynchronous), tool skill (number of beads transferred), and recentness of tool use (tool-skill before vs. after the illusion), was fitted to the data. The model also included a random effect of participant and a between-subjects factor of response to the tweezers user question (never vs. rarely vs. occasionally vs. frequently vs. regularly). The main effect of timing of visuo-tactile stimulation was significant: Wald Chi-Square (1) = 4.85, p = 0.028, with synchronous stimulation (M = 0.24, SE = 0.24) yielding significantly higher proprioceptive drift than asynchronous stimulation (M = −0.39, SE = 0.24). In addition to the main effect, the interaction between viewed object, timing of visuo-tactile stimulation and tweezers use response was significant: Wald Chi-Square (1) = 10.22, p = 0.037.

To ease in the interpretation of this interaction, we divided participants into two categories: frequent tweezers use (n = 35) vs. little or no tweezers use (n = 37). The choice to employ a median split in our analysis was made on the basis of our desire to compare the means of two groups as a more direct method of addressing the research question: whether or not frequent use of a tool facilitates proprioceptive drift in the RHI. We fit a new model to the data identical to the model described above except that the between-subjects effect of tweezers use was dichotomous (frequent use vs. little to no use). Again the main effect of timing of visuo-tactile stimulation was significant: Wald Chi-Square (1) = 4.65, p = 0.031. As before, the interaction between viewed object, timing of visuo-tactile stimulation and tweezers use/little or no use was significant: Wald Chi-Square (1) = 5.57, p = 0.018. The interaction is plotted in **Figure 7**.

Although not significant, the interaction between selfreported frequent use of tweezers and tweezers skill trended in the predicted direction: Wald Chi-Square (1) = 3.19, p = 0.074, with frequent tweezers users transferring more beads (M = 146, SE = 24) than non-frequent users (M = 130, SE = 24). Recentness of tool use was not significant and did not interact with any of the other parameters, and no other main effects or interactions were significant (all p's > 0.01). A follow-up analysis that included a factor for participant sex was conducted to ensure that the tweezers use was not sex-dependent. Indeed, sex did not significantly affect proprioceptive drift (p > 0.3), nor did it interact with any other factor in the model (all p's > 0.25).

#### Rubber Hand Illusion Questionnaire

fnhum-12-00537 January 7, 2019 Time: 19:45 # 9

Following Longo et al. (2008), mean ratings for the five components of the rubber hand illusion questionnaire (Embodiment, Loss of one's hand, Movement, Affect, and Deafference) were submitted to an ANOVA with the four illusion conditions (Tweezers Rubber Hand Synchronous, Tweezers Rubber Hand Asynchronous, Wooden Block Synchronous, and Wooden Block Asynchronous) as within-subjects factors. The ANOVA revealed a significant effect of questionnaire component [F(4,67) = 40.53, p < 0.001] and a trending effect of condition [F(4,68) = 2.45, p = 0.062]. The interaction was not significant. To follow-up this finding, an ANOVA examining differences in participants' endorsement of Embodiment-related questions in the four conditions was conducted. This ANOVA revealed a significant effect of condition: F(3,68) = 3.86, p = 0.010, with the synchronous tweezers condition resulting in slightly more positive endorsement of embodiment items (M = −0.26, SE = 0.21) than the other conditions (asynchronous tweezers: M = −0.61, SE = 0.22; synchronous wooden block: M = −0.92, SE = 0.26; asynchronous wooden block: M = −1.33, SE = 0.24). This finding is consistent with previous studies, which similarly find a small advantage for the synchronous condition in which the viewed object matches the object held by the participant. The interaction of component of the RHI Questionnaire and the illusion condition is plotted in **Figure 8**.

The significant interaction between viewed object, timing of visuo-tactile stimulation and tweezers-use status adds credence to the idea that morpho-functional match and sensorimotor match is an important component for the success of the illusion, and suggests that it is only the tools that match on these dimensions (chopsticks and tweezers) that integrate sufficiently with the body representation to affect an illusion of body ownership like the RHI. Although a median split was used in the analysis, the results suggest that the illusion only succeeds for individuals who report actual experience using the tweezers on a regular basis. Chopsticks are a relatively complicated tool to use, and so only those with chopsticks experience succeed at the tool skill task. On the other hand, tweezers are very simple to use and so even participants with very little real-world tweezers experience were able to transfer many beads. Therefore, the effects of the illusion emerge when participants' real world experience with tweezers are taken into account, rather than when examining their success at a somewhat arbitrary measure of tool-skill.

### CROSS-EXPERIMENT COMPARISON

To compare the amount of proprioceptive drift experienced by participants in the pliers and tweezers versions of the illusion, the self-reported responses to the tool-use question in Experiment 1 was similarly divided into two groups that resulted in a frequent pliers use group (n = 8) and a little or no pliers use group (n = 58). This allowed the data from the synchronous and asynchronous tool conditions from both experiments to be

combined. A linear mixed-effects model that included parameters for Experiment (pliers vs. tweezers), Condition (synchronous tool vs. asynchronous tool), and self-reported tool use (tool user vs. non-user) was fitted to the data. The model also included a random effect of participant. As expected, the main effect of condition was significant: Wald Chi-Square (3) = 6.89, p = 0.009, with synchronous tool (M = 0.86, SE = 3.00) yielding significantly higher proprioceptive drift than asynchronous tool (M = −0.02, SE = 2.98). There was also a significant effect of Experiment, with the pliers experiment yielding significantly higher proprioceptive drift (M = 0.87, SE = 2.72) than the Tweezers experiment (M = 0.01, SE = 2.23) across both conditions and levels of self-reported tool use: Wald Chi-Square (1) = 5.83, p = 0.016. Finally, the interaction between condition and self-reported tool use, plotted in **Figure 9**, was trending toward significance: Wald Chi-Square (1) = 3.80, p = 0.051. No other main effects or interactions reached significance (all p's > 0.3).

This finding suggests that tool experience is an important variable to consider in future investigations of the tool version of the RHI, as well as studies employing other paradigms to investigate the effects of tools on their wielders.

### CONCLUSION

The effects of tools and rubber hands on body representations have been reported in disparate literatures since both fields began to gain traction in the past 20 years. Similarly, for over 100 years, researchers have recognized a sharp divide between the body schema, a body representation for action, and the body image, a body representation for perception and identification (Head and Holmes, 1911; Anema et al., 2009). Many have argued that the embodiment of external limbs in the RHI is fundamentally

different from the type of embodiment experienced by tool users (De Preester and Tsakiris, 2009; De Vignemont and Farné, 2010). Although both skilled tool users and individuals who experience the RHI report that the tool or rubber hand feels like it is a part of their body, the effects of tool-incorporation and rubber-hand incorporation on subsequent behavior are markedly different.

Barring brain injury or the isolated study of a particular type of embodiment through illusion or tool training studies, the body schema and the body image must work in harmony for one to experience a coherent sense of control over and identification with one's physical form. Thus it seems likely that the two representations are not entirely separate. Weser et al. (2017) set out to examine the link between the embodiment of tools and rubber hands by adapting the classic RHI illusion to include a handheld tool. Skillfully using chopsticks prior to experiencing a RHI in which chopsticks receive the visuo-tactile stimulation increases the experience of the illusion, as measured behaviorally through proprioceptive drift (Weser et al., 2017).

Proprioceptive drift is the difference between a participant's estimate of the position of his or her own hand before and after the visuo-tactile stimulation of the real and rubber hands. Proprioceptive drift is believed to be a behavioral measure of the RHI that indexes the effect of visuo-tactile stimulation of non-corporeal objects on the body image. Since the RHI is performed with passive tactile stimulation and measured with introspective report and visual judgments of the location of one's hand, it is believed to be a purely perceptual (as opposed to motoric) illusion that only alters the body image, not the body schema (De Vignemont and Farné, 2010). In contrast, practice using a tool results in real time updates to one's capacity for action that is captured by changes to the body schema, as typically measured in changes in reach-to-grasp kinematics (e.g., Cardinali et al., 2012; Baccarini et al., 2014), changes in tactile acuity suggesting a perceived lengthening of the arm or widening of the hand, depending on the type of tool used (e.g., Cardinali et al., 2011; Canzoneri et al., 2013; Miller et al., 2017), and a tendency to overestimate the length of the arm following reach-extending tool use as measured in the forearm bisection task (e.g., Sposito et al., 2012; Romano et al., 2018). The findings of Weser et al.'s (2017) study using chopsticks and Experiment 2 presented here were novel because the motoric changes to the body schema following chopstick use or tweezer use manifested in perceptual drift in the RHI, a purely perceptual measure of the body image. Consistent with the tool-use literature, perceptual drift was larger for participants in Weser et al.'s (2017) study who had a chance to use chopsticks prior to experiencing the illusion. In the present work, the finding that only frequent tweezers users experienced the illusion adds nuance to the burgeoning literature and suggests future avenues of research should solicit expert tool users as participants. Although recent or frequent use of the tool facilitated the illusion, the lack of success of Weser et al.'s (2017) study using a teacup as the tool held by the rubber hand indicates the mere familiarity of the tool is not enough to facilitate its embodiment in this paradigm. This suggests that even if the participants in Experiment 1 had been familiar with pliers, they likely would not have experienced significant proprioceptive drift.

Most researchers report a strong correlation between subjective reports of the experience of the illusion (e.g., "it felt like the rubber hand was part of my body") and proprioceptive drift toward the rubber hand (e.g., Tsakiris and Haggard, 2005). In other words, participants estimate that their hand is closer to the rubber hand when they have a stronger feeling that the rubber hand is part of them. However, in both Weser et al. (2017) and the studies presented here, synchronous stimulation of a rubber hand holding a tool matching the participant's own held tool did not result in high self-reported rubber hand embodiment, even when synchronous visuo-tactile stimulation did cause high proprioceptive drift. So although tool-versions of the RHI do provide evidence for cross-talk between the body models for perception and action, the introspective aspect of this perceptual illusion seems to be less susceptible to modification from tools. The studies presented here were designed to investigate the difference in the behavioral outcome of the chopsticks and teacup version of the RHI conducted in Weser et al. (2017), but together they also contribute to the mounting evidence that proprioceptive drift and the introspective questionnaires used in the RHI literature do not necessarily measure the same phenomena, as they are not always strongly or even positively correlated (e.g., Holmes et al., 2006; Makin et al., 2008; Rohde et al., 2011).

The effect of morpho-functional and sensorimotor match on the proprioceptive drift outcome of the tool-versions of the RHI was the driving force behind the studies presented here. The morpho-functional component of a tool refers to its shape and the action it affords—the tool's output. Sensorimotor is the wielder's actions and grip while using the tool—the wielder's input. It has been speculated that the match or lack of match determines whether or not a modulation of the wielder's body representation occurs (Miller et al., 2014; Cardinali et al., 2016). The experiments presented here examined the difference between tools that both act on the environment in a precision manner, and therefore have the same morpho-functional output, but the tools are operated with either a power grip (pliers) or a precision grip (tweezers), and therefore differ in their sensorimotor input. Only the tool with a morpho-functional and sensorimotor match (tweezers: precision action, precision grip) resulted in a successful toolversion of the RHI, confirming that the same match found in chopsticks may play a deciding role in the illusion's success. That said, the tweezers version of the illusion only succeeded for participants who reported frequent tweezers use, suggesting that tool experience also effects whether or not a tool will alter one's body representation.

When given the opportunity to use chopsticks prior to experiencing the tool-version of the RHI, the changes to the body schema manifested in increased proprioceptive drift relative to the drift experienced by individuals who used the tool following the illusion. Moreover, only individuals who frequently used tweezers experienced a tweezers version of the RHI, suggesting that long term tool use facilitated body image modification during synchronous visuo-tactile stimulation of real and rubber hands holding tools. Taken together, this indicates that the body

image remains distinct from the body schema when it comes to introspective self-identification, but that taking action with tools can alter perceptual models of the body. The exploration of the mechanisms that contribute to and are responsible for tool-effects on body representations makes an important contribution to the literature: it is an investigation of the complex interplay between bottom-up effects such as simultaneous multisensory integration and tool experience with more top-down knowledge about body appearance, identity, position, tool function, appropriate grip, and tool expertise.

### ETHICS STATEMENT

This study was carried out in accordance with the recommendations for studies involving human subjects and the protocol was approved by the Institutional Review Board for Social and Behavioral Sciences at the University of Virginia. All participants gave written informed consent in accordance with the Declaration of Helsinki.

### REFERENCES


### DATA AVAILABILITY STATEMENT

The datasets generated for these experiments can be found in figshare data repository. Pliers data: 10.6084/m9.figshare.7177118 Tweezers data: 10.6084/m9.figshare.7177115.

### AUTHOR CONTRIBUTIONS

VW conceived of the presented idea. VW and DP designed the studies. VW created the materials and collected and analyzed the data under the supervision of DP. VW and DP discussed the results and contributed to the final manuscript.

### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnhum. 2018.00537/full#supplementary-material



handtool. Conscious. Cogn. 56, 150–164. doi: 10.1016/j.concog.2017. 07.002


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Weser and Proffitt. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Effectiveness of Robot-Assisted Upper Limb Training on Spasticity, Function and Muscle Activity in Chronic Stroke Patients Treated With Botulinum Toxin: A Randomized Single-Blinded Controlled Trial

Marialuisa Gandolfi1,2 \* † , Nicola Valè1,2†, Eleonora Kirilova Dimitrova1,2, Stefano Mazzoleni <sup>3</sup> , Elena Battini <sup>3</sup> , Mirko Filippetti 1,2, Alessandro Picelli 1,2, Andrea Santamato<sup>4</sup> , Michele Gravina<sup>4</sup> , Leopold Saltuari 5,6 and Nicola Smania1,2

#### Edited by:

Mariella Pazzaglia, Sapienza University of Rome, Italy

#### Reviewed by:

Giorgio Scivoletto, Fondazione Santa Lucia (IRCCS), Italy Geert Verheyden, KU Leuven, Belgium

#### \*Correspondence:

Marialuisa Gandolfi marialuisa.gandolfi@univr.it

†These authors have contributed equally to this work

#### Specialty section:

This article was submitted to Neurorehabilitation, a section of the journal Frontiers in Neurology

Received: 02 October 2018 Accepted: 14 January 2019 Published: 31 January 2019

#### Citation:

Gandolfi M, Valè N, Dimitrova EK, Mazzoleni S, Battini E, Filippetti M, Picelli A, Santamato A, Gravina M, Saltuari L and Smania N (2019) Effectiveness of Robot-Assisted Upper Limb Training on Spasticity, Function and Muscle Activity in Chronic Stroke Patients Treated With Botulinum Toxin: A Randomized Single-Blinded Controlled Trial. Front. Neurol. 10:41. doi: 10.3389/fneur.2019.00041 <sup>1</sup> Department of Neurosciences, Biomedicine and Movement Sciences, University of Verona, Verona, Italy, <sup>2</sup> UOC Neurorehabilitation, AOUI Verona, Verona, Italy, <sup>3</sup> Polo Sant' Anna Valdera, Scuola Superiore Sant' Anna, The BioRobotics Institute, Pontedera, Italy, <sup>4</sup> Physical Medicine and Rehabilitation Section, OORR Hospital, University of Foggia, Foggia, Italy, <sup>5</sup> Research Department for Neurorehabilitation South Tyrol, Bolzano, Italy, <sup>6</sup> Department of Neurology, Hochzirl Hospital, Zirl, Austria

Background: The combined use of Robot-assisted UL training and Botulinum toxin (BoNT) appear to be a promising therapeutic synergism to improve UL function in chronic stroke patients.

Objective: To evaluate the effects of Robot-assisted UL training on UL spasticity, function, muscle strength and the electromyographic UL muscles activity in chronic stroke patients treated with Botulinum toxin.

Methods: This single-blind, randomized, controlled trial involved 32 chronic stroke outpatients with UL spastic hemiparesis. The experimental group (n = 16) received robot-assisted UL training and BoNT treatment. The control group (n = 16) received conventional treatment combined with BoNT treatment. Training protocols lasted for 5 weeks (45 min/session, two sessions/week). Before and after rehabilitation, a blinded rater evaluated patients. The primary outcome was the Modified Ashworth Scale (MAS). Secondary outcomes were the Fugl-Meyer Assessment Scale (FMA) and the Medical Research Council Scale (MRC). The electromyographic activity of 5 UL muscles during the "hand-to-mouth" task was explored only in the experimental group and 14 healthy age-matched controls using a surface Electromyography (EMGs).

Results: No significant between-group differences on the MAS and FMA were measured. The experimental group reported significantly greater improvements on UL muscle strength (p = 0.004; Cohen's d = 0.49), shoulder abduction (p = 0.039; Cohen's d = 0.42), external rotation (p = 0.019; Cohen's d = 0.72), and elbow flexion (p = 0.043; Cohen's d = 1.15) than the control group. Preliminary observation of muscular activity showed a different enhancement of the biceps brachii activation after the robot-assisted training.

**20**

Conclusions: Robot-assisted training is as effective as conventional training on muscle tone reduction when combined with Botulinum toxin in chronic stroke patients with UL spasticity. However, only the robot-assisted UL training contributed to improving muscle strength. The single-group analysis and the qualitative inspection of sEMG data performed in the experimental group showed improvement in the agonist muscles activity during the hand-to-mouth task.

Clinical Trial Registration: www.ClinicalTrials.gov, identifier: NCT03590314

Keywords: upper limb, rehabilitation, robotics, electromyography, spasticity

### INTRODUCTION

Upper limb (UL) sensorimotor impairments are one of the major determinants of long-term disability in stroke survivors (1). Several disturbances are the manifestation of UL impairments after stroke (i.e., muscle weakness, changes in muscle tone, joint disturbances, impaired motor control). However, spasticity and weakness are the primary reason for rehabilitative intervention in the chronic stages (1–3). Historically, spasticity refers to a velocity-dependent increase in tonic stretch reflexes with exaggerated tendon jerks resulting from hyperexcitability of the stretch reflex (4) while weakness is the loss of the ability to generate the normal amount of force.

From 7 to 38% of post-stroke patients complain of UL spasticity in the first year (5). The pathophysiology of spasticity is complicated, and new knowledge has progressively challenged this definition. Processes involving central and peripheral mechanisms contribute to the spastic movement disorder resulting in abnormal regulation of tonic stretch reflex and increased muscle resistance of the passively stretched muscle and deficits in agonist and antagonist coactivation (6, 7). The resulting immobilization of the muscle at a fixed length for a prolonged time induces secondary biomechanical and viscoelastic properties changes in muscles and soft tissues, and pain (8–11). These peripheral mechanisms, in turn, leads to further stiffness, and viscoelastic muscle changes (2, 8). Whether the muscular properties changes may be adaptive and secondary to paresis are uncertain. However, the management of UL spasticity should combine treatment of both the neurogenic and peripheral components of spasticity (9, 10).

UL weakness after stroke is prevalent in both acute and chronic phases of recovery (3). It is a determinant of UL function in ADLs and other negative consequences such as bone mineral content (3), atrophy and altered muscle pattern of activation. Literature supports UL strengthening training effectiveness for all levels of impairment and in all stages of recovery (3). However, a small number of trials have been performed in chronic subgroup patients, and there is still controversy in including this procedure in UL rehabilitation (3).

Botulinum toxin (BoNT) injection in carefully selected muscles is a valuable treatment for spastic muscles in stroke patients improving deficits in agonist and antagonist coactivation, facilitating agonist recruitment and increasing active range of motion (6–8, 12–14). However, improvements in UL activity or performance is modest (13). With a view of improving UL function after stroke, moderate to highquality evidence support combining BoNT treatment with other rehabilitation procedures (1, 9, 15). Specifically, the integration of robotics in the UL rehabilitation holds promise for developing high-intensity, repetitive, task-specific, interactive treatment of upper limb (15). The combined use of these procedures to compensate for their limitations has been studied in only one pilot RCT reporting positive results in UL function (Fugl-Meyer UL Assessment scale) and muscular activation pattern (16). With the limits of the small sample, the results support the value of combining high-intensity UL training by robotics and BoNT treatment in patients with UL spastic paresis.

Clinical scales are currently used to assess the rehabilitation treatment effects, but these outcome measures may suffer from some drawbacks that can be overcome by instrumental assessment as subjectivity, limited sensitivity, and the lack of information on the underlying training effects on motor control (17). Instrumental assessment, such as surface electromyography (sEMG) during a functional task execution allows assessing abnormal activation of spastic muscles and deficits of voluntary movements in patients with stroke.

Moreover, the hand-to-mouth task is representative of Activities of Daily Life (ADL) such as eating and drinking. Kinematic analysis of the hand-to-mouth task has been widely used to assess UL functions in individuals affected by neurological diseases showing adequate to more than adequate test-retest reliability in healthy subjects (18, 19). The task involves flexing the elbow a slightly flexing the shoulder against gravity, and it is considered to be a paradigmatic functional task for the assessment of spasticity and strength deficits on the elbow muscles (17, 20). Although sEMG has been reported to be a useful assessment procedure to detect muscle activity improvement after rehabilitation, limited results have been reported (16, 21).

The primary aim of this study was to explore the therapeutic synergisms of combined robot-assisted upper limb training and BoNT treatment on upper limb spasticity. The secondary aim was to evaluate the treatment effects on UL function, muscle strength, and the electromyographic activity of UL muscles during a functional task.

The combined treatment would contribute to decrease UL spasticity and improve function through a combination of training effects between BoNT neurolysis and the robotic treatment. A reduction of muscle tone would parallel improvement in muscle strength ought to the high-intensity, repetitive and task-specific robotic training. Since spasticity is associated with abnormal activation of shortening muscles and deficits in voluntary movement of the UL, the sEMG assessment would target these impairments (2, 8–11, 15).

## MATERIALS AND METHODS

### Trial Design

A single-blind RCT with two parallel group is reported. The primary endpoint was the changes in UL spasticity while the secondary endpoints were changes in UL function, muscle strength and the electromyographic activity of UL muscles during a functional task. The study was conducted according to the tenets of the Declaration of Helsinki, the guidelines for Good Clinical Practice, and the Consolidated Standards of Reporting Trials (CONSORT), approved by the local Ethics Committee "Nucleo ricerca clinica–Research and Biostatistic Support Unit" (prog n.2366), and registered at clinical trial (NCT03590314).

### Patients

Chronic post-stroke patients with upper-limb spasticity referred to the Neurorehabilitation Unit (AOUI Verona) and the Physical Medicine and Rehabilitation Section, "OORR" Hospital (University of Foggia) were assessed for eligibility.

Inclusion criteria were: age > 18 years, diagnosis of ischemic or hemorrhagic first-ever stroke as documented by a computerized tomography scan or magnetic resonance imaging, at least 6 months since stroke, Modified Ashworth Scale (MAS) score (shoulder and elbow) ≤3 and ≥1+ (22), BoNT injection within the previous 12 weeks of at least one of muscles of the affected upper limb, Mini-Mental State Examination (MMSE) score ≥24 (23) and Trunk Control Test score = 100/100 (24).

Exclusion criteria were: any rehabilitation intervention in the 3 months before recruitment, bilateral cerebrovascular lesion, severe neuropsychologic impairment (global aphasia, severe attention deficit or neglect), joint orthopedic disorders.

All participants were informed regarding the experimental nature of the study. Informed consent was obtained from all subjects. The local ethics committee approved the study.

### Interventions

Each patient underwent a BoNT injection in the paretic limb. The dose of BoNT injected into the target muscle was based on the severity of spasticity in each case. Different commercial formulations of BoNT were used according to the pharmaceutical portfolio contracts of our Hospitals (Onabotulinumtoxin A, Abobotulinumtoxin A, and Incobotulinumtoxin A). The dose, volume and number of injection sites were set accordingly. A Logiq <sup>R</sup> Book XP portable ultrasound system (GE Healthcare; Chalfont St. Giles, UK) was used to inject BoNT into the target muscle.

Before the start of the study authors designed the experimental (EG) and the control group (CG) protocols. Two physiotherapists, one for each group, carried out the rehabilitation procedures. Patients of both groups received ten individual sessions (45 min/session, two sessions/week, five consecutive weeks). Treatments were performed in the rehabilitative gym of the G. B. Rossi University Hospital Neurological Rehabilitation Unit, or "OORR" Hospital.

### Robot-Assisted UL Training

The Robot-assisted UL Training group was treated using the electromechanical device Armotion (Reha Technology, Olten, Switzerland). It is an end-effector device that allows goal-directed arm movements in a bi-dimensional space with visual feedback. It offers different training modalities such as passive, active, passive-active, perturbative, and assistive modes. The robot can move, drive or oppose the patient's movement and allows creating a personalized treatment, varying parameters such as some repetitions, execution speed, resistance degree of motion. The exercises available from the software are supported by games that facilitate the functional use of the paretic arm (25). The robot is equipped with a control system called "impedance control" that modulates the robot movements for adapting to the motor behavior of the patient's upper limb. The joints involved in the exercises were the shoulder and the elbow, is the wrist fixed to the device.

The Robot-assisted UL Training consisted of passive mobilization and stretching exercises for affected UL (10 min) followed by robot-assisted exercises (35 min). Four types of exercises contained within the Armotion software and amount of repetitions were selected as follows: (i) "Collect the coins" (45–75 coins/10 min), (ii) "Drive the car" (15–25 laps/10 min), (iii) "Wash the dishes" (40–60 repetitions/10 min), and (iv) "Burst the balloons" (100–150 balloons/5 min) (**Figure 1**). All exercises were oriented to achieving several goals in various directions, emphasizing the elbow flexion-extension and reaching movement. The robot allows participants to execute the exercises through an "assisted as needed" control strategy. For increment the difficulty, we have varied the assisted and

FIGURE 1 | The upper limb robot-assisted training setting.

non-assisted modality, increasing the number of repetitions over the study period.

### Conventional Training

The conventional training consisted of UL passive mobilization and stretching (10 min) followed by UL exercises (35 min) that incorporated single or multi-joint movements for the scapula, shoulder, and elbow, performed in different positions (i.e., supine and standing position). The increase of difficulty and progression of intensity were obtained by increasing ROM, repetitions and performing movements against gravity or slight resistance (26). Training parameters were recorded on the patient's log.

### Outcomes

Clinical and demographic data were collected at enrollment. The primary outcome was the changes on the UL MAS score computed as the sum of evaluation of shoulder, elbow, and wrist muscles (single-joint score range, 0–4; with higher scores indicating worse spasticity; total score, 0–12; with higher scores indicating worse spasticity) (16).

Secondary outcomes included changes on the Fugl Meyer Upper Limb Assessment scale (FMA) (27) (score range, 0–66; with higher scores indicating better performance), a widely used measure of UL function composed by 33 items assessing reflex activity, muscle strength, and movement control. Moreover, changes on the UL muscle strength were assessed by the Medical Research Council Scale (MRC) computed as the sum of the score from shoulder flexion/abduction/external rotation, elbow flexion/extension and wrist supination/flexion/extension (singlemovement score range, 0–5; with higher scores indicating muscle strength against full resistance; total score, 0–40; with higher scores indicating muscle strength against full resistance) (28, 29).

Patients were assessed by a blinded rater before (T0) and posttreatment (T1). A subgroup of patients in the experimental group was investigated by surface Electromyography (EMGs) during the "hand-to-mouth" motor task (ARAT sub-item).

### EMG Protocol

The subject seated in a comfortable position on a chair with a backrest, the feet resting on the floor and the knees and hips flexed at 90◦ . The start position consisted of the hand of the examined side lying on the distal third of the thigh. Then, the patient was asked to touch his mouth with the palm at average speed and return to the starting position. The patient was instructed not to move the head toward the hand. No other indications regarding how to move the arm for not to influence the spontaneity of the movement. The EMG activity of 5 upper limb muscles of the affected side (deltoid scapular, deltoid clavicular, pectoralis major–clavicular head, triceps brachii, biceps brachii) was measured using pairs of self-adhesive surface electrodes. Disposable Ag-AgCl electrodes were placed according to SENIAM guidelines with an interelectrode spacing of 0.02 m. Before electrode placement, the skin was shaved with a disposable, single-use razor and cleaned with alcohol (30). Raw EMG signals were collected using BTS FREEEMG 300 wireless surface EMG sensors (BTS spa, Milan, Italy) at a sampling rate of 1,000 Hz. Raw EMG signals were processed with a customized routine developed in MATLAB environment (MathWorks, USA). The raw EMG signal was bandpass filtered at 20–450 Hz and then smoothed using a 20 ms root mean square (RMS) algorithm to obtain the envelope. Signals were recorded in three conditions: 30 s during resting position (basal), 5 s of maximal voluntary isometric contraction (MVIC), and during the hand-to-mouth task. The hand-tomouth task has been widely used to assess UL functions in individuals affected by neurological diseases showing adequate to more than adequate test-retest reliability in healthy subjects (18, 19).

The task was divided into two sub-phases through the definition of three time-events: (1) start of the movement, (2) the moment when the hand touches the mouth, and (3) return to the initial position. The first sub-phase, named "elbow flexion phase," was defined as the interval between the movement onset and the maximum elbow flexion. The second sub-phase, named "return phase," refers to the interval between the maximum elbow flexion until movement offset after returning to the starting position. Normative data were collected from 14 healthy agematched controls undergoing one EMGs acquisition. The timeevents were determined using an accelerometer (BTS spa, Milan, Italy).

### Sample Size

Sample calculation took into account that in a similar study by Pennati et al. a difference in the MAS total score of 2.75 was detected between the experimental and the control group (pooled SD 2.77) because of the experimental rehabilitation (16). According to this study, a sample of at least 28 patients (14 per group) was estimated to have 80% power and an alpha (probability of type I error) of 5%. Assuming a 10% drop-out rate, 31 patients were necessary to perform the study.

### Randomization

The patients considered eligible were randomly assigned to the experimental group or control group (allocation ratio 1:1) by using a computer-generated random numbers list with simple randomization (www.randomization.org). Group allocation and the randomization list was kept concealed.

### Blinding

The same blinded examiner measured primary and secondary outcomes at each evaluation session. Another blinded assessor performed the EMG protocol.

### Statistical Analysis

An intention to treat analysis was used. Descriptive statistics included means, standard deviation and graph. The Shapiro-Wilk test was used to test data distribution. Parametric or nonparametric tests were used for inferential statistics, accordingly. The T-Test for unpaired data (or the Mann-Whitney test) was used for testing between-group differences at T0 and T1. For this purpose, we computed the changes of the score (1) between T0–T1. The T-Test for paired data (or Wilcoxon signed rank tests) was used to compare within-group changes over time. The effects size measures between the two independent groups (Cohen's d calculation) were used to evaluate the magnitude of the between-group treatment effects. The Pearson correlation test was performed for testing the association between FMA and MRC scores. The EMG signals were processed by using an adaptive pre-whitening filter and the approximated generalized likelihood-ratio (AGLR) algorithm to detect the muscle activity. The onset and offset of muscle activity were analyzed as the percentage of the movement cycle. A qualitative analysis of sEMG graphic records was carried out. Training efficiency was calculated as the number of repetitions divided by the number of minutes of active therapy (31). The level of significance was set p < 0.05. Software statistics SPSS 20.0 (IBM SPSS Statistics for Windows, Version 20.0, Armonk, NY, USA).

### RESULTS

Forty nine patients were evaluated for eligibility between February 2017 and April 2018. Twelve patients were excluded because they did not meet the inclusion criteria and five declined to participate. Thus, 32 patients were randomly allocated to the EG (n = 16) or CG (n = 16). All patients complete the study (**Figure 2** and **Table 1**). A mean training efficiency of 7.3 repetitions/min and 2.2 repetitions/min was computed in the EG and CG, respectively. Matching in initial between-group conditions

#### TABLE 1 | Demographic and clinical characteristics of treated subjects.


SD, standard deviation; n.s, not significant.

Frontiers in Neurology | www.frontiersin.org

no significant differences as to age, disease duration, and all baseline clinical measures at T0 were measured (**Tables 1**, **2**).

### Primary Outcome Measure

Both groups reduced UL spasticity significantly without reporting significant between-group differences (**Table 2**).

### Secondary Outcome Measures

Both groups improved UL function significantly without significant between-group differences (**Table 2**). The experimental group reported significantly greater improvements on UL muscle strength (delta T1–T0 = 3.62, p = 0.004, Cohen's d = −0.49), shoulder abduction (delta T1–T0 = 0.62, p = 0.039, Cohen's d = −0.42), external rotation (delta T1–T0 = 0.53, p = 0.019, Cohen's d = −0.72) and elbow flexion (delta T1–T0=0.59, p = 0.043, Cohen's d = −1.15) than the control group (**Table 2**). Changes in UL muscle strength were significantly correlated with changes in FMA (r = 0.49, p = 0.05). The results of sEMG analysis are reported in **Figures 3**–**5**. The mean envelopes of sEMG signals of the healthy subjects and patients of the EG are shown as a function of the movement progression.

Based on the results, the mean sEMG envelope computed on the healthy subjects' recordings (**Figure 3**) shows a low but constant recruitment of scapular deltoid and triceps brachii during the entire movement. On the other hand, the activation of the biceps brachii, clavicular deltoid, and pectoralis major is higher during the first phase and decreases during the second phase. It corresponds to the typical muscular recruitment occurring during the execution of the required reaching movement (32).

The same mean sEMG enveloped computed on stroke patients at T0 (**Figure 4**, left) showed a similar trend, especially in the first phase: during the second phase the decrease of the biceps brachii, clavicular deltoid and pectoralis major is higher than that observed in the healthy subjects.

At T1 (**Figure 4**, right) the time of sustained activation of the biceps brachii is much longer if compared to the activation observed at T0: in fact, the decrease which starts at about 70% of the movement corresponds to a double value if compared to T0 (∼0.06 vs. 0.03 mV). In addition, the value observed in the healthy subjects at the transition of the two phases is ∼0.035 mV, the corresponding value observed in the subjects at T1 is ∼0.7 mV: this higher mean sEMG activation value corresponds to a different abnormal recruitment pattern of this muscle in the recruited patients.

As regards the analysis of the mean sEMG envelope of single subjects, patient #1 at T0 (**Figure 5** top left) shows a poor modulation of recorded muscles which is more regular at T1 (**Figure 5** bottom left). The activation of patient #2 at T0 is almost absent (**Figure 5** top right), while at T1 (**Figure 5** bottom right) the modulation of muscles is clear. In both cases the plots at T1 highlight a change of the muscular recruitment probably due to the proposed upper limb training program.

## DISCUSSION

The main finding of this work is that robot-assisted training is as effective as conventional training on muscle tone reduction when combined with Botulinum toxin in chronic stroke patients with UL spasticity. However, only the robot-assisted UL training contributed to improving muscle strength with a moderate positive correlation with UL function. Preliminary observation of muscular activity in the experimental group showed, in some subjects, enhancement of the agonist muscles activity during the hand-to-mouth task. It might suggest a task-specific effect of the robot-assisted approach on muscle activity during the functional task. However, the sEMG protocol was not focused on investigating the relationship between abnormal activation of agonists and antagonists' muscles as it was limited to a qualitative analysis of sEMG envelope.

The combined use of BoNT injection and robot-assisted UL training appears to be a promising combination to target the different sensorimotor impairments because spasticity is associated with abnormal activation of shortening muscles and deficits in the UL voluntary movements (2, 8–11, 15). On the one hand, BoNT appears to be effective in the reduction of the neural components of the spastic movement disorders facilitating agonist recruitment and decreasing co-contraction of the antagonist's muscles. On the other hand, robotic devices can reduce muscle tone and motor impairment (1).

Given the multiplicity of symptoms that often need to be addressed in the UL rehabilitation of stroke patients, the integration of robotics holds promise for developing highintensity, repetitive, task-specific, interactive treatment (15). Despite significant heterogeneity in the robotic system design (exoskeletons and end-effectors) and clinical research paradigms used, a consensus exists on the safety and value of robotassisted UL therapy in reducing motor impairment, mainly at the shoulder and elbow (15, 33). The effects of robot-assisted therapy on muscle tone remains uncertain as only two reviews have specifically addressed this topic (34).

A recent systematic meta-analysis in 38 trials evaluated the effects of robot-assisted UL training in patients after a stroke on outcomes of motor control of the paretic upper limb, upper limb capacity, and basic ADL, in comparison with non-robotic treatment. Secondary outcomes were muscle strength and muscle tone. No serious adverse events were reported. Results reported significant improvements in UL motor control (about 2 points FMA UL sub-items) and muscle strength after robot-assisted training. Shoulder/elbow robotics showed small but significant effects on both motor control and muscle strength, while elbow/wrist robotics had small but significant effects only on motor control. Uncertain effects were reported for muscle tone assessed by the Modified Ashworth Scale (34). No effects were found for upper limb capacity and basic ADL. In the review by Bertani and colleagues 14 randomized controlled trials, two systematic reviews, and one meta-analysis were included (35). The Fugl-Meyer and Modified Ashworth scale were selected to measure primary outcomes, a measure of motor function and muscle tone, respectively. Functional independence measure and motor activity log were selected to measure secondary

#### TABLE 2 | Clinical outcome measures and inferential statistics.


<sup>γ</sup>non-parametric statistics.

P≤ 0.05.

FMA, Fugl Meyer Assessment; MAS, Modified Ashworth Scale; MRC, Medical Research Council; ES. Effect size; EG, Experimental Group; CG, Control Group; CI, Confidence Interval; SD, Standard Deviation; Q1, 25◦ interquartile; Q3, 75◦ interquartile; LB, Lower Bound; UB, Upper Bound; n.s., not significant.

outcomes, such as activities of daily living. According to previous findings, the robot-assisted UL rehabilitation was more effective in improving upper limb motor function recovery, especially in chronic stroke patients than conventional therapy. No significant improvements were observed in the reduction of muscle tone or daily living activities (35).

Few studies have explored the combined effects of pharmacological treatments of UL spasticity and robot-assisted rehabilitation, so far (16, 36, 37). These preliminary results agree that greater improvements from the combined approach are expected in UL function, as assessed by the FMA. However, the training effects on spasticity and UL use in the ADL have found disagreement between the different studies (16, 36, 37).

With the limits of methodological differences among studies, the present study corroborates some elements of the existing literature (16). Some equivalence between the robot-assisted and

conventional approaches was noticed in UL function, while no differences in the MAS were reported. Interestingly, the UL robot-assisted training increased UL muscle strength in specific muscles involved during functional tasks, outcome not included in earlier studies (16, 37). Historically, strengthening training has been a subject of controversy in stroke rehabilitation. However, this procedure is now included in post-stroke rehabilitation programs given the absence of adverse effects on spasticity and the positive consequences reported on strength and activity (3, 38). Strength training is commonly considered to be progressive resistance exercise. However, any intervention that involves attempted repetitive effortful muscle contractions can result in increased motor unit activity and increase strength (37). Besides, exercising entailing more numerous repetitions but with a reduced workload are recommended in post-stroke patients (31). There is growing evidence suggesting the crucial role of the treatment dose in functional recovery. However, a lack of consensus on the quantification of training intensity

FIGURE 5 | Lesions displayed on the magnetic resonance imaging brain template and electromyographic activity of the five upper limb muscles during the "hand-to-mouth" task was explored using a surface Electromyography (EMGs) in two patients. (A) Top left: Lesion Mapping (patient 1). Magnetic resonance imaging showed a lesion involving the left fronto-parietal lobes. Brodmann areas 48, 40, 39, 6, 44, 45, 3, 7, 22, 41, 2, 42, 4, 19, 43, 37, 21,47,9, 46, 1, 18, 23, 10 (MRIcron software, http://www.mricro.com/mricron). The electromyographic activity of the 5 UL muscles during the "hand-to-mouth" task was explored using a surface Electromyography (EMGs). (B) Top right: Lesion Mapping (patient 2). Magnetic resonance imaging showed a lesion involving the left frontal lobe. Brodmann areas 48, 45, 44, 6, 46, 43, 4, 3, 47, 32, 9, 38, 22, 10 (MRIcron software, http://www.mricro.com/mricron). The electromyographic activity of the 5 UL muscles during the "hand-to-mouth" task was explored using a surface Electromyography (EMGs).

and its relationship with UL recovery patterns exists. Within this perspective, the ratio between the number of repetitions divided by the total time dedicated to the training has been reported to be a useful parameter to quantify training intensity and efficiency (31). These data could be relevant to evaluate the cost-effectiveness of technology-mediated rehabilitation as robot-assisted training. Robotics, in fact, allows implementing high-standardized training in term of numbers of repetition and progression of intensity and workload over time (15). The knowledge of the underlying neuromuscular mechanisms is of great interest in neurorehabilitation. With the limitations of the single-group analysis and the qualitative inspection of data, sEMG can reveal myographic activity reflecting the physiological processes that occur following cortical damage and changes promoted by rehabilitation training (39). The surface EMG assessment in this perspective allowed to observe specific impairments in proximal agonist muscle activity in the hand-tomouth task, as reported in **Figure 4** and as an example in two patients in **Figure 5**. Before treatment, the decrease of the biceps brachii, clavicular deltoid, and pectoralis major was higher than that observed in the healthy subjects suggesting an impairment in the eccentric contraction of elbow flexors and modulation of the internal shoulder rotator during the return (second) phase of the task. Five distinctive UL spasticity patterns has been identified for the position of the shoulder, elbow, forearm, and wrist joints (39). The most frequent (41.8%) is characterized by internal rotation and adduction of the shoulder and flexion at the elbow coupled with a neutral positioning of the forearm and wrist (40). The significance to recognize these patterns is essential to guide specific pharmacological and rehabilitative interventions (7, 39). The instrumental assessment of these patterns during UL movement using the sEMG analysis can reveal dynamic pattern of altered activation during specific task phases to further customize rehabilitation procedures. The results suggested that Armotion could act selectively on these proximal muscles. At the end of the treatment, the mean time of sustained activation of the biceps brachii was much longer if compared to the activation observed pre-treatment. The analysis of the mean sEMG envelope of single subjects is an example of how the robot-assisted training might improve the modulation of elbow flexors muscles (**Figure 4**, top left) and increase their activation (**Figure 4** top right) during active task. Similarly, Pennati et al. performed a qualitative analysis of sEMG graphic records during two gestures: reaching the wall and return to the mouth and reaching movement to a visual target (16). Their results showed an improvement in muscle activation pattern and a reduction of co-contraction of agonist-antagonist muscles after the robotic exercises.

The strengths of the present study are the relatively large patient sample and the low drop-out rate, which suggest the feasibility of robotic training. The EMG analysis using a standardized experimental protocol to investigate the training effects on muscle activity are further strengths of this study, albeit only in a subgroup of patients (experimental group). The study limitations are the lack of a functional assessment after the treatment, measurement on activities of daily living and participation and patient related outcome. In addition, any patient stratification by the degree of impairment, the lack of follow-up assessment and neuroimaging support were reported.

This preliminary study has implications for practice and research. These findings suggest that UL robot-assisted strengthening interventions combined with pharmacological neurolysis increase strength and do not increase spasticity. Thus, strengthening programs should be part of rehabilitation after stroke to improve function and activity. Robot-assisted rehabilitation offers a wide range of training modalities that

### REFERENCES


can be chosen for an individualized treatment in terms of assistance (passive, active-assisted, active) and perturbation. Research would be oriented toward ideal training models (i.e., number of repetitions, progression, duration) and how long the training effects last after the intervention. Surface EMG should be part of this multidisciplinary intervention to characterize specific pattern and to focus the training exercises to specific muscle impairment. It could help researchers to design studies with accurate patients' selection and stratification with specific impairments and similar likelihood of responding to rehabilitation. It is essential to draw up recommendations for a therapeutic guide of UL spasticity management in chronic stroke patients with UL spasticity.

To conclude, upper limb robot-assisted training holds promise when combined with botulinum toxin in improving upper limb strength and muscular activation pattern in patients with chronic stroke and spasticity.

### AUTHOR CONTRIBUTIONS

MaG and NV have made substantial contributions to conception and design. MF and AP participated in the enrollment phase and carried out BoNT treatment procedures. ED and MiG carried out the clinical assessment. NV carried out the EMG assessments. SM and EB designed the algorithm for EMG data analysis. NV and ED participated in the manuscript draft and revision process. MaG and NV participated in the study design and coordination, and statistical analysis. MaG drafted the manuscripts and revision process. NS, LS, and AS participated in the manuscript revision and gave the final approval of the version.

### ACKNOWLEDGMENTS

Veronica Urbani and Niccolò Ramponi for participating in the rehabilitation procedures, and Elisa Battistuzzi for participating in the assessment procedures.

with post-stroke spasticity. Clin Neurophysiol. (2018) 129:2065–74. doi: 10.1016/j.clinph.2018.07.007


affected arm after stroke: a single-blinded randomized trial in two centers. Stroke (2005) 36:1960–6. doi: 10.1161/01.STR.0000177865.37334.ce


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Gandolfi, Valè, Dimitrova, Mazzoleni, Battini, Filippetti, Picelli, Santamato, Gravina, Saltuari and Smania. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# The Overlooked Outcome Measure for Spinal Cord Injury: Use of Assistive Devices

Giorgio Scivoletto1,2 \*, Giulia Galli <sup>1</sup> , Monica Torre1,2, Marco Molinari <sup>1</sup> and Mariella Pazzaglia1,3 \*

*<sup>1</sup> Spinal Cord Unit, IRCCS Fondazione Santa Lucia, Rome, Italy, <sup>2</sup> Spinal Rehabilitation Lab, IRCCS Fondazione Santa Lucia, Rome, Italy, <sup>3</sup> Department of Psychology, La Sapienza University of Rome, Rome, Italy*

Although several outcome measures are used to assess various areas of interest regarding spinal cord injuries (SCIs), little is known about the frequency of their use, and the ways in which they transform shared knowledge into implemented practices. Herein, 800 professionals from the International Spinal Cord Society, especially trained for caring in patients with SCI, were invited to respond to an Internet survey collecting information on the use of standardized measures in daily clinical practices. We asked both clinicians and researchers with different areas of interest about their use of functional outcome measures, and, in particular, which scales they habitually use to assess various aspects of clinical practice and rehabilitation. We selected a set of rating scales, which were validated for measuring SCIs (http://www.scireproject.com/outcome-measures). The results show that the areas of interest assessed by most of the participants were neurological status, upper limb, lower limb gait, pain, spasticity, self-care, and daily living. The most widely used rating scales were the spinal cord independence measure, the functional independence measure and the International Standards for Neurological Classification of Spinal Cord Injury. Instead, the majority of respondents did not evaluate the use of assistive technology. Despite the availability of several outcome scales, the practice of evaluating SCIs with standardized measures for assistive technologies and wheelchair mobility is still not widespread, even though it is a high priority in the rehabilitation of SCI patients. The results emphasize the need for a more thorough knowledge and use of outcome scales, thus improving the quality of assistive device evaluation.

Keywords: spinal cord injury, rehabilitative tools, survey, outcome measures, embodiment

### INTRODUCTION

Traumatic spinal cord injury (SCI) is a condition that affects nearly one out of every 1,000 people each year (0.721–0.906 per thousand people in the United States) and represents one of the leading causes of disability worldwide (1). Young adults are at the highest risk (2, 3). As all areas of the disfunctions of patients with SCI are capable of rehabilitation, therapies must be coordinated comprehensively and effectively to treat the medical, physiologic and psychological consequences of the injury. As such, SCI rehabilitation is complex and resource demanding with a costs that may vary from \$53,000–88,000 USD (including the first admission and readmissions within the first

#### Edited by:

*Giovanni Abbruzzese, University of Genoa, Italy*

#### Reviewed by:

*Domenico Antonio Restivo, Ospedale Garibaldi, Italy Marco Paoloni, Sapienza University of Rome, Italy*

\*Correspondence:

*Giorgio Scivoletto g.scivoletto@hsantalucia.it Mariella Pazzaglia mariella.pazzaglia@uniroma1.it*

#### Specialty section:

*This article was submitted to Neurorehabilitation, a section of the journal Frontiers in Neurology*

Received: *07 December 2018* Accepted: *28 February 2019* Published: *22 March 2019*

#### Citation:

*Scivoletto G, Galli G, Torre M, Molinari M and Pazzaglia M (2019) The Overlooked Outcome Measure for Spinal Cord Injury: Use of Assistive Devices. Front. Neurol. 10:272. doi: 10.3389/fneur.2019.00272*

**31**

two years after the lesion) or more, depending on the country and the severity of the lesion (4, 5). Given these high costs, it is necessary to monitor the efficacy and efficiency of rehabilitation with appropriate functional measure outcomes.

Furthermore, in the last two decades, research in the field of SCI included more than 900 clinical trials (https://clinicaltrials. gov). It is clear that the results of these trials need to be assessed with appropriate and validated instruments.

Accordingly, several studies (6, 7) recommended the routine collection of standardized outcome measures that represent a common language, allowing for the evaluation of the impact of rehabilitation, the outcomes of different populations and different intervention protocols. However, an agreement has yet to be made over which outcome measures should be used. It is evident that, due to the complexity of SCI rehabilitation, one single outcome measure is not sufficient to measure all the benefits of intervention, meaning a pool of instruments is needed. These outcome measures should be, if not specifically designed for SCI subjects, at least validated for this population, demonstrating good psychometric characteristics and ease of use and scoring. Moreover, although several outcome measures are used to assess various areas of interest to SCI patients, little is known about the frequency of their use.

Thus, this study had two main aims. First, we sought to evaluate how frequently validated assessment scales are used in SCI rehabilitation. We used the outcome measures suggested in the Spinal Cord Injury Rehabilitation Evidence (SCIRE) project available at http://www.scireproject.com/outcome-measures to assess and synthesize the frequency of use of outcome measures for rehabilitation interventions in SCI. SCIRE provides the measures of best practices applicable for reliability, sensitivity and validity to rehabilitation in individual with SCI, evaluating 1,600 published studies.

Second, we investigated the medical community's interest in the use and evaluation of alternative measures that report on best practices available for SCI rehabilitation. This raises an important question about how specialists measure outcomes and whether assessments should focus on outcomes that are relevant to the majority of patients as well as individual patients. Currently, studies have not been conducted on the instruments that experts in different centers have found to be the most useful as outcome indicators in the transitional care of SCI patients. Many centers select and combine several clinical outcome measures depending on their aims, but only some are related to SCIs. This reduces the efficiency of routine care, leads to inconsistent outcome measurements, and complicates the comparison of medical data.

A shared knowledge of the clinical outcome measures and implemented practices would not only improve interdisciplinary communication and support clinical education, it would also facilitate the planning and implementation of treatment for SCI patients.

### METHODS

### Participants

All professionals in the ISCoS registry were sent an invitation to participate in the survey. Of the total 800 invitations sent, 70 were rejected (emails returned). A total of 110 surveys were returned on the first request and 33 after a reminder, giving a total of 143 and a response rate of 18%. Nine had complete demographic information but did not answer the questions and were therefore not considered for the present analyses. Within the overall sample of 134 respondents (49 ± 10.9 years), 51% of the participants were female (47 ± 8.5 years), 49% were male (51.5 ± 12.4 years), and they were comparable in age (p > 1). Participants came from all fields of rehabilitation expertise, with physicians representing the majority (81), as well as 29 physical therapists, 7 occupational therapists, 6 nurses, and 4 psychologists. For the remaining seven respondents, the occupation was unknown. The region returning the greatest number of surveys was Europe (65 respondents), 30 from America, 18 from Australia and New Zealand, 15 from Asia, and 7 from Africa. Almost two thirds (64%) of respondents were clinicians and researchers, 11% only researchers, and 25% only clinicians.

The study was approved by the ethics committee of Santa Lucia Foundation and was performed in accordance with the Declaration of Helsinki. Informed consent was obtained from participants.

### Survey Description

To select the outcome measures for these surveys, we used the Outcome Measures section of the SCIRE Project (http://www. scireproject.com/outcome-measures). This section presents 107 measures examined for psychometric properties (i.e., reliability, validity, and responsiveness) and recommended for use in treating SCI. We then prepared a survey using SurveyMonkey <sup>R</sup> , asking the participants which outcome measures they use, within different areas of interest, including assistive technology, community reintegration, lower limb gait, mental state, neurological status, other affected systems, quality of life, pain, sexual function, spasticity, secondary conditions, self-care, daily living, skin status, upper limb, and wheelchair mobility. For each area, we listed all the outcome measures identified by the SCIRE, allowing participants to indicate more than one measure if needed. We added two possible answers: "I do not assess this area" and "I do assess this area with other measures." When choosing the last answer, a box opened where the respondent could specify which measure they used for that area. We also asked for some demographic information, including the age, gender, geographical area, and type of work of the respondent as well as the center in which the respondent worked.

### Analysis

The survey responses were analyzed using descriptive statistics, such as frequencies and proportions. To better characterize the rehabilitation outcome measures identified by SCIRE, the distribution of different responses was subjected to hierarchical cluster analysis in which the responses were sorted according to their adhesion to the following responses: (i) "I do not assess this area," (ii) "I assess with measures identified by SCIRE," and (iii) "I assess with other measures." Using cluster analysis, similar responses were grouped into homogeneous subsets. In this case, the analysis identified the measures with minimal within-response variation and maximal betweenresponse variation. Hierarchical cluster analysis does not require pre-specify selection of the number of clusters, and the dendrogram provides a simple and comprehensive image of the number of clusters. The analysis of the contingency table using the chi square test was employed to compare the frequency of each response.

### RESULTS

### Cluster Analysis

A hierarchical cluster analysis of different responses was conducted to identify the varying degrees of (dis)similarity in the responses using a (dis)similarity matrix. The distribution of the different measures is shown in **Figure 1**.

According to the described sorting process, we identified three clusters that had the greatest difference between the responses, and we compared them to the outcome measures identified by SCIRE: one cluster with three measures (nos. 7– 9) had a high consensus (mean = superior of 80%), another cluster with six measures (nos. 1–6) had a moderate consensus (mean = 58%), and one cluster with three measures had a low consensus (nos. 10–12). **Figure 2** shows the means of the participants' responses for the three different clusters. While Pain and Assistive Technology were present, these were clustered as separate profiles. Pain presented as being separate from the other clusters for a moderate consensus (mean = 46%) of the respondents in their use of the SCIRE measures, but also for a moderate consensus (mean = 35%) in the use of alternative measures. This pattern is not present in the other clusters. Remarkably, Assistive Technology clustered as a separate profile mainly because the respondents did not assess this measure (mean = 85%). The analysis of the contingency table using the chi-square test indicated a significant difference among the three responses in the five clusters that were identified (χ <sup>2</sup> = 175, p < 0.0001).

### Objective Outcome Measures Used in Patients With SCI

We determined the frequency of use of each measure indicated in set of outcome measurement of SCIRE Toolkit. The most commonly used objective outcome measures were: the International Standards for Neuorlogical Classification of Spinal Cord Injury designed to assess the Neurological Impairment, and the MAS developed to evaluate Spasticity in more of 80% of the responders, the SCIM and FIM in clinical area of Self Care and Daily Living and 6MWT, 10 MWT, WISCI for clinical area lower limb and walking in more of 50% of participants. **Table 1** presents a list of the measures that the respondents most commonly selected as their first and second choice for each clinical area using the SCIRE toolkit.

### The Use of Measures Outcome in Europe and America

We also assessed the differences in outcome measures identified by SCIRE when considering regions with universal healthcare (Canada and Europe) vs. fee-for-service healthcare (USA). An analysis of the contingency table using chi square test indicated a significant difference among the three responses in four measures identified:

For Community (χ <sup>2</sup> = 4.21, p < 0.04), Self-care (χ <sup>2</sup> = 3.75, p = 0.05), and Quality of life (χ <sup>2</sup> = 4.48, p < 0.03), the difference is explained given that compared to USA, Europe has less centers that assess the outcomes in these measures.

For Pain, meanwhile, American centers had a high consensus of using the SCIRE outcome measures. The data also showed that

reported along the *y*-axis. Measures with the most similar performance are closer to each other. Three main clusters are apparent with relatively homogeneous measures. Pain and Assistive Technology clustered separately and seem to have a different profile.

Europe centers had a substantial proportion of the respondents answer that they use "other measures." In other measures, respondents indicated the Visual Analog Scale or the Numeric Rating Scale, quick, and easy measures to use to assess the pain.

No differences in Assistive Technology were present between the regions with and without fee-for-service healthcare because the outcome measure was not evaluated in rehabilitation centers.

### DISCUSSION

The data suggested that, for motor and sensory impairment, the consensus among professionals is to assess SCI by using the outcome measures identified by the SCIRE. Although better standardized outcome measures have been selected by the SCIRE in the treatment of SCI patients, a number of the identified areas are under-evaluated with regards to some recommended measures, while other areas are not evaluated at all in rehabilitation centers. The survey's most striking result is that the use of assistive devices is not currently studied by rehabilitation specialists as an outcome measure.

### Less Widely Used Rating Scales for Spinal Cord Injury

The majority of respondents did not evaluate the use of Assistive Technology as an outcome measure identified by the SCIRE. We investigated whether the respondents chose another measure specifically to do with the use of assistive devices, different from those selected by the SCIRE. Interestingly, no other measures were used to study the use of assistive devices, suggesting that this outcome is of little interest to medical professionals and researchers. The reasons for this could include this outcome measure's failure to adequately reflect recovery after SCI, the lack of valid instruments to measure this outcome, the requirement to have a primary outcome area to support selfcare in SCI treatment, and the short time period allotted to study this outcome measure and to care for patients. Unlike other outcome measures, the decision to use a functional measure for the use of assistive devices seems mainly to be at the discretion of the treating medical professional or researcher, instead of rehabilitation specialists. This could represent a bias for several reasons.

First, providing mobility equipment that meets the individual needs of an SCI patient encourages that patient to be independent and to participate in society while reducing that patient's behavioral challenges and reliance on assistance (8, 9). Conversely, inappropriate mobility equipment can restrict an individual's independence and opportunities for a social life (10–12). Therefore, assigning assistive devices to SCI patients is challenging. Selecting an appropriate prosthetic device, training for its initial use, and evaluating for a possibly more suitable devices and overall outcomes are essential aspects of rehabilitation.

Furthermore, the strong connection between tool and body perception, often termed embodiment (8, 9, 11, 13–17), may be one of the most crucial factors affecting functional recovery (18– 22). Recent studies have demonstrated that establishing a sense of embodiment for prostheses in patients with limb amputation is associated with enhanced competence and patients' ease of use of such devices. Conversely, a low level of embodiment impedes the efficient use of assistive tools and contributes to their rejection (11, 23). Therefore, surveys to measure technology acceptance should always be conducted on SCI patients (17).

TABLE 1 | Rank order of preferred outcome measure by clinical area of use.


*(Continued)*

#### TABLE 1 | Continued


Third, in recent years, the focus on assistive devices that use robotic technologies to aid in recovery and rehabilitative treatment has increased (24–26). Adequate provisions, for example walking with exoskeleton, can reduce clinical complications resulting from life in a wheelchair, decrease the intensity of pain and spasticity, increase bone density, and improve well-ness and the overall quality of life (27). While substantial advancements have been made in terms of the portability and safety of assistive devices, little attention has been devoted to the outcome measures that must be studied for their usage. Despite great progress from a technological standpoint, as well as SCI patients facing medical and societal pressures to move in wheelchairs or use other assistive tools, only a small number of centers assess outcome measures related to assistive devices.

Besides Assistive Technology, the results of the present survey show that a number of areas identified by the SCIRE are evaluated less than others (Other Affected Systems, Sexuality, and Reproduction). It is still possible that being physicians the majority of respondents, they might not have the same comprehensive knowledge for all the areas surveyed.

Possible other reasons for not collecting these outcome measures may include: resource constraints and the lack of a consensus among professionals regarding which outcome measures should be used (28). Furthermore, some instruments are never or are seldom used due to inadequate measures. A different possible explanation is that the extent of the use of one measure could be considered an indicator of its usefulness (29). Although this assumption has yet to be proven, the little use of some measures might suggest that clinicians do not rely on these measures to assess the outcome of an intervention (30).

### Most Widely Used Outcome Measures for Spinal Cord Injury

At present, no comprehensive survey has been conducted within the field of SCI other than the one conducted in the current study. The only possible comparisons are a Canadian survey on amputees (30) and a United Kingdom (28) survey on rehabilitation centers. According to these two surveys, the selfcare measure is the most frequently assessed (75–80% of the respondents in the previous studies, 87% in the present one). The functional independence measure (FIM) and the spinal cord independence measure (SCIM) were the most frequently used measures for self-care and daily living, with the SCIM being the most popular at 58% and the FIM close behind at 50%. This likely reflects the different origins of the respondents, with most respondents being European. This could also be due to the increasing success of the SCIM in the field of SCI. In fact, not only did the SCIM prove to be more sensitive to the changes of SCI patients than the FIM (31), it is also recognized as the first choice outcome measure (32, 33). Although the FIM requires significant time for training and data collection, it remains a widely used measure in some countries. However, it is possible that this popularity is partly because the FIM must be used in some countries due to administrative requirements.

The International Standards for Neurological Classification of Spinal Cord Injury (ISNCSCI) published by the American Spinal Injury Association is a well-established international outcome measure utilized by both researchers and clinicians to quantify the level of neurological impairment resulting from an SCI. One of the common complications for SCIs is spasticity, which has received some attention with the Ashworth and Modified Ashworth scales (MAS). The MAS appears to provide a valid scale for qualitative assessments that can easily be used in practice.

Other areas that are frequently assessed include lower limb and walking, which was evaluated by more than 78% of the respondents. In this area, the measures used most were two time tests (i.e., the 10-meter walk test and the 6-min walk test) and the Walking Index for Spinal Cord Injury (WISCI). This is in agreement with what was suggested by the SCIRE and several guidelines (34, 35). The WISCI assesses walking capacity based on the need for orthosis, walking aids, and assistance (36). However, it does not offer any information on walking speed and suffers from a ceiling effect (i.e., those who are level 20 cannot improve further) that is reached by SCI subjects within the first 6 months after the lesion (37). The timed tests describe walking in terms of speed and endurance but suffer from a floor effect, as patients who are unable to walk cannot be assessed. Therefore, the combination of these three measures (WISCI, MAS, ASIA) seems to be the best method to assess the walking and walking improvements of SCI patients.

The Upper Limb area is frequently assessed as well, but using a variety of measures that makes it difficult to suggest a preferred measure. However, it should be noted that half of the respondents who answered "Other" declared that they assessed the upper limb area using a strength assessment. Together with the 18% of respondents who used a handheld myometer (38), in total, 30% of respondents assessed the upper limb area using a strength assessment. This is probably because most of the proposed tests require a number of staff or resources as well as time, training, and special equipment that is not always free available.

### Outcome Measures for Pain

One peculiar result of this survey concerns the use of outcome measures for pain. In total, 40% of the respondents answered that they used measures other than those proposed. Most of them used the visual analog scale [VAS (39)] or the numeric rating scale [NRS (40)] instead of or in addition to other rapid measures to assess pain. American centers showed a high consensus in using the SCIRE. At present, no measure (at least within those listed by SCIRE) takes all aspects of pain into account. Some examine the nature and localization of the pain, some examine the impact of the pain on a patient's life, and some examine only particular aspects of pain [e.g., the Wheelchair User's Shoulder Pain Index (WUSPI)]. Furthermore, most of these tests are time consuming. Although the International Spinal Cord Injury Pain Classification indicates several distinct types of pain, including neuropathic, nociceptive, other, and unknown pain (41), only 25% of the respondents indicated using the Classification System for Chronic Pain in SCI and Donovan SCI Pain Classification System proposed by SCIRE to distinguish between neuropathic and nociceptive pain. Most of respondents answered that they used DN4 than those proposed. Most of respondents answered that they used DN4 instead of the measures that were proposed.

However, the VAS and the NRS can be conducted quickly and thus could be utilized for repeated assessments during the day to gauge the severity of the pain and the effects of drugs and treatments.

It should be noted that the SCIRE does not list between the outcome measures for pain in the International Spinal Cord Injury Pain Basic Data Set (42), probably because it is still quite new and is not widespread or commonly utilized in research settings. This instrument, produced by the ISCoS, has the advantage of encompassing different aspects of pain (i.e., type, localization, intensity, and impact on daily life activities and mood), thus reconciling the flaws of other instruments listed by the SCIRE. This type of evaluation may be very useful in patients with SCI who may present with pain due to a primary neurological involvement or to other non-neurological causes, or even mixed pain (for example pain associated with spasticity or painful tonic spasms). To define the exact type of pain it is necessary to target a more specific pharmacological treatment.

### Limitations

This study has some limitations. Of the 800 SCI clinicians and researchers who were invited several times to participate in the survey, only 143 completed it. Although they represented clinicians from most countries around the world (most of the respondents were from Europe), the low number of participants could limit the generalizability of our results. However, this response rate is typical of surveys that examine clinical practice (43–45). The second limitation is the different experiences of respondents (mostly physicians, but also physical therapists, occupational therapists, nurses, and psychologists). It is also difficult to say whether the respondents had comprehensive knowledge of all the areas surveyed. Future studies addressing multiple confounding factors are necessary to establish which factors improve the outcomes of SCI patients. However, this study suggests the need to generate new knowledge regarding the outcome measures of assistive devices as well as the impact of adopting a new

### REFERENCES


approach when using assistive devices in cases of brain– body disconnection, opening the door to an innovative clinical prospect in terms of user-centered neuroprosthetic technologies (46).

### DATA AVAILABILITY

The datasets generated for this study are available on request to the corresponding author.

### AUTHOR CONTRIBUTIONS

GS and MP conceived and designed research, interpreted the results, drafted the manuscript. GS, GG, and MP edited and revised the manuscript. MT and MM critical revision of manuscript for intellectual content. All authors have approved the final version of the manuscript.

### ACKNOWLEDGMENTS

The authors would like to thank SIMS (Società Italiana del Midollo Spinale), CNOPUS (Coordinamento Nazionale Operatori Professionali Unità Spinali) and SIPLES (Società Italiana di Psicologia della Lesione Spinale) for the suggestions and support to the study.

This work was supported by the Italian Ministry of Health (RF-2018-12365682 to MP), the Italian Ministry of Health (RC08G to GS) the European Commission in the Seventh Framework Program (ICT-2013-611626 SYMBITRON to GS), the (ERA-NET NEURON JTC 2016 Program SILENCE to GS).

embodiment of assistive devices-from wheelchair to exoskeleton. Phys Life Rev. (2016) 16:191–4. doi: 10.1016/j.plrev.2016.02.004


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The reviewer MP declared a shared affiliation, with no collaboration, with one of the authors, MP, to the handling editor at time of review.

Copyright © 2019 Scivoletto, Galli, Torre, Molinari and Pazzaglia. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Touching! An Augmented Reality System for Unveiling Face Topography in Very Young Children

Michiko Miyazaki 1,2\*, Tomohisa Asai <sup>3</sup> and Ryoko Mugitani 2,4

<sup>1</sup>Department of Social Information Studies, Otsuma Women's University, Tokyo, Japan, <sup>2</sup>NTT Communication Science Laboratories, Atsugi, Japan, <sup>3</sup>Advanced Telecommunications Research Institute International (ATR), Kyoto, Japan, <sup>4</sup>The Faculty of Integrated Arts and Social Sciences, Japan Women's University, Kanagawa, Japan

Developmental body topography, particularly of the face, is a fundamental research topic in the current decade. However, empirical investigation of this topic for very young children faces a number of difficulties related to the task requirements and technical procedures. In this study, we developed a new task to study the spatially-sensed position of facial parts in a self-face recognition task for 2.5- and 3.5-year-old children. Using the technique of augmented reality (AR) and 3D face tracking technology, we presented participants with their projected self-image on a screen, accompanied by a digital mark located on parts of their face. We prepared a cheerful visual and auditory reward on the screen when participants showed correct localization of the mark. We then tested whether they could indicate the position of the mark on their own faces and remain motivated for task repetition. To assess the efficacy of this task, 31 2.5- and 11 3.5-yearold children participated in this study. About half of the 2.5-year-olds and 80% of the 3.5 year-olds could perform more than 30 trials. Our new task, then, was to maintain young children's motivation for task repetition using the cheerful visual and auditory reward. The analysis of localization errors suggested the uniqueness of spatial knowledge of self-face in young children. The efficacy of this new task for studying the development of body image has been confirmed.

#### Edited by:

Giulia Galli, Fondazione Santa Lucia (IRCCS), Italy

#### Reviewed by:

Alejandro José Estudillo, University of Nottingham Malaysia Campus, Malaysia Brianna Beck, University of Kent, United Kingdom

\*Correspondence:

Michiko Miyazaki myzk@otsuma.ac.jp

Received: 02 March 2019 Accepted: 22 May 2019 Published: 11 June 2019

#### Citation:

Miyazaki M, Asai T and Mugitani R (2019) Touching! An Augmented Reality System for Unveiling Face Topography in Very Young Children. Front. Hum. Neurosci. 13:189. doi: 10.3389/fnhum.2019.00189 Keywords: psychology, development, body image, augmented reality, face, body topography

### INTRODUCTION

Children begin to learn about their own bodies from early in life. They learn several methods for body representation and organize these representations among various modalities, including names of body parts (semantic or conceptual), body topography (spatial or structural), and body schema (somatosensory or controllability; Schwoebel and Coslett, 2005). Several interesting behaviors that derive from the immature emergence or organization of body representations are observed in young children. For example, from the age of 2 years, children draw ''tadpole humans'' that typically consist of circles with some facial features, representing the head as well as the body (Freeman, 1975; Cox, 2013). Another interesting behavior is called the scale error (Deloache et al., 2004), whereby, after playing with a body-sized large toy car, young children aged around 2 years may attempt to enter and drive a miniature toy car ignoring their body size. Yet another interesting exploration error is the rear-search error (Miyazaki and Hiraki, 2009), which occurs in children aged around 2 years during body part localization of their mirrored self-body. In Miyazaki and Hiraki's experiment, participants initially attempted to localize a target on the rear of their heads, even though it was placed on their forehead, an error that was observed in over one-third of the 2-year-old participants. These behaviors suggest that young children may have specific and immature body representation(s).

Beyond these observations, it is generally challenging to empirically examine young children's immature body representations. Compared to adults, children: (1) cannot fully understand verbal instructions; (2) cannot keep themselves motivated; and therefore (3) cannot repeat trials. Since it is necessary to describe the developmental trajectory and differences in body representation between young children and adults, it is useful to seek a solution to the aforementioned difficulties. Although we aimed to explore the development of whole-body representation(s) for future, in this study, as a first step, we developed a new task for evaluating the emergence of face topography in young children, using an augmented reality (AR)-based procedure.

Several previous studies have examined the development of body topography in young childhood (Witt et al., 1990; Brownell et al., 2010, 2012; Camões-Costa et al., 2011; Herold and Akhtar, 2014; Waugh and Brownell, 2015). For example, Brownell et al. (2010) examined the body topography of 20- and 30-month-olds using a sticker placing task. An experimenter demonstrated the task to the children by placing stickers on another experimenter's specific body parts. The children were then asked to place a sticker on an unnamed body location on themselves. The sticker task began on the nose, then proceeded to include 12 body locations (nose, hand, foot, head, back, neck, forehead, wrist, elbow, calf, temple, and nape). The results revealed an immature body structural knowledge. On average, 20-month-olds were able to locate only two or three of the locations, while 30-month-olds (2.5 year-olds) were able to locate four or five (Brownell et al., 2010). Herold and Akhtar (2014) examined body structural knowledge using a similar sticker task in 2.5-year-olds. They demonstrated the task to the children by placing a sticker on a life-sized drawing of a child. The children were then asked to place a sticker on four body parts (hair, stomach, arm, and foot). The results demonstrated that 16 out of 48 children correctly placed the sticker more than three times (Herold and Akhtar, 2014).

Based on the findings of these previous studies, young children have little knowledge of their own body topography. However, it is possible that children's knowledge of own body is underestimated. As mentioned above, verbal instruction might be tricky for young children and the task difficulty and complexity could keep children from revealing their potential knowledge about their body structure. That is, they might have an implicit body topography without having any explicit knowledge. In Brownell et al.'s (2010) study described above, after demonstrating the task with another experimenter, children were given the following verbal instruction: ''Now you put your sticker on you right there, so it's just like [name]. You put your sticker right there on you.'' This phrasing may be slightly difficult and complex for young children because it requires referential and conceptual inference from the phrases ''right there'' and ''just like [name].'' Furthermore, this task required stickers to be placed 12 times and too many repetitions is likely to be boring for young children (Brownell et al., 2010). In our pilot examination for this study, we found that young children were bored by task repetition without any reward. Thus, a new task for evaluating body structural knowledge requires a simple task rule and exciting feedback in order to ensure that children remain motivated for task repetition.

To overcome the problems mentioned above, using the technique of AR and 3D tracking technology, we presented participants with their projected self-image on a screen with one of several famous cartoon characters (digital images). The cartoon character then appeared on various parts of the children's bodies, and we tested whether the participants could demonstrate correct localization by touching the same parts of their own bodies. We also tested whether they would remain motivated throughout the task despite the repetition.

In this study, as the first step before exploring whole body topography, we developed a new task for evaluating face topography in children aged 2.5 and 3.5 years. Previous studies examining body topography in adult participants have used several perspectives for the estimation: face (Fuentes et al., 2013b; Serino et al., 2015; Estudillo and Bindemann, 2017; Mora et al., 2018; Porciello et al., 2018), hands (Longo and Haggard, 2010; Longo, 2017), and the whole body (Fuentes et al., 2013a). Many studies demonstrated interesting distortions or plasticity of face, hands, and whole body topography in adults. In particular, several methods have been employed in studies examining face topography in adults; however, the methods used in these studies are not directly applicable to young children. For example, Mora et al. (2018) developed a proprioceptive pointing task to locate face landmarks in the first-person perspective. A vertical acrylic sheet was placed in front of the participants, very close to their face. Participants were asked to place their face on the chin rest and locate 11 face landmarks by finger pointing. Their findings suggested that size distortions are intrinsic to self-face representation. This task enables us to identify the features of face topography in adult participants; however, it is challenging to carry out the same task for young children because of difficulties such as providing verbal instructions and asking the children to maintain posture.

Our task was based on the mark test, a well-known test of mirror self-recognition (Gallup, 1970; Amsterdam, 1972), wherein children perform the required task without any instruction. However, while the mark test simply examines whether children recognize themselves in a mirror reflection, our task can additionally visualize their body topography in terms of their spatial error pattern and reaction time—quantifications that can be measured for children aged 2.5 and 3.5 years during the task repetition.

In this study, children aged 2.5 and 3.5 years were targeted because the ability to recognize oneself in a mirror reflection (mirror self-recognition) develops at approximately 24 months of age (Amsterdam, 1972; Nielsen and Dissanayake, 2004). Thus, evaluating face parts localization for children of these ages is important because their localization reflects the first organization of face topography. Among these, face topography is particularly noteworthy because we only see our faces reflected in the mirror. Therefore, correct localization of face parts may require matching proprioceptive and visual information for each face part. Since 2.5-year-olds are known to generally pass the mark test, we hypothesized that their initial face topography (without verbal instruction) could be examined using this AR task. We also hypothesized that the results of 2.5- and 3.5-year-olds could be compared quantitatively based on their developmental stages.

Taken together, the aim of this study was to develop a new task and assess its efficacy. We also examined whether this task would maintain children's motivation for task repetition without difficult verbal instruction and whether it could be used to evaluate body topography in 2.5- and 3.5-year-olds.

## EXPERIMENT

### Participants

Forty-three 2.5-year-olds and 12 3.5-year-olds participated in this study. The final sample comprised 31 2.5-year-olds (15 females) and 11 3.5-year-olds (five females). Fourteen participants were excluded from the analysis (attrition rate = 25%) due to fussiness or embarrassment (N = 10), experimental error (N = 4), failure of the video recording, or because Kinect failed to properly identify their bones. The children were recruited from the participants' pool of the NTT Communication Science Laboratories. This study was carried out in accordance with recommendations from the NTT Communication Science Laboratories Ethical Committee and with written informed consent from all participants' parent(s). All parents gave written informed consent in accordance with the Declaration of Helsinki. The study protocol was approved by the NTT Communication Science Laboratories Ethical Committee.

### Apparatus and Task

We developed a task called ''Touching!'' using a motionsensing input device (Microsoft, Kinect v2) to track participants' faces and using an AR technique to present participants with their projected self-image (EPSON, EB485WT) on the screen (KIMOTO, RUM60N1). Cartoon characters (digital image) then appeared on various parts of the children's faces (nose, right/left cheek, lower/upper forehead, and chin; **Figure 1**). Participants' bodies were presented in a mirror-like (ipsilateral) relationship. A motion-sensing input device was used to track participants' faces in 3D. The program used to present digital images was written in Processing 3.0 and Kinect v2 for Processing library. We used a device with Graphics Processing Unit (GPU; DOSPARA, GALLERIA GAMEMASTER NX) for presenting the self-image with a maximum of 150 ms temporal delay (approximately four frames). A time delay of about 150 ms causes peculiarity, but it does not affect movement accuracy (Katayama et al., 2018).

FIGURE 1 | Experimental setup. Participants were asked to touch their real face with reference to their face image on projected digital images. If participants correctly touched the corresponding part of their real face, a cheerful visual and auditory reward was presented. To track participants' face in 3D, a motion-sensing input device (Microsoft, Kinect v2) was used. The program to present digital images was written in Processing 3.0 and Kinect v2 for Processing library. We obtained permission from the participant's parent for the publication of this image.

The movie via the USB camera of Kinect v2 was recorded by a monitor capturing device (Avermedia, AVT-C878) and laptop PC (Panasonic, CF-MX3).

Participants were asked to touch their real faces with reference to their face image on the projected digital images. First, a digital image was displayed on participants' face part with a beep sound by the key input of the experimenter. This is the onset of each trial. If participants correctly touched the corresponding part of their real face, a cheerful visual and auditory reward was presented by the experimenter's manual key press, and the digital image disappeared. If participants failed to touch, the digital image was displayed again on the same site but these responses were not used for analysis because such responses had considerable individual difference. If the experimenter judged that the participants had lost the motivation to touch, the trial was silently ended (the image disappeared), and the next trial began.

The first location presented by the digital image was always the nose because previous studies have shown that the nose is one of the first body parts children learn (Witt et al., 1990). The order of the following locations was randomized by Latin square design. In the experiment, to assess their motivation for task repetition, we prepared a relatively large number of trials: 37 trials maximum (1 example + 6 face places × 3 characters × 2 blocks).

To assess participants' knowledge of vocabulary related to body parts, we asked their caregivers to complete a questionnaire, which included 60 words related to body parts (see **Supplementary Table S1**). Caregivers were requested to check whether their child could comprehend or comprehend and produce each word. To assess participants' development of sensory profile, we asked the caregivers to answer the Japanese translation version of the Infant Toddler Sensory Profile (ITSP; Dunn and Daniels, 2002; Tsujii et al., 2015). The ITSP is a 48-item caregiver questionnaire that measures sensory modulation abilities as reflected in daily experiences in children aged 7 months to 36 months. Although ITSP is an assessment tool for evaluating sensory modulation behaviors in toddlers with autism spectrum disorders (ASDs), in this study, we used this tool to capture participants' sensory modulation state only for exploring correlation with the task performance. We only asked the caregivers of 2.5-year-olds to answer the ITSP because the maximum age of eligibility for ITSP is 36 months (3 years). To assess participants' experience of self-images in their daily life (mirror, video, pictures, etc.), we asked the caregivers how frequently the participants played with self-images and how they played with them.

### Procedure

To demonstrate the task rules, we first asked caregivers to play the ''Touching!'' game. The caregivers stood in front of the screen (height = 1.12 m) and the Kinect camera (height = 0.55 m from the ground, distance from participants' position = 1.50 m; see **Figure 1**). Once the program was ready to capture their bones, the ''Touching!'' game began. They were asked to touch their real faces with reference to their face image on the projected digital image. At the beginning of the task, the caregivers demonstrated how to play the game and encouraged their children to participate in the game. A maximum of 13 trials were conducted during this pre-experiment phase. When the participants began to engage in the task spontaneously without caregiver's guidance, we considered it as participants fully understanding the task and commenced the experiment. During the game, caregivers were asked not to say the names of body parts aloud because we wanted to assess body localization using visual and proprioceptive information and thus this instruction prevented the effect of semantic body knowledge.

After the practice phase by their caregivers, the children participated in the game. A total of 36 experimental trials were conducted (see ''Task Repetition'' section). The number of trials in the present experiment was based on those in previous studies (Witt et al., 1990; Brownell et al., 2010; Camões-Costa et al., 2011). In Brownell's study, there were 12 trials for the task for 20- and 30-month-olds. In Witt et al. (1990) study, there were 20 trials for children aged 11–25 months. In Camoes-Costa's study, 100 trials were conducted, although the participants were relatively older (age range: 26–41 months; mean age: 35 months). We considered more than 35 trials to be relatively high and thus, the experiment continued until the participants stopped participating.

### Analysis

The children's responses were recorded in video clips. To analyze the error pattern and response time (RT) of their first touches, we coded participants' correct/incorrect responses in each trial using frame by frame coding. The two coders were blind to the study's goal. Error was defined as a failure of initial touch/pointing on the target body parts. Even if the participants correctly touched the target body part in their final touch after several explorations, we did not count such trials as correct. Body positions that the participants touched in error include mouth, eyes, temple, lip, neck, chest, etc. Inter-coder reliability based on correct/incorrect

responses was calculated at 55% for all data. The coder agreement was κ = 0.88. The coders reached mutual agreement for the trials in which there were disagreements.

After discussing and seeking agreement in several cases, the second coder's score was used. RT was defined as the duration between the appearance of the digital mark to participants' first touch. In addition, we coded each trial for persistent response (more than two times repetitive touch across the trials), LR-error (left-right reversal error), and rear error (touch on the back of their head).

### RESULTS

### Task Repetition

To evaluate participants' motivation for task repetition, we summarized the number of executed trials aggregately in **Figure 2**. More than 60% of the 2.5-year-olds executed more than 30 trials, while more than 80% of the 3.5-year-olds did so. Thus, both age groups maintained their motivation for the task despite the high level of repetition.

### Error Analysis

In **Figure 3**, the error rates are summarized in each age group panel **Figure 3A** and in each face position panel **Figure 3B**. In the 2.5-year-olds, the error rates varied widely, while in the 3.5-year-olds, they did not vary widely. A Welch-Satterthwaite t-test revealed a significant difference between the two age groups (t(23.61) = 4.47, p < 0.001, Cohen's d = 1.46).

To explore toddlers' accuracy of face localization, we summarized the error rates for first touch in each face position (see **Figure 3B**) as follows: in the 2.5-year-olds, right cheek (45.5%), left cheek (40.2%), chin (39.8%), nose (33.1%), forehead (39.5%), and upper forehead (39.7%); in the 3.5-year-olds, right cheek (16.1%), left cheek (15.1%), chin (19.3%), nose (20.0%), forehead (11.1%), and upper forehead (7.0%). Although age differences were clear in all face positions, no clear differences were found between the error rates in each face position.

To analyze localization of initial touch in each age group and among the face positions, we summarized the heat map matrix in **Figure 4** in each age group. Blue-colored cells refer to correct touch rate (maximum correct: 1.0). Red cells refer to error rate (maximum error: −1.0). Yellow-colored cells in (**Figure 4C**) refer to the subtraction of error between the two age groups. Darker yellow indicates a larger difference in error rate between the two age groups.

Next, to visualize the relationship among target positions and touched locations, we summarized **Figure 4** into network plots (i.e., digraph) in **Figure 5** in each age group. This visualization ascertained the initial touch pattern of each face position in each age group. Circles indicate each target face position and a cooler color refers to higher rate of touching in initial touch (correct response). Hotter arrows indicate incorrect touch, and the tops of the arrows indicate the error position while the bottom side of the arrows indicates the target position. Broader lines indicate higher frequency of error touch. In the 2.5-year-olds, the accuracy of localization was relatively low (i.e., the node colors are whiter); localization errors varied both along the horizontal- and verticalaxes. In the 3.5-year-olds, the accuracy of localization increased (i.e., the node colors are bluer); the variation of localization errors are limited between adjacent parts.

In 2.5-year-olds, RT ranged from 33 ms to 177,100 ms. The median was 2,233 ms, while mode was 1,500 ms. To describe the distinctive features of the RT, we excluded the RTs that exceeded 4,000 ms (as a result, 83% of the overall data was included. This exclusion was made for RTs only). In **Figure 6**, mean RTs are summarized in correct/incorrect trials and in each target position. To capture the relationship among RTs, touch error, and age, we ascertain these relationships as heat maps in **Figure 7**. For the network analysis and heat map analysis, we summarized these error positions into the distances from the target positions (see **Figure 7A**). The x-axis refer to RTs, while the y-axis refers to the relative distance from each target position (see **Figure 4** for the distance definition). For example, when the target is Rcheek (''Rch'') but the touched location is nose (''nos''), the distance is 1. When the target is Rcheek but

FIGURE 4 | Heat map matrix between targeted- and touched-location in each age group. (A) 2.5-year-olds, (B) 3.5-year-olds. Blue-colored cells refer to correct touch rate (maximum correct: 1). Red cells refer to error rate (maximum error: −1). Yellow-colored cells in (C) refer to the subtraction of each rate between the two ages. Darker yellow indicates a larger difference between the two ages (maximum subtraction was −0.5 in this plot). The sum of absolute columns should be 1 (or less than 1, if other uninteresting parts like the foot are touched), while the sum of rows could exceed 1 since this matrix is asymmetric. For example, the nose was their favorite touched part regardless of the target (especially for 2.5-year-olds). Therefore, the sum of rows was more than 1 for the nose as the touched position, while the chin was not much preferred, and therefore, the sum of rows was less than 1. Note. Lch, left cheek; Rch, right cheek; Ufr, upper forehead; frh, forehead; nos, nose; chn, chin.

the touched location is forehead (''frh''), the distance is 1.4 in Euclidean distance. The negative y is simply for visualization of the distribution tail. Therefore, the distance 0 indicates the correct responses. A hotter color refers to a higher frequency of responses. The initial touch responses to each target position in the 2.5-year-olds have a single peak, while responses to right/left cheek, nose, forehead, and upper-forehead in the 3.5-year-olds have a double peak. The spread of y-oriented error responses (refer to whiter colors) in each figure suggests localization errors. In general, variation of error position (y-axis) narrowed from the 2.5-year-olds to the 3.5-year-olds, as did the variation in reaction time (x-axis).

### Word Acquisition of Body Parts

In **Table 1**, the mean number of acquired words related to body parts was summarized in each age group. The number of acquired words (comprehension + production) was greater in the 3.5-year-olds (M = 47.1, SD = 10.5) than the 2.5-year-olds (M = 34.5, SD = 11.9), t(38) = −3.08, p = 0.0001. The number of acquired words did not relate to the rates of initial touch errors


TABLE 1 | Mean number of lexical acquisition of body parts (60 words) and mean acquisition rate of the target position based on caregiver report.

Note. Two of the 2.5-year-olds were excluded from the analysis due to data unavailability (N = 2).

in both age groups (r = −0.274, p = 0.087). The acquired rate of each target face position word was as follows: in the 2.5-year-olds, cheek (90%), forehead (59%), nose (97%), and chin (38%); in the 3.5-year-olds, cheek (100%), forehead (100%), nose (100%), and chin (73%).

### Sensory Profile [Japanese Translation Version of Infant Toddler Sensory Profile (ITSP)]

We explored the relationship between the task performance and sensory profile in the 2.5-year-olds, excepting the 3.5 year-olds because the ITSP is only suitable up to 36 months. We calculated Spearman's rank correlation coefficient among the task performances (number of correct/incorrect trials, error rate), sensory profile (auditory, visual, tactile, vestibular, and oral sensory), and sensory types (low registration, sensation seeking, sensory sensitivity, and sensation avoiding; see **Supplementary Table S2**). No significant correlations were found between the task performance and sensory profile.

### Participants' Experience of Play Using Self-Images

To analyze the relationship between frequencies of play using self-image and the accuracy of body localization, we collected data of frequency of play using self-images (see camera, video, mirror, cell phone applications using selfimages). The frequencies of play using self-image did not relate to the performance of body localization in either age group (2.5-year-olds: ρ(<sup>n</sup> <sup>=</sup> <sup>31</sup>) = 0.058, p = 0.760; 3.5-year-olds: ρ(<sup>n</sup> <sup>=</sup> <sup>11</sup>) = −0.052, p = 0.880).

### SUMMARY AND FUTURE DIRECTIONS

In this study, we developed a new face localization task to overcome the issue of maintaining young children's motivation for task repetition. Using the AR technique and 3D face-tracking technology, we presented 2.5- and 3.5-year-olds with their projected self-image on a screen accompanied by a digital mark located on positions of their face, and then required them to touch these marks on their own bodies. Nearly half of 2.5-year-olds repeatedly executed more than 30 trials and almost all 3.5-year-olds executed all 36 trials.

Although the body localization tasks used in previous studies may underestimate young children's knowledge of body structure due to the high level of repetition without any reward (Brownell et al., 2010, 2012), our new task was able to maintain young children's motivation for task repetition. Our task is also expected to reveal the details of young children's face topography because repetitive data collection for the same face parts enables us to calculate the error rate in each face position. Furthermore, the analyses, which comprised combined multiple measurements such as error position of initial touch, relative distance from the target position, and RT, helped us to reveal the characteristics of face topography in young children.

In the present study, we found a clear age contrast for localization accuracy between 2.5- and 3.5-year-olds. For example, errored positions were broader in the 2.5-year-olds than in the 3.5-year-olds. This finding suggests that face topography in 2.5-year-olds is relatively more blurry than in 3.5-year-olds. From analysis of the RTs, we also captured the difference between the two age groups. The RTs of initial touch in the 2.5-year-olds showed a single peak, while the responses to right/left cheek, nose, forehead, and upper-forehead in the 3.5-year-olds showed a double peak. It is likely that 2.5-year-olds' touch demonstrates ballistic touching, which is relatively fast, a straight path, and without adjustment, while 3.5-year-olds' touch includes visual proprioceptive motor control as well as ballistic touching. Ballistic touching might reflect proprioceptive localization of face parts; therefore, the children, particularly the 2.5-year-olds, demonstrated incorrect touch without modification of their initial touch. On the other hand, 3.5-year-olds showed touches with relatively longer RTs. This might include visual-proprioceptive motor control with reference to visual feedback from own hands on the screen; therefore, incorrect localizations might be modified by the way of touching.

How can the findings of the present study be considered in adult studies of face topography? As stated above, in Mora et al. (2018) study, adult participants were asked to point to 11 face landmarks (i.e., hairline, corners of each eye, tip of nose, lateral side of both nostrils, corners of the mouth, and chin). The results showed overestimated of the width of the nose and mouth (Mora et al., 2018). This is the first task to evaluate proprioceptive-based face topography in adults. The participants were asked to point to their face parts according to the verbal instruction without visual feedback. However, it is difficult to carry out the same task among young infants. In this study, we used a mirror test as a hint to develop a pointing task with visual feedback. By introducing visual feedback and enabling pointing by visuomotor control, purely proprioceptive-based face topography cannot be evaluated; however, the main research question in the present study was whether young children can maintain their motivation for task repetition. Therefore, a direct comparison with the Mora et al. (2018) findings is a task for the future. However, if we can sophisticate our task to distinguish visuomotor control and measure the mistouched points in more detail, we will be able to quantify the distortion of the face topography.

In recent years, research on the plasticity of the face representation, known as the enfacement illusion (Sforza et al., 2010), has also attracted attention (Porciello et al., 2018 for a review; Serino et al., 2015; Estudillo and Bindemann, 2017). Similar to the rubber hand illusion (RHI; Botvinick and Cohen, 1998) and the out-of-body illusion (Ehrsson, 2007), it is an approach to examine the plasticity of self-face representation using the self-other discrimination task by controlling multisensory stimulation between the self and others. The boundaries of self-other distinctions might be more ambiguous in young children than in adults, and several theories suggest that the state of undifferentiated self and-other promotes sociality (the like-me theory; Meltzoff, 2007; the socialbiofeedback model; Gergely and Watson, 1999). Based on these perspectives, it is intriguing to examine the likelihood and development of the enfacement illusion in young children. In particular, it is worth noting whether to remap tactile/motor information, or whether it can be initially processed in a supramodal manner.

What advantage does the ''Touching!'' task have for examining the development of body recognition? Before discussing this issue, let us introduce three types of body representation as proposed by Schwoebel and Coslett (2005). The first, termed the body schema, comprises on-line sensorimotor representation of the body. Actual and mentally simulated movements depend on the body schema; this can be estimated by the hand imagery/action task (Sirigu et al., 1996) and the hand laterality task (Parsons et al., 1995). The second type, termed body topography, comprises a topological map of the body. This does not require verbal knowledge of the body. A typical example of impairment of body topography is autotopagnosia, which is characterized by an inability to localize body parts on one's own or others' bodies (Buxbaum and Coslett, 2001). Body topography has two types of representation: one based on tactile sensation and the other based on proprioceptive sensation. The third, termed body image, comprises a semantic and lexical representation of the body. Following an examination among brain-injured patients, Schwoebel and Coslett proposed these putative three types of body representation that assume independent neural pathways.

The developmental transitions of tactile-based body topography during the first year of life emerge consecutively (Somogyi et al., 2018; Meltzoff et al., 2018, 2019). At the neural level, the somatosensory brain map differentiates from early in life. Meltzoff, Saby, and Marchall examined the neural representation of the body in 60-day-old human infants. Electroencephalography (EEG) was recorded while infants received tactile stimulation of three body parts: hand, foot, and lip. Tactile stimulation of these body parts elicited distinguishable signatures (Meltzoff et al., 2019). Interestingly, however, concerning aspects of body localization, infants do not touch their hand(s) to the correct body positions until 7.5-months-old (Somogyi et al., 2018). Somogyi et al. (2018) examined the ability for body localization during the first months of life by examining localization of vibrotactile stimulation on infants' limbs. The dissociation between neural differentiation of tactile sensation and practical use in localization is interesting and important to reveal the developmental transition of body topography.

In this study, we assume that ''Touching!'' can be used to evaluate body topography based on proprioceptive sensation because the participants could not find projected marks without using proprioceptive information related to their body. We also found that accuracy of face localization did not correlate with the word acquisition of body parts. This finding supports the fact that acquisition of body topography and the emergence of semantic and lexical knowledge of the body are independent of each other. Further research is necessary, however, to confirm this finding.

Considering the development of proprioceptive body topography, few studies have examined this topic in young children. Most relevant literature on this topic examines the RHI in childhood, which is a famous experimental paradigm to reveal the nature of body ownership. The subjective sense of body ownership is constructed by multimodal integration among visual, proprioceptive, and tactile information (Botvinick and Cohen, 1998). Recent works suggest that there are two dissociable processes of body representation; a process based on visual-tactile information and a later-maturing process based on visual-proprioceptive information (Biko et al., 2012; Cowie et al., 2013). These examinations using the RHI are helpful to clarify the cognitive background of body ownership based on visual-proprioceptive information, whereas it would be too difficult for children under four to complete the RHI task due to the difficulty of verbal instruction.

Our task does not require the report of illusion or an understanding of complex verbal instruction. Therefore, it is helpful for examining the development of proprioceptive face topography in very young children.

This study has several limitations. The first limitation is related to face and bone detection. In this study, we used Microsoft Kinect for the detection of face and bones. However, the face and bone model used for basic programming might be adjusted for adults' size. Thus, the face and bone detection were sometimes off the correct position because toddlers are small and have short limbs. Nevertheless, the current system could detect the face and bones correctly in most trials. A body localization task should be developed in further research using other detection devices. The second limitation is also related to face and bone detection. There were cases in which bone detection was difficult due to the temperament of the participants. For example, some children have a strong bonding need and cannot stay away from their mothers. Thus, sampling bias is inevitable. If there is a technique to detect the bone even if the child is near their mother, this limitation can be overcome to some extent. The third limitation is that there are several possible explanations for the touching. One possibility is that the development of the proprioceptive face topography affects touch, while another possibility is that simple visuo-motor control without the knowledge of self-face affects touch. We consider that the former is highly possible because differences in reaction patterns are seen in each digital image presentation location, but it is difficult to completely separate these two possibilities in this study. In future, it is necessary to compare tasks that can distinguish visuo-motor control, such as tasks that involve touching of one's own body parts and tasks that involve touching the parts of a toy.

In future research, we would like to extend the ''Touching!'' task to estimate whole-body localization. Furthermore, it is also important to reveal the developmental transition and relationship between the tactile and proprioceptive body topographies.

### DATA AVAILABILITY

The datasets generated for this study are available on request to the corresponding author.

### ETHICS STATEMENT

This study was carried out in accordance with the recommendations of the NTT Communication Science Laboratories Ethical Committee with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the NTT Communication Science Laboratories Ethical Committee.

### REFERENCES


### AUTHOR CONTRIBUTIONS

MM, RM, and TA conceptualized, designed the study and analyzed the data and the results were interpreted and examined by all authors. MM and RM collected the data. MM wrote the first draft of the manuscript. All authors agreed on the final version of the manuscript.

### FUNDING

MM was supported by JSPS KAKENHI Grant Numbers 15H02735, 16K21341 and 19H04019. TA was supported by JSPS KAKENHI Grant Number 17K13971, ''Research and development of technology for enhancing functional recovery of elderly and disabled people based on non-invasive brain imaging and robotic assistive devices'' of NICT, and AMED grant number JP18dm0307008. This study was also supported by NTT Communication Science Laboratories. The funder was not involved in the study design or collection, analysis, or interpretation of the data.

### ACKNOWLEDGMENTS

We would like to thank Makiko Ishikawa, Ayana Shiino, Kouichi Matsuda, and Norihiro Ban for their help with game programming, and Mami Seto and Emi Fujita for their help with data collection and analysis. We would also like to thank all caregivers and children who participated in this study.

### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnhum. 2019.00189/full#supplementary-material


**Conflict of Interest Statement**: MM was an unpaid guest researcher at NTT Communication Science Laboratories and RM was employed as a research scientist by NTT Communication Science Laboratories.

The remaining author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Miyazaki, Asai and Mugitani. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Corrigendum: Touching! An Augmented Reality System for Unveiling Face Topography in Very Young Children

#### Michiko Miyazaki 1,2 \*, Tomohisa Asai <sup>3</sup> and Ryoko Mugitani 2,4

*<sup>1</sup> Department of Social Information Studies, Otsuma Women's University, Tokyo, Japan, <sup>2</sup> NTT Communication Science Laboratories, Atsugi, Japan, <sup>3</sup> Advanced Telecommunications Research Institute International (ATR), Kyoto, Japan, <sup>4</sup> The Faculty of Integrated Arts and Social Sciences, Japan Women's University, Kanagawa, Japan*

Keywords: psychology, development, body image, augmented reality, face, body topography

Approved by:

**A Corrigendum on**

*Frontiers Editorial Office, Frontiers Media SA, Switzerland*

> \*Correspondence: *Michiko Miyazaki myzk@otsuma.ac.jp*

Received: *24 June 2019* Accepted: *25 June 2019* Published: *12 July 2019*

#### Citation:

*Miyazaki M, Asai T and Mugitani R (2019) Corrigendum: Touching! An Augmented Reality System for Unveiling Face Topography in Very Young Children. Front. Hum. Neurosci. 13:236. doi: 10.3389/fnhum.2019.00236* **Touching! An Augmented Reality System for Unveiling Face Topography in Very Young Children**

by Miyazaki, M., Asai, T., and Mugitani, R. (2019). Front. Hum. Neurosci. 13:189. doi: 10.3389/fnhum.2019.00189

In the Acknowledgments section of the original article, the names of two individuals were spelled incorrectly. The first error is **Ayano Shiina**; the correct spelling is **Ayana Shiino**. The second error is **Koichi Matsuda**; the correct spelling is **Kouichi Matsuda**.

The authors apologize for the oversight and confirm that this does not change the scientific conclusions of the article in any way. The original article has been updated.

Copyright © 2019 Miyazaki, Asai and Mugitani. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# SAE+LSTM: A New Framework for Emotion Recognition From Multi-Channel EEG

#### Xiaofen Xing<sup>1</sup> , Zhenqi Li <sup>1</sup> , Tianyuan Xu<sup>1</sup> , Lin Shu<sup>1</sup> \*, Bin Hu<sup>2</sup> and Xiangmin Xu<sup>1</sup>

<sup>1</sup> School of Electronic and Information Engineering, South China University of Technology, Guangzhou, China, <sup>2</sup> School of Information Science and Engineering, Lanzhou University, Lanzhou, China

EEG-based automatic emotion recognition can help brain-inspired robots in improving their interactions with humans. This paper presents a novel framework for emotion recognition using multi-channel electroencephalogram (EEG). The framework consists of a linear EEG mixing model and an emotion timing model. Our proposed framework considerably decomposes the EEG source signals from the collected EEG signals and improves classification accuracy by using the context correlations of the EEG feature sequences. Specially, Stack AutoEncoder (SAE) is used to build and solve the linear EEG mixing model and the emotion timing model is based on the Long Short-Term Memory Recurrent Neural Network (LSTM-RNN). The framework was implemented on the DEAP dataset for an emotion recognition experiment, where the mean accuracy of emotion recognition achieved 81.10% in valence and 74.38% in arousal, and the effectiveness of our framework was verified. Our framework exhibited a better performance in emotion recognition using multi-channel EEG than the compared conventional approaches in the experiments.

#### Edited by:

Jan Babic, Jožef Stefan Institute (IJS), Slovenia

#### Reviewed by:

Sung Chan Jun, Gwangju Institute of Science and Technology, South Korea Oluwarotimi WIlliams Samuel, Shenzhen Institutes of Advanced Technology (CAS), China

#### \*Correspondence:

Lin Shu shul@scut.edu.cn

Received: 26 September 2018 Accepted: 24 May 2019 Published: 12 June 2019

#### Citation:

Xing X, Li Z, Xu T, Shu L, Hu B and Xu X (2019) SAE+LSTM: A New Framework for Emotion Recognition From Multi-Channel EEG. Front. Neurorobot. 13:37. doi: 10.3389/fnbot.2019.00037 Keywords: EEG, emotion recognition, neural network, Stack AutoEncoder, LSTM

### 1. INTRODUCTION

Emotion has a great influence on human cognition (Yoo et al., 2014), behavior and communication. Since emotion can reflect information of hobbies, personality, interests and even health, recognition of human emotions can help machines and robots in improving the reliability of human-machine interaction (Yin et al., 2017) and also help them in action processing and social cognition (Urgen et al., 2013). Therefore, research on EEG-based automatic emotion recognition is very important and significance for brain-inspired robots and machines, as it enables them to read people's interactive intentions and states through the wirelessly acquired EEG.

As a subjective feeling, emotion is difficult to be represented by a quantitative model. Researchers often use a two-dimensional space to model emotions (Lang, 1995), where different emotion points can be plotted on a 2D plane consisting of a Valence axis and Arousal axis. Compared with facial expression (Zhang et al., 2016) and speech (Mao et al., 2014), emotion recognition based on physiological signals such as EEG, ECG (electrocardiogram), and EMG (electromyography) (Alzoubi et al., 2012; Chen et al., 2015a; Shu et al., 2018) are more objective and reliable. The main component of the EEG signals are brain rhythm signals from different brain regions, which reflect the activity of the region (Niedermeyer and da Silva, 2005).

The electrical activities of the cortex were propagated through the anatomical structures to the scalp. Therefore, the acquired EEG was a mixture of the source signals from different brain regions, which carried a great deal of redundant information with a low SNR (signal to noise ratio) (Korats et al., 2012). Additionally, the asymmetry features regarding brain regions, such as DASM (differential asymmetry), RASM (rational asymmetry) and DCAU (differential causality) have been explored in the literature on emotion recognition (Zheng et al., 2016; Li et al., 2018), indicating that the spatial information of EEG signals is useful. Decomposing the source signals from different brain regions in the collected EEG could extract useful spatial information while reducing the redundant information in EEG signals, which was considered as one of the key issues in this paper.

On the other hand, the extraction of temporal correlations of spontaneous EEG signals in the context of emotion recognition referred to another key issue. Emotions were affective phenomena varying with time that are caused by a result of stimuli. The context correlation of EEG time sequence reflected the emotion variation. However, most of the commonly used classifiers could only conduct emotion recognition using independent EEG segments, like SVM (support vector machine) or kNN (k-Nearest Neighbor) (Mohammadi et al., 2017). Although there is substantial literature on scalp ERPs, which were highly correlated temporally in the research area of motor control, only a few studies have considered the temporal correlations of spontaneous EEG signals in emotion recognition (Soleymani et al., 2016), and their recognition rate was not adequate. Considering the context correlation of EEG time sequence, making use of the temporal correlation features might provide more effective means in automatic emotion recognition.

In this paper, we present a novel framework for EEG emotion recognition, where SAE is used (Hinton and Salakhutdinov, 2006) to build the linear EEG mixing model and decompose the EEG source signals from the collected EEG signals. Then, followed by the feature extraction, feature sequences of the EEG source signals are obtained. Finally, to explore the temporal correlations in EEG source signal feature sequences, LSTM-RNN (Bengio et al., 2002) is elected as the emotion classifier.

### 2. RELATED WORK

Some recent studies have been working on emotion recognition using EEG signals.

Khosrowabadi et al. presented a biologically inspired feedforward neural network named ERNN to recognize human emotions from EEG. To simulate the short term memory of emotion, a serial-in/parallel-out shift register memory was used in ERNN to accumulate the EEG signals. Compared with other feature extraction methods and feedforward learning algorithms, ERNN achieved the highest accuracy when using the radial basis function (Khosrowabadi et al., 2014).

Soleymani et al. studied how to explore the emotional traces of videos and presented an approach in instantaneously detecting the emotions of video viewers from EEG signals and facial expressions. They utilized LSTM-RNN and continuous conditional random fields (CCRF) to detect emotions automatically and continuously. The results showed that EEG signals and facial expressions carried adequate information for detecting emotions (Soleymani et al., 2016).

Li et al. explored the influence of different frequency bands and number of channels of the EEG signals on emotion recognition. The emotional states were classified into the dimensions of valence and arousal using different combinations of EEG channels. The results showed that the gamma frequency band was preferred and increasing the number of channels could increase the recognition rate (Li et al., 2018).

Independent Component Analysis (ICA) approaches for multi-channel EEG processing are popular, especially for artifact removal and source extraction.

You et al. presented a method of blind signal separation (BSS) for multi-channel EEG, which combined the Wavelet Transform and ICA together. The high-frequency noises were removed from the collected EEG by using the noise filtering function of wavelet transform, so that the ICA could extract the EEG source signals without regard to the problem of noise separation. The experimental results approved the effectiveness of this method in the BBS of multi-channel EEG (You et al., 2004).

Brunner et al. compared three ICA methods (Informax, FastICA and SOBI) with other preprocessing methods (CSP) find out whether and to what extent spatial filtering of EEG data can improve single trial classification accuracy. The results showed that Informax outperformed the other two ICA algorithms (Brunner et al., 2007).

Korats et al. compared the source separation performance of four major ICA algorithms (namely FastICA, AMICA, Extended InfoMax, and JADER) and defined a low bound of data length for robust separation results. AMICA showed an impressive performance with very short data length but required a lot of running time. FastICA took very little time but required twice the data length of AMICA (Korats et al., 2012).

In recent years, autoencoder has drawn more and more attention in biological signal processing, especially in signal reconstruction and feature extraction.

Liu et al. presented a multimodal deep learning approach to construct affective models with the DEAP and SEED datasets to enhance the performance of affective models and reduce the cost of acquiring physiological signals for real-world applications. Using EEG and eye features, the approach achieved mean accuracies of 91.01 and 83.25% on the SEED and DEAP datasets. The experiment results demonstrated that high-level representation features extracted by the BDAE (Bimodal Deep AutoEncoder) network were effective for emotion recognition (Liu et al., 2016).

Majumdar et al. proposed an autoencoder-based framework that simultaneously reconstructed and classified biomedical signals. Using an autoencoder, a new paradigm for signal reconstruction was proposed. It has the advantage of not requiring any assumption regarding the signal as long as there was a sufficient amount of training data. The experiment results showed that the method was better in reconstruction and more than an order of magnitude faster than CS (Compressed Sensing)-based methods. It was capable of providing real-time operations. The method also achieved a satisfactory classification performance (Majumdar et al., 2016).

In these reviewed studies, EEG-based emotional classification has been studied extensively, and corresponding achievements have been realized in the aspects of EEG signal preprocessing, feature extractions, and classifiers. However, decomposition of EEG signals is still a challenge. The current mainly used ICA method assumes the source signals that constitute the mixed EEG signals are independent of each other and do not conform to the normal distribution. But the physiological structure of the brain does not support this hypothesis, as the interconnected cerebral cortex makes the EEG signals have a natural correlation among each other. On the other hand, feature extractions in this area have seldom considered the association and contextual relationships between frames of different EEG signals, which leads to an inadequate utilization of multi-domain information of EEG signals in space-time and the frequency domain. In this work, we tried to explore the method in decomposing EEG signals to source signals and adopt the context correlation of EEG feature sequences to improve emotion recognition.

### 3. METHODOLOGY

### 3.1. Framework Design

As shown in <sup>1</sup>**Figure 1**, our new framework is made up of three sequential parts, including source signal decomposition, feature extraction and emotion classifier. The details of each part are given below.

In the proposed framework, SAE was used in a linear EEG mixing model to decompose the source signals from the collected EEG signals. LSTM+FC was the main component that was used in the emotion timing model to recognize emotion using the correlation of the EEG feature sequence based on the EEG source signals decomposed by SAE.

### 3.2. Source Signal Decomposition

### 3.2.1. Linear EEG Mixing Model

The EEG signal reflects the electrophysiological activity of the cerebral cortex. However, under existing hardware conditions, EEG signals are collected at the scalp instead of the cortex, and there is a skull barrier between the cerebral cortex and the scalp. In fact, the collected EEG signals are the mixture of the EEG source signals. Researchers proposed a linear mixing model to simulate the mixing process, which is widely acknowledged in medical areas (Sanei and Chambers, 2013). In this work, we presented a new method to solve the EEG linear mixing model. The linear EEG mixing model is presented in **Figure 2**.

The mixture of EEG signals can be written as <sup>2</sup> (1):

$$X = A\mathcal{S} \tag{1}$$

### 3.2.2. AutoEncoder

Autoencoder is an unsupervised neural network consisting of two components, an encoder and a decoder, whose completely symmetrical structure is given in **Figure 3**. If the reconstructed data is equal to the input data, the output of the "encoder" should be the "code," which contains all the information about the input data.

When using the linear activation function, the mathematical expression of the encoder is given in <sup>3</sup> (2) (Ignore the bias).

$$H = W I \tag{2}$$

From (2), we observe that autoencoder network and linear EEG mixing model have similar expressions. Therefore, we have tried to build and solve the linear EEG mixing model using autoencoder.

### 3.2.3. Linear EEG Mixing Model Based on Stack AutoEncoder

The purpose of this work is to determine an encoder that allows us to decompose the source signals from the collected EEG signals. To achieve a better performance, an autoencoder is used that consists of multiple layers, called a stacked autoencoder (SAE). The formula of SAE has the same form as the formula of standard autoencoder. The structure and the hyper parameters for the SAE we designed are shown in <sup>4</sup>**Figure 4**.

We assumed that the source signals came from 12 different functional brain regions based on previous research (Keil et al., 2002). The 12 regions were formed by crossing hemispheres (left, right) with a horizontal plane (anterior, lateral, posterior) and a vertical plane (inferior, superior) based on recording sites of the international 10–20 system. We made some investigations on the effect of a different number of source channels, such as 6 and 7 source channels. However, the results were not as good as 12 source channels, which was one of the reasons why 12 source channels was selected in our study.

Specifically, X is a 32-dimensional vector as a 32-channel EEG signal is used as the input and S is a 12-dimensional vector as we discussed before.

### 3.2.4. Decomposition Results

To conduct the training of our linear EEG mixing model, minibatch gradient descent was used as the optimizer algorithm, which was an upgraded version of traditional stochastic gradient descent (SGD) and was generally used as the optimizer of the neural network. Mini-batch gradient descent randomly selected a mini batch of data to calculate gradient of the loss

<sup>1</sup> x<sup>1</sup> ∼ x<sup>n</sup> represent the nth-channel EEG signals, s<sup>1</sup> ∼ s<sup>m</sup> represent the mthchannel EEG source signals, FBP<sup>θ</sup> i , FBP<sup>α</sup> i , FBP<sup>2</sup> β and FBP<sup>γ</sup> i represent the frequency band power of theta, alpha, beta and gamma band in the ith frame, respectively, PCCs<sup>i</sup> represent the Pearson correlation coefficient of each channel in the ith frame, I<sup>1</sup> ∼ I<sup>t</sup> represent each frame of the feature sequence, L<sup>1</sup> ∼ L<sup>t</sup> represent each step of the unrolled LSTM layer, F<sup>1</sup> ∼ F<sup>t</sup> represent each unit of the Full-Connect layer, O represents the output layer of the classifier.

<sup>2</sup>X refers to the collected EEG signals, A refers to the mixing matrix and S denotes the EEG source signals.

<sup>3</sup> I is the input data, H is the output of the hidden layer, W is the transformation matrix of the encoder.

<sup>4</sup>X refers to the collected EEG signals, H refers to the output signals of the first hidden layer, S refers to the EEG source signals, Xˆ and Hˆ refers to the reconstructed data of X and H.

function at every step, leading to a fast convergence speed and computational efficiency. As for other optimizers, batch gradient descent needed all data samples to calculate the gradient, which was time-consuming and complicated. Stochastic gradient descent used one sample at each step to reduce the computational complexity and improve the speed, but the drawback was related to its instability and possibility in causing fluctuations. Adam optimizer was faster than SGD and exhibited the advantage of adaptive learning rate, although it might have a convergence problem due to the unstable learning rate (Reddi et al., 2018). In this work, we applied learning rate attenuation in mini-batch gradient descent method to make the model more stable, which turned out to be better than Adam optimizer and other gradient descent methods.

Mean square error (MSE) was used as the loss function. Then, the training data is fed into the model, where the adjusted R-squared between the test data and its reconstructed data is calculated to validate the model. The expression of the adjusted R-square is shown in (3).

$$R\_{adjusted}^2 = 1 - \frac{(1 - R^2)(N - 1)}{N - p - 1} \tag{3}$$

Where R 2 is the sample R-square and p is the number of predictors. The value of P was set to 32 in this work since the channel number of EEG signals is 32 in the database of DEAP. N is the total sample size. The expression of R 2 is given in (4).

$$R^2 = 1 - \frac{\sum (X - \hat{X})^2}{\sum (X - \bar{X})^2} \tag{4}$$

When the training data (the 32 channel EEG signals) were fed into the model, the adjusted R-square between the test data and the reconstructed data was calculated to validate the model. Once the adjusted R-square exceeded 0.9, it meant the "code" output by the encoder of our model almost retained all the

information of the source EEG signals. In other words, the "code" could represent the EEG source signals successfully and the decomposition was done successfully. The process of EEG source signal extraction is shown in **Figure 5**.

### 3.3. Feature Extraction 3.3.1. Signal Framing

As a central nervous physiological signal, EEG signal is nonstationary and chaotic. To facilitate signal processing, the EEG signals are always divided into short time frames and it is assumed that the signal within the frame is stationary (Soleymani et al., 2014). Therefore, some signal processing methods for stationary signals are applicable for EEG signal processing. The EEG signal processing steps are shown in **Figure 6**, where a 1 s window with 50% overlap is applied to the EEG source signals to divide the signals into 125 frames of data. In this work, we also tried sliding data by a 2 s window, 5 s window and so on, while the results turned out no better than the 1s window. The reason might be that the neural network required a larger amount of data, and the 1 s window with 50% overlap could obtain more data than the 2 s window and others. If different experiment settings or models were configured, the choices might be changed flexibly.

After signal framing, the EEG features are extracted from each frame and arranged into a feature sequence. Finally, the feature sequences with 125-frame EEG features are obtained.

### 3.3.2. Frequency Band Power Feature

Biologically speaking, EEG signals are composed of brain rhythm signals, event related potentials (ERP) and spontaneous electrical activity signals. Many studies (Niedermeyer and da Silva, 2005; Whitten et al., 2011) have proved that changes in brain states are often characterized by rhythmic signals from different brain regions. According to the frequency range from low to high, the EEG signals are divided into five frequency bands of delta waves (δ: 0.5–3.5 Hz), theta waves (θ: 4–7 Hz), alpha waves (α: 8–13 Hz), beta waves (β: 14–30 Hz) and gamma waves (γ : 31–50 Hz). As seen in **Figure 7**, we applied the Hanning window to each EEG

channel and the power spectral density (PSD) was calculated by Welch's method. Then, four frequency band powers (FBP) of the EEG signals were chosen in our experiment.

The Hanning window is most often used in random signals to avoid spectrum leakage. Since EEG signals are typical random signals, the Hanning window was used in this work for data segmentation and band power feature extraction. As the reviewer commented, we plotted the amplitude responses of a rectangular window and a Hanning window for a comparison. In **Figure 8**, the narrower main window of the rectangular window is more conducive to identifying the specified frequency, however the sidelobe gain is higher and the spectrum leakage is severe, resulting in amplitude information misalignment. The major advantage of the Hanning window is that the spectrum leakage is small, and the main features extracted in this paper are relevant to frequency band energy, so it is appropriate to choose the Hanning window. Of course, other window functions with small spectrum leakage can be also considered.

### 3.3.3. Channels' Pearson Correlation Coefficient

After receiving stimuli, the brain needs to integrate information to understand correctly the emotional significance of the stimuli. According to the 'binding problem hypothesis' (Singer and Gray,

1995), neurons with similar feature properties will synchronize their discharges under certain specific circumstances, and the functional connectivity of the brain can be estimated using the measure of the synchrony (Gupta et al., 2016). The Pearson correlation coefficient is a measurement on linear correlation between two signals and can be used to measure the inter-channel EEG correlations (Bonita et al., 2014; Chen M. et al., 2015). As seen in **Figure 9**, one of the frame signals is selected as the reference signal and the Pearson correlation coefficients (PCC) between signals can be calculated by (5).

$$PCC = \frac{\sum\_{i=1}^{N} (\chi\_i - \bar{\chi})(\boldsymbol{\wp}\_i - \bar{\boldsymbol{\wp}})}{\sqrt{\sum\_{i=1}^{N} (\chi\_i - \bar{\boldsymbol{\wp}})^2} \sqrt{\sum\_{i=1}^{N} (\boldsymbol{\wp}\_i - \bar{\boldsymbol{\wp}})^2}} \tag{5}$$

### 3.4. Classifier 3.4.1. Emotion Timing Model

In emotional situations, the hippocampal complex and amygdala interact in subtle but important ways. Specifically, the hippocampal complex can influence the amygdala's response when emotional stimuli are encountered (Phelps, 2004). Therefore, we assume that the present emotional status is influenced by the previous emotional status, and EEG under previous emotion status might have correlations with those under present emotion state as EEG could reflect the emotion status while EEG context information could also be adopted

in emotion recognition (Li et al., 2017b), so the EEG feature sequence was viewed as containing information on emotion changes in this paper. Based on this assumption, we then propose an emotion timing model. To simulate our emotion timing model, a classifier is needed which can take full advantage of the context correlations in EEG feature sequences.

### 3.4.2. Long Short-Term Memory Network

The Long Short-Term Memory network (LSTM) is applied to do the emotion classification, which is an improvement on the Recurrent Neural Network (RNN). RNN has the problem of long-term dependencies (Bengio et al., 2002) so it is not suitable for time series analysis, while LSTM can solve the problem due to the design of its repeating module. LSTM is thus adopted in our work to calculate the context correlations of EEG feature sequence. The structure of a regular RNN and our LSTM model in this study is shown in **Figure 10**.

### 3.4.3. Emotion Classifier Based on LSTM

To recognize emotion using the correlation of the EEG feature sequence, a deep neural network for emotion recognition based

on LSTM is proposed. The first layer of the deep neural network is the LSTM layer, which is used to mine the context correlation in the input EEG feature sequence. The second layer is the fullconnect layer, which is used to integrate information and act as the major role of the classifier.

The detailed hyper parameter settings for our neural network model are illustrated in **Figure 11**. In the LSTM layer, 125 LSTM cells are set to correspond to 125 frame features in each sequence. In the full-connect layer, connection units are set with the same number. Finally, the sigmoid activation function is used in the output layer. For classifier training, the mini-batch gradient descent optimizer and the MSE loss function have been also used.

"Dropout" was added in the LSTM and full-connect layers to avoid over-fitting. The training epochs of LSTM were set to a few thousand. In the first few hundred epochs, a high learning rate was set to speed up the training procedure, and then it was slowly changed to a lower rate to achieve more robust results. When the training AUC met the set goal, the training was completed.

### 3.5. Model Training

### 3.5.1. Hyper-Parameter Tuning

The SAE and LSTM models were trained separately, and the parameters were set or tuned according to certain rules or bases to ensure their optimization.

1) The SAE model was an unsupervised model trained via the back propagation of the reconstruction error. The hyperparameter setting is described below: the input layer contained 32 units determined by the number of EEG data channels in

the DEAP dataset. There were 64 units in the first hidden layer which were tuned by the reconstruction error. The second hidden layer had 12 units, which was consistent with the 12 functional brain zones. To conduct the training of our linear EEG mixing model, the mini-batch gradient descent was used as the optimizer algorithm and the mean square error (MSE) was applied as the loss function.

2) The LSTM model was a supervised model. Its time step was set to 125, as 125 data segments were achieved under the conditions whereby each EEG data in DEAP had a length of 63 s, and a 1 s time window with 0.5 s step size was adopted. The hidden layer of LSTM had 125 units, which was tuned by the reconstruction error.

To show how our proposed method handled the data during network training, the training procedure was visualized by plotting the feature clustering in different epochs. In **Figure 12**, the features in connections with positive and negative emotions have been clustered into two categories which are represented by two colors. It can be observed that after a few thousand epochs,

FIGURE 14 | The results comparison among relevant methods.

The proposed framework could effectively solve the over-fitting issue. The SAE model was trained using the reconstruction error, and the sparse and penalty constraints were added to avoid the over-fitting problem. In the training of the LSTM+FC model, three aspects of work had been conducted to handle the overfitting/over-training issues: (1) A 1 s window with 50% overlap was applied to the EEG data segmentation, which augmented the size of data samples and guaranteed the amount of data used in model training. (2) "Dropout" operations were added in the training of the LSTM and full-connect layers to avoid overfitting, which can be seen in **Figure 11**. (3) Regularization items of the parameters had been added. (4) 10-fold cross validation was used to verify our approach, and the result of cross validation could be considered that these results were highly probable

3.5.2. Over-Fitting Handling

without over-fitting.

3.5.3. Training Visualization

### 4.1. Experimental Dataset

We used the EEG data from the DEAP dataset to validate our framework (Koelstra et al., 2012). DEAP is a database using different kinds of physiological signals for human affective state analysis. It contains 32-channel electroencephalogram (EEG) and 8-channel peripheral physiological signals of 32 subjects. Each subject was required to watch 40 one-minute excerpts of music videos during which their signals were recorded. Subjects rated each video in terms of valence, arousal, dominance, liking with the rating distributed from 1 to 9 in each dimension.

The EEG signals in the DEAP database were downsampled to 128 Hz, and a 4.0–45 Hz band-pass filter was applied. The data were then segmented into several 63 s trials, where the 3 s pre-trials were removed and the following 60 s trials were kept for further processing. Since EEG signals might be contaminated by other signals such as EOG (Li et al., 2017a; Samuel et al., 2017), the EOG noise was eliminated by ICA in the DEAP dataset to ensure that EEG data can better represent the emotions of the subjects.

As shown in **Figure 13**, the experimental datasets were selected from DEAP, where we divided the trials into two classes based on the value of valence (or arousal) and labeled "High" if the valence (or arousal) value was higher than 5.5 and "Low" if it was lower than 4.5. Then, the downsampling method was used to balance the number of samples of both "High" and "Low" and we obtained the valence (or arousal) dataset.

### 4.2. Emotion Recognition Results

We selected some relevant studies which had similar experimental settings for a comparison. We used the 10-fold cross validation method to validate results in our classification.

the features were clearly classified by our model.

4. EXPERIMENTAL AND RESULTS

TABLE 1 | First experiment: significance test results.


The 10-fold cross-validation method applied in our work was the regular cross-validation method which was normally adopted in relevant studies (Li et al., 2017b, 2018). Specifically, the data samples contained with all subjects' information were randomly split by 10-fold cross-validation method into 10 folds, where 9 folds were for training and 1 for testing. The validation process was repeated ten times to achieve an average result. The results were calculated by <sup>5</sup> (6).

$$AUC\_{mean} = \frac{1}{10} \sum\_{k=1}^{10} \frac{N\_{correct}^k}{N\_{test}} \tag{6}$$

The average accuracy results of our new framework with a comparison of other conventional methods are shown in **Figure 14**. The results show that our framework exhibits an effective performance.

Compared with relevant methods, our framework achieves the best performance in emotion recognition using both valence (81.10%) and arousal (74.38%). The reason might be as follows. Chen et al. used HMM to build the relationship between the present and previous emotion states (Chen et al., 2015b). However, each step output of HMM was only related to some of the previous states, thus the classifier could not automatically learn like LSTM. Li et al. proposed a CRNN framework for emotion recognition (Li et al., 2017b), but CNN required a large quantity of training data to extract features and the DEAP dataset cannot satisfy that. P Arnau-Gonzlez (2017) studied the method of EEG feature fusion and achieved the best accuracy using SVM (Arnau-Gonzlez et al., 2017), but the SVM classifier was not able to explore the context correlations of the EEG feature sequence, therefore its performance was limited.

The main purpose of setting up this framework was for valence classification. Theoretically, the effect of EEG spatial information on valence classification was more obvious. The results in **Figure 14** showed that our classification accuracy in valence (81.10%) was better than relevant studies. Meanwhile, the framework did not affect and even slightly improved the arousal classification performance. The innovative point was that this framework effectively utilized the time domain and space domain information of EEG signals by a linear EEG mixing model based on SAE and an emotion timing model based on LSTM, which significantly improved the valence classification and did not affect, or even slightly improved, the arousal classification.

### 4.3. Verification Experiment

In order to further verify the effectiveness of our framework, we designed three sets of experiments and made a comparison

<sup>5</sup>AUCmean represents the average recognition accuracy, Ntest represents the number of the testing samples, N k correct is the number of samples that have been classified correctly.

TABLE 2 | Second experiment: significance test results.


of the performance on different emotional dimensions by statistical analysis.

#### 4.3.1. First Experiment - SVM and LSTM Classifier

The first experiment was designed to demonstrate the validity of our LSTM classifier. The experiment settings and results are shown in **Figure 15** and the significance test results are shown in **Table 1**.

The values in **Table 1** are p-values. The p-value in different experiment setups was the probability of paired sample t-tests for different experiment setups, which was calculated by results of the average and standard deviation of the 10-fold cross validation. The main idea of the t-test was to state recognition results under two conditions to get the approximate distribution of each condition and to calculate the probability that two distributions have significant difference. When p < 0.01 (or 0.05), it can generally be concluded the emotion recognition rate of our method was significantly higher than other methods by using different EEG features in both valence and arousal.

Compared with SVM, the emotion recognition accuracy of LSTM was significantly (p < 0.01) higher in both valence and arousal, which proves that exploring the correlations in the EEG feature sequence was more effective than merely integrating the recognition result of each EEG feature frame.

Using LSTM can model emotion in the time dimension and extract the emotion feature of each time step, so that our classifier can integrate the entire feature sequence information. This result agrees with our previous assumption that the change of emotion is continuous.

### 4.3.2. Second Experiment - ICA and SAE

The second experiment was designed to evaluate the performance of our SAE based model for EEG source signal decomposition, which contained two parts.

In the first part, we compared the classification performances among methods with EEG decomposition via SAE or ICA or without EEG decomposition. The results in **Figure 16** showed that the SAE based EEG source signal decomposition method achieved better performance than the ICA methods or nondecomposing methods, especially in the case of using FBP features. The statistically significance test results in **Table 2** further verified the results (p < 0.01).

Using SAE for EEG source signal decomposition, in fact, was to encode the EEG channel. The spatial characteristics of the EEG signal, in other words, EEG channel correlations, were also extracted at this time, which was the reason why using EEG source signals could improve the emotion recognition accuracy.

EEG source signal decomposition was an important step in our framework, which took extra time cost. Luckily, using SAE

FIGURE 17 | Second experiment: Computation Time.

for EEG source signal decomposition would reduce the number of channels of EEG signals that need to be processed, and saved time for feature extraction.

In the second part, we counted the total computation time of EEG source signal decomposition and feature extractions, as can be seen in **Figure 17**, where it was observed that although the decomposition process costs extra time, it reduces the time spent in feature extraction, especially for complex features. The operation speed of SAE is two orders of magnitude faster than that of ICA. The result is explained as follows:<sup>6</sup>

According to the experimental results, using SAE for EEG source signal decomposition could improve the emotion recognition accuracy while ensuring fast recognition speed.

The number of parameters in both the SAE and LSTM models were recorded, where the values were 2040376 and 5804 for LSTM and SAE, respectively. We tried to estimate the parameter number of other models in the relevant literature. However, many of them did not provide the whole parameter settings, especially the parameters regarding hidden layers, so the numbers could not be calculated, and hence a comprehensive comparison on the computational complexity using the number of parameters has not been achieved.

### 4.3.3. Third Experiment - FBP and PCC

The third experiment was designed to compare the performance of FBP and PCC in our new framework. The experiment settings and results are illustrated in **Figure 18** and the significance test results are shown in **Table 3**.

We can see that in **Figure 16** and **Table 3**, compared with the PCC feature, the FBP feature performs better (p < 0.01). The reason may be that FBP is a frequency-domain feature while PCC is a spatial-domain feature. Combining EEG source signal with FBP, "SAE+FBP" can reflect the features of EEG in the spatialfrequency domain, like DASM feature and RASM feature (Lin et al., 2010). While "SAE+PCC" can only reflect EEG in the spatial domain. Therefore, we view the frequency-domain features more suitable for our framework.

### 5. DISCUSSION

In this work, we obtained EEG-based emotion recognition rates of 81.10% in valence and 74.38% in arousal. The current recognition rates of EEG-based emotion recognition methods are still not adequate for real applications. One of the major problems is related to individual differences, which can be minimized via experiment paradigm design or calibrations that can remove the effects of EEG baseline variations on different subjects. On the other hand, the emotion classification accuracies of these methods are difficult to evaluate in an objective way, since there are no universal standard test datasets in the area, and the evaluation steps of the related work in the literature are different. Eliminating individual differences and establishing standard test sets represent important future work for EEG-based emotion recognition.

Compared with valence, our framework does not exhibit high recognition accuracy in arousal. On the one hand, the EEG features we used might not be enough. Trying more complex features, such as the EEG spectral asymmetry index (SASI) (Orgo et al., 2015), the derived features of bispectrum (Kumar et al., 2016) and the wavelet entropy features (Hosseini and Naghibi-Sistani, 2011), may be more effective. On the other hand, our classifier network may be not complex enough. Using Bidirectional recurrent neural networks (Schuster and Paliwal, 1997), like Bidirectional LSTM (Sak et al., 2014), as classifier may

<sup>6</sup>FBP\_32 represents the FBP feature extraction of 32-channel collect EEG signals, PCC\_12 represents the PCC feature extraction of 12-channel EEG source signals, SAE represents the EEG source signal decomposition using SAE based approach and ICA represents the EEG source signal decomposition using an ICA-based approach.

When using ICA to decompose EEG source signals, the EEG data needed to be whitened. Since there existed significant variations in EEG amplitudes among different subjects or among different trials under the same subject, it was necessary to resolve the ICA model when decomposing different EEG data records. However, the SAE method required only one solution, therefore its time consumption was much lower than ICA.



achieve better recognition performance. All of our experiments were conducted on the DEAP dataset. In order to evaluate our framework more systematically and comprehensively, an additional EEG dataset with emotional tags is needed.

In this work, we focused on valence and arousal based on the literature (Chen et al., 2015b; Li et al., 2017b; Mohammadi et al., 2017). Since most relevant studies made the same choice, it is fairer to compare their results with ours in the valence and arousal dimensions. Of course, dominance and other dimensions would be considered in the future work.

### 6. CONCLUSIONS

In this paper, we present a novel emotion recognition framework consisting of a linear EEG mixing model and an emotion timing model. The SAE-based linear EEG mixing model can be used for decomposition of EEG source signals and extracting EEG channel correlations, and it can also improve computation efficiency in feature extraction and upgrade the emotion recognition performance. The emotion timing model is simulated by LSTM, which increased the recognition accuracy by exploring the context correlations

### REFERENCES


of the EEG feature sequence. The comparison results in our experiment approved the effectiveness of our framework, especially in the valence recognition task. This work can promote the development of brain-inspired robots, especially in human-robot interaction.

### AUTHOR CONTRIBUTIONS

XfX, ZL, and LS designed the framework, conducted experiments and wrote the manuscript; ZL and TX carried out experiments; XfX, BH, and XmX analyzed the results and presented the discussion and conclusion parts.

### FUNDING

This work was supported by the Natural Science Foundation of China (Nos. U180120050, 61702192 and U1636218), Science and Technology Program of Guangzhou (201704020043), Natural Science Foundation of Guangdong Province of China (2018A030310407) and the Fundamental Research Funds for the Central Universities (2018ZD11).


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Xing, Li, Xu, Shu, Hu and Xu. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Subthreshold Vibrotactile Noise Stimulation Immediately Improves Manual Dexterity in a Child With Developmental Coordination Disorder: A Single-Case Study

Satoshi Nobusako1,2 \*, Michihiro Osumi 1,2, Atsushi Matsuo1,2,3, Emi Furukawa<sup>1</sup> , Takaki Maeda<sup>4</sup> , Sotaro Shimada<sup>5</sup> , Akio Nakai <sup>6</sup> and Shu Morioka1,2,3

*<sup>1</sup> Neurorehabilitation Research Center, Kio University, Koryo, Japan, <sup>2</sup> Graduate School of Health Science, Kio University, Koryo, Japan, <sup>3</sup> Department of Physical Therapy, Faculty of Health Sciences, Kio University, Koryo, Japan, <sup>4</sup> Department of Neuropsychiatry, Keio University School of Medicine, Tokyo, Japan, <sup>5</sup> Department of Electronics and Bioinformatics School of Science and Technology, Meiji University, Kawasaki, Japan, <sup>6</sup> Graduate School of Clinical Education & The Center for the Study of Child Development, Institute for Education, Mukogawa Women's University, Nishinomiya, Japan*

#### Edited by:

*Mariella Pazzaglia, Sapienza University of Rome, Italy*

#### Reviewed by: *Christian Hyde,*

*Deakin University, Australia Jun Ueda, Georgia Institute of Technology, United States*

> \*Correspondence: *Satoshi Nobusako s.nobusako@kio.ac.jp*

#### Specialty section:

*This article was submitted to Neurorehabilitation, a section of the journal Frontiers in Neurology*

Received: *16 April 2019* Accepted: *17 June 2019* Published: *02 July 2019*

#### Citation:

*Nobusako S, Osumi M, Matsuo A, Furukawa E, Maeda T, Shimada S, Nakai A and Morioka S (2019) Subthreshold Vibrotactile Noise Stimulation Immediately Improves Manual Dexterity in a Child With Developmental Coordination Disorder: A Single-Case Study. Front. Neurol. 10:717. doi: 10.3389/fneur.2019.00717* Developmental coordination disorder (DCD) is the most common childhood movement disorder. It is characterized by clumsiness of fine and gross motor skills in developing children. Children with DCD have low ability to effectively use tactile information for movements, instead relying on visual information. In addition, children with DCD have deficits in visuo-motor temporal integration, which is important in motor control. These traits subsequently lead to clumsiness of movements. Conversely, however, imperceptible vibrotactile noise stimulation (at 60%-intensity of the sensory threshold) to the wrist provides stochastic resonance (SR) phenomenon to the body, improving the sensory and motor systems. However, the effects of SR have not yet been validated in children with DCD. Thus, we conducted a single case study of a 10-year-old boy with a diagnosis of DCD to investigate the effect of SR on visual dependence, visuo-motor temporal integration, and manual dexterity. SR was provided by vibrotactile noise stimulation (at an intensity of 60% of the sensory threshold) to the wrist. Changes in manual dexterity (during the SR on- and off-conditions) were measured using the manual dexterity test of the Movement Assessment Battery for Children-2nd edition. The point of subjective equality measured by visual or tactile temporal order judgment task served as a quantitative indicator reflecting specific sensory dependence. The delay detection threshold and steepness of delay detection probability curve, which were measured using the delayed visual feedback detection task, were used as quantitative indicators of visuo-motor temporal integration. The results demonstrated alleviated visual dependence and improved visuo-motor temporal integration during the SR on-conditions rather than the SR off-conditions. Most importantly, manual dexterity during the SR on-conditions was significantly improved compared to that during the SR off-conditions. Thus, the present results highlighted that SR could contribute to improving poor movement in children with DCD. However, since this was a single case study, a future interventional study with a large sample size is needed to determine the effectiveness of SR for children with DCD.

Keywords: delayed visual feedback detection task, DCD, manual dexterity, sensory-dependence, stochastic resonance (SR), temporal order judgment (TOJ) task, vibrotactile noise stimulation, visuo-motor temporal integration

### INTRODUCTION

Developmental coordination disorder (DCD), which is characterized by clumsiness in fine and gross motor skills, affects ∼6% of school-aged children, making it the most common childhood movement disorder (1–4). Children with DCD have lower ability to effectively use tactile information for movement, instead relying on visual information. Several studies have shown that increased visual dependence in children with DCD has a negative impact on the success of motor tasks (5–12). In addition, children with DCD have deficits in sensory-motor integration. Many previous studies have shown that deficits in sensory-motor integration have been linked to clumsy movements (13–23). In the current case study, we focused on the manual dexterity of a child with DCD. Recent research and review articles have shown that the clumsiness of manual dexterity in individuals with DCD is associated with the reduced activity of the premotor cortex and inferior parietal lobe, i.e., the frontal-parietal network (24–27). Therefore, it is suggested that the effective activation of the frontal-parietal network may improve manual dexterity in DCD.

On the other hand, sensory subthreshold mechanical noise stimulation to the body is known to improve the sensory-motor system. This improvement is related to stochastic resonance (SR), a phenomenon described as a "noise benefit" to various sensory and motor systems (28). SR application has been shown to improve the sensitivity of the visual (29), auditory (30), vestibular (31), and tactile (32–36) sensory systems. In addition, previous studies have demonstrated the immediate improvement in posture balance, walking, and hand movements following the application of SR (32, 34, 35, 37–41). These improvements were observed not only in healthy participants but also in older adults and patients with diabetes, stroke, and Parkinson's disease, and children with cerebral palsy (32–34, 38–40, 42, 43).

Vibrotactile noise stimulation to the wrist at an intensity of 60% of the sensory threshold generates SR phenomena in the hand, which in turn improve the tactile sensitivity of the fingertips and manual dexterity (34, 36, 40, 41). This improvement is thought to be caused by SR acting on the peripheral and central nervous systems. Vibrotactile noise stimulation can also enhance sensory sensitivity by directly stimulating peripheral sensory receptors (35). In addition, vibrotactile noise stimulation increases cortical and spinal neuronal activity (44–46). Importantly, this increase is not only limited to the sensorimotor cortex but also extends to the premotor and posterior parietal cortices (46, 47), which are important for tactile sensitivity (46), visuo-motor temporal integration (48), and manual dexterity (24–27). Further, studies have shown that vibrotactile noise increases the synchronization of neuronal firing between the spinal cord and sensorimotor cortex and between different brain areas (44, 45, 49–51). This increased neural synchronization can facilitate neural communication for perception between spinal and cortical levels (49, 52). Therefore, the application of SR to children with DCD may improve the clumsiness of movements; however, this has not yet been verified.

In the current case study, we hypothesized that the application of vibrotactile noise stimulation to the wrist with an intensity of 60% of the sensory threshold in children with DCD could reduce visual dependence by enhancing tactile sensitivity, promoting visuo-motor temporal integration, and improving poor manual dexterity. To verify this hypothesis, we applied SR to a 10-yearold boy with DCD and measured changes in manual dexterity, sensory dependence, and visuo-motor temporal integration.

### MATERIALS AND METHODS

### Case

A 10-year-old boy was examined by a neuro-pediatrician specialist 1 year before the current study and was diagnosed with DCD according to the Diagnostic and Statistical Manual of Mental Disorders 5th edition (DSM-5) (1). The boy had no other diagnosis of a general medical condition (e.g., cerebral palsy, hemiplegia, and muscular dystrophy), other developmental disorder (e.g., autism spectrum disorder, attention deficit hyperactivity disorder, and learning disorder), or intellectual disability. The experimental procedures were approved by the local ethics committee of the Graduate School and Faculty of Health Sciences at Kio University (approval number: 15–33). There were no foreseeable risks to the patient. No personal identification information was collected. We explained the study to the patient and his parents. The patient and his parents provided written informed consent for participation in this study and publication of this study. The procedures complied with the ethical standards of the 1964 Declaration of Helsinki regarding the treatment of human participants in research.

The boy's motor function and depression tendency were evaluated using the Movement Assessment Battery for Children-2nd edition (M-ABC-2) (53) and Depression Self-Rating Scale for Children (DSRS-C) (54), respectively, 1 day before carrying out the current study (**Table 1**).

The patient's parents also completed the Japanese version of the Developmental Coordination Disorder Questionnaire (DCDQ) (55), Social Communication Questionnaire (SCQ)


*M-ABC2, Movement Assessment Battery for Children-2nd Edition; DCDQ, Developmental Coordination Disorder Questionnaire; SCQ, Social Communication Questionnaire; ADHD-RS, Attention-Deficit Hyperactivity Disorder Rating Scale; DSRS-C, Depression Self-Rating Scale for Children; PSE, point of subjective equality; DDT, delay detection threshold: Steepness, steepness of the probability curve for delay detection.*

(56), and Attention-Deficit Hyperactivity Disorder Rating Scale (ADHD-RS) (57), 1 day prior to conducting the current study to evaluate the patient's motor function (55), autism spectrum disorder (ASD) traits (56), and ADHD traits (57), respectively (**Table 1**). In addition, the patient performed temporal order judgment (TOJ) and delayed visual feedback detection tasks to evaluate sensorydependent tendency and visuo-motor temporal integration, respectively (**Table 1**).

M-ABC-2 is an international standard evaluation battery for evaluating DCD diagnostic criteria A of DSM-5 (53) and DCDQ is a parent's rating scale for evaluating DCD diagnostic criterion B (55). In order to satisfy the DCD diagnostic criteria A of DSM-5, it is recommended that it be less than the 16th percentile as measured by M-ABC-2. The Japanese version of M-ABC-2, which is now being developed (58), has not been standardized. Thus, the original UK data were used when raw scores were converted to a standardized score or percentile. In order to satisfy the DCD diagnostic criterion B of DSM-5, it is recommended that it is 57 points or less as measured by DCDQ. The patient was in the 9th percentile of the M-ABC-2 and had 29 points according to the DCDQ; thus, he was diagnosed with DCD. The score of SCQ was nine points, ASD traits were low. The percentile of the ADHD-RS was 88th percentile for the inattention item, 84th percentile for the hyperactivity-impulsivity item, and 87th percentile for the total. The score of DSRS-C was three points, and no depression tendency was observed. He was not receiving any ongoing habilitation or medication therapy at the time of participating in the current study.

### Procedures

**Figure 1A** outlines the block design of the experimental protocol. There were three blocks each of the SR on-condition and SR offcondition (in order of SR on-off-on-off-on-off), with six blocks in total. Blocks 1, 3, and 5 were the SR on-condition, while blocks 2, 4, and 6 were the SR off-condition. This order was designed to offset the learning effects of repeating the test. Each block contained two manual dexterity tests, with a total of 12 manual dexterity tests performed throughout the study. That is, a total of 12 manual dexterity tests were performed six times each under the SR on-condition (Blocks 1, 3, and 5) and SR-off condition (Blocks 2, 4, and 6). The temporal order judgment task and delayed visual feedback detection task was administered once each during Block 1 and 5 (first and last SR on-condition) and Block 2 and 6 (first and last SR off-condition), respectively. This design was intended to reduce the burden on the patient.

### Stochastic Resonance

Vibrotactile noise was applied using four compact devices (vertical, 10 mm; width, 18 mm; height, 2 mm; Vibration Actuator Sprinter α; Nidec Seimitsu, Nagano, Japan) attached to the volar and dorsal areas of the child's right and left wrists, respectively, using contact tape (i.e., two devices on the right wrist and two devices on the left wrist). The resonance frequency of the device was 170 ± 10 Hz (average ± SD); low-pass filters at 500 Hz were used as per previous studies (34, 36, 40, 41, 46). A digital amplifier (FX Audio D802; North Flat Japan, Osaka, Japan) was used to output the white noise signals to the SR device (a vibrotactile noise device). Consistent with previous protocols (34, 36, 40, 41, 46), we attached the device to the wrist to minimize manual interruption while affecting the tactile sensation of the fingers. The intensity of the vibrotactile noise was set to 60% of the sensory threshold at the start of the test—the optimum level to affect the sensory system (33, 34, 36, 40, 41, 46). The sensory thresholds of the vibrotactile noise were measured immediately before starting each of the six blocks, irrespective of

in the hole of the vibration box and contacted the vibrotactile stimulator. Therefore, the case could not observe the tactile stimulus. The child was requested to watch the LED panel. The TOJ task device was to set conditions for the synchronized presentation of the visual and tactile stimuli, and the presentation of the visual stimulus at 50 and 100 ms earlier than the tactile stimulus (or vice versa). In addition, the setup included a blackout curtain so that the child would not be able to see outside the experimental chamber. (C) Delayed visual feedback detection task. The child's right hand was placed under a two-way mirror so that he was unable to see his right hand directly. The image of the hand, which was reflected in the two-sided mirror, was filmed with a video camera (FDR-AXP35, Sony, Tokyo, Japan). The movie of the photographed hand was further reflected from an installed monitor (LMD-A240, Sony, Tokyo, Japan) onto the two-sided mirror via a video delay-inserting device (EDS-3306, FOR-A YEM ELETEX, Tokyo, Japan). Thus, the child observed the delayed image of his right hand reflected in the mirror at the position where his right hand would be. In addition, the setup included a blackout curtain so that the child would not be able to see outside the experimental chamber. The intrinsic delay of the visual feedback in this experimental setting was 33.7 ms as measured by a time lag check device (EDD-5200, FOR-A YEM ELETEX, Tokyo, Japan).

whether it was an SR on- or off-condition. The vibrotactile noise device was attached at all times during testing and was turned on or off at the beginning of each block according to the SR on-/offconditions used. The patient was blinded to the condition as he could not feel the noise vibrations.

## Manual Dexterity Test

The manual dexterity test of the M-ABC-2 is a standardized, ageadjusted test to evaluate the DCD diagnostic criteria A of DSM-5 (53). Since the patient was 10 years old, we conducted three subtests of age band-2 to evaluate manual dexterity; placing pegs test (Manual dexterity 1), threading lace test (Manual dexterity 2), and drawing trail II test (Manual dexterity 3). The patient was wearing vibrotactile noise devices on the right and left wrists during this test. This test was conducted twice in each block (Blocks 1, 3, and 5 as the SR on-conditions, and Blocks 2, 4, and 6 as the SR off-conditions), with a total of 12 tests conducted throughout the whole experiment. The component score, standard score, and percentile were then calculated from the obtained raw scores. An increase in the component score, standard score, and percentile represented an improvement in manual dexterity. This assessment was administered by a specifically trained, certified physical therapist.

### Temporal Order Judgment Task

Sensory dependence was measured using the temporal order judgment (TOJ) task (59–63) (**Figure 1B**), where two stimuli (visual-flashes; tactile-vibrations) were presented in several stimulus onset asynchronies (SOA). The child was then required to determine which stimulus (visual or tactile) was presented first. This visuo-tactile TOJ task was carried out using a TOJ task device (Keio method, UT-0021, Medical Try System, Tokyo, Japan). Visual stimulation was elicited by a green LED in an LED panel (UT-0021-2, Medical Try System, Tokyo, Japan). The luminance of the visual stimulus was 40 cd/m<sup>2</sup> and the duration of visual stimulation was 1 ms. A 1-ms tactile stimulus (converted to vibration by pneumatic pressure) was administered to the right index finger controlled by a 1-V signal from the vibration box (UT-0021-1, Medical Try System, Tokyo, Japan). The stimulation condition included the following five conditions: (at −100, −50, 0, 50, 100 ms), i.e., four conditions where visual or tactile stimulation was administered 50 or 100 ms earlier than the other (i.e., tactile first, −100, −50 ms; visual first, 50, 100 ms), and a synchronous condition of visual and tactile stimulation (0 ms). During each block, the five stimulation conditions were considered a set and the child performed five sets; the trial order was randomized. Therefore, the child completed 100 trials with four blocks. This task was performed with the child's right hand attached to the vibrotactile noise devices.

Before starting the TOJ task, simple stimulus tests were used to confirm that the patient had no problems with vision and touch. First, the visual stimulus was not presented, and only the tactile stimulus was given five times to determine if there was a problem with tactile input. Subsequently, no tactile stimulus was presented, and only five visual stimuli were given to confirm whether there was a problem with visual input. The visual and tactile stimuli used for confirmation were the same as the stimuli used in the TOJ task. These simple stimulus tests confirmed that the patient was able to perceive tactile and visual stimuli.

For the TOJ task, the "visual first" response probability for each several SOA conditions (−100, −50, 0, 50, 100) was then calculated. Logistic curves were fitted to the "visual first" response probability in the TOJ task on the basis of following formula (23, 64, 65).

$$P(t) = \frac{1}{1 + \exp(-a(t - t\_{\rm{PSE}}))}$$

where t is the SOA; P(t) is the probability of "visual first" response; a indicates the steepness of the fitted curve; and tPSE indicates the observer's point of subjective equality (PSE), which demonstrates the SOA where "visual first" and "tactile first" judgment probabilities are equal (50%). Data were fitted using a non-linear least squares algorithm in MATLAB R2014b (MathWorks, MA, USA). Further, the PSE of each of the four blocks including the SR on-conditions (Blocks 1 and 5) and SR off-conditions (Blocks 2 and 6) was calculated. The PSE was a sensory-dependent quantitative indicator, where a large negative PSE value showed visual dependence and a large positive PSE value showed tactile dependence. Therefore, a PSE value approaching 0 ms demonstrated no biased sensory dependence. As baseline data, this TOJ task was also conducted 1 day before the current study, with the SR devices not attached (**Table 1**).

### Delayed Visual Feedback Detection Task

The delayed visual feedback detection task was carried out using the same setting as previous studies (23, 65–67) (**Figure 1C**). The child performed the task with his right hand, which was connected to the vibrotactile noise devices. After the experimenter had informed him orally that the trial had started, the child opened and closed his right hand once in a continuous and smooth manner, according to his own volition. The selfgenerated movements were observed under the following 18 delay conditions using a video delay-inserting device: 33, 67, 100, 133, 167, 200, 233, 267, 300, 333, 367, 400, 433, 467, 500, 533, 567, and 600 ms. The child had to determine if the visual feedback was synchronous or asynchronous relative to the movement of his right hand. Immediately following the trial, the child had to state orally if the visual feedback was "delayed" or "not delayed" by using the forced-choice method. In each block, all 18 delay conditions were treated as one set; their presentation order was randomized. Four sets were performed in total. The task was carried out once during each of the first and last SR on-block (Blocks 1 and 5) and SR off-block (Blocks 2 and 6), respectively; a total of four tests were carried out. Therefore, the child completed a total of 72 randomized trials with 18 delay conditions per set of four per block. Since there were four blocks in total, with or without SR, a total of 288 randomized tests were completed. Before the task, we confirmed that the patient could distinguish between a minimum delay of 33 ms and a maximum delay of 600 ms. That is, before the task, he reported "not delayed" for the minimum delay of 33 ms and reported "delayed" for the maximum delay of 600 ms.

The delay detection threshold (DDT) and steepness of the probability curve for delay detection, which will be referred to herein as "steepness," were determined using this task. Shortened DDT and/or increased steepness represented high visuomotor temporal integration, while prolonged DDT and/or decreased steepness represented poor visuomotor temporal integration. A logistic curve was fitted to the child's response on the visual feedback delay detection task, using the following formula (23, 64, 65):

$$P(t) = \frac{1}{1 + \exp(-a(t - DDT))}$$

where t was the visual feedback delay length (independent variable); P(t) was the probability of delay detection (observed value); a was the steepness of the fitted curve; and DDT was the observer's DDT representing the delay length at which the probability of delay detection was 50%. The curve was fitted using a non-linear least squares method (a trust-region algorithm) with MATLAB R2014b (MathWorks, Inc., Natick, MA, USA) to estimate a and DDT. DDT and the steepness of each of the four blocks including the SR on-conditions (Blocks 1 and 5) and SR off-conditions (Blocks 2 and 6) were calculated. As baseline data, this task was also conducted 1 day before the current study, with the SR devices not attached (**Table 1**).

### Statistical Analysis

The results of the manual dexterity test under the SR onconditions (a total of six test results of twice each in Blocks 1, 3, and 5) and the results of the manual dexterity test under the SR off-conditions (a total of six test results of twice each in Blocks 2, 4, and 6) were compared. Manual dexterity test scores (component score, standard score, percentile) were compared using the Wilcoxon signed-rank test, since they were not normally distributed by the Shapiro-Wilk test. In addition, the effect size was calculated (68). The significance level was set at P < 0.05. All statistical analyses were performed using SPSS ver. 24 (SPSS, Chicago, IL, USA).

### RESULTS

**Table 2** outlines the measurement results of each index of each block. **Figure 2A** shows a comparison of the manual dexterity test scores (component score, standard score, and percentile) of the SR on-conditions (a total of six test results of twice each in Blocks 1, 3, and 5) and the SR off-conditions (a total of six test results of twice each in Blocks 2, 4, and 6). The manual dexterity test scores were higher during the SR on-conditions compared with the SR off-conditions (component score, z = −2.207, P = 0.027, effect size (r) = −2.21; standard score, z = −2.214, P = 0.027, effect size (r) = −2.21; percentile, z = −2.207, P = 0.027, effect size (r) = −2.21; **Figure 2A**).

**Figure 2B** shows the "visual first" response probability curves of the SR on-conditions (average of two results in Blocks 1 and 5) and the SR off-conditions (average of two results in Blocks 2 and 6). On average, the PSE of the SR off-condition was −16.092 ms, whereas the average PSE of the SR on-condition was −0.096 ms (**Table 2**; **Figure 2B**). Therefore, the PSE of the SR on-condition approached 0 ms as compared with the SR off-condition, which showed a reduction of visual dependence (**Table 2**; **Figure 2B**).

**Figure 2C** shows the delay detection probability curves of the SR on-conditions (average of two results in Blocks 1 and 5) and SR off-conditions (average of two results in Blocks 2 and 6). DDT and steepness of the SR on-condition were 219.4 ms and 0.049 on average, respectively, whereas DDT and steepness of the SR off-condition were 272.9 ms and 0.028 on average, respectively (**Table 2**; **Figure 2C**). Thus, DDT and steepness of the SR on-condition shortened and increased, respectively, as compared with the SR off-condition, which in turn indicated the improvement of visuo-motor temporal integration during the SR on-condition (**Table 2**; **Figure 2C**).

### DISCUSSION

The present results showed that in this one case of DCD, manual dexterity under the SR-on conditions significantly improved immediately, compared with the SR off-conditions. Generally, children with DCD have visual dependence (5–12). In the current case, the PSE of the TOJ task on the day before the experiment was −24.77 ms (**Table 1**) and the average PSE under the SR off-conditions was −16.092 ms (**Table 2**; **Figure 2B**), which indicated visual dependency. However, the average PSE of the TOJ task under the SR on condition was −0.096, indicating a mitigation of visual dependence (**Table 2**; **Figure 2B**). Tactile sensation of the hand is a prerequisite for manual dexterity such as object grasping, object manipulation, and handwriting (69– 72). Previous studies showed that vibrotactile noise stimulation to the wrist with an intensity of 60% of the sensory threshold improves fingertip tactility and manual dexterity in the affected limbs of patients with stroke (32, 34, 40). Therefore, the improvement of manual dexterity under the SR on-conditions in the current case may have been due to the improvement of tactile sensitivity in the child's hand, which is important for manual dexterity, and the accompanying relief of visual dependency.

In addition, visuo-motor temporal integration is a very important function for manual dexterity (23, 66). In the current case, the DDT and steepness of the delayed visual feedback


*SR (+), stochastic resonance on-condition; SR (–), stochastic resonance off-condition; MD 1, manual dexterity test one (placing pegs test); MD 2, manual dexterity test two (threading lace test); MD 3, manual dexterity test three (drawing trail II test); PSE, point of subjective equality; DDT, delay detection threshold: Steepness, steepness of the probability curve for delay detection.*

FIGURE 2 | Results of the manual dexterity test and experimental tasks under the SR on-and SR off-conditions. (A) Comparison results of manual dexterity test scores between the SR on- and SR off-conditions. SR (+), stochastic resonance on-condition; SR (–), stochastic resonance off-condition; Blue box, SR on-condition; Pink box, SR off-condition. \**P* < 0.05. Lines represent the range of the minimum and maximum. Boxes represent the lower, median, and upper quartiles. (B) The "visual first" response probability curves of the SR on- and the SR off-conditions in the TOJ tasks. Blue curve, SR on-condition; Pink curve, SR off-condition. (C) Delay detection probability curves of the delayed visual feedback detection tasks in the SR on- and off-conditions. Blue curve, SR on-condition; Pink curve, SR off-condition.

detection task on the day before the experiment were 275.7 ms and 0.0267, respectively (**Table 1**), and the average DDT and steepness under the SR off-conditions were 272.9 ms and 0.028, respectively (**Table 2**; **Figure 2C**). In contrast, the average DDT and steepness under the SR on-conditions were 219.4 ms and 0.049, respectively (**Table 2**; **Figure 2C**). This suggested that the improvement of visuo-motor temporal integration under the SR on-conditions, which, in addition to the reduction of visual dependency, could have contributed to the improvement of manual dexterity following imperceptible vibrotactile noise stimulation in the current case. Therefore, it is possible to hypothesize that the improvement of manual dexterity by SR in the current case was because SR reduced visual dependency and promoted visuo-motor temporal integration.

We did not measure the patient's brain activity; therefore, although the following is completely speculative, the results observed in the current case may have been brought about by the effects of SR on the activity of the central nervous system. Seo et al. (46, 47) demonstrated that imperceptible vibrotactile noise on the wrist increases not only sensorimotor cortex activity but also the activity of the premotor and parietal cortices, which are responsible for tactile sensitivity (46), visuo-motor temporal integration (48), and manual dexterity (24–27). Thus, the positive effects observed in the current case may have been due to activation of the frontal-parietal network in addition to activation of the sensorimotor cortex.

The current case study has several limitations that should be noted. The data could not be analyzed statistically since only a few sensory dependence (TOJ task) and visuo-motor temporal integration (delayed visual feedback detection task) measurements were acquired during the SR on- and offconditions. Therefore, we cannot conclude that the reason for the significant improvement of manual dexterity under the SR onconditions shown in the current case was via an improvement of visual dependence and visuo-motor temporal integration. In addition, this was a single case study; thus, future interventional studies with a large sample size are needed to determine the effectiveness of SR for children with DCD. Furthermore, the verification of retention effects after the end of SR administration is also required. In the current study, the SR on- and offconditions were performed alternately, but the effects obtained under the SR on-condition disappeared under the next offcondition. Therefore, there may be no retention effects after removing SR devices. Thus, future studies designed to investigate retention effects after removing SR devices are also needed. The advantage of the SR phenomenon is that children only wear the devices, the stimulation is below the detection threshold, and children do not need special efforts to use the devices. The combined use of SR with highly effective interventions (73), such as the cognitive orientation to daily occupational performance approach and neuromotor task training, may provide additional benefits to children with DCD.

### DATA AVAILABILITY

The raw data supporting the conclusions of this manuscript will be made available by the authors, without undue reservation, to any qualified researcher.

### REFERENCES


### ETHICS STATEMENT

The experimental procedures were approved by the local ethics committee of the Graduate School and Faculty of Health Sciences at Kio University. There were no foreseeable risks to the case; no personally identifying information was collected. The child and parents provided background information and written informed consent. The procedures complied with the ethical standards of the 1964 Declaration of Helsinki regarding the treatment of human participants in research.

### AUTHOR CONTRIBUTIONS

SN designed the study, collected and analyzed the data, and wrote the manuscript. MO, TM, SS, and AN provided experimental equipment and evaluation battery and helped with data analyses. AM, EF, and SM supervised the study. All authors read and approved the manuscript.

### FUNDING

This work was supported by JSPS KAKENHI, Grant-in-Aid for Young Scientists (Grant Numbers 16K16453, 18K17700), JSPS KAKENHI, Grant-in-Aid for Scientific Research (C) (Grant Number 16K09981), and JSPS KAKENHI, Grantin-Aid for Scientific Research on Innovative Areas (Grant Number 17H05915).

### ACKNOWLEDGMENTS

The authors wish to acknowledge and thank the case and parents who participated in this study.

touch information for quiet standing. Gait Posture. (2011) 34: 183–90. doi: 10.1016/j.gaitpost.2011.04.007


patients with diabetic neuropathy. Arch Phys Med Rehabil. (2002) 83:171–6. doi: 10.1053/apmr.2002.28025


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Nobusako, Osumi, Matsuo, Furukawa, Maeda, Shimada, Nakai and Morioka. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Recent Technology-Aided Programs to Support Adaptive Responses, Functional Activities, and Leisure and Communication in People With Significant Disabilities

Giulio E. Lancioni <sup>1</sup> \*, Marta Olivetti Belardinelli <sup>2</sup> , Nirbhay N. Singh<sup>3</sup> , Mark F. O'Reilly <sup>4</sup> , Jeff Sigafoos <sup>5</sup> and Gloria Alberti <sup>6</sup>

*<sup>1</sup> Department of Neuroscience and Sense Organs, University of Bari, Bari, Italy, <sup>2</sup> Interuniversity Center for Research on Cognitive Processing in Natural and Artificial Systems (ECONA), Sapienza University of Rome, Rome, Italy, <sup>3</sup> Medical College of Georgia, Augusta University, Augusta, GA, United States, <sup>4</sup> Department of Special Education, University of Texas at Austin, Austin, TX, United States, <sup>5</sup> School of Education, Victoria University of Wellington, Wellington, New Zealand, <sup>6</sup> Lega F. Doro Research Center, Osimo, Italy*

#### Edited by:

*Giulia Galli, Fondazione Santa Lucia (IRCCS), Italy*

#### Reviewed by:

*Giorgio Scivoletto, Fondazione Santa Lucia (IRCCS), Italy Michael L. Wehmeyer, University of Kansas, United States*

> \*Correspondence: *Giulio E. Lancioni giulio.lancioni@uniba.it*

#### Specialty section:

*This article was submitted to Neurorehabilitation, a section of the journal Frontiers in Neurology*

Received: *17 April 2019* Accepted: *31 May 2019* Published: *02 July 2019*

#### Citation:

*Lancioni GE, Olivetti Belardinelli M, Singh NN, O'Reilly MF, Sigafoos J and Alberti G (2019) Recent Technology-Aided Programs to Support Adaptive Responses, Functional Activities, and Leisure and Communication in People With Significant Disabilities. Front. Neurol. 10:643. doi: 10.3389/fneur.2019.00643* This paper presents an overview of recent technology-aided programs (i. e., technology-aided support tools) designed to help people with significant disabilities (a) engage in adaptive responses, functional activities, and leisure and communication, and thus (b) interact with their physical and social environment and improve their performance/achievement. In order to illustrate the support tools, the paper provides an overview of recent studies aimed at developing and assessing those tools. The paper also examines the tools' accessibility and usability, and comments on possible ways of modifying and advancing them to improve their impact. The tools taken into consideration concern, among others, (a) microswitches linked to computer systems, and aimed at promoting (i.e., through positive stimulation) minimal responses or functional body movements in individuals with intellectual disabilities and motor impairments; (b) computer systems, tablets, or smartphones aimed at supporting functional activity engagement of individuals with intellectual disabilities or Alzheimer's disease; and (c) microswitches with computer-aided systems, elaborate communication devices, and specifically arranged smartphones or tablets, directed at promoting leisure, communication, or both.

Keywords: technology-aided programs, support tools, disabilities, cognition, adaptive responses, functional tasks, leisure, communication

## INTRODUCTION

Technology-aided programs are increasingly recognized as essential means for supporting people with significant disabilities (e.g., congenital intellectual, motor, or sensory impairments, and possible combinations of them, and neurodegenerative or post-traumatic disorders) within their daily contexts (1–5). Technology-aided programs are designed to build functional links between user, technology and environment, and thus ensure that a satisfactory (goal-directed) interaction with the environment is possible also for individuals with extensive levels of disabilities (6).

**76**

Such interaction is considered critical to promote the individual's general achievement, personal satisfaction (quality of life), social image, and cognition (2, 7–10).

Technology-aided programs for people with significant disabilities have largely focused on providing them with support in main problem areas, such as passivity and detachment, failure to carry out functional activities, and failure to engage in leisure and communication. The first area (i.e., passivity and detachment) is concerned with the inability to engage in simple responses/movements functional to interact with the environment and, possibly, carry out forms of physical activity with potential health benefits (4, 11–14). For example, people with pervasive motor impairments and intellectual disability (or consciousness disorders) tend to be passive and detached and thus fail to reach any control of environmental stimulation and improve their general alertness and awareness (12, 15). Similarly passive and detached may also be people with severe/profound intellectual disabilities who have less extensive (or no specific) motor impairments, as well as people who are affected by advanced Alzheimer's disease (4, 16, 17).

The second area (i.e., failure to carry out functional activities) is concerned with the inability to engage in complex, relevant tasks, such as vocational, domestic, and self-care tasks. For example, people with moderate intellectual disabilities frequently fail to independently perform vocational tasks because they cannot remember the steps and materials involved in those tasks (18, 19). The same people as well as people affected by mild/moderate Alzheimer's disease or acquired brain injury may be unable to independently perform relevant domestic tasks because they do not recall the time of the day when those tasks are due and/or the task steps (20–22).

The third area (i.e., failure to engage in leisure and communication) is concerned with the inability to manage leisure activities and communication interactions independently. For example, people with severe intellectual and developmental disabilities may be unable to start and engage in leisure activities on their own and thus remain dependent on staff or caregivers (23–25). The same people may also be unable to communicate their needs or desires, that is, to make clear requests to and have some basic interaction with communication partners (i.e., staff and family members) present in their immediate surrounding or distant from them (26–29). Serious leisure and communication problems also occur among people with neurodegenerative diseases (e.g., amyotrophic lateral sclerosis) and post-traumatic multiple disabilities (30–33).

Technology-aided programs set up to address the aforementioned problem areas are generally designed to function as support tools aimed at bridging the gap between the individual's actual skills and the skill level required to reach meaningful goals. Support tools can be expected to work effectively only if arranged in line with the individual's specific condition and the goal set for him or her. For example, if the individual's specific condition is severe intellectual disability and extensive motor impairment, support tools might be designed to help him or her carry out various forms of adaptive responses. In practice, support tools might help the individual to (a) make one or few small responses available in his or her repertoire effective to produce a relevant environmental change (i.e., a change that the response or responses per se would be inadequate to produce), or (b) use a functional body movement to reach relevant environmental events and, at the same time, manage some form of mild, potentially beneficial physical exercise (4, 12, 34).

If the individual's specific condition is moderate intellectual disability or mild to moderate Alzheimer's disease with inability to carry out basic/functional daily tasks, support tools might be designed to provide (a) time cues (i.e., reminders when the tasks are due), and step instructions (i.e., verbal or pictorial instructions concerning the material and responses required for each single step of the task to carry out) (20, 35, 36). Such tools would allow the individual to have a positive role in the environment and to gain social appreciation and respect (34).

If the individual's specific condition is moderate intellectual disability with sensory and/or motor impairments, or emergence from a minimally conscious state with motor and speech disabilities, support tools might be designed to promote leisure engagement, communication, or both. In the first case, the support tools would serve to present the individual with different leisure options and allow him or her to choose among them with simple responses (24, 37). In the second case, the support tools would serve to offer communication options (e.g., the opportunity to make verbal requests or send text messages) and allow the individual to choose among those options and activate them (29). In the third case, the support tools would serve to offer the individual leisure and communication options. The individual would then be allowed to choose between those types of options as well as among the alternatives available within each option (24, 38).

This paper is an effort to illustrate some of the support tools mentioned above (i.e., in relation to the three main problem areas) by providing an overview of recent studies aimed at developing and assessing those tools. Given the specific, and rather circumscribed, scope of the paper (i.e., as just indicated) the tools illustrated and the studies reported to describe their applicability and impact represent a non-systematically selected group of the tools and studies available. The selection was made by the authors based on two simple, intuitive criteria. The criterion used for selecting the tools was their perceived technical and practical relevance. Essentially, the tools selected were deemed to represent innovative (challenging) and effective resources for fostering favorable changes within the main problem areas identified above. The criterion used for selecting the studies concerned the fact that they represented recent evidence in the field (i.e., had been published within the last few years) and provided clear illustrations of the tools' technical aspects and support potential. In summary, the paper describes the characteristics and impact of a series of support tools (i.e., the technology components involved, the intervention strategies set up to make those technology components an integral part of the individuals' response/interaction with the environment, and the results obtained) with the help of studies reporting such tools. The paper is also providing comments on the tools' accessibility and usability, and possible ways of modifying and advancing them to improve their impact.

## SUPPORT TOOLS TO PROMOTE ADAPTIVE RESPONSES

As suggested above, adaptive responses may involve (a) small/minimal movements (e.g., small limb or head movements as well as finger, lip, and eyebrow movements) (14, 39) and (b) large and functional body movements (e.g., leg lifting and ambulation steps) (4, 17, 40). Minimal movements/responses are not considered suitable to allow the individual to have any impact on the environment or produce some form of physical exercise useful for his or her condition. Yet, the same responses may become critical to enable the individual to interact with the environment and control relevant events if adequate support tools are used within a suitable intervention process (13, 14, 41). Functional body movements can be instrumental to control environmental events and, at the same time, can constitute beneficial forms of physical exercise (4, 12, 17, 40). Making minimal responses or functional body movements instrumental to control environmental events is critical for two reasons. First, it may be the only way to enable the individual to reach those events and control their occurrence independently. Second, the occurrence of those events can motivate the individual to repeat the emission of the responses/movements over time and thus ensure consistent contact with the environment, arousal and engagement, and possibly relevant physical exercise (6, 40, 42–44).

The support tools required to help individuals develop and strengthen small responses or functional body movements and make them relevant consist of devices that can (a) monitor the emission of those responses and (b) ensure that they trigger the occurrence of environmental events preferred by the individuals (e.g., music, videos, and familiar voices addressing them). Six recent studies are summarized in this section of the paper to provide (a) illustrative examples of the support tools available for fostering small responses and functional body movements, and (b) an informative basis for commenting on those support tools (12, 16, 44–46). Two of the six studies were specifically directed at individuals who could only produce small responses (12, 45), while the other four studies were directed at individuals who could perform functional body movements (see **Table 1**).

## Small Responses

#### Studies

The first study dealing with small responses (12) focused on the development and assessment of a support tool relying on microswitches for four participants who were affected by a massive paralysis of their body and allegedly severe/profound intellectual disability, subsequent to congenital encephalopathy, or spinocerebellar ataxia. None of the participants had any form of communication or interaction with the environment. Two participants used lip movements as their response. The other two participants used prolonged eyelid closure and eyelid opening, respectively. The responses were monitored via an experimental optic microswitch involving an infrared light-emitting diode and a mini infrared light-detection unit fixed on the participant's face or via a camera-based microswitch placed in front of the participant. Each response activated the microswitch, which in turn triggered a computer that delivered 10 s of preferred stimulation (e.g., audio and video recordings with music and familiar voices). All participants showed a clear increase in response frequencies when the support tool was in use.

The second study (45) focused on the application and assessment of a microswitch-based support tool similar to that described above with 10 participants who were in a minimally conscious state and presented with extensive motor impairment and lack of speech or any other functional communication following brain injury and coma. Their responses included eyelid closures, and small head, hand/finger, foot, or lip movements. Eyelid closures and lip movements were detected via optic microswitches (such as the one mentioned for the previous study). Head and foot responses were detected through simple pressure microswitches. Hand or finger movements were detected through microswitches sensitive to touch and pressure. As in the study summarized above, each response activated the related microswitch and this triggered a computer, which delivered 10 s of preferred stimulation. All participants showed meaningful increases in their responding during the intervention phases of the study in which the support tool was in use, thus showing improved levels of alertness, attention, and activation.

### Comments on the Support Tools

The support tools described above can be considered the most immediate instruments for enabling individuals with pervasive impairments to have self-determined contact with the environment, control their stimulation input, and improve their social image (2, 47). From a technical standpoint, it should be underlined that those tools mostly relied on the use of experimental or adapted microswitches capable of effectively and reliably monitoring the small responses that the participants could produce (41).

From a practical standpoint, two considerations are in order. First, these tools were not designed to be easily portable. Indeed, while the microswitches were readily wearable and portable, the computer to which they were connected was not necessarily easy to carry around. The lack of tools' portability is not deemed a real drawback in this context, given that the participants are confined in bed or in a wheelchair. The second consideration is that one may need to look at these tools as temporary solutions for a number of participants. For example, some participants with neurodegenerative diseases (e.g., amyotrophic lateral sclerosis) may lose the responses on which the tools initially relied and require the identification of new responses and new microswitches. Participants with acquired brain injury may improve their general level of functioning over time, and thus require more advanced support tools.

New research could be directed at (a) identifying additional microswitches that would allow the possibility of monitoring a larger variety of minimal responses with reliability and low intrusion, and (b) developing and assessing portable support tools. With regard to the latter objective, for example, one could conceive the use of smartphones for both monitoring specific responses (i.e., via the proximity and light sensors) and delivering stimulation contingent on the responses. The smartphones could be automated through available applications (e.g., MacroDroid) TABLE 1 | Studies using support tools to promote small responses and functional body movements.


so as to control the specificity and duration of the stimulation in relation to the responses (46).

Functional Body Movements

#### Studies

The four studies summarized in this section (16, 44, 46, 48) were aimed at individuals who could perform functional body movements and developed and assessed technology-aided support tools directed at promoting those movements. In particular, Shih and Chiu (48) set up a support tool to promote walking in place for two adolescents, who were affected by severe or mild intellectual disability and obesity, and tended to be largely passive. The technology used to detect the participants' step responses was a standard dance pad. The pad was connected to a computer device that could switch on and off specific videos considered to be preferred for the participants. In practice, the system turned on the videos when two sensor sections of the pad were activated in succession (i.e., as expected to occur during walking) and kept the videos on, as long as the sequential activation continued. Conversely, the system would not turn on, or would turn off the videos when the two sensor sections were simultaneously activated (i.e., as it occurs when an individual stands still). Data showed that the participants' levels of walking (i.e., step frequencies) increased drastically during the intervention when the support tool was in use.

Stasolla et al. (44) set up and assessed a support tool that was aimed at promoting assisted ambulation in two children, who were considered to function within the severe/profound intellectual disability range, had no speech abilities, and were performing only a few steps when provided with a walker device. The technology used to detect the participants' step responses consisted of a photocell, which was fixed onto the low lateral frame of the walker device and faced a reflecting panel. Any time the participant performed a step (moved the foot forward), the light beam produced by the photocell did not reach the reflecting panel and thus was not reflected back to the photocell. This lack of beam return triggered an electronic control device, which in turn activated 3 s of preferred stimulation (e.g., music and familiar voices). Data showed that the participants' step frequencies increased considerably during the intervention with the support tool and they were also accompanied by increases in indices of happiness.

Lancioni et al. (16) set up and assessed a support tool for promoting leg lifting movements in 11 participants with advanced Alzheimer's disease, who were sedentary and largely static particularly with regard to their lower limbs. The leg responses were monitored through one or two commercial tilt sensors/microswitches, which were linked to a computer device. Leg responses activated the microswitch(es), and in turn the computer, which delivered 10 s of preferred stimulation. The computer would provide a verbal encouragement to respond if the participant was passive for a preset period of time. Results showed that all participants had a clear increase in response frequency [thus, reaching a useful level of physical activity (49, 50)], which was accompanied by indices of positive participation/mood.

Lancioni et al. (46) set up and assessed a support tool aimed at fostering two functional responses (e.g., arm stretching to push a panel and leg-foot forward moving to push a box) for each of seven participants, who were characterized by severe or moderate/severe intellectual disability and extensive motor impairment confining them to a wheelchair. The technology used for monitoring the responses and providing stimulation contingent on their occurrences consisted of a smartphone device with Android operating system. The smartphone's functioning was regulated through the MacroDroid application, which ensured the recording of the responses and the delivery of 10 s of preferred stimulation for each response occurrence. With the use of the support tool, all participants showed large increases in response frequencies and indices of happiness. They also showed significant increases in heart rates, indicating that response performance represented a beneficial level of physical exercise (49, 51).

### Comments on the Support Tools

The support tools reported for promoting functional body movements (a) relied on four types of response sensors (i.e., dance pad, photocell, tilt microswitch, and smartphone), (b) served participants with different levels of cognitive and motor functioning, and (c) focused on different responses. From a technical standpoint, one can underline the fact that the sensors

on which the support tools relied were all commercially available and readily accessible in terms of costs. It may also be emphasized that while the dance pad, the photocell and the tilt device were connected to a computer system, which was to deliver stimulation, the smartphone had the dual function of sensor and stimulation device. Using a single instrument like a smartphone rather than two or more instruments (sensors and computers) would be seen as advantageous and could be easily viable in interventions such as those targeting footstep responses or leg movements.

From a practical standpoint, one can (a) reflect on the lack of portability of the support tools except the one relying on the smartphone, and at the same time (b) argue that portability might be a relatively marginal aspect given that the tools are typically employed in specific intervention settings rather than across settings. It might also be noted here that the advantage of using a single instrument like the smartphone should not be taken to suggest that its employment would be immediate (i.e., it would not require an appropriate preparation through suitable applications, such as the MacroDroid).

Future research may focus on extending the intervention to new functional movements, to new sensors, and eventually new support tools that would be easy to arrange in addition to being commercially available. For example, one might consider the possibility of targeting functional movements such as pulling oneself to a standing position and remaining in that position for brief periods of time (52) for people with congenital or acquired motor impairments with or without serious intellectual disability. Similarly interesting might be any initiative to foster those movements through support tools based on smartphones and thus commercially available and completely portable.

### SUPPORT TOOLS TO PROMOTE FUNCTIONAL TASKS

People with mild to severe intellectual disabilities with or without additional sensory or motor impairments, people with neurodegenerative diseases, in particular mild and moderate Alzheimer's disease, as well as people with acquired brain injury are often unable to carry out functional, multistep activities (21, 22, 35, 53). Indeed, they may fail to perform self-care sequences (e.g., morning routine), daily domestic tasks (e.g., preparing coffee, setting the table, or making a snack), and work tasks (e.g., assembling and packaging commercial products) (35, 54, 55). An increasingly popular approach to help these people concerns the use of technology-aided support tools providing step instructions and other types of assistance (56, 57). These tools can be defined as forms of cognitive prostheses bridging the gap between the people's skill level and the task demands (6, 58).

Six recent studies adopting a technology-based approach are summarized in this section (19, 21, 35, 59–61). The support tools used for helping the participants to perform the tasks varied on multiple aspects. Yet, the studies reported below are divided into two groups based on only one of those aspects, that is, on whether the support tools were non-portable (19, 60, 61), or portable (21, 35, 59) (see **Table 2**).

### Tasks With Non-portable Support Tools Studies

Mihailidis et al. (19) developed and tested a new technologyaided prompting/instruction tool to help individuals with intellectual and developmental disabilities carry out an assembly task. The core technology included a LCD touchscreen that provided the instructions verbally and visually (via pictures and videos), cameras for monitoring the participant's performance, an animated job coach providing encouragement and positive stimulation, and a complex software package. This technology system was able to determine the appropriate type of instruction to be presented to the participant based on his or her performance. The prompts/instructions could become more detailed (e.g., including extra visual elements), and plausibly helpful, based on the participant's difficulties. The timing of the instructions could also change based on the participant's progression. Whenever the technology system was unable to identify an instruction solution, a human job coach was alerted so that human supervision would be applied. The system was assessed in a pilot study with four adults with mild to moderate intellectual disability and a task including 18 steps. All adults showed an improvement in their task performance during the few intervention trials carried out.

Lin et al. (60) set up and assessed a technology-aided tool regulating video prompting to support activity skills in three adolescents with moderate intellectual disability. The technology involved two dance pads and two notebook computers. The notebooks were on two separate tables and the dance pads were placed before the tables. When the participant stood on the dance pad in front of the first table, the notebook on that table presented a video clip of the step response the participant was to perform (e.g., take the teacup) and accompanied the clip with the verbal instruction matching the video. When the participant moved to the second table and thus walked on the dance pad in front of it, the notebook on that table displayed the video of the step response the participant was to perform (e.g., put the teacup on the table, in the upper right corner) accompanying it with the matching verbal instruction. Then the participant was to continue to use the two tables with the two dance pads and notebooks until the task was completed. The task used for the three participants consisted of Chinese table setting and included 16 steps. With the support of the technology-aided tool, the participants managed to perform all 16 steps correctly with only sporadic exceptions.

O'Neill et al. (61) reported the set up and assessment of a micro-prompting device ("Guide") to support the morning routine of people with acquired brain injury. The Guide system involved a computer with voice tracker, speech recognition software, activity protocols and activity protocol player. In essence, the computer involved audio-verbal interactive microprompting that was to emulate the verbal prompts and questions normally delivered by staff. Twenty-four adults with acquired brain injury were recruited for the study and randomly assigned to either the experimental (technologyaided) group or the control group. The morning routine sequence included seven main steps (e.g., getting up, showering, shaving, and dressing) and was supported via a plurality of TABLE 2 | Studies using support tools to promote functional tasks.


step-related checks (questions) and instructions/prompts by the system. Data indicated that the experimental group required a significantly smaller number of direct staff interventions than the control group for carrying out and completing the morning routine accurately.

### Comments on the Non-portable Support Tools

The support tools described above were developed with the objective of providing high levels of supervision and/or specific forms of interaction (closely simulating staff interaction) so as to ensure high levels of correct performance. The tool developed by Mihailidis et al. (19), in particular, was also capable of reorganizing the instruction sequence and the amount and type of instruction guidance so as to lead the participant to complete the task in a satisfactory manner. The tools used by Mihailidis et al. (19) and by O'Neill et al. (61) were fairly complex in terms of design and components. A slightly different consideration can be made for the self-prompting tool reported by Lin et al. (60). In fact, while it involved two laptop computers and two dance pads with relative interfaces, its working was fairly simple and thus required minimal software arrangement.

From a practical standpoint, one can make at least two considerations. First, the tools developed by Mihailidis et al. (19) and O'Neill et al. (61) appear quite expensive and relatively difficult to manage [i.e., compared to the one reported by Lin et al. (60)], with a potentially reduced affordability for and applicability within many contexts. Second, all support tools were tested on a single task, although the components/steps of the task reported by O'Neill et al. (61) were rather large and composite. A narrow testing provides an evidence base that does not allow one to determine the versatility of the tools for other tasks.

New research might be focused on setting up and assessing simpler versions of the tools to find out whether one can still ensure significant levels of success with those simplified (more accessible) versions. Another research point could be the assessment of those new tools' versions over a number of relevant tasks so as to establish their overall applicability within rehabilitation and occupation/work contexts.

### Tasks With Portable Support Tools Studies

Cullen et al. (59) developed and assessed a technology-aided support tool that was to help three young adults manage a cleaning task and improve their performance of similar (generalization) tasks. The participants were diagnosed with autism or intellectual disability and visual impairment or traumatic brain injury, but their IQs were in the mild or above the mild intellectual disability range. The technology consisted of an iPad 4 with the MyPicTalk application allowing the use of video clips with voice-over for the single task steps. Prior to the start of the intervention on the target task (i.e., cleaning a table which included 12 steps), the participants were trained on how to use the iPad and related application so as to ensure that they would be ready for self-directed video prompting. All three participants showed a drastic performance improvement on the target task. They also showed variable levels of improvement on three generalization tasks.

Lancioni et al. (21) set up and assessed a support tool aimed at promoting the performance of daily tasks of eight participants with mild or moderate Alzheimer's disease. The tool reminded the participants of those tasks at the appropriate times and provided them with verbal instructions concerning the steps of those tasks. The technology included a tablet with Android operating system as well as a wireless Bluetooth earpiece, which allowed the participants to receive reminders and instructions without carrying the tablet. For each participant, 12 or 14 daily tasks, including means of 14 or 18 steps, were used. Six or seven tasks were scheduled per morning and/or per afternoon over periods of 2 or 3 h. When the time for a task was reached, the participant was reminded to start that task and thereafter he or she was presented with the instructions for it. With the use of the support tool, the participants managed to start virtually all the tasks scheduled independently, and reached high percentages of correct step performance.

Lancioni et al. (35) designed and assessed a support tool for promoting the performance of daily tasks of eight participants with mild/moderate or moderate intellectual disability and visual or hearing impairments. The technology consisted of a smartphone with Android operating system, which included standard functions and was fitted with the Easy Alarm YouTube application as well as with audio and video files for the single tasks. As soon as the time scheduled for a task was reached, the smartphone emitted a verbal reminder or a visual and vibratory reminder. The reminder was then followed by each of the step instructions (verbal or visual) arranged for the task. Ten to 12 tasks of 20–25 steps were available for each participant. Six tasks were scheduled per morning and/or per afternoon. With the use of the support tool, all participants managed to start the tasks at the appropriate times (following the reminders), and had high percentages of correct performance.

### Comments on the Portable Support Tools

From a technical standpoint, the aforementioned support tools, which were used for a single task (59), or a plurality of tasks (21, 35), may be considered easily accessible. In fact, the tablets or smartphones and the applications required to automate their functioning are commercially available and readily acquirable. The availability of devices and applications should not be interpreted, however, as if all those support tools were ready for use by staff and caregivers. In fact, automating a tablet or smartphone to provide reminders and step instructions for a variety of tasks spread over specific periods of time requires a certain amount of preparatory work as well as a level of technical competence.

From a practical standpoint, four considerations may be in order. First, those support tools are easily affordable in terms of costs and thus suitable for daily contexts. Second, their usability for a variety of tasks makes them highly helpful in supporting the participants over large parts of the day, with minimal external supervision and maximum impact on the participants' functional engagement/interaction with their physical and social environment. Third, the tools are suitable for presenting verbal instructions as well as visual instructions. This versatility makes them appropriate also for people who have hearing impairment and/or verbal comprehension problems. Fourth, participants who use only verbal reminders and instructions do not have to carry the smartphone or tablet with them. Indeed, they can receive the reminders and instructions through wireless Bluetooth earpieces linked to those devices (21).

A primary objective of new research may be to gather additional evidence on the suitability and effectiveness of the aforementioned tools in (a) reminding participants with intellectual disabilities, Alzheimer's disease, and acquired brain damage about their daily tasks, and (b) supporting their performance of those tasks through verbal or visual instructions. Another research goal might be that of comparing the effectiveness of the tools' different instruction options (i.e., verbal, visual through static pictures, and visual through video prompts/clips) (62, 63).

## SUPPORT TOOLS TO PROMOTE LEISURE, COMMUNICATION, OR BOTH

One common trait of many people with intellectual and other disabilities, neurodegenerative diseases, and acquired brain injury is the inability to engage in leisure activities independently (23, 37, 64, 65). This apparent inability may be largely due to difficulties in reaching and operating devices typically used to access leisure activities (e.g., television, computer, and music instruments). Another common trait concerns communication problems (29, 66–70). These problems may be characterized by the people's inability to (a) express their requests for caregivers and staff's attention or make other types of requests, and/or (b) reach relevant partners (e.g., preferred family or staff members) not immediately available in their context. The aforementioned problems may be related to lack of speech and alternative communication options and/or inability to use telephones or other devices to interact with distant partners (2, 37, 70–72). Studies have typically addressed one of the problems (i.e., either the leisure or the communication problem) at a time (1, 24, 29, 37, 73, 74). Recently, some studies have also been reported, which have addressed both problems within the same intervention program, thus allowing the participants to freely switch between the two types of engagement (32, 72).

The eight studies summarized in this section represent illustrative examples of interventions with support tools relevant for this area (see **Table 3**). Specifically, the first two studies addressed the leisure problem (75, 76); the following two studies addressed the communication problem (1, 77); and the final four studies targeted both problems within the same intervention context (32, 72, 78, 79).

## Leisure

### Studies

Stasolla and De Pace (76) reported the use of a support tool with two participants who had emerged from a minimally conscious state, and presented with extensive motor impairment, and lack of communication and interaction with the environment. The technology included a computer connected to a microswitch (i.e., a touch-sensitive device) through a specific interface. The computer would present visually and verbally different stimuli considered to be preferred for the participants (e.g., songs and mother's voice). The stimuli were presented in sequence and the participants could choose the one they wanted to access through microswitch activation. The selection of a stimulus led to the computer's presentation of several variations of that stimulus so that the participants could be more specific in their final choices and access what they most preferred at the time. During the intervention with the support tool, both participants managed to make choices thus accessing their preferred stimuli independently.

Lancioni et al. (75) set up and assessed a support tool, which technically resembled that used by Stasolla and De Pace (76), to enable 11 participants with mild or moderate Alzheimer's disease to engage in leisure activities independently. The participants did not have the ability to operate a computer or other device for leisure engagement. Yet, they discriminated verbal TABLE 3 | Studies using support tools to promote leisure, communication, or both.


questions/instructions and visual images concerning preferred people and events, and were capable of activating a pressure microswitch for operating their choices. The technology (a) included a laptop computer with screen and sound amplifier, which was linked to a pressure microswitch, and (b) allowed the participants to choose among music, comedy, films, and television shows. The participant could choose any option by activating the microswitch when that option was highlighted. After choosing an option, the participant was presented with a variety of stimuli connected to it, so he or she could select the one to access. During the intervention (i.e., when the support tool was available), all participants displayed successful choice performance, thus accessing a variety of stimuli and remaining positively engaged throughout most of the time allotted.

### Comments on the Support Tools

Individuals whose condition (e.g., intellectual and motor disabilities or Alzheimer's disease) precludes them from reaching/accessing preferred stimuli and engaging with them freely need the help of a support tool to bypass those limitations. The tools described above were aimed at supporting individuals with traumatic brain damage and individuals with Alzheimer's disease. Similar tools have also been used with people with intellectual and multiple disabilities, with amyotrophic lateral sclerosis or various forms of brain injury (37, 80). From a technical standpoint, the aforementioned tools can be considered relatively simple as they included (a) a computer presenting the stimuli available for choice and delivering the stimuli that the individuals eventually chose, and (b) a microswitch connected to the computer that allowed the participants to carry out their choice responses. Although rather simple, those tools are not readily accessible/available and need to be prepared for the single participants so as to respond to their preferences. It is also noteworthy that experimental microswitches may be needed for participants whose motor repertoire is very poor (80).

From a practical standpoint, one can consider the tools fairly friendly for the participants, manageable for staff and caregivers, and reasonable in terms of costs. A possible question as to their accessibility for daily contexts may arise whenever an experimental microswitch is required. The fact that those tools are not always easy to carry around may not be a serious drawback given that most of the individuals using those tools are confined to specific contexts, and thus do not need to carry the tools across settings (2, 24).

New research could focus on arranging and testing new technology packages so as to have a range of different solutions to address the needs of individuals with different characteristics. It is also important to recognize that support tools might be more valuable if they do not only allow leisure engagement, but also provide the conditions for communication (see below) (72, 79). Tools supporting different forms of engagement would promote performance variability and thus ensure longer periods of profitable occupation.

### Communication

### Studies

The main body of intervention studies concerning communication have addressed the participants' inability to make (verbal) requests and assessed the impact of tools such as speech-generating devices (e.g., iPods and iPads) (26, 28, 67, 68). The two studies summarized below (1, 77) seem to add to the main body of the literature available in terms of technology used and/or participants involved. In particular, Simacek et al. (77) arranged two types of communication support tools to foster request making in two participants with Rett syndrome. The older participant had a diagnosis of atypical Rett syndrome and was able to ambulate, although with assistance, and could use her hands to press, grasp, hold, and release objects. The younger participant had a diagnosis of typical Rett syndrome, spent most of her time in a sitting position, and did not manipulate objects. She was reported, however, to gaze at objects she presumably wanted. The technology consisted of (a) a speech-generating device that could be activated by touching the image of the object requested (older participant) or (b) an eye-tracking device that could be activated by looking at the object requested for a prefixed amount of time (younger participant). Both participants were taught to request three preferred items through various intervention steps that also involved increasing the size of the images of the preferred items. Data showed that each participant learned to use the relative communication support tool to acquire requesting for the three preferred items. The older participant was relatively fast in her acquisition. The younger participant needed more time. Yet, she seemed to represent the first case with Rett syndrome to acquire multiple requests through an eye-gaze response.

Davies et al. (1) set up and assessed a new technology system (i.e., GeoTalk) as support tool for facilitating request making across different settings with a group of 37 people. The people's average IQ score was in the low region of the mild range. The technology involved a hand-held communication device that integrated a global positioning system (GPS) and other sensors, which allowed the vocabulary/communication symbols appearing on the device's screen to change automatically based on the geographic zone. When the participant was in a zone such as the school, the symbols were those normally used for communication requests made in that zone. If the participant moved to a grocery store, the symbols available on the device changed accordingly. To determine the usability and effectiveness of the GeoTalk, the participants' performance with such tool was compared with their performance with two other communication technology solutions. One involved a speechgenerating device in which, the participant was to change the communication-symbol layer independently when he or she entered a new context. The other involved a device based on a palmtop computer in which the symbol sets were to be changed through a screen operation. Data showed that the GeoTalk compared favorably with the other devices. The participants made fewer errors, required fewer prompts, and needed shorter time to make the requests.

### Comments on the Support Tools

The support tools described above, although generally defined as speech-generating devices, differ from those previously available in the area in that they include components that can make their use more effective (1) and/or feasible also for people with no reliable hand movements (77). From a technical standpoint, the tools reported [except for the one used by Simacek et al. (77) for the older Rett participant] can be considered relatively complex. Essentially, the tool reported by Davies et al. (1) is an experimental arrangement of available technology components, which was specifically designed to assist people across settings. The eye-gaze tool reported by Simacek et al. (77) for the younger Rett participant is based on commercial eye-tracking technology, which (a) supports multiple activities besides communication, and (b) is typically used with individuals with amyotrophic lateral sclerosis or other conditions of pervasive motor impairment.

From a practical standpoint, two considerations are in order. First, those tools are not designed to be easily portable. Indeed, the Davies et al.'s tool involves a number of sensors that are tied to specific settings and work properly within those settings. The tool reported by Simacek et al. for the younger participant involves a relatively sophisticated computer-aided system that is generally fixed to a table stand or the participant's wheelchair and cannot always be freely moved across settings or used across individuals. Second, the cost of those tools is expected to be considerable and their set up requires a certain amount of competence.

New research work could advance the development of the Davies et al.'s tool with new/cheaper commercial components (e.g., with smartphones functioning as speech-generating devices) so as to make it more affordable and easier to set up and use as a real resource for individuals attending school, work and other community settings. A simpler and cheaper eye-gaze device may also need to be developed (to replace the expensive commercial versions; e.g., Tobii C12) specifically for individuals who only use it for limited purposes.

### Leisure and Communication Studies

Borgestig et al. (78) set up and assessed eye-tracking support tools for 10 participants who presented with extensive motor impairments. Five of them were also reported to have an unspecified level of cognitive impairment. All participants were said to be able to communicate (show interest) with facial expressions and eye gazing. Some participants could also express yes/no eye movements. The technology consisted of Tobii C12 or Tobii P10 eye-tracking devices mounted on a floor stand, table stand, or the wheelchair. Following a protracted intervention phase, during which parents, teachers and technology experts were involved in promoting the use of the technology, there was a follow-up assessment during which expert supervision was no longer available. Follow-up data showed that all 10 children managed to use the eye-tracking devices. Most children learned to talk with others via the devices. However, communication was the main purpose in using the devices only for two children. Other children used the devices predominantly or exclusively for playing games, watching photos, or listening to music.

Lancioni et al. (32) set up and assessed customized support tools to meet the leisure and communication needs of seven participants with acquired neurological damage and multiple disabilities. Participants presented with lack of expressive communication and pervasive motor disabilities that prevented them from having any direct, independent contact with environmental stimuli. The technology involved a computer, which (a) showed the leisure and communication options available for the single participants (e.g., songs, television, direct requests, text messages, or writing) and (b) supported any of those options, thus allowing the participants to access and engage in any of them. For example, if the participant chose text messages, the computer presented (a) the names of various persons to whom messages could be sent, and (once a person was selected), (b) the messages available for that person. The participants could make their choices and manage any leisure or communication event through the use of a microswitch suiting their motor condition. All participants managed to use the support tools successfully, and consequently engaged in leisure and communication consistently throughout the study.

Lancioni et al. (72) set up and assessed a support tool that was aimed at promoting leisure activities and telephone calls in eight participants who presented with mild to moderate intellectual disability and sensory or sensory-motor impairments. The technology consisted of a smartphone with Android operating system, which was (a) supplied with audio or audiovisual files concerning the leisure activities and telephone partners for communication and (b) automated through the MacroDroid application. The participants made their requests for leisure activities or telephone calls by placing cards or mini objects fitted with frequency code tags on the back of the smartphone. Recognition of the cards and mini objects' tags led the MacroDroid to activate the related activities or telephone calls thus allowing the participants to access them. With the help of the support tool, all participants learned to make requests and access leisure events and telephone calls successfully and maintained their positive performance over time.

Lancioni et al. (79) set up and assessed an additional support tool to ensure access to leisure activities and video calls to eight participants who presented with moderate intellectual disability and had very poor speech skills or had no speech and no receptive verbal skills due to hearing loss. Video calls seemed the only way or the preferred way for the participants to interact/communicate with distant partners. The technology involved a tablet with Android operating system, which was fitted with a SIM card and two specific applications, that is, WhatsApp Messenger for making video calls and MacroDroid for automating the functioning of the tablet. The tablet typically presented pictures representing leisure activities and pictures concerning preferred partners. The participant could select any of them by touching/approaching the tablet's proximity sensor when the activity or partner was lit. Selection of an activity led the tablet to present several variations of it among which the participant could choose. Selection of a partner led the tablet to start a video call with that partner. Data showed that all participants learned to use the support tool and were successful in accessing leisure activities and making video calls independently.

### Comments on the Support Tools

Four different support tools were described in this section, that is, commercial eye-tracking devices, a combination of computer system and microswitch, a specially arranged smartphone, and a specially arranged tablet. From a technical standpoint, the tools differ substantially, with the first two being much more complex than the other two. Indeed, the eye-tracking devices are highly sophisticated and powerful instruments that require a careful tuning with the participants under expert supervision, and thus are not readily/immediately accessible. The microswitch used in combination with the computer system is frequently an experimental device or an adaptation of a commercial device (i.e., to suit participants affected by pervasive motor impairment), and this requires extra preparation time and costs. The last two tools are based on common commercial technology and thus are more easily accessible and comparatively simple. Even so, adapting them for the participants' use requires, as suggested in previous sections of this paper, a certain amount of preparation work and expertise involving the management of specific applications, such as the WhatsApp Messenger and the MacroDroid.

From a practical standpoint, a few considerations are in order. Eye-tracking devices may not be easily affordable in daily contexts given their complexity and costs. Their employment with individuals with pervasive motor disabilities might often be successfully replaced by the use of support tools involving the combination of computer and microswitch. These latter tools, albeit not immediately accessible (as observed above), (a) can be adapted to individuals with minimal response repertoire, and (b) are much less expensive than the eyetracking devices. The other two support tools, based on smartphone and tablet technology, have the advantage of being fairly inexpensive/affordable and easily portable. Yet, they are applicable only when the participants have control of the basic motor responses necessary to manipulate cards or mini objects and to activate the tablet's proximity sensor, respectively.

New research may be focused on the development of alternative support tools that can promote leisure activities, audio and video calls, and message exchanges, and are easily affordable and portable. One might also explore the possibility of using smartphones as microswitches for participants with no use of their hands (81). For example, a smartphone could (a) monitor, through its light or proximity sensors, small movements of the participant's head, and (in relation to those movements) (b) operate leisure or communication choices on a second smartphone or tablet.

### CONCLUSIONS

The paper has analyzed a number of support tools, which were used to help people with significant disabilities bypass the limits imposed by their condition and reach important objectives. The generally positive results reported by the studies summarized above are encouraging as to the beneficial role of those tools. They were successfully used (a) as extension of the individuals' body, so the individuals could engage with the environment effectively irrespective of their response limits, and (b) as extension of the individuals' cognitive dimension, so the individuals could act in a more accomplished manner and improve their occupational achievement, social contacts and communication. In essence, technology-based support tools were reported to be instrumental in fostering goal-directed interactions of the individual with disability with his or her physical and social environment, and possibly in promoting the individual's cognition and development (8–10, 82, 83). Notwithstanding the above, some caution might still be needed in drawing general conclusions, due to the fact that (a) the studies reported to illustrate the tools included few participants, (b) the relatively limited data available do not allow sophisticated statistical analyses and do not provide specific evidence of the tools' use in daily contexts (i.e., under the supervision of regular staff), and (c) the rapidly changing field of technology might shortly present new scenarios with new, alternative tools.

The appropriateness and friendliness of any tool are high when the tool's operation fits the individual's physical conditions and/or cognitive skills and when the goal set to be reached is feasible for the individual [i.e., in line with his or her embodied experience (84)]. In light of this statement, one could also maintain that support tools are to be adapted in terms of their operation requirements and/or their content (the goal set to be reached) to the single individuals exposed to the tools (85, 86). Adaptations/customizations would ensure that all individuals are able to manage the use of the tools provided and reach the results expected easily and rapidly (i.e., without failures and frustration and without physical strain) (32).

One additional element that would have a decisive impact on the final results obtained with any tool is the individual's motivation to use such tool (i.e., the individual's motivation to reach the input/consequences available for the tool-mediated performance). The individual may be strongly motivated to produce high response rates (e.g., high levels of small responses or functional body movements) if the stimulation following the performance of those responses is significant (highly preferred) for him or her (12, 21, 42, 87). Similarly, the individual may be strongly motivated to have high levels of task accuracy and high levels of leisure and communication engagement if his or her performance encounters success and satisfaction. Satisfaction could be here interpreted as (a) the availability of positive feedback (higher level of social enclosure) for correct performance, (b) the possibility to access leisure events meeting the individual's interests, and (c) communication opportunities involving the individual's preferred partners and preferred topics (2, 87).

Acceptance and regular use of support tools within daily contexts cannot be considered an easy-to-reach objective for a variety of reasons, some of which were discussed above. Indeed,

### REFERENCES


one would assume that accessibility, affordability and friendliness of the tools represent main variables favoring their application outside of the research environment. The presence of all these variables alone, albeit essential, might not yet guarantee a positive decision of daily contexts in relation to the use of support tools. There are situational/practical issues, in fact, that may interfere with a positive decision (88–91). A major issue is the knowledge (competence) gap that exists between the experimental world in which tools are developed and assessed and the daily reality. The gap cannot be bridged by simply relying on technology experts, who have no specific competence in the field of education and rehabilitation, and thus cannot ensure the identification of a suitable tool for each single individual (32, 72). The gap may be reduced, however, when technology experts are called to work in close collaboration with education/rehabilitation personnel who can identify the skills and limitations of the individuals to serve and the objectives to target with them.

In conclusion, the support tools presented and discussed above seem to have great potential for improving the situation of individuals with serious disabilities by providing those individuals with new opportunities for (a) engaging in goal-directed interaction with the environment and thus (b) enhancing cognition and development. The possibility of having those tools largely available in daily contexts may greatly depend on (a) the accessibility, affordability, and friendliness of the tools, and (b) the availability of areas of competence and responsibility in the daily contexts that would ensure a successful set up and application of the tools, and possibly a positive intervention outcome.

### AUTHOR CONTRIBUTIONS

GL was responsible for conceiving and writing the paper. MOB, NS, MO, JS, and GA were involved in defining the scope of the paper and the material to include, and contributed in the writing and editing of the manuscript.


programs. Front Public Health. (2017) 5:205. doi: 10.3389/fpubh.2017. 00205


intellectual and multiple disabilities. Int J Dev Disabi. (2017) 63:238–45. doi: 10.1080/20473869.2017.1288888


and choice. Disabi Rehabi. (2014) 36:1626–35. doi: 10.3109/09638288.2013. 863389


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Lancioni, Olivetti Belardinelli, Singh, O'Reilly, Sigafoos and Alberti. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Learning of Artificial Sensation Through Long-Term Home Use of a Sensory-Enabled Prosthesis

Ivana Cuberovic1,2, Anisha Gill<sup>3</sup> , Linda J. Resnik3,4, Dustin J. Tyler1,2 and Emily L. Graczyk1,2 \*

<sup>1</sup> Department of Biomedical Engineering, Case Western Reserve University, Cleveland, OH, United States, <sup>2</sup> Louis Stokes Cleveland VA Medical Center, Cleveland, OH, United States, <sup>3</sup> Providence VA Medical Center, Providence, RI, United States, <sup>4</sup> Department of Health Services, Policy, and Practice, Brown University, Providence, RI, United States

#### Edited by:

Mariella Pazzaglia, Sapienza University of Rome, Italy

#### Reviewed by:

Silvestro Micera, Sant'Anna School of Advanced Studies, Italy Caroline Dietrich, Friedrich Schiller University Jena, Germany Marco D'Alonzo, Campus Bio-Medico University of Rome, Italy

> \*Correspondence: Emily L. Graczyk emily.graczyk@case.edu

#### Specialty section:

This article was submitted to Neuroprosthetics, a section of the journal Frontiers in Neuroscience

Received: 31 May 2019 Accepted: 30 July 2019 Published: 21 August 2019

#### Citation:

Cuberovic I, Gill A, Resnik LJ, Tyler DJ and Graczyk EL (2019) Learning of Artificial Sensation Through Long-Term Home Use of a Sensory-Enabled Prosthesis. Front. Neurosci. 13:853. doi: 10.3389/fnins.2019.00853 Upper limb prostheses are specialized tools, and skilled operation is learned by amputees over time. Recently, neural prostheses using implanted peripheral nerve interfaces have enabled advances in artificial somatosensory feedback that can improve prosthesis outcomes. However, the effect of sensory learning on artificial somatosensation has not been studied, despite its known influence on intact somatosensation and analogous neuroprostheses. Sensory learning involves changes in the perception and interpretation of sensory feedback and may further influence functional and psychosocial outcomes. In this mixed methods case study, we examined how passive learning over 115 days of home use of a neuralconnected, sensory-enabled prosthetic hand influenced perception of artificial sensory feedback in a participant with transradial amputation. We examined perceptual changes both within individual days of use and across the duration of the study. At both time scales, the reported percept locations became significantly more aligned with prosthesis sensor locations, and the phantom limb became significantly more extended toward the prosthesis position. Similarly, the participant's ratings of intensity, naturalness, and contact touch significantly increased, while his ratings of vibration and movement significantly decreased across-days for tactile channels. These sensory changes likely resulted from engagement of cortical plasticity mechanisms as the participant learned to use the artificial sensory feedback. We also assessed psychosocial and functional outcomes through surveys and interviews, and found that self-efficacy, perceived function, prosthesis embodiment, social touch, body image, and prosthesis efficiency improved significantly. These outcomes typically improved within the first month of home use, demonstrating rapid benefits of artificial sensation. Participant interviews indicated that the naturalness of the experience and engagement with the prosthesis increased throughout the study, suggesting that artificial somatosensation may decrease prosthesis abandonment. Our data showed that prosthesis embodiment was intricately related to naturalness and phantom limb perception, and that learning the artificial sensation may have modified the body schema. As another indicator of successfully learning to use artificial sensation,

**90**

the participant reported the emergence of stereognosis later in the study. This study provides the first evidence that artificial somatosensation can undergo similar learning processes as intact sensation and highlights the importance of sensory restoration in prostheses.

Keywords: neural prosthesis, touch perception, proprioception, learning, embodiment/bodily experience, amputation – rehabilitation, home use, phantom limb experience

### INTRODUCTION

Tool use is a ubiquitous human trait. Prostheses for upper limb amputees are considered to be a special case of tool use, because their purpose is to replace a missing body part rather than to augment normal human capabilities. A classically studied goal of upper limb prosthesis rehabilitation has been recovery of grasping and dexterous manipulation. To that end, considerable efforts have gone into the development of mechanically dexterous prosthetic limbs (Weir and Sensinger, 2009; Belter et al., 2013; Bajaj et al., 2018) and intuitive algorithms to control them (Scheme and Englehart, 2011; Dalley et al., 2012; Hargrove et al., 2017; Segil et al., 2017), with some of these technologies becoming commercially available. More recently, somatosensation has also been restored in upper limb prostheses, either through noninvasive electrocutaneous and vibrotactile techniques (Dietrich et al., 2012, 2018; Clemente et al., 2016) or through implanted neural interfaces (Raspopovic et al., 2014; Tan et al., 2014; Davis et al., 2016). Multiple groups have investigated direct electrical stimulation of the remaining nerves as a means of restoring sensation of the missing hand to upper limb amputees (Raspopovic et al., 2014; Tan et al., 2014; Davis et al., 2016). In addition to quantifying the evoked percepts (Tan et al., 2014; Graczyk et al., 2016, 2018a), these groups have also shown marked improvements in performance of functional tasks, such as object identification (Schiefer et al., 2016), object feature discrimination (Horch et al., 2011; Raspopovic et al., 2014; Oddo et al., 2016; Schiefer et al., 2018), and closed-loop control (Wendelken et al., 2017; Valle et al., 2018), when sensory feedback is provided.

Like all tool use, skilled prosthesis use develops over time and often requires training. The relationship between training and general skill acquisition has been quantified through the learning curve (Newell and Rosenbloom, 1981; DeKeyser, 2015). In its simplest form, the learning curve shows that markers of skill, such as increases in accuracy, decreases in errors, and decreases in cognitive effort, improve with training over extended durations of time (Newell and Rosenbloom, 1981; DeKeyser, 2015). Active learning occurs as the task or skill is explicitly practiced through a training regimen. In contrast, passive learning occurs as the task or skill is performed as needed during daily activities.

Learning skilled tool use involves neural changes in both the motor and sensory systems. Perceptual learning is enabled by plasticity in the sensory cortex (Gilbert et al., 2001) and involves increases in the size of the representation of a stimulus in the sensory cortex, narrowing of the selectivity of tuned cells, changes in the temporal relationships of neuronal responses, and shifts in processing from higher to lower sensory cortices (Gilbert et al., 2001; Hoffman and Logothetis, 2009). Indeed, numerous studies have shown expansion of somatosensory cortical regions to enable trained sensorimotor skills, such as reading Braille (Pascual-Leone and Torres, 1993), playing instruments (Elbert et al., 1995), or understanding speech with a cochlear prosthesis (Fallon et al., 2008).

Tool embodiment is also intricately related to sensory learning, as they both involve similar neural mechanisms. Multiple groups have shown that both referred sensations on the residual limb and localized percepts on the missing limb can increase embodiment of a prosthesis (Ehrsson et al., 2008; Dietrich et al., 2012; D'Alonzo et al., 2015; Graczyk et al., 2018b, 2019; Marasco et al., 2018; Page et al., 2018; Valle et al., 2018). Prosthesis embodiment is an important psychosocial outcome that is related to positive prosthesis outcomes (Murray, 2004; Graczyk et al., 2019). It is defined as both the conscious perception of tool inclusion within one's bodily borders and the preconscious sensorimotor processing of the tool as if it belonged to the body (Gallagher, 2005; Haggard and Wolpert, 2005; Arzy et al., 2006; Giummarra et al., 2008; Longo et al., 2008; de Vignemont, 2010b, 2011). Although the criteria necessary for embodiment to occur vary through the literature, embodiment is generally understood to emerge from multisensory integration among spatiotemporally coincident stimuli (de Vignemont, 2010b). Embodiment alters the processing of sensory events in both peri-personal and personal space (Iriki et al., 1996; Galli et al., 2015; Miller et al., 2017) and modifies the body schema, which is the preconscious, dynamic sensorimotor representation of the body (Cardinalli et al., 2009; Jovanov et al., 2015). Because they replace lost body parts, prostheses may be truly incorporated into the body schema, rather than operating on the level of bodily extension like most tools (de Preester and Tsakiris, 2009; de Vignemont, 2010b).

The role of learning in interpreting artificial sensory feedback has been studied most extensively in audition with the advent of cochlear prostheses (Watson, 1991; Fu et al., 2005; Fu and Galvin, 2007, 2012; Oba et al., 2011). With cochlear prostheses, learning is defined as improvements in auditory perception and interpretation over time, and differences between passive and active perceptual learning have been quantified (Fu and Galvin, 2012). Both passive and active learning regimens are associated with corresponding cortical plasticity (Tremblay et al., 1997, 1998; Fu and Galvin, 2007; Gentner and Margoliash, 2009; Sharma and Dorman, 2012). While auditory perception through cochlear prostheses can improve with passive learning, these improvements typically plateau in 3–12 months (Watson, 1991; Fu and Galvin, 2007, 2012). Additional improvements in perception require active training interventions, such as training in particular noise environments (Fu et al., 2005;

Fu and Galvin, 2007; Oba et al., 2011) or training to recognize specific phonemes (Fu et al., 2005).

However, the effect of sensory learning on the perception and utilization of artificial somatosensory percepts evoked by electrical stimulation in upper limb prostheses has not been studied. For artificial somatosensation in prosthetic limbs, studies primarily investigate sensation in controlled laboratory environments, where laboratory visits are sporadic and only last for a few hours or days at a time. Given prior studies on passive learning in cochlear prostheses, which indicate that passive learning can continue shaping auditory perception for up to a year (Watson, 1991; Fu and Galvin, 2007, 2012), studies of sensory learning in upper limb prostheses may similarly require months to years of extended usage. To our knowledge, there has only been one study in which participants received artificial somatosensation from implanted nerve interfaces for multiple days in a row. This study involved independent usage of a prosthesis with stimulation-evoked sensory feedback in home and community settings for up to 2 weeks in two participants (Graczyk et al., 2018b). We found that sensory detection thresholds were stable over this duration, but did not quantify whether any aspects of the sensations changed over time with continued exposure to stimulation. Sensory learning may also influence functional and psychosocial outcomes. In our prior home use study, we also found that sensory feedback impacted the psychosocial experience of prosthesis embodiment, confidence, and perceived efficiency. While we found that function was better with sensation than without in our prior home study, it did not improve over the short duration of the study. Because the study was brief relative to the passive learning interval, we were not able to investigate the time course of these changes or whether they had plateaued.

Thus, the purpose of this study is to investigate whether artificial somatosensory feedback can be learned over time. We define learning as changes in the perception and utilization of artificial somatosensation and changes in user outcomes due to prolonged exposure to sensory stimulation. We examined the impact of extended daily usage of the sensory-enabled prosthesis on quantitative metrics of the perception of evoked somatosensation, perception of the phantom limb, psychosocial outcomes, and functional outcomes. We also conducted a qualitative analysis of data from in-person interviews. We hypothesized that extended usage of a sensory-enabled prosthesis would correspond with sensory percept changes to better align with information transduced by the prosthesis sensors. We further hypothesized that extended usage would yield better functional and psychosocial outcomes.

### MATERIALS AND METHODS

### Subject

One adult male with unilateral transradial limb loss participated in this trial. He sustained a traumatic right transradial amputation approximately 3 inches distal to the elbow in a workplace accident in 2004 and was right hand dominant prior to amputation. The participant was implanted with 8-channel Flat Interface Nerve Electrodes (FINEs) around his median and radial nerves in 2013. He participated in a previous short-term home study in months 41–42 post-implant (Graczyk et al., 2018b). Data for the current home study was collected in months 71– 75 post-implant. The participant did not report any instances of phantom pain during this time. All study devices and procedures were reviewed and approved by the U.S. Food and Drug Administration Investigational Device Exemption, the Cleveland Department of Veterans Affairs Medical Center Institutional Review Board, and the Department of the Navy Human Research Protection Program. All study procedures and experiments were performed in accordance with relevant guidelines and regulations of these institutions. Written informed consent was obtained from the subject.

### Home Use System

The home use system included a portable neurostimulator and associated hardware for providing artificial somatosensory feedback with a single degree-of-freedom (DOF) prosthesis (**Figure 1A**). The subject wore his own, clinically fit prosthetic socket and used his standard settings for agonist/antagonist myoelectric control. An OttoBock VariPlus Speed prosthetic hand was augmented by embedding force-sensitive resistors in the tips of the thumb, index, and middle fingers using medical grade silicone adhesive. An aperture sensor was mounted in the prosthetic hand to measure the opening span of the hand's single DOF. The installed sensors sent analog signals via a cable to the neurostimulator, which was worn about the waist in a small pack. The neurostimulator translated the incoming pressure and aperture data into stimulation pulse trains and sent the stimulation trains to the subject's implanted FINEs via a cable to his percutaneous leads. Stimulation pulses were cathode-first, biphasic, and charge-balanced. Each sensor in the prosthetic hand corresponded to a single electrode contact in the FINEs. A set of three electrode contacts inside the FINE were used as the return current path for all active contacts. Increases in pressure or decreases in hand aperture were linearly scaled to stimulation pulse frequency. Full details of the home use system are presented in an earlier publication (Graczyk et al., 2018b).

### Study Design

The study used a quasi-experimental time series design with three periods of home use, each separated by 2 days of inlab testing (**Figure 1B**). The participant also completed in-lab testing at the start and end of the study. The total duration of the study was 115 days, including in-lab testing. Each period of home use was intended to be 1-month in duration. However, due to interruptions throughout the study, the durations of home use periods were modified (**Figure 1C**). Interruptions included system component breakages, illnesses, and other personal emergencies (**Supplementary Table S1**). The participant wore the system a home for 9, 21, and 19 days in intervals 1, 2, and 3, respectively. The average wear time of the system, based on onboard system logs, was 6.7 ± 0.25 h/day (mean ± SEM). There was no difference in wear time based on study interval (1-way ANOVA, p = 0.065).

FIGURE 1 | Home system for sensory restoration and usage throughout the study. (A) Home system for sensory restoration. Implanted peripheral nerve cuff electrodes delivered artificial somatosensory feedback of touch and proprioception corresponding to sensors on a hand prosthesis. This panel is reproduced from Graczyk et al. (2018b) with minor revisions (CC BY 4.0, https://creativecommons.org/licenses/by/4.0/). (B) Study design. The three at-home study intervals were flanked by in-lab testing sessions. (C) Usage of the restored sensation varied throughout the study (n = 115 days in total). Days of wear were interspersed with days of non-wear, either due to interruptions or choice. Interruptions included breakage of system components that necessitated repair by the study team, illnesses, and urgent personal matters (Supplementary Table S1).

### Outcome Measures

Onboard usage logs, surveys, and functional outcomes were obtained at various time courses throughout the study (**Table 1**). Two general categories of metrics, at-home and in-lab, were collected. At-home metrics included surveys administered daily or weekly on days the participant wore the system and onboard usage logs, which recorded timestamps for system on and off periods and continuously recorded prosthesis sensor activity. In-lab measures consisted of surveys and functional tests administered during the participant's laboratory visits. The metrics utilized in this study were a subset of the metrics obtained during our prior home study, and details of each metric can be found in the previous publication (Graczyk et al., 2018b).

Sensation location, intensity, and quality were assessed twice daily throughout the study. The participant filled out surveys about each of the four sensations immediately after donning and calibrating the system and immediately prior to doffing the system. The participant reported the perceived sensation location for each sensor by outlining the location on a hand diagram (Tan et al., 2014). He reported the perceived intensity of the sensation and the degree to which several quality descriptor words (**Supplementary Table S2**) described the sensation using a series of visual-analog scales (VAS). The descriptors selected for this study were a subset of those presented in the prior home use study (Graczyk et al., 2018b) (descriptors that were never rated by the participant during the previous home trial were excluded here). The participant also reported the position of the phantom limb twice daily, prior to donning and doffing the system. The participant drew the perceived position of his phantom fingertips relative to their expected anatomic location on an arm diagram (Graczyk et al., 2018b).

Psychosocial outcomes were evaluated at two time scales. First, the participant completed the Take-Home Experience Diary (THED) in the evenings on days that he wore the system. The THED consisted of the short form of the Patient Experience Measure (PEM, see below) and free response questions to describe any notable experiences or circumstances that day (Graczyk et al., 2018b). During the monthly laboratory visits, the participant completed additional psychosocial surveys. The PEM consisted of five subscales including embodiment of the prosthesis, self-efficacy, social touch, body image, and prosthesis efficiency. The embodiment subscale measured the perception of ownership of the prosthesis (e.g., the prosthesis is a part of me), the self-efficacy subscale measured confidence in using the prosthesis for functional tasks, the social touch subscale measured the perceived ability to use the prosthesis in social situations (such as shaking hands), the body image subscale measured the impact of the prosthesis on the conscious perception of the body, and the prosthesis efficiency subscale measured the perceived speed and focus required to use the prosthesis (Graczyk et al., 2018b). Note that the short form of the PEM (administered daily) was an abridged version of the PEM and did not include the body image subscale. The Rubber Hand Illusion (RHI) questionnaire evaluated prosthesis embodiment immediately after performing a functional task in the laboratory setting (see below) (Ehrsson et al., 2008; Marasco et al., 2011; Schiefer et al., 2016). Two other surveys evaluated holistic psychosocial outcomes: the Orthotics and Prosthetics Users' Survey (OPUS) Quality of life (QoL) metric was used to assess overall quality of life (Heinemann et al., 2003; Jarl et al., 2014), and the Quick Disabilities of the Arm, Shoulder and Hand (QuickDASH) survey evaluated the participant's perception of his disability (Gummesson et al., 2006; Polson et al., 2010; Resnik and Borgia, 2015).

#### TABLE 1 | Administered measures and time courses of data collection.


The schedule for administering specific measures is indicated by the check marks.

Functional outcomes were evaluated using surveys, standardized functional tests performed in the laboratory, and system logs. The participant's ability to interpret sensory feedback was evaluated using the foam block task (Graczyk et al., 2018b; Schiefer et al., 2018). Briefly, the participant was asked to identify the size or compliance of foam blocks presented to the prosthesis without visual or auditory feedback. The blocks had three sizes (small, medium, large) or three compliances (soft, medium, hard). The participant completed the tasks both with and without sensory feedback during each laboratory visit. In addition to this objective measure of function, three measures were used to assess perceived function, which is the participant's subjective view of their abilities with the prosthesis. Before each trial set of the foam block task, the participant was asked to report on his confidence in his ability to perform the upcoming foam block task (Graczyk et al., 2018b; Schiefer et al., 2018). Two standard clinical metrics were also used to evaluate perceived function. The Patient Specific Functional Scale (PSFS) required the subject to identify five tasks he had difficulty performing prior to the study, then tracked his perceived ability to perform those same tasks throughout the study (Stratford et al., 1995; Hefford et al., 2012; Resnik and Borgia, 2012). The modified OPUS Upper Extremity Functional Status (UEFS) survey evaluated the participant's willingness and perceived ability to perform a set of 28 standard activities of daily living (ADL) (Heinemann et al., 2003; Jarl et al., 2012; Graczyk et al., 2018b).

Finally, the participant's use of the sensory-enabled system was tracked through system logs. The system logs recorded system settings and button presses, such as enabling or disabling sensory feedback, and all activity from the prosthesis pressure and aperture sensors (Graczyk et al., 2018b).

### Qualitative Analysis

We conducted a qualitative analysis of in-person interview data to explore how the participant's experiences with the sensoryenabled prosthesis changed over time. In each laboratory session, the participant completed a 20–40 min semi-structured interview with authors EG and IC. Interview questions explored the experience of sensation, the experience of the prosthesis, changes

in sensation over time, and changes in prosthesis experience over time. See **Supplementary Data Sheet S1** for interview questions. EG was the primary interviewer, while EG and IC both probed responses for clarification or expansion. The interview data were video recorded and transcribed by IC.

After completing the interviews, four of the investigators (IC, LR, AG, and EG) performed a modified grounded theory analysis using constant comparison methods (Strauss and Corbin, 1990, 1994; Creswell, 2007). We utilized the grounded theory approach to perform open coding, axial coding, and selective coding (Strauss and Corbin, 1998). NVivo 12 software was used to organize the data (NVivo qualitative data analysis Software; QSR International Pty Ltd., Version 12.3.0). In the open coding phase, the analysis team performed line-by-line open coding of the transcript from the interview at the end of interval 1. The investigators then established a preliminary set of codes through consensus coding (Haverkamp and Young, 2007). After creating the initial codebook, LR stepped away from the analytical discussions to serve as an external auditor of the analytic process and findings. Each successive interview transcript was then coded, and the code list and code definitions were iteratively finetuned through consensus of the analytic team (EG, AG, and IC). In the axial coding phase, the codes were separated into categories and sub-categories. Descriptions of each code were generated and supported with rich text exemplars. These code descriptions were then used to assist with the process of selective coding, in which the axial codes were organized into overarching themes. Throughout the analytic process, the team maintained an audit trail to track their decisions (Charmaz, 1996), and LR reviewed the audit trail.

The analysis was conducted from a primarily post-positivist epistemological framing, with some constructivist leanings (Mills et al., 2006; Creswell, 2007; Staller, 2013). Note that EG and IC are experts in neural engineering and sensory neuroprostheses, and each have over 5 years of experience working with this participant in associated research studies. LR and AG conduct research related to rehabilitation outcome assessment with a focus on upper limb prostheses, and have never met the participant. LR, EG, and AG had previous experience with grounded theory analysis of participant perspectives on sensory prostheses from the prior short-term home study (Graczyk et al., 2019), whereas IC did not.

### Statistical Analysis

Analysis was performed after all data was collected. Throughout, an alpha-value of 0.05 was used. All values are reported as mean ± standard error of the mean.

We studied both within-day and across-days changes in sensory perception (location, quality, and phantom limb length). Within-day changes were compared using paired t-tests. Acrossday trends were evaluated using linear regressions over hours with sensation. Data for each sensory channel was only included for days on which its corresponding prosthesis sensor operated correctly, as determined by analysis of the onboard log and corroborated by participant daily diary reports. Specifically, data is shown for days 1–111 for channel 1, 1–36 for channel 2, 1–111 for channel 3, and 1–58 channel 4 (**Supplementary Figure S1A**). Analyses of within-day changes were performed only on days for which both morning (AM) and evening (PM) data is available. Across-day analyses include data from the morning surveys on days that sensors failed. Across-day analyses of location alignment are binned such that each point represents 5 days of collected data. However, the last point in each time series contains between 3 and 5 data points, based on the number of days the participant wore the system before the study ended. Additional statistical analyses for channels 1 and 4 are described in **Supplementary Data Sheet S2**. Further, two outliers, whose values were more than 3 standard deviations away from the mean, were removed from the analysis of the phantom limb length data.

Changes in psychosocial survey outcomes between in-lab sessions were performed by comparisons of successive data points. Surveys for which normative data exists (QuickDASH and OPUS QoL) were evaluated using their respective minimum detectible change values (MDC). The PEM was analyzed using paired t-tests of subscale items between successive in-lab sessions and between the first and last in-lab sessions. Agreement with embodiment vs. control statements in the RHI was analyzed using 2-sample t-tests, and trends across the study were analyzed using linear regression. Changes to the short form PEM scores over time were analyzed using linear regression.

Changes in functional outcomes over time, including foam block performance and confidence, modified UEFS task difficulty and completion rate, the PSFS, and active sensor usage, were evaluated using linear regression. As normative data exists for the PSFS, successive data points were also evaluated using its MDC.

Functional outcomes with and without sensation were also compared (see **Supplementary Figure S2**). The foam block test was evaluated both with and without sensation during this study, and comparisons were made using paired t-tests. In contrast, at-home measures, including the UEFS and active prosthesis usage, were always collected with sensation-enabled during this study. To compare performance with sensation vs. without sensation in these measures, data from this study, in which sensation was always enabled, was compared to sensationdisabled data from our previous study (Graczyk et al., 2018b). In addition, we compared sensation-enabled data from this study to the sensation-enabled data from the prior study. Statistical comparisons between the two studies were made using 1-way ANOVA followed by Tukey test.

### RESULTS

The effects of learning on sensory perception (sensation location, sensation quality, and phantom limb) were evaluated withinday and across-day. Because we hypothesized that working prosthesis sensors were required for sensory learning, data for each sensory channel was only analyzed for days on which the sensor was known to have operated correctly, as determined by corroboration of the onboard log data with the participant's daily diary. Note that channel 2 failed after 13 days of wear. The within-day comparison was used to study the immediate effects of actively using sensation on perception, whereas across-day trends over the 115-day study were examined to investigate whether any

changes in sensation were retained over time. Across-day trends were evaluated independently from within-day changes, as we believe they may have different mechanisms or implications.

### Perceived Sensation Locations Become Aligned With Prosthesis Sensor Locations Over Time

We defined the perceived sensory location to be aligned with the prosthesis sensor if the location reported on the hand diagram overlapped the defined prosthesis sensor location. Within-day changes of sensory percept location were evaluated by comparing sensation locations between morning and evening drawings. We categorized sensations as either moving toward the sensor (**Figure 2A**, left), staying constant (**Figure 2A**, middle), or moving away from the sensor (**Figure 2A**, right) based on whether the drawings of the percept changed between morning and evening and whether the drawings overlapped with the prosthesis sensor position. Channel 4 was excluded from this analysis because the aperture sensor with which it was associated was mounted underneath the prosthesis cosmetic cover and thus did not have a defined location on the surface of the hand.

with the sensor location at the beginning of the day (AM), before any functional usage occurred (n = 49, 14, 48 days of wear for channels 1, 2, and 3, respectively). For (B,C), stars denote statistically significant trends over time (p < 0.05). (D) Bar plot showing relative proportion of within-day movements that were aligned with sensor use (green) vs. not aligned with sensor use (orange) (n = 47, 12, 45 days of wear for channels 1, 2, and 3, respectively). Asterisks denote significant differences between groupings (p < 0.05).

We found that stimulation-evoked sensation locations changed both within a single day of use and over the course of the study. There also were interactions between these two time courses: the types of within-day location changes exhibited across-day trends over the course of the study. For channel 1, the frequency at which the perceived sensation moved toward the sensor decreased over the course of the study (line slope test, p = 0.045) (**Figure 2B**, left, green), while the frequency at which perceived locations remained constant increased (line slope test, p = 0.006) (**Figure 2B**, left, yellow). Interestingly, there were no significant trends in sensations becoming misaligned with the sensor across the study (line slope test, p = 0.126) (**Figure 2B**, left, red). The reported sensation locations for channels 2 and 3 followed similar trends, although none were statistically significant (**Figure 2B**, middle and right).

These long-term trends of within-day change occurred as a result of the sensory alignment becoming more permanent through the study. To examine changes in sensation location across days, we tracked the proportion of days that the morning percept overlapped with the sensor position across the 115-day trial. Across days, the sensation location became significantly more frequently aligned with the sensor location upon donning the system (line slope test, p < 0.001 and p = 0.002 for channels 1 and 3, respectively) (**Figure 2C**). The perceived location for channel 2 was aligned with the sensor location throughout its brief period of functionality.

Throughout the study, the percept locations for the working sensors were more likely to be aligned with the sensor than not (test of 2-proportions, p < 0.001 for all channels) (**Figure 2D**). This was driven by within-day shifts toward the sensor at the beginning of the study and by the constantly aligned sensation later in the study. Interestingly, this trend is reversed for channel 2 on the days that the prosthesis sensor was malfunctioning (**Supplementary Figure S1B**). For channel 2, the percept location was significantly less likely to be aligned with the sensor when the sensor was malfunctioning than when it was working (test of 2-proportions, p = 0.001).

### Perceived Sensation Quality Became More Congruent With Transduced Information and More Natural Over Time

The reported stimulation-evoked sensation qualities also changed over time. The subject rated the extent to which each of thirteen descriptor words characterized the perceived sensation on a VAS (**Supplementary Table S2**). The subject rated seven of the words, including "unpleasant," "tingling," "rough," "electrical," "sharp," "cramping," and "edged," in less than 5% of the completed surveys, so these words were excluded from the analysis. It is interesting to note that these infrequently rated words are primarily qualities that could not be transduced by the simple sensors on the prosthesis, such as "rough," or those with negative valence, such as "unpleasant." Words that were rated in more than 5% of the surveys included "intense," "natural," "pressure," "contact touch," "vibrating," and "movement." For the purpose of this analysis, intensity is considered a dimension of sensation orthogonal to quality (**Figure 3A**, left), and natural is considered a judgment about the holistic experience of sensation (**Figure 3A**, middle). The other four descriptors can be categorized as tactile ("pressure," "contact touch," and "vibrating") or proprioceptive ("movement") (**Figure 3A**, right). Based on the definitions provided to the participant (**Supplementary Table S2**), "pressure" was associated with compression or weight, while "contact touch" involved light touch sensation from a voluntary action. It is interesting to note that the participant consistently rated the proprioceptive descriptor significantly lower than the tactile descriptors for the tactile channels 1–3 (1-way ANOVA followed by Tukey, p < 0.001 for all three channels), but rated the proprioceptive descriptor equivalently to the tactile descriptors for channel 4, which was associated with the aperture sensor.

As with sensory percept location, we examined within-day (**Figure 3B**) and across-day changes (**Figure 3C**) in sensory percept intensity, naturalness, and quality. There were no significant within-day changes in either sensation intensity, naturalness, or quality for channels 2, 3, and 4, but there were significant changes for channel 1 (**Figures 3B,D**). For channel 1, the intensity, naturalness, pressure, and movement of the sensation all significantly increased within a day of usage (paired t-test, p < 0.001).

Across-day changes in the sensation intensity, quality, and naturalness over the course of the study were more prevalent across channels (**Figures 3C,E**). The reported intensity of the percepts increased across the 115-day study for two of the tactile channels (line slope test, p = 0.022 and p = 0.027 for channels 1 and 3, respectively) and decreased for the proprioceptive channel (line slope test, p = 0.03). The participant also reported increases in naturalness for the two working tactile channels (line slope test, p < 0.001 and p = 0.001, for channels 1 and 3, respectively). Quality descriptors also changed over the duration of the study. Ratings of contact touch significantly increased throughout the study for the working tactile sensors (line slope test, p = 0.019 and p < 0.001, for channels 1 and 3, respectively). The contact touch descriptor is likely most similar to the information transduced by the pressure sensors on the prosthesis during functional use. Conversely, the participant's ratings of words that did not align with the use cases of the tactile sensors, including pressure, vibrating, and movement, either decreased or did not significantly change (line slope test, channel 1: p < 0.001, p < 0.001, and p = 0.017, respectively; channel 3: p = 0.73, p < 0.001, and p = 0.071, respectively) over the course of the study. The across-day changes in sensation naturalness and quality reflect increased congruency with transduced sensor information (**Figure 3E**).

Throughout the study, there were no significant trends for channel 2, the non-working tactile sensor. Unlike the perceived sensation location, there also were no significant differences in the reported VAS ratings for descriptor words for days when the sensor was working vs. non-working (**Supplementary Figure S1C**).

Quality and naturalness ratings for channel 4, which was associated with the prosthesis aperture sensor, did not change

(n = 49, 14, 48, 29 days of wear for channels 1, 2, 3, and 4, respectively). Horizontal bars denote statistical groupings based on a one-way ANOVA with Tukey pairwise comparisons. Bars that do not share a letter are significantly different (p < 0.05). (B) Bar plots depicting within-day changes between the morning (AM) and evening (PM) self-reported ratings of quality descriptor words for channel 1 (n = 47 days of wear). Asterisks denote significant differences between groupings (p < 0.05). (C) Scatter plot showing changes over the course of the study in self-reported ratings of quality descriptor words for channel 1 (n = 49). Descriptor words are color-coded as in the tables in (D,E). Stars denote statistically significant trends over time (p < 0.05). (D) Table depicting within-day changes for all channels, based on paired t-tests comparing AM to PM ratings (n = 47, 12, 45, 28 days of wear for channels 1, 2, 3, and 4, respectively). Significant increases (↑) and decreases (↓) from AM to PM are shown with their corresponding p-values, and insignificant changes are indicated by (→). (E) Table depicting across-day changes for all channels, based on a linear regression of the AM descriptor ratings vs. hours with working sensation (n = 49, 14, 48, 29 days of wear for channels 1, 2, 3, and 4, respectively). Significant increases (↑) and decreases (↓) are shown with their corresponding p-value, and insignificant changes are indicated by (→).

over time, although this may be an artifact of the lower sample size due to only analyzing data from intervals 1 and 2 of the study. Indeed, analysis of all three intervals (see **Supplementary Data Sheet S2**) shows significant trends in four of the descriptor words (**Supplementary Figure S1E**). Over the full 115-day study, both intensity and naturalness significantly increased for channel 4 (linear regression, p = 0.001 and p < 0.001, respectively), as they did for the two working tactile channels. The participant's ratings of tactile descriptors did not trend together: his rating of "pressure" increased although not significantly (linear regression, p = 0.072), his rating of "contact touch" significantly increased (linear regression, p = 0.006), and his rating of vibration significantly decreased (linear regression, p = 0.006). Surprisingly, the participant's rating of the proprioceptive descriptor, "movement," which trended upward, did not significantly increase over the course of the study (linear

regression, p = 0.162). There were no significant differences in within-day changes for channel 4, even when analyzing the full data set (**Supplementary Figure S1D**).

### Perceived Phantom Position Aligned With the Prosthesis Over Time

The position of the phantom limb also changed over time. The participant reported the position of the phantom fingertips relative to the position of the prosthesis fingertips using a drawing, and the error in phantom limb length was measured. The phantom limb became significantly more aligned with the location of the prosthetic hand within-days (paired t-test, p < 0.001) (**Figure 4A**). The alignment of the phantom limb with the prosthetic hand also improved across-days for the evening drawings, as shown by the decrease in limb length error over time (line slope test, p = 0.026) (**Figure 4B**, open circles). However, the within-day improvements in alignment did not carry over to the next morning (**Figure 4B**, filled circles). In fact, the phantom limb length error increased over time for the morning drawings (line slope test, p = 0.005).

### Psychosocial and Functional Outcomes Improve With Use of a Sensory Prosthesis

Beyond changes in sensory perception, we also investigated the effects of long-term usage of a sensory enabled prosthesis on psychosocial and functional outcomes. The QuickDASH and OPUS QoL are holistic measures of the subject's perceived disability and quality of life, respectively. These surveys trended toward improvement but did not reach significance. The participant's score on the QuickDASH improved by 13.6 points over the course of the study [MDC-95 = 17.4 (Resnik and Borgia, 2015)] (**Figure 5A**). Similarly, the participant's score on the OPUS QoL improved by 3.24 points over the course of the study [MDC-95 = 7.4 (Jarl et al., 2014)] (**Figure 5B**).

The participant had significant improvements in several subscales of the PEM (**Figure 5C**). The participant's ratings on the self-efficacy, embodiment, and social touch subscales all significantly improved within the first month (paired t-test, p = 0.001, p = 0.005, and p < 0.001, respectively), then stabilized. Prosthesis efficiency also significantly improved within the first interval (paired t-test, p = 0.038), but then continued improving through the remaining intervals. In contrast, the participant's ratings on the body image subscale did not improve significantly within any single interval but did improve significantly over the entire study (paired t-test, p = 0.013). The improvement in embodiment shown by the PEM was supported by the RHI (**Figure 5D**). While the RHI scores did not significantly improve over time (line slope test, p = 0.593), the participant did significantly embody his sensory-enabled prosthesis throughout the study, as evidenced by the significant difference between the embodiment and control statements (two-sample t-test, p ≤ 0.015). Interestingly, the short form of the PEM, which was administered daily, only showed a significant increase in the prosthesis efficiency subscale over time (line slope test, p < 0.001) (**Figure 5E**). There were no other significant trends over time on the PEM short form.

In addition, we examined changes in functional outcomes over the course of the study. We found that the participant's perceived ability to use his prosthesis for functional tasks improved over time. First, we evaluated the participant's confidence in his ability to identify the size or compliance of a foam block using his prosthesis without visual or auditory feedback. As previously reported (Graczyk et al., 2018b; Schiefer et al., 2018), he was significantly more confident in performing this laboratory task when using the prosthesis with sensation compared to without (paired t-test, p < 0.001) (**Supplementary Figure S2A**). Further, his confidence in performing this laboratory task significantly improved over the duration of the study (**Figure 6A**, line slope test p = 0.011). Second, the participant reported his perceived ability to do standard ADLs and self-identified,

end of the study and between successive intervals of the study. For the RHI, comparisons were made between embodiment and control statements within each session. (A) The QuickDASH is a measure of the participant's perceived disability. For this measure, a higher score indicates a worse outcome (n = 4 scores). (B) The OPUS QoL is a holistic measure of the participant's quality of life (n = 4 scores). A higher score indicates a better quality of life. (C) The PEM measures the participant's perception of various psychosocial outcomes (n = 10, 8, 11, 9, 3 items in the self-efficacy, embodiment, social touch, body image, and prosthesis efficiency sub-scales, respectively). For each sub-scale, a higher score indicates a better outcome. (D) The RHI survey is a measure of the extent to which the participant embodied the sensory-enabled prosthesis (n = 3, 6 items in the embodiment and control groups, respectively). Positive values indicate more agreement with the survey statements and negative values indicate more disagreement with the survey statements. For (A–D), asterisks denote significant differences between groupings (p < 0.05). (E) A short form of the PEM was administered daily at-home (n = 48 days of wear). Longitudinal trends were analyzed. Stars denote statistically significant trends over time (p < 0.05).

personally relevant tasks at-home through the modified OPUS UEFS difficulty rating and the PSFS, respectively. The modified OPUS UEFS difficulty rating did not improve with sensation (2-sample t-test p = 0.584) (**Supplementary Figure S2C**) or over time (line slope test, 0.355) (**Figure 6B**). However, his ratings on the PSFS improved significantly across the duration of the study (MDC-95 = 1.3, total improvement = 5.4 points) (**Figure 6C**). The most dramatic improvement in PSFS scores occurred within the first month (3.4 points), but continued to improve significantly within the last month of the study (1.6 points). The participant's self-identified tasks in the PSFS included peeling vegetables, cutting food and meal

administered approximately weekly at-home (n = 9 scores). (C) The PSFS measured the participant's perceived ability to do a set of functional activities chosen by the participant. The PSFS was administered prior to the start of the study and then approximately monthly during the participant's visits (n = 4 scores). Crosses denote significant differences between groupings based on MDC-95 values. (D) The foam block task objectively measured the participant's ability to identify objects with the prosthesis during laboratory sessions (n = 4 scores). (E) The proportion of tasks on the modified UEFS that the participant reported completing each week with the prosthesis (n = 9 scores). (F) The participant's active usage of the prosthesis at-home was measured by the number of presses on the prosthesis sensors logged by the system each day (n = 48 days of wear).

preparation, holding someone's hand, folding clothes, and putting away dishes.

Surprisingly, the participant's actual ability to perform functional tasks with the sensory-enabled prosthesis, as measured by laboratory performance of the foam block test, did not improve over time (line slope test p = 0.562) (**Figure 6D**). However, as with the participant's confidence in the task, his objective ability to perform the task was significantly better with sensation compared to without (**Supplementary Figure S2B**, paired t-test p = 0.003).

Finally, we measured the participant's engagement with the prosthesis using the modified OPUS UEFS completion rate and daily sensor presses. The modified OPUS UEFS completion rate assessed how many tasks out of the 28 task UEFS list the participant chose to attempt each day. The daily sensor presses indicated how many times the participant actively interacted with the prosthesis by pressing on the sensors or performing a grasp with the prosthesis. Neither of these measures significantly increased over the course of the study (line slope test, p = 0.199 and p = 0.975, respectively) (**Figures 6E,F**). However, the participant did use the sensory-enabled prosthesis significantly more actively than a standard prosthesis, as indicated by the increase in modified UEFS tasks completed and sensor presses compared to data without sensation from a previous study (2-sample t-test, p < 0.001 in both) (**Supplementary Figures S2D,E**).

### Qualitative Findings of Sensation Experience, Prosthesis Engagement, and Embodiment Change Over Time

The qualitative analysis resulted in the identification of four primary themes: sensation experience, learning, prosthesis engagement, and embodiment. The sub-categories within each theme and a brief description of each is provided in **Table 2**. We present each of the themes below with rich text exemplars. Brackets denote interjected text to improve readability, and parentheses denote gestures. Each exemplar's transcript number is indicated at the end of the quotation.


TABLE 2 | Primary themes and associated sub-categories generated through the qualitative analysis.

All themes were modulated by time, in that continued exposure to the sensory feedback and extended usage of the sensory-enabled prosthesis led to long-term changes in experiences. The qualitative analysis also produced a secondary theme, system operation, which is described in **Supplementary Data Sheet S3**. This theme included comments related to technical functionality of system components, and as it did not change over time, it did not address our primary research question of long-term changes in sensation or experience. Full definitions for all themes and sub-categories are provided in **Supplementary Table S3**.

In short, the participant described key changes in his experience over time within the four primary themes (**Figure 7**). These changes occurred either within-days or across the duration of the study. Within individual days, the sensation intensity became stronger, the sensation location became more focal, and the phantom limb became better aligned with the prosthesis. Over the course of the study, the two most prominent changes were increases in the naturalness of the experience and increases in prosthesis engagement. The naturalness of the experience included both the sensation experience and the prosthesis experience. In addition, over time, his perception of the sensation experience began to include perceptions of stereognosis. Stereognosis is defined as "the ability to perceive the form of an object by using the sense of touch" and includes the perception of object features such as shape, size, and weight (Carlson and Brooks, 2009). The increase in prosthesis engagement was exemplified by the participant's reported increased willingness to do functional and bilateral tasks with the prosthesis and to use the prosthesis in social interactions. Through both passive and active learning mechanisms, the sensations became more useful in accomplishing tasks over time and the ease of using the prosthesis increased. Throughout the study, the participant described his preference for sensation and demonstrated embodiment of the prosthesis.

### Sensation Experience

The sensation experience theme explains how the participant perceived and interacted with sensation over the course of the study. The participant described three dimensions of sensation – intensity, location, and quality. He described the intensity of the sensation as providing information about his grip strength with the prosthesis.

"The stronger it gets, along with the pressure and everything, tells me how hard I'm touching something." [T3]

The participant also described how perceived sensation location provided information about which prosthesis sensor was engaged.

"When I press [the sensors] individually they still feel the individual spots and everything . . . according to which one I press on, where sensation is at." [T2]

He stated that the intensity of sensation would generally increase and that the location would sometimes become more focal over a day of use.

"Usually intensity just gets stronger throughout the day." [T2]

"Some days it would localize a little more, like umm sometimes in the morning it'd feel like it was index and the thumb and later in

the day or when I'd take it off and do the paperwork in the evening, it felt like it was localized in just like the index finger." [T1]

Interestingly, he did not report across-day changes in sensation type or location over the duration of the study. However, he did report that he was learning how to understand the sensations better with time.

"Not really different every day because they're pretty much in the same place, same sensation every day. So it's just a matter of learning how to try to hone it in better." [T2]

The participant reported that the sensory experience became more natural over the course of the study.

"It just felt more natural, like you were actually grabbing and holding something like that (picks up water bottle with intact hand and transfers to prosthesis to hold)." [T3]

In later interviews, the participant began describing how the individual sensations provided by the sensors began to "work together" such that the sensory experience became more holistic and complete.

"It felt more like one large global, it just felt like [the sensations] were all working together, merging together. It was my hand grabbing it, not just these fingers and then I could feel the other. It just kinda, all was kinda together." [T2]

Further, he indicated that he began to have the experience of holding an object in his hand (stereognosis), because the sensation experience reflected the shape of the object he grasped.

". . . it feels like (grasps water bottle with prosthesis) something's right in that area right through there (gestures at thumb through index finger). So I can feel the light pressure, the vibration and everything and it- it just- some of it just feels natural because it feels like (gestures at prosthesis) something is right there, that shape, that you're grabbing. Which a lot of times it's something round I'm grabbing anyway, so that's what it feels like. Holding a cup? I'm holding a cup." [T3]

The sensory feedback altered his perceived ability to do tasks and his ability to engage in social interactions.

"So, that's one of the best experiences with it, is being able to play with them [grandchildren] and know that I'm not hurting them. I'm grabbing and squeezing, and I can tell how hard I'm holding them and things. Umm, we've been out places, and people come up and shake hands and stuff." [T3]

As a result of having the sensory feedback, the participant was more willing to use his prosthesis rather than only relying on his intact, left hand.

"It's just umm, it's [the sensation] helped me to start using my right arm more than what I have been and everything with the prosthetic on, because I can actually feel when I grab and touch stuff." [T3]

The participant reported an overall positive experience with sensory feedback. He frequently reported that the "sensory feedback was good" [T3] and that he "liked feeling my hand" [T2]. He reported a preference for having sensory feedback, saying:

"I prefer having the sensory feedback. It gives me that feedback I need to know what I'm doing with that [prosthetic] hand, and how hard I'm squeezing on things, how hard I'm touching people. And the feedback's just great." [T1]

At the end of the study, the participant affirmed that he would like to wear the sensory feedback system again in the future, stating:

"Oh yeah. Yeah. Def-def- definitely. Because, like I said, I feel better with the sensation than without it." [T3]

Although the participant experienced several extended illnesses and breakages of components that interrupted his study, he explained that these problems did not change his desire to have the sensory feedback.

"I mean, yeah, it'd be nice if that all worked right, but I still prefer having [sensory feedback] as not having it."

#### Learning

The theme of learning explains how the participant improved his ability to use his sensory-enabled prosthesis over the course of the study. When asked if he felt like he was learning to use the sensory feedback, he replied:

"That's the perfect way to put it. Umm, gaining skills, umm learning how the sensation, basically, which sensation is. How strong, how weak, and those sort of things. To make me use it even better." [T2]

Throughout the study, the participant described sensory feedback as being useful for performing functional tasks or interacting with others. He predominantly described using the sensation intensity to regulate his grip force:

"I could tell if it [sensation] started to lighten up to squeeze a little harder and everything and hold [the dishes] while I opened the cabinet door (mimes opening an overhead cabinet with intact arm) and reach up and put them in (mimes bimanually putting plate in overhead cabinet) and all." [T3]

He learned to interpret the intensity of the sensation as related to his applied grip strength, saying:

"But I can do it real light like that (gently pinches index sensor, then middle finger sensor), so it feels like that, or I can press real hard (increases pressure on middle finger sensor). . . I'll do it to umm feel the difference in the pressure, to feel how hard I'm grabbing things, to know how hard I'm grabbing things or how lightly I'm grabbing them." [T3]

He described the process of learning to use sensory feedback as happening both passively and actively. In passive learning, the improvements in sensory feedback discrimination occurred through exposure to the sensory feedback over time.

"The more I wear [the prosthesis] with the sensory feedback, the more feedback I get, the easier it is to use, the better it is to use, tell the difference in the sensation or the pressure.''

In active learning, the participant described how he purposefully practiced receiving different levels of sensory feedback and interpreting it correctly as grip strength.

"And then there's times, like I say, I just sit down and want to feel my hand so I'll do like this (squeezes on intact hand with prosthesis). How light can I go? How strong can I go? And work on feeling the difference." [T2]

"A lot of it was practice . . . Pressing and start squeezing to feel how strong it would go, how light I could get it and that (repeats squeezing motion on sensors). That's where a lot of it, for me, came from and everything. And actually doing it (makes reach and grasp motion with prosthesis). Actually grabbing things." [T3]

As he practiced interpreting the sensation, he described improvements in several hallmarks of learning. He described an improved ability to perform tasks with a decrease in errors or accidents:

"And umm less apt to have those accidents of dropping something with this (makes grasping motion with prosthesis), because I feel [the sensation] start to lighten up a little bit (mimes startle reflex – twitches shoulders up) and I'll squeeze back tight again and all. So that's one difference." [T3]

As another indicator of learning, he described decreases in effort to perform the task:

"I got a light grip there to a heavy grip there. And learning the difference in that. Which, you need to think about it a little bit, but more I wear it, the less I have to think about them. the more it becomes natural." [T2]

"I can tell where I'm at and what I need to and I can hold it there. It's just easier with the sensory cause I don't have to watch it as close." [T2]

### Prosthesis Engagement

fnins-13-00853 August 19, 2019 Time: 18:8 # 16

The theme of prosthesis engagement addresses the participant's views on his usage of the prosthesis and the extent to which he wanted to or was willing to wear and use the prosthesis. The participant described how having sensory feedback made him want to wear and use the prosthesis more. Prior to the study, he described a lack of confidence in his standard device:

"Just grabbing dishes and putting them up in the cabinets and everything with my left [intact] hand a lot of times and not the right because it was just easier, faster, and didn't feel as confident. . ." [T1]

When asked if sensation changed the way he used the prosthesis, he said:

"I started using it more and started wearing it more." [T2]

The participant described certain tasks that he was more likely to do or try now that he had sensation in the prosthesis.

"I was more apt to grab ahold the grandkids and grab other things around the house that I normally don't and was getting to where I'm using the hand more and more than what I have in the past years without sensation, and wearing it a lot more." [T2]

Having sensation also motivated the participant to want to engage the prosthesis more actively in everyday tasks. Instead of using only his intact hand for tasks, he was more willing to try tasks with the prosthesis.

"It's just starting to feel more normal to me, and using it more often. Using it to try to do things, umm, using it to try to grab dishes out of the dishwasher, plates and that kind of stuff, and put away rather than using my one – doing my one hand. Used to be, even when I had it on, I'd, I don't know why, I'd just hold back on it, and grab with my good hand. Now, cause I can feel, I mean, I have a tendency to use it more." [T2]

In addition to using the prosthetic hand more frequently in skilled unilateral activities, the participant was also more willing to use the prosthesis in bilateral activities.

"I'm more apt to put a knife in here (points to prosthesis). Or a fork in here and hold the fork and cut my food up and stuff with the knife and all. Umm even being able to hold a potato and tell that I'm holding it and how hard I'm squeezing it while trying to peel a potato (mimes peeling a potato while holding the potato in the prosthesis and a peeler in the intact hand) and stuff. It's all different when you can feel it and when you can't." [T3]

He also was more likely to use the prosthesis to interact with other people, such as when shaking hands.

"We've been out places, and people come up and shake hands and stuff. And that's a big difference too because used to be, it'd be my left hand (gestures with intact hand), and now I'm more apt to put my right arm out there (gestures with prosthesis) to shake somebody's hand." [T3]

His willingness to use the prosthesis more actively appeared to change over the course of the study.

"The more I wear the system, the more I use my right hand to help the left hand, whereas before I used to just keep it off to the side more." [T1]

Using the prosthesis more also helped him to learn how to interpret the sensory feedback, which then led to him wanting to use the prosthesis more.

"A lot of it was practice just like this (squeezes repeatedly on thumb sensor). . . Actually grabbing things, and just make myself use it more. And then want to use it more." [T3]

The participant also explained how his willingness to wear the prosthesis increased across months of the home study. He described this change as being directly related to his increased confidence in using the device correctly with sensory feedback:

"I would use the hand even more this past month than what I did the previous month to do stuff and everything with the sensation and all. Because, confidence was building. It was becoming more normal." [T3]

### Embodiment

The theme of embodiment addresses concepts related to limb ownership, body schema, body image, and perception of the phantom limb. The participant reported that the sensory experience facilitated his experience of device embodiment, and that both of these factors contributed to his prosthesis experience becoming more natural over time.

The prosthesis felt more like a part of his body when the participant had sensation, as if it were his actual hand rather than a tool.

"Well, it feels more like it's my hand and more like a part of me than just a tool that I'm using." [T1]

The participant expressed ownership of the prosthesis as if it belonged to his body. The prosthesis was viewed as being part of the body even though its attachment to his body was not permanent.

"Yeah, cuz, view it [the prosthesis] more of being a part of me. Me, my limb, even though I take it off at night and everything, still being my limb when I'm wearing it. Because I can feel what I'm doing with it, or at least tell that I'm touching things." [T2]

The perception that the prosthesis was a part of him was associated with the position of his phantom hand. The participant explained that, when he had sensation, the position of his perceived phantom became more congruent with the prosthesis position. It appears that the sensory feedback led to decreases in limb telescoping, as the length of his phantom limb began to extend toward the fingertips of the prosthetic device.

"Yeah, because it feels more like my hand and not just a tool extended from my arm. Because I can actually feel it. And, wearing it with the sensation actually, proprioception is putting it right

about where the hand is at (gestures with prosthesis) with the prosthetic. It feels like, my fingertips feel like they're about right in here (points to distal interphalangeal joint of prosthetic fingers), which is pretty close to where they are. Umm, when I'm not wearing [the prosthesis], getting up in the mornings and everything, and I think about [the phantom], then it feels like it's back where it is (gestures to residual limb). . . I think it kinda changes when I start feeling the sensation in the fingers (grasps thumb sensor with intact hand) when I put it on and start calibrating it in the morning. For some reason, it just feels like that's where it's at, and not back where I'm amputated at (gestures at residual limb)." [T3]

The experience of embodiment was interconnected with the experience of normalcy or naturalness of the prosthesis.

"I mean, like I say, when you got the sensation it just it- it feels like my hand, so it feels more natural." [T3]

The naturalness of the experience was also related to the position of the phantom limb. When asked to define the way he was using the word "natural" in the interview, the participant explained:

"It feels like my hand's actually there, and it's not just, like I said, a tool added on. It umm brings the sensation out to where my hand was, in the fingers, wearing it and all. And it gives me sensation of my fingers that I can feel." [T3]

The participant remarked repeatedly that "the more I have the system on and wear it, the more natural it becomes to me" [T3]. In fact, the participant reported on the increase in naturalness or normalcy of the sensation over time at least a dozen times throughout the interviews.

"The more I used it, the more natural, more or less, it became to me. The more, felt more normal, not necessarily like my left hand, but more normal to me." [T1]

The participant also described how the sensation experience became more natural throughout the study.

"[The sensations] just felt more natural to me. Sometimes it felt more like pressure than necessarily the vibration and sometimes movement and everything. And it just, like I said, felt like somebody squeezing on those three fingers or just grabbing and holding them or like holding something in that area (gestures at water bottle)." [T2]

In the third month, the participant described that, although the sensation itself wasn't changing, the naturalness of the experience continued to improve.

"Sensation didn't really change other than the fact that, to me, it just became more and more normal and natural." [T3]

### DISCUSSION

The participant in this case study used the sensory-enabled prosthesis for a total of 49 days over the course of the 115-day study and reported overall positive experiences with the sensory restoration system at-home. To our knowledge, this is the longest home use trial of a sensory-enabled prosthesis and is also the first study to examine the effects of learning on artificial sensation produced by electrical stimulation of the peripheral nerves. While the participant was not blinded to the presence of sensation over the course of the study, he was blinded to the objectives and hypotheses of the study in order to limit potential biases in his responses. In this study, we showed that perception of both evoked sensations and the phantom limb changed with prolonged exposure to sensory stimulation to become more congruent with the information provided through prosthesis use. Further, we found that psychosocial survey scores of self-efficacy, prosthesis embodiment, body image, social touch, and prosthesis efficiency were significantly higher while using the sensory feedback prosthesis at-home than at pre-test. Finally, we showed that perceived function significantly improved with sensation and usage. These findings were corroborated by the outcomes from the qualitative analysis, which described the subjective experiences of sensation, learning, prosthesis engagement, and embodiment. While limited conclusions can be drawn about generalizability from this case study, our findings agree with results reported in the sensory substitution (Dietrich et al., 2012, 2018) and cochlear prosthesis literature (Fu and Galvin, 2007, 2012), lending credence to our results. Our findings suggest that both passive and active learning modulate the perceptual and psychological experience of a sensory-enabled prosthesis over time.

### Implications of Changes in Sensory Perception Over Time

The participant reported changes in the sensory percepts produced by stimulation both within single days of use and across the duration of the study. Within-days, the perceived sensations changed to become more congruent with transduced sensor information. Several cognitive processes may drive these observed short-term sensory changes. In the intact system, information acquired from multiple sensory modalities, such as vision and touch, is integrated to form global percepts of the environment or specific objects within the environment (Ernst and Bülthoff, 2004; Lederman and Klatzky, 2009; van Atteveldt et al., 2014). When there are significant discrepancies in information between modalities, tactile perception is momentarily altered such that it aligns with the presented visual information in a phenomenon known as visual capture (Lederman et al., 1986; Shimojou, 1987; Pavani et al., 2000; Ernst and Banks, 2002; Deneve and Pouget, 2004; Kersten et al., 2004; Körding et al., 2007; Gallace and Spence, 2008; Giummarra et al., 2008; Stanford et al., 2008; Lederman and Klatzky, 2009; van Atteveldt et al., 2014). When grasping objects with the prosthesis, the participant would have received visual cues about how the prosthesis sensors were contacted while feeling somatosensory feedback. Thus, multi-sensory integration and visual capture likely influenced the participant's tactile perception during object interactions.

However, visual cues were minimized when the participant was completing the morning and evening surveys about his sensory experience, because he directly activated sensation through button presses on the neurostimulator rather than triggering sensation via the prosthesis sensors. Thus, visual

capture alone cannot explain the increase in congruency of the evening sensation reports. Prior studies in short-term sensory learning have shown that repeated association of coincident stimuli can induce transient changes in both sensory cortical mapping and cortical activity levels. While these changes persist for a few hours after the presentation of the associated stimuli, they are fully reversible and disappear within 8–24 h (Rossini et al., 1994; Godde et al., 1996; Oliver et al., 2007). Thus, the within-day location changes we observed in early stages of this study may have arisen from reversible cortical changes which occurred because of the reinforcement of the visually captured sensory percepts as the participant interacted with various objects throughout the day. Given the sparse withinday trends for sensation quality, sensation quality may not be as influenced by visual capture or short-term cortical changes as sensation location.

Over the course of the 115-day study, the prevalence of within-day location changes decreased and long-term changes in location, intensity, naturalness, and quality emerged. The sensations became permanently aligned with the prosthesis sensors, such that the sensations matched the transduced information immediately upon system donning. The accumulation of experience with prosthesis-object interactions throughout the study enabled repeated multisensory associations of the evoked sensations with other sensory modalities and promoted passive learning of the sensation (Bogacz, 2007). In the interview, the participant also described active learning strategies, in which he purposefully practiced interpreting sensation. Over time, these passive and active learning mechanisms led to the long-term entrenching of the congruous sensory percepts.

The long-term somatosensory learning observed over the course of the study could have been driven by mechanisms related to cortical plasticity. The somatosensory cortex remains highly plastic through adulthood, and changes in sensory cortical representation following amputation are well-studied (Merzenich et al., 1984; Pons et al., 1991; Lund et al., 1994; Borsook et al., 1998; Huttenlocher, 2002; Weiss et al., 2004). Adult somatosensory plasticity is also a key driver of recovery after nerve injury (Bach-y-Rita, 1990; Lundborg, 2000; Fraser et al., 2002; Navarro et al., 2007; Knox et al., 2015), and, depending on the extent of the injury, the time courses associated with cortical reorganization can range from weeks (Merzenich et al., 1984; Borsook et al., 1998) to a year (Pons et al., 1991; Lund et al., 1994). In addition, prior studies have shown that both passive and active learning regimens in cochlear prostheses are associated with cortical plasticity (Tremblay et al., 1997, 1998; Fu and Galvin, 2007; Gentner and Margoliash, 2009; Sharma and Dorman, 2012). Analogous cortical plasticity mechanisms to those observed in cochlear prosthesis use may have driven the long-term ingraining of congruous artificial tactile percepts over the study duration. Future neuroimaging studies would be necessary to confirm whether the long-term perceptual changes shown here correspond to cortical changes indicative of plasticity.

As the participant became more familiar with the sensory feedback over the course of the study, higher-level sensory experiences began to emerge. The qualitative analysis demonstrates that the participant began interpreting sensory feedback from all sensors holistically as opposed to separately, providing the experience of stereognosis. Stereognosis is considered a complex, emergent property of the sensory experience that requires integration of tactile sensations, proprioception, and motor intent (Carlson and Brooks, 2009). Further, the ability to merge somatosensory information into a single object percept is dependent on familiarity with the object: people are better at identifying common objects like hammers than novel nonsense shapes (Klatzky et al., 1985; Ernst and Bülthoff, 2004). In addition, stereognosis of novel objects can improve upon training (Simmons and Locher, 1979). At early time points, the participant did not appear to experience stereognosis. However, in later interviews, the participant reported "global" percepts that matched the shape of objects he held most frequently, such as cups and bottles. Despite being familiar with these everyday objects prior to entry into the study, stereognosis only emerged over time after gaining repeated exposure to these object interactions and learning the sensory feedback. We hypothesize that the experience of stereognosis emerged as learning modified the participant's sensory experience and the sensory feedback integrated with his prosthesis motor control.

We also observed an increase in the perceived naturalness of the sensation experience over the course of the study. Sensation naturalness is an important and controversial concept in somatosensory neuroprostheses. Typically, neuroprosthetics research groups discuss naturalness in the context of sensation quality. Electrical stimulation of the nerves typically results in unnatural feelings of paresthesia or "tingling" (Schady et al., 1983; Macefield et al., 1990; Kaczmarek et al., 1991; Geng et al., 2012; Perovic et al., 2013; Clark et al., 2014; Ortiz-Catalan et al., 2014; Tan et al., 2014; Saal and Bensmaia, 2015), although several stimulation approaches have been shown to improve the perceived naturalness of the artificial sensation quality (Tan et al., 2014; Valle et al., 2018). The participant rarely indicated the word "tingling" on the quantitative surveys throughout this study. Thus, the significant increases in sensation naturalness that we observed for two channels over time suggest that the concept of naturalness is related to familiarity with a particular sensation, regardless of its quality. In fact, while the participant occasionally described the naturalness of the sensation quality in the interviews, he more frequently described the naturalness of the sensation experience or of the prosthesis experience. He discussed how the more he used the prosthesis, the more natural the experience became. He also used the words "natural" and "normal" interchangeably, further corroborating the interaction of naturalness and familiarity or expectation. Based on this evidence, we believe that the naturalness of a sensation is a top-level interpretation of the normalness or familiarity of sensory information, and as such, may be impacted by other cognitive factors. Importantly, this increase in naturalness over time, which is likely related to passive learning of the artificial sensation, indicates that neuroprostheses do not have to perfectly reproduce sensory percepts to be useful. By simply approximating the correct inputs with electrical stimulation, cortical plasticity may assist in producing interpretable and natural sensations.

It is encouraging that the participant was able to undergo learning within the time course of this study. Unlike cochlear prostheses, whose users receive continuous auditory feedback from the time of implant, the participant in this study only received somatosensory feedback when he chose to wear the device. These choices and various interruptions throughout the study limited the continuity of his sensory feedback exposure. Because this is the first long-term home study of artificial somatosensory feedback, we do not know the extent or time course of washout due to intermittent periods of non-use. Despite these periods, we still observed significant changes in sensory perception. We hypothesize that the effects observed in this study would be further strengthened if there were fewer periods of non-use, but additional studies are needed to confirm or refute this hypothesis. Current studies in the neuroprosthetics field are attempting to understand the effects of manipulating stimulation parameters on sensation independently of any long-term learning effects (Clark et al., 2014; Graczyk et al., 2016, 2018a; Oddo et al., 2016; Wendelken et al., 2017; Valle et al., 2018). The results from this study indicate that learning strongly modulates sensory perception and should be controlled for in the laboratory setting. In addition, these changes in sensory feedback due to learning indicate that prosthesis users could benefit from training paradigms to promote active integration of sensory feedback into prosthesis utilization strategies.

We also do not know if there is a difference between periods of non-use in which the participant received no sensory feedback and periods of non-use in which the participant received erroneous or random sensory feedback due to malfunctioning system components. Our post hoc comparison of data from days in which the index sensor functioned at the beginning of the study to days in which it malfunctioned later in the study indicates that receiving incongruent or random feedback promotes maladaptive learning. This suggests that future studies should be careful to reduce or eliminate exposure to incongruent or erroneous sensory feedback in order to maximize useful sensory learning. Further, only limited information, pressure or aperture, was transduced from four sensors in this study. Additional work is necessary to quantify whether perceptual changes can extend to additional sensors, such as those required for a dexterous hand, or sensation types, such as textural features of an object.

### Implications of Psychosocial and Functional Changes Over Time

As artificial sensation was assimilated into the prosthetic hand experience and learned over time, the participant's perceived functional ability and psychosocial outcomes improved, but with varying time courses. Several outcomes appeared to be predominantly influenced by the presence or absence of sensation instead of accumulation of sensation experience. For example, the participant performed better identification of foam blocks with sensory feedback than without at every testing interval. Similarly, specific subscales of psychosocial outcomes evaluated by the PEM, including self-efficacy, embodiment, social touch, and body image, reached a plateau in improvement within the first interval of usage. Given that our prior 2-week home study also showed improvement on these measures (Graczyk et al., 2018b), future studies of these types of specific psychosocial outcomes may not require extended home usage.

Embodiment also appeared to be more influenced by the presence of sensation than by time. Both the quantitative and qualitative analyses indicated that using the prosthesis with sensation led to increases in prosthesis embodiment. Embodiment is a complex phenomenon that includes both conscious and unconscious processes (de Vignemont, 2010b). The conscious perception of the body image (Gallagher and Cole, 1995; Gandevia and Phegan, 1999) was measured with several surveys, including the PEM and RHI, and demonstrated that the participant viewed the prosthesis as part of the body (Murray, 2004). The conscious perception of self-attribution, or ownership of the prosthesis as belonging to the self (Tsakiris and Haggard, 2005; Tsakiris, 2010), was also indicated by the participant's usage of possessive pronouns to refer to the prosthesis as "my hand" in the interviews. Although we utilized primarily self-report surveys and interviews, which inherently measure conscious perception, our data also indicates that unconscious aspects of embodiment improved throughout the study. The reported changes in the perceived position of the phantom hand over time likely indicate a change to the body schema, which is the sensorimotor internal model of the body and an unconscious aspect of embodiment (Maravita and Iriki, 2004; Gallagher, 2005; Giummarra et al., 2007; de Vignemont, 2010a). The extension of the phantom toward the prosthetic fingertips could be considered a type of proprioceptive drift, a common indicator of embodiment in which the proprioceptive sense of the hand position moves to become aligned with the tool (Tsakiris and Haggard, 2005; Longo et al., 2008; de Vignemont, 2010b; Kalckert and Ehrsson, 2014). The alignment of the phantom with the prosthesis is evidence that the phantom successfully merged with the prosthesis (Giummarra et al., 2010).

Given that both tool embodiment and perceptual learning involve similar neural changes in sensory cortices (Iriki et al., 1996; Hoffman and Logothetis, 2009; Miller et al., 2017), the experience of embodiment may have been intricately linked to the sensory learning of the artificial somatosensory percepts. As the participant learned the artificial somatosensation, he may have refined his ability to integrate the artificial sensation into his existing body representation, leading to more efficient and accurate prosthesis actions. The participant's reports of decreases in the attention required to use the prosthesis could indicate incorporation of the prosthesis into the body schema, since peri-personal space is prioritized in attention (Reed et al., 2006) and embodiment expands peripersonal space to include the peri-tool space (Galli et al., 2015; Miller et al., 2017). Interestingly, the decrease in phantom telescoping experience may have also been influenced by cortical changes that occurred through these learning mechanisms. Prior research has shown that telescoped limbs are associated with cortical remapping of distal limb segments onto nearby regions of the somatosensory cortex (Giummarra et al., 2007). The decrease in phantom telescoping could have occurred as sensory learning reorganized the sensory

cortex to again distinguish the hand area from the arm area in the cortex.

In contrast to these fairly immediate effects of sensation, the holistic measures of the participant's quality of life (OPUS QoL) and perceived disability (QuickDASH) improved but had not yet plateaued by the end of the 115-day study. Several measures of perceived prosthesis function, such as the PSFS and foam block confidence, similarly significantly increased through the study without reaching a plateau. This aligns with results from previous short-term training studies in which participants' perceived function improved after two weeks of training with a sensory-substitution prosthetic system (Dietrich et al., 2012, 2018). Surprisingly, the participant's in-lab performance of the foam block identification task with sensory feedback did not significantly improve over the course of the study, although it did trend upward. This may be due in part to the choice of functional task. Blindfolded recognition of foam blocks does not have a direct analog in the home setting and thus may not have been influenced by passive learning. The foam block test also does not have established validity or reliability, and it may not be sensitive enough to detect changes due to learning. Further, work in cochlear prostheses suggests that passive learning may take up to 12 months to occur (Watson, 1991; Fu and Galvin, 2007, 2012). If passive learning of somatosensory feedback occurs on a similar time course, it is possible that improvements in functional and psychosocial outcomes that are driven by sensory learning will not be detectable until later points in the learning process. Studies with longer-term follow-up are needed to confirm or refute this hypothesis.

The qualitative analysis demonstrated that the participant's engagement with his prosthesis increased as he learned to utilize sensory feedback. He reported being more willing and more likely to use the prosthesis in performing skilled single-handed or bilateral tasks and in engaging in social interactions. He frequently described performing tasks with the sensory-enabled prosthesis that he did not do previously due to a lack of confidence in his ability to perform them correctly. For example, he became more willing to handle dishes with his prosthesis when unloading the dishwasher, because he believed that the sensation would give him information about slip and allow him to correct grasping errors before dropping a dish. However, quantitative indicators of prosthesis usage, such as the modified UEFS Task Completion metric, did not show significant increases over the course of the study. This may be because the specific tasks on the modified UEFS metric did not reflect the types of tasks the participant became more willing to do throughout the study. For example, the modified UEFS does not include any tasks related to social interactions (Jarl et al., 2014). Further, many tasks in the modified UEFS, such as buttoning a shirt or writing, require extensive mechanical dexterity of the hand and wrist. Performing these tasks with the single DOF hand provided to the participant could be difficult regardless of sensory feedback. In fact, the participant discussed that the mechanics of the sensory-enabled hand limited his ability to perform some tasks and mentioned a desire for a sensory-enabled dexterous hand.

If sensory feedback can increase the willingness to use a prosthesis or lead to more active use, this could have implications for the health and well-being of upper limb amputees. Approximately 28% of upper limb amputees reject their prosthesis, stating that they feel more functional without any prosthetic device than with one (Biddiss and Chau, 2007a,b). Sensory feedback could reduce the visual attention required to perform tasks with the device, which is a desired prosthesis improvement (Atkins et al., 1996). Thus, sensation may help reduce abandonment of prostheses by providing additional motivation to use a prosthetic device and improving perceived prosthesis function. In addition, relying solely on the intact contralateral limb often leads to overuse injuries in the contralateral limb and trunk (Burger and Vidmar, 2016; Postema et al., 2016). Active engagement of the prosthesis in activities through artificial sensory feedback could reduce overuse injuries, leading to enhanced quality of life and lower healthcare costs for this population.

### CONCLUSION

We studied the effects of 115 days of home-use of a sensory-enabled prosthesis on sensation experience, psychosocial outcomes, and prosthesis function in a single participant with acquired upper limb loss. Using mixed methods, we found that many aspects of the participant's experience changed over the course of the study. Perception of sensation location and quality changed over time to better align with the multisensory information acquired through repeated prosthesis usage. These sensory changes likely resulted from active and passive learning mechanisms, indicating that cortical plasticity can mediate sensory learning even for artificial sensations produced by electrical stimulation. In addition, prosthesis embodiment, confidence, and other psychosocial measures improved significantly over the course of the study. These psychosocial impacts often appeared within a month of athome usage, suggesting that sensory-prostheses can have rapid benefits for persons with upper limb loss. Finally, the perceived function of the prosthesis and active engagement of the prosthesis in tasks increased over the home trial. This study provides the first evidence that artificial somatosensation can undergo similar learning processes as intact sensation and highlights the importance of sensory restoration in prostheses.

### DATA AVAILABILITY

The raw data supporting the conclusions of this manuscript will be made available by the authors, without undue reservation, to any qualified researcher.

### ETHICS STATEMENT

All study devices and procedures were reviewed and approved by the U.S. Food and Drug Administration Investigational Device Exemption, the Cleveland Department of Veterans Affairs Medical Center Institutional Review Board, and the Department of the Navy Human Research Protection Program. All study procedures and experiments were performed in accordance with relevant guidelines and regulations of these institutions. The patients/participants provided their written informed consent to participate in this study.

## AUTHOR CONTRIBUTIONS

fnins-13-00853 August 19, 2019 Time: 18:8 # 21

IC designed the study, designed the analyses of quantitative sensory data, collected the data, performed the quantitative and qualitative analyses, interpreted the data, and wrote the manuscript. AG and LR designed the qualitative analysis, analyzed the qualitative data, and revised the manuscript. DT contributed to the study design and data interpretation, and revised the manuscript. EG designed the study, designed the analyses of quantitative and qualitative data, collected the data, performed the quantitative and qualitative analyses, interpreted the data, and wrote the manuscript.

### FUNDING

This work was supported by the DARPA Biological Technologies Office (BTO) HAPTIX program through the Space and Naval Warfare Systems Center (Pacific Contract No. N66001-15- C-4014), and by the U.S. Department of Veterans Affairs Rehabilitation Research and Development Service Program (Center #C3819C). The content is solely the responsibility of the

### REFERENCES


authors and does not represent the official views of the listed funding institutions.

### ACKNOWLEDGMENTS

We thank M. Schmitt for clinical management of the participants, as primary liaison to the local institutional review board, and for management of the study records. We thank M. Keith, J. R. Anderson, and K.J. Malone for evaluation and primary medical responsibility for the participants, surgical implementation of the system, and valuable clinical advice. We thank L. Yu for help with data entry and preliminary analysis. Special thanks to the Cleveland FES Center for medical illustration and the Cleveland APT Center for technical resources. We thank D. Weber and A. Emondi, Defense Advanced Research Projects Agency (DARPA) Hand Proprioception and Touch Interfaces (HAPTIX) program managers, for their support during the performance of these experiments. We also thank the participant for his patience and countless hours in the laboratory dedicated to improving the future of those with limb loss.

### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnins. 2019.00853/full#supplementary-material




changes in sensory cortex somatotopy following ischemic anesthesia. Brain Res. 642, 169–177. doi: 10.1016/0006-8993(94)90919-9


**Conflict of Interest Statement:** DT has patents on the electrodes (US Patent #6456866B1) and stimulation patterns (US Patent #9421366B2) related to sensory restoration. DT and EG also have a patent application on stimulation patterns related to sensory restoration (PCT/US2017/056070).

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Cuberovic, Gill, Resnik, Tyler and Graczyk. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Immersive Virtual Reality and Virtual Embodiment for Pain Relief

Marta Matamala-Gomez1,2 \*, Tony Donegan<sup>3</sup> , Sara Bottiroli4,5, Giorgio Sandrini1,2 , Maria V. Sanchez-Vives3,6,7† and Cristina Tassorelli1,2†

<sup>1</sup> Neurorehabilitation Unit, IRCCS C. Mondino Foundation, Pavia, Italy, <sup>2</sup> Department of Brain and Behavioral Sciences, University of Pavia, Pavia, Italy, <sup>3</sup> Institut d'Investigacions Biomèdiques August Pi i Sunyer (IDIBAPS), Barcelona, Spain, <sup>4</sup> Faculty of Law, Giustino Fortunato University, Benevento, Italy, <sup>5</sup> Headache Science Center, IRCCS Mondino Foundation, Pavia, Italy, <sup>6</sup> ICREA, Barcelona, Spain, <sup>7</sup> Departament de Cognició, Desenvolupament i Psicologia de l'Educació, Facultat de Psicologia, Universitat de Barcelona, Barcelona, Spain

A significant body of experimental evidence has demonstrated that it is possible to induce the illusion of ownership of a fake limb or even an entire fake body using multisensory correlations. Recently, immersive virtual reality has allowed users to experience the same sensations of ownership over a virtual body inside an immersive virtual environment, which in turn allows virtual reality users to have the feeling of being "embodied" in a virtual body. Using such virtual embodiment to manipulate body perception is starting to be extensively investigated and may have clinical implications for conditions that involve altered body image such as chronic pain. Here, we review experimental and clinical studies that have explored the manipulation of an embodied virtual body in immersive virtual reality for both experimental and clinical pain relief. We discuss the current state of the art, as well as the challenges faced by, and ideas for, future research. Finally, we explore the potentialities of using an embodied virtual body in immersive virtual reality in the field of neurorehabilitation, specifically in the field of pain.

#### Edited by:

Mariella Pazzaglia, Sapienza University of Rome, Italy

#### Reviewed by:

Eleonora Borelli, University of Modena and Reggio Emilia, Italy Bigna Lenggenhager, University of Zurich, Switzerland

#### \*Correspondence:

Marta Matamala-Gomez marta.matamala10@gmail.com †These authors share senior authorship

Received: 30 May 2019 Accepted: 29 July 2019 Published: 21 August 2019

#### Citation:

Matamala-Gomez M, Donegan T, Bottiroli S, Sandrini G, Sanchez-Vives MV and Tassorelli C (2019) Immersive Virtual Reality and Virtual Embodiment for Pain Relief. Front. Hum. Neurosci. 13:279. doi: 10.3389/fnhum.2019.00279 Keywords: embodiment, virtual reality, pain, ownership illusion, body illusion

## INTRODUCTION

Embodiment is defined as the sense of having a body, and the body can be considered to be both the subject and object of medical science and practice (Gallagher, 2001). One of the main goals in the field of cognitive neuroscience is to investigate how we experience ourselves inside a body as it interacts continuously with the environment. Historically, the bodily self has been described as "obvious and unproblematic" (James, 1890) and connected to a single somatic sensory system such as visceral interception (Damasio, 2000); however, more recently, embodiment has been described as being composed of several different structurally organized subjective components (Longo et al., 2008), as opposed to a single dimension. Hence, we feel our self as being inside a body, a body that moves according to our intentions (Kilteni et al., 2012a) and that interacts with the environment. Indeed, the sense of embodiment is thought to emerge from a complex interaction between bottom–up and top–down signals (Longo et al., 2008).

At first glance, experimental manipulation of embodiment might seem problematic; however, in the last few years, many studies have investigated bodily perception and revealed alternative

ways of manipulating embodiment by using fake body parts. One example of this is the rubber hand illusion (RHI) study, in which synchronous visuotactile stimulation of both a rubber hand located within the visual field of the participant, and the participant's real hand, located outside the visual field of the participant, confers an illusion of ownership over the rubber hand (Botvinick and Cohen, 1998). Since this study, many researchers have investigated how to manipulate body perception through the use of fake bodies such as a mannequins (Ehrsson and Petkova, 2008), mirrors (Ramachandran et al., 2009a), and virtual reality (VR) (Slater et al., 2008, 2010). Slater et al. (2008) were the first to replicate the RHI study in VR inducing ownership of a virtual hand based on visuo-tactile correlations, in an experience termed the "virtual hand illusion," while a similar ownership was successfully induced by means of visuomotor correlations in Sanchez-Vives et al. (2010) (see **Figure 1**). A number of studies have focused on the use of body illusions to address pathological conditions such as chronic pain, with the focus being on the analgesic effects of cross-modal perception (e.g., pain and vision) (for reviews, see Boesch et al., 2015, 2016; Martini, 2016).

Chronic pain, where the symptoms last beyond normal tissue healing times, is the most burdensome health issue worldwide in terms of years lived with disability (Vos et al., 2012) and economic cost (Gaskin and Richard, 2012). In some cases, the negative emotional experience of pain can even lead to suicidal intention (Campbell et al., 2016). Current management strategies including physical activity/exercise and psychological interventions such as cognitive behavioral therapy show shortterm effects only, with small effect sizes (Williams et al., 2012; Geneen et al., 2017), while pharmacological agents, such as opioids, have limited efficacy and carry significant risks and side effects (Hofmann et al., 2012; Carter et al., 2014). Indeed, the economic burden of prescription opioid misuse alone in the United States is estimated at \$78.5 billion a year, including healthcare costs, lost productivity, addiction treatment, and criminal justice system involvement (Florence et al., 2016). Many investigators have therefore attempted to look for new ways to manage pain states via non-pharmacological means (Carter et al., 2014). This paper presents a review of experimental and clinical studies that have explored the manipulation of an

FIGURE 1 | Experimental setups for (A) the rubber hand illusion (RHI), (B) the virtual hand illusion in non-immersive virtual reality, and (C) the virtual hand illusion in immersive virtual reality. Part (C) taken from Martini et al. (2015), reprinted with permission from Springer Nature.

embodied virtual body in immersive VR for both experimental and clinical pain relief.

## WHAT IS EMBODIMENT?

The capability of our brain of having a representation of our body results in a mental construction composed of perceptions and ideas about the dynamic organization of our own body, involving vision, touch, proprioception, interoception, motor control, and vestibular sensations (Maselli and Slater, 2013). In this regard, embodiment is defined as the sense of having a body. But to what are we referring when we talk about having a body? Longo et al. (2008) described it as follows:

The sense of [having] one's own body, variously termed "embodiment" (Arzy et al., 2006), "coenaesthesia" (Critchley, 1953), "bodily self-consciousness" (Bermúdez, 1998; Legrand, 2006), or "corporeal awareness" (Critchley, 1979; Berlucchi and Aglioti, 1997), has often been described as a non-conceptual, somatic form of knowledge, different in kind from other types of knowledge (e.g., Kant, 1965; Bermúdez, 1998).

Longo et al. (2008, p. 978)

These different descriptions of embodiment refer to the fact that we are able to feel the sense of having a body by integrating the different sensory signals arriving to our body, which our brain interprets to create a coherent representation of our self. In this regard, Longo et al. (2008) discuss the fact that others have described embodiment as the "storm-center of experience" arriving to our body, resulting in an essential factor for the construction of our internal life (James, 1905), and that other authors support the idea that embodiment is key for the construction of our inner self representation by demonstrating that the sense of embodiment is also closely related to the sense of self, and is strongly related to our individual psychological identity (Edelman, 2005; Cassam, 2012).

However, some investigations have shown that embodiment is divided into different subcomponents that form our body representation, such as body image and body schema (Gallagher and Cole, 1995). In this regard, it is known that body image and body schema play a fundamental, but clearly differentiated, role in understanding the sense of self and in individual psychological identity.

### CONCEPTUAL CLARIFICATIONS OF BODY IMAGE AND BODY SCHEMA

Gallagher (2001) has described body image as "an intentional content of consciousness that consists of a system of perceptions, attitudes, and beliefs pertaining from one's own body." In contrast, body schema has been described as an "automatic system of processes that constantly regulates posture and movement" and is mostly controlled by the sensorimotor system (Gallagher, 2001). One clear example of the difference between body image and body schema is the difference between perception of movement (conscious awareness of movement),

related to body image, and the final execution of that movement (motor performance), related to body schema.

Studies aimed at analyzing body image have distinguished three different intentional elements: (1) the subject's perceptual experience of his/her own body, (2) the subject's conceptual understanding of the body, and (3) the subject's emotional attitude toward his/her own body (Cash and Brown, 1987; Powers et al., 1987; Gardner and Moncrieff, 1988). The body image relies in the congruent inputs for all sensory and motor systems, and it has been described that experimental asynchronous multisensory stimulation results in distortion of body image (Perez-Marcos et al., 2018). In contrast, body schema is not the result of mental perception, beliefs, or attitudes, involving instead a system of motor functions or programs that operate "below" the level of self-referential intentionality, playing a dynamic role in governing posture and movement in a close automatic/subconscious way (Gallagher, 2001). While subconscious and automatic, body schema is not just a matter of mere reflex. Actions controlled by the body schema can be precisely shaped by the intentional experience or goal-directed behavior of one's own body (Gallagher, 2001). Therefore, once one becomes aware of perceptual limb position, movement, posture, pleasure, pain, and kinesthetic experience, such awareness contributes to the perceptual aspect of one's body image and such awareness may interact with one's body schema (Gallagher, 2001).

### THE BODY IN THE BRAIN

According to Melzack (1990), the body schema is controlled by a distributed neural network, or neuromatrix, mostly prewired by genetics, but flexible and open to the continuous shaping influence of experience. This network includes the somatosensory system, reticular afferents to the limbic system, and cortical regions that are important for self-recognition and recognition of external objects and entities. Somatosensory inputs to the brain from different modalities are essential for bodily awareness, especially those from proprioceptors, as demonstrated by Lackner (1988), in which he showed changes in body awareness using muscle vibration and other somatic manipulations. The sense of vision is also very important, as demonstrated by the evident anatomical distortions when congenitally blind subjects attempt to draw their own and other people's bodies (Critchley, 1979). Further, visual information regarding the hand's position is normally in accordance with the proprioceptive information regarding its position (van Beers et al., 1999). Tactile events regarding the body are strongly coupled with visual information (if available) of the same event (Pavani et al., 1999). Similarly, execution of movements is normally corroborated by congruent visual and tactile feedback (Janczyk et al., 2009).

### Brain Lesions and Body Representation

In addition to body perception disturbances in congenitally blind subjects, it has also been shown that brain lesions can induce profound changes in body perception and body representation (Aglioti et al., 2016). For example, some patients with righthemisphere lesions report the delusional perception that their contralateral limb or side of their body does not belong to them—a syndrome called "somatoparaphrenia" (Vallar and Ronchi, 2009; Jenkinson et al., 2013). These types of lesions allow us to explore the relationship between patients' subjective delusory perceptions and their structural brain deficits (de Vignemont, 2011), especially if those deficits concern areas that are traditionally considered to be multisensory. Further, some brain lesions, such as stroke and/or the resultant neuroplastic changes in the brain, might result in a specific alteration of the body schema or parts of it, as for example in stroke patients who have anosognosia (lack of self-awareness) for their motor and sensory defects and refuse to believe they are affected at all (McGlynn and Schacter, 1989; Levine et al., 1991), or stroke patients with personal neglect (Guariglia and Antonucci, 1992). Disownership of affected body parts can occur after right-sided brain damage (Loetscher et al., 2006), and has also been observed in chronic pain patients suffering from complex regional pain syndrome (CRPS) (Birklein and Schlereth, 2015). In addition, brain-damaged patients without amputations have reported the presence of multiple supernumerary body parts, mostly hands or feet (Halligan et al., 1993; Ramachandran and Blakeslee, 1999). Regarding neuropathic pain patients, limb amputee patients often present with body perception disturbances, such as the affected limb changing in size and form over time (Halligan et al., 1999). Body perception disturbances have also been demonstrated in patients with CRPS (Pleger et al., 2006; Lewis et al., 2007), chronic low back pain (Moseley, 2008), and other chronic pain conditions (Lotze and Moseley, 2007). Finally, body perception disturbances, specifically affecting body image, have been demonstrated in patients with spinal cord injury without brain damage (Fuentes et al., 2013). Part of these body perception disturbances are caused by alterations in the afferent inputs. When a body part is deafferented (deprived of sensory input), the feeling of an increased size of that body part often occurs. Such an effect is observed under local anesthesia, as well as in patients with spinal cord injury that perceived their torso and limbs elongated (Fuentes et al., 2013). Similarly, anomalous multisensory information provided experimentally on the body have been found to elicit a recalibration of the body image with an elongation of the stimulated body part (Perez-Marcos et al., 2018).

In order to study the mechanisms of body perception disturbances, early investigations were conducted in healthy people using devices such as fake limbs, prisms, mirrors, and cameras, which permitted the manipulation of body-related visual cues relative to other body-related sensory information, for example, tactile and proprioceptive cues. On the basis of these techniques, experimental studies on body perception used scenarios in which an external non-self-object was experienced as part of one's own body through multisensory and/or sensorimotor correlations between the real and the fake body or body part. For many psychologists and neuroscientists, these so-called body ownership illusions (BOIs) have constituted the main experimental method for disentangling body perception in healthy adults over the last 15 years (Blanke et al., 2015).


### BODY OWNERSHIP ILLUSIONS

How the brain represents our body is a fundamental question in cognitive neuroscience (Slater and Sanchez-Vives, 2016). How can we tell that our hand is part of our body and a physical object like a book is not? We generally believe that our own internal body representation is stable; however, some investigations have elicited the illusion of body ownership over objects that are not part of the body at all, which suggests that our body representation is actually highly malleable. In addition, out-of-body illusion research was reignited by Botvinick and Cohen (1998) with their RHI study. In the RHI study, perceived ownership of the rubber hand occurs because the brain's perceptual system resolves the sensory conflict between the congruent visuotactile information (the visual position of the rubber hand together with the tactile stimulus from the stroking) and the proprioceptive input (which indicates the position of the real hand) by prioritizing the importance of the visuotactile input over the proprioceptive input, integrating the two separate but synchronous inputs (visual and tactile) into a single prediction, as a result of which participants have the perceptual illusion that the rubber hand is their real hand. The visuotactile input is sufficient to override any contradicting proprioceptive input and produce the (incorrect) prediction that

the real hand is located closer to where the rubber hand is, a phenomenon known as "proprioceptive drift." Interestingly, if the visual and tactile stimulation are asynchronous, the illusion does not occur, suggesting that congruous multisensory input is required to produce the illusion. Later, Armel and Ramachandran (2003) demonstrated than when the rubber hand is threatened, there is a strong skin conductance response (SCR), indicating a physiological response to the threat. In this study, they argue that our body representation is continuously updated based on the stimuli being received. With synchronous multisensory perception, we can feel that a rubber hand is our real hand because the brain quickly generates the corresponding illusion as a way of resolving the contradiction between the visuotactile and the proprioceptive inputs (Slater and Sanchez-Vives, 2016).

Further, it has been shown that BOI may also be induced over the entire body in healthy subjects by using a mannequin (Petkova et al., 2011). In this study, healthy subjects observed an artificial body (a mannequin) through a head-mounted display connected to a two-synchronized-color video cameras oriented down at the mannequin body. As in the RHI study and in order to induce a BOI, participants received synchronous visuotactile stimulation at the same place in both the artificial and the real body. This whole body illusion is commonly known as the full BOI (Slater et al., 2010; Maselli and Slater, 2013). The full body ownership illusion from a first-person perspective is described as the feeling of owning an artificial body, which substitutes the real body as the origin of perceptual sensations. In this regard, some investigations have demonstrated that in order to induce a BOI, first-person visual perspective of the artificial body part or full body is key (Ehrsson and Petkova, 2008; Slater et al., 2010; Petkova et al., 2011). In addition to visuotactile stimulation and visual perspective, it has been shown that subjects may also experience the illusion when visuotactile stimulation is substituted by other modalities of multisensory and/or sensorimotor stimulation, such as sensorimotor contingencies in active or passive movements (Tsakiris et al., 2006; Sanchez-Vives et al., 2010; Kalckert and Ehrsson, 2012).

Hence, in the context of full-body illusions, self-location can be advantageously regarded as the combination of two parallel spatial representations: (1) an abstract allocentric representation of the body, mainly associated with visual perspective (first-or third-person visual perspective), and (2) an egocentric mapping of somatosensory sensations (visuotactile or visuomotor sensations) into the external space, mainly associated with peripersonal space. As reported by specific experimental paradigms adopted to induce out-of-body illusions, if these spatial representations are selectively or simultaneously altered, this could have implications for the sense of ownership of an artificial body (Maselli, 2015).

### EMBODIMENT IN VR

Nowadays, the integration of technology in the field of applied neuroscience such as VR systems allows the replacement of a person's real body with a virtual body representation, allowing the subject to feel embodied in a virtual body. In this regard, several investigations demonstrate that one may experience the sense of ownership over a virtual limb (Slater et al., 2008) and even an entire virtual body (Slater et al., 2010) by using immersive VR. In the latter study, Slater and colleagues demonstrated a full-body transfer illusion in which male subjects were able to embody a virtual female body. This finding was demonstrated subjectively (by questionnaire) and physiologically (through heart-rate changes) in response to an attack on the virtual body.

In addition, VR has been defined as a way to simulate reality and real-life situations (Slater and Sanchez-Vives, 2016). For example, it has been demonstrated that when a virtual knife stabs an embodied virtual body in an immersive VR environment, participants demonstrate an autonomic response and motor cortex activation in preparation to move the hand out of the way, just as they would in real life (González-Franco et al., 2014). Hence, anything that can happen in reality can be programed to happen in VR and be experienced as a real situation (Slater and Sanchez-Vives, 2016).

VR allows the experimenter to manipulate not only the virtual environment but also the embodied virtual body in ways that would be impossible in physical reality (Bohil et al., 2011). For example, immersive VR allows the manipulation of body representation in terms of structure, shape, size, and color, in ways that can contrast sharply with our own body image (Kilteni et al., 2012a,b; Banakou et al., 2013; Peck et al., 2013). Further, it has been shown that manipulating the characteristics of the virtual body may influence the physiological responses of the real body (Martini et al., 2013; Bergström et al., 2016), and may also modulate behavioral responses of the subjects (Osimo et al., 2015; Seinfeld et al., 2018). For this reason, immersive VR has been shown to have many potential applications in the fields of psychotherapy, rehabilitation, and behavioral neuroscience (for reviews, see Tarr and Warren, 2002; Martini, 2016; Riva et al., 2018), and even consciousness studies (for a review, see Sanchez-Vives and Slater, 2005).

### VR AND PAIN MANAGEMENT

At the beginning of the 21st century, VR was introduced to the field of pain management (Hoffman et al., 2000a). The first application of VR in clinical pain was a video game in which adolescent and adult burnt patients experienced less pain while they were playing (Hoffman et al., 2000b). Later, Hoffman and colleagues conducted an fMRI brain scan study in which they found that VR greatly and significantly reduced pain in five brain regions of interest related to pain (the anterior cingulate cortex, primary and secondary somatosensory cortex, insula, and thalamus) in healthy subjects exposed to thermal stimulation (Hoffman et al., 2004). Some years later, a second fMRI study demonstrated that the pain reduction experienced by using VR was comparable to the analgesic effect of a moderate dose of hydromorphone pain medication (Hoffman et al., 2007). Up to this point, the analgesic properties of VR had been mostly attributed to its powerful distractive capacity. However, its effectiveness has been demonstrated in the management of mild and severe pain states (Doctor et al., 2002; Hoffman et al., 2011,

2014). In addition, the positive pain-relieving effects of VR may also be mediated through a reduction in anxiety and through the user experiencing positive emotions such as a sense of fun (Triberti et al., 2014).

One reason in favor of the distractive effect on pain associated to VR in the studies from Hoffman and colleagues is because of the lack of embodiment in a virtual body in the VR scenarios of their studies, in which patients were observing fun and distractive situations in a display instead of being embodied in a virtual environment through an immersive VR system. In addition, Malloy and Milling, in a review on the effectiveness of VR intervention for pain relief, reported that immersive VR is more effective in promoting analgesia than non-immersive VR systems (Malloy and Milling, 2010). The difference between these two systems is the lack of embodiment in the non-immersive VR systems, whereas using immersive VR systems, one may be embodied in a virtual body and immersed in the virtual world, feeling present in the generated VR scenario (Sanchez-Vives and Slater, 2005). It has been reported that this "transportation of consciousness to another place" involved in the sense of presence in a virtual environment might be strong enough to diminish sensations of pain (Sanchez-Vives and Slater, 2005). Hence, although Hoffman and colleagues used an immersive VR system in their pain studies, these early pain studies using VR did not include embodiment in a virtual body.

### IMMERSIVE VR AND PAIN

The sense of being present in an immersive VR scenario while being embodied in a virtual body offers the possibility of modulating pain perception by observing the embodied virtual body from a first-person perspective (for a review, see Martini, 2016). The representation of the body is modulated by the integration of different sensory signals, and this has been extensively investigated (Macaluso and Maravita, 2010; Medina and Coslett, 2010; Serino and Haggard, 2010; Wesslein et al., 2014). In this regard, in IVR, we can therefore act on the virtual body seen from a first-person perspective and experimentally manipulate the multisensory integration in a highly controlled way.

### The Vision of the Body in Pain

It has been shown that watching clips of another person's hand receiving painful stimuli, while concomitantly receiving painful laser stimulations on one's one hand, modulates the pain system in the second somatosensory area that reflects the sensory qualities of pain (Valeriani et al., 2008). Later, Longo and colleagues demonstrated, again using laser-evoked potentials, that the vision of one's painful part of the body is analgesic (Longo et al., 2009). In this study, they conducted three different experiments in which they showed that when participants observed their own painfully stimulated hand (without observing the painful stimulation), they felt less pain compared to when they were looking at a box or at someone else's hand. The authors postulated that reduction of pain perception while observing one's own hand was due to a visually induced activation of inhibitory GABAergic interneurons in somatosensory areas. Similarly, Cardini and coworkers showed that vision of the hand, compared to vision of a box, caused a suppression of the early somatosensory potential when electrical stimulation was applied to two fingers at the same time, thus revealing an augmented inhibitory interneuronal activity within the somatosensory cortex (Cardini et al., 2011). This finding was supported by an EEG study by Mancini et al. (2013), in which they demonstrated that vision of the body, compared to vision of a neutral object, increased noxiousrelated beta oscillatory activity bilaterally in sensorimotor areas, which probably reflects cortical inhibitory activity of nociceptive stimuli processing.

Other neuroimaging studies have found that vision of the painful body part (subjected to painful mechanical stimulations) increases the functional connectivity between brain areas of the so-called "pain matrix" and the posterior parietal and occipitotemporal brain areas related to vision of the body (Longo et al., 2012). Further, in this study, the authors observed that the vision of one's own hand led to a reduction in the activation of the primary somatosensory cortex and the operculo-insular cortex following painful stimulation (Longo et al., 2012). Specifically, the analgesic effects of the vision of the body part seem to be sitespecific, which means that less pain is perceived when looking at the body region where the painful stimuli is applied (Diers et al., 2013). Another factor that modulates pain perception while observing the painful part of the body is visual size modification. One example of this is the study by Mancini et al. (2011), in which the authors found a direct correlation between thermal pain threshold and hand size. Specifically, they found that enlargement of the stimulus-receiving hand enhanced analgesia (i.e., increased the pain threshold), whereas visual reduction of the hand decreased analgesia (reduced the pain threshold). However, there are contradictory results about how visual size modification affects pain perception. For instance, while enlargement of the affected hand had an analgesic effect in healthy subjects (Mancini et al., 2011; Romano et al., 2015), the opposite occurred in patients with chronic arm pain (Moseley et al., 2008), while enlarging the hand had no effect in patients with hand osteoarthritis (Preston and Newport, 2011). In addition, when visual enlargement is shown in a single direction (i.e., a "stretch" illusion) and is accompanied by tactile feedback (emphasizing the stretch by simultaneously pulling on the limb), there is a marked analgesic effect in both hand (Preston and Newport, 2011) and knee osteoarthritis (Stanton et al., 2018). It is worth noting that in both these aforementioned studies, a minority of subjects experienced a greater analgesic effect when the opposite (i.e., a shrink/compression) illusion was shown. The authors suggest that the effect may be specific to the individual (Stanton et al., 2018), which raises the intriguing possibility that greater analgesic effects may be achieved with tailored VR experiences that address cognitive aspects of the patient's unique pain experience. For example, in osteoarthritis, if patients believe that their pain is caused by compression of the bony surfaces, a stretch illusion may be effective; in other patients who believe that swelling is the primary driver of their pain, a shrink illusion may be more effective.

It has been also shown that the observation of a downscaled back in chronic back pain patients reduced their pain perception, while no effect was reported for the enlarged back visual condition (Diers et al., 2016). The latter study supports the results found in a case study of phantom limb pain conducted by Ramachandran et al. (2009b), in which by using mirrors, they found that minimizing the size of the lost left forearm reduced the patients' pain perception, while magnifying it had no effect. One explanation for the contradictory results between pain-free participants and chronic pain patients is the complex relationship between pain and the neural representation of the body (Lotze and Moseley, 2007; Gilpin et al., 2015). Related to this, while the temporary painful stimulation in pain-free participants for experimental purposes does not modulate the representation of the body, it is known that patients suffering from chronic pain have associated changes in the central neural system, including a modified cortical representation of the painful part of the body (Moseley and Flor, 2012).

Taken together, these studies demonstrate an important modulatory effect of the vision of one's own painful part of the body, both in healthy subjects and in subjects with chronic pain. However, it has been recently suggested that, in order to be effective at decreasing pain perception, the visual feedback has to be "realistic" by using real-time video or realistic representations of the painful part of the body, instead of a static or neutral image, at least with chronic lower back pain patients (Diers et al., 2016). For this reason, pain management using immersive VR, which allows subjects to be embodied in a virtual body capable of movement, seems to be a potential alternative for studying pain perception in both healthy and clinical populations.

### Embodiment in VR for Pain Relief

In the context of these studies, Martini and colleagues investigated the effect of virtual body ownership on pain perception and found that looking at one's own virtual hand also had analgesic properties, as described for the real hand (Longo et al., 2009) (**Figure 2**). An increased experimental pain threshold was found when compared with the observation of either a real or a virtual object (Martini et al., 2014). Further, they found that the feeling of ownership over the virtual arm was crucial to accomplish the analgesic effect. Regardless, the analgesic effect experienced while observing one's own body seems to be effective even when observing an embodied virtual body if participants experienced high levels of ownership of the body. The fact that looking at one's own "rubber hand" (after inducing the RHI) is not analgesic (Mohan et al., 2012) opened up a debate regarding the extent to which looking at a surrogate body was actually analgesic. This issue was sorted out by Nierula et al. (2017), who demonstrated the relevance of the position of the surrogate with respect to the real hand. While the rubber hand cannot be co-located with the real hand (since they both occupy physical space), the virtual hand can be co-located (or not) with the real hand. Nierula et al. (2017) demonstrated that as the distance between the real and the virtual hand increases, the analgesic effect decreases (**Figure 3A**). In agreement with this, previous findings by Romano and colleagues also reported reduced physiological responses to painful stimuli measured via SCRs, when participants observed a virtual body from a first-person perspective co-located with their real body compared with observing the virtual body turned 90◦ from the real body (Romano et al., 2015). Moreover, in the same study, the authors observed that physiological responses were negatively correlated with the size of the virtual body: the bigger the virtual body, the lower the SCRs (Romano et al., 2015). These results are in line with the observation of a magnified body part increased experimental heat pain thresholds (Mancini et al., 2011; Romano and Maravita, 2014).

Visual manipulations of the body modulate pain perception. One example is the study conducted by Martini et al. (2013) in which the color of a virtual arm was modified and the pain threshold was measured in healthy subjects (see **Figure 3B**). Specifically, observation of a bluish "cold" virtual arm increased

heat pain thresholds, whereas observation of a reddened "hot" virtual arm decreased heat pain thresholds. Co-location of the virtual body with the real one seems to be another key factor for increasing pain thresholds in healthy subjects (Nierula et al., 2017).

Although evidence suggests that observing one's own body while experiencing a painful stimulus reduces pain perception, what would happen if the painful part of the body were to fade away? To answer this question, Martini and co-workers conducted an experimental study in which the virtual body was rendered with different levels of transparency while participants were exposed to a painful heat stimulus. They found that the higher levels of transparency were inversely correlated with levels of ownership, but where the body was semi-transparent, higher levels of ownership over a see-through body resulted in an increased pain sensitivity (Martini et al., 2015). Nevertheless, in clinical populations, the effect of transparency is less clear. In this regard, in a study by Matamala-Gomez et al. (2018), two different groups of chronic arm pain patients [CRPS and peripheral nerve injury (PNI)] were immersed in VR and the virtual arm was observed by the patients at four different transparency levels (transparency test) and three different sizes (size test). In contrast to the study conducted on healthy subjects by Martini et al. (2015), Matamala-Gomez et al. (2018) found that increasing transparency levels of the observed virtual arm decreased pain ratings in CRPS, but this did not occur in PNI. Size increase slightly increased pain ratings only in CRPS patients. Further, the authors found that patients with chronic pain can achieve levels of ownership and agency over a virtual arm similar to healthy participants. Moreover, the VR exposure to all of the conditions globally decreased the mean pain ratings by half by the end of the experiment compared to pain ratings at baseline (see **Figure 4**). This study highlights the possibility that embodiment in VR decreases, at least temporarily, pain ratings in patients with chronic pain. The specific underlying mechanisms of each type of pain probably have a role in the type of strategy that is more effective for reducing pain perception in clinical populations. Further research is required to ascertain optimal dosage and duration of the effects.

Other investigations have also used embodiment in a virtual body to modulate pain perception in clinical populations. In a recent study by Solcà et al. (2018), 24 CRPS patients were immersed in VR, embodied in a virtual body, and observed their affected virtual limb flashing in synchrony with their own detected heartbeat, or asynchronously in the control condition. Here, the authors observed reduced pain ratings and improved motor limb function while observing the synchronous heartbeat condition compared with the asynchronous control condition. Moreover, in another recent study that attempted to modulate neuropathic pain in spinal cord injury patients, the authors showed that VR exposure using multisensory stimulation is associated with mild analgesia, to suggest potential implications for spinal cord injury neurorehabilitation protocols (Pozeg et al., 2017). Finally, Llobera et al. (2013) used body ownership illusions induced using immersive VR combined with a brain–computer interface (BCI) system in a single patient with dystonia of the upper limb suffering from chronic pain. The patient was

FIGURE 3 | (A) Experimental setup of co-location experiment by Nierula et al. (2017). The participant wore a head-mounted display that provided an immersive virtual environment including a virtual own body that was perceived from a first-person perspective. The transparent arm outlined with a white dashed line indicated the positions of the virtual arm. Position of participant during (left panel) co-location, where the virtual and real arm were co-located, and (middle panel) when there was a distance of 30 cm between the real and virtual arm (right panel). The virtual body from the participant's point of view. Reprinted with permission from Elsevier. (B) Participant's view of virtual arm in the experiment by Martini et al. (2013). The right arm is co-located with the virtual arm, with congruent finger movements, in order to induce embodiment of the virtual limb. Heat stimulation is provided to the wrist while the skin color changed. Pain threshold was increased in the blue arm condition (left) versus the red arm condition (right).

embodied in a virtual body while observing a virtual hand opening either automatically or through a cognitive task assessed using a BCI that required patient effort. The evaluation was conducted also on a group of five healthy controls. The authors found that embodiment in the virtual body induced changes in

electromyography and BCI tasks in the patient that were different from those observed in the controls (see **Table 1** for a review).

### DISCUSSION

This review has discussed the potentialities of using an embodied virtual body in immersive VR for pain modulation. Specifically, we have discussed the use of multisensory integration applications, by means of body ownership illusions, to decrease pain perception in healthy and clinical populations.

In a systematic review conducted by Boesch et al. (2016) of non-virtual body illusions (illusory changes of size, mirror therapy, etc.) on clinical pain, they found that there is limited evidence to suggest that bodily illusions can alter pain, but some illusions, namely, mirror therapy, bodily resizing, and use of functional prostheses, show therapeutic promise. Concerning the effects of embodiment on clinical pain, the authors discuss two studies of patients with chronic pain that showed no effect of embodiment on pain levels and suggest that a potential explanation is that embodiment and pain modulation may be separate processes. However, the review did not examine any studies that used immersive VR studies to induce embodiment. Here, we show that through an embodied virtual body, we may modulate body representation and change pain perception in healthy and clinical populations.

Regarding the importance of body representation in pain perception, it is known that many chronic pain patients have a distorted representation of the affected part of the body (Lewis et al., 2007; Moseley, 2008; Senkowski and Heinz, 2016). Further, misrepresentations of the body have been associated with pain (Lotze and Moseley, 2007), and several reports support structural and functional differences between people with and without pain, both at a cortical or at a subcortical level, in brain areas involved in body awareness and body perception (Flor et al., 1997; Pleger et al., 2006; Gwilym et al., 2010). Distortions of body perception involving a painful part of the body (i.e., the body part feeling larger than it really is) have also been demonstrated (Moyer, 2005; Lewis et al., 2007). There is some evidence that treatment directed at changing these functional brain alterations, such as graded motor imagery and sensorimotor retraining (Moseley, 2004, 2006; Pleger et al., 2006), reduces pain, which suggests that there is a bidirectional link between pain and body perception. In addition to this, it has been shown that pain perception is reduced with a corresponding restoration of functional cortical representation of the painful part of the body in CRPS patients (Pleger et al., 2006).

### REFERENCES


### FUTURE RESEARCH

These studies support a link between body perception and clinical disorders such as pain, highlighting the advantages of using embodiment through VR systems in neurorehabilitation and pain management. Nonetheless, robust and suitably powered randomized control trials are needed to further explore the full potential of body illusions and embodied technologies to modulate pain perception, especially with the use of immersive VR. Furthermore, further investigations aimed at modulating pain perception through an embodied virtual body with larger sample sizes will allow a better understanding of the contribution that the subjective feeling of ownership over an embodied virtual body has on pain perception. Moreover, future studies on this topic may make use of brain imaging techniques, which will allow better identification of the neural structures underlying the complex link between modification of body perception and pain.

Interestingly, virtual body embodiment may also allow the study empathy in pain. It is known that the mere observation of other people in pain tends to elicit empathic responses regarding pain perception in one's body (Lamm et al., 2011; Benuzzi et al., 2018). Hence, what will happen if we use embodiment to create a pain-free representation of the body? Although some authors have started to investigate how to use empathy for pain relief by using embodiment (Fusaro et al., 2016), further investigations are needed to create new behavioral and cognitive training methods for modulating pain perception in clinical populations.

### CONCLUSION

The studies commented throughout this narrative review, especially those conducted with chronic pain patients, pave the way for the design of new rehabilitation protocols with prolonged and repeated doses of embodied virtual body in immersive VR to tackle chronic pain disorders, and enable the integration of such "digital therapy" with existing conventional pain treatments.

### AUTHOR CONTRIBUTIONS

MM-G contributed to the bibliographic review and writing of the manuscript. TD contributed to the writing and review of the manuscript. SB and GS contributed to the bibliographic suggestions and review of the manuscript. MS-V and CT contributed to the supervision of the manuscript.

Arzy, S., Overney, L. S., Landis, T., and Blanke, O. (2006). Neural mechanisms of embodiment: asomatognosia due to premotor cortex damage. Arch. Neurol. 63, 1022–1025. doi: 10.1001/archneur.63.7.1022


facial expressions of pain. Front. Psychol. 9:1825. doi: 10.3389/fpsyg.2018. 01825


James, W. (1890). The Principles of Psychology. New York, NY: Holt.


discrimination and pain intensity in complex regional pain syndrome. Neuroimage 32, 503–510. doi: 10.1016/j.neuroimage.2006.03.045


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Matamala-Gomez, Donegan, Bottiroli, Sandrini, Sanchez-Vives and Tassorelli. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Is Bodily Experience an Epiphenomenon of Multisensory Integration and Cognition?

Josselin Baumard<sup>1</sup> \* and François Osiurak 2,3

*<sup>1</sup> Normandie Univ, UNIROUEN, CRFDP (EA7475), Rouen, France, <sup>2</sup> Laboratory for the Study of Cognitive Mechanisms (EA 3082), University of Lyon, Lyon, France, <sup>3</sup> French University Institute, Paris, France*

Keywords: action, bodily awareness, body schema, body image, epistemology, multisensory integration, sense of ownership, sense of agency

### INTRODUCTION

Having a physical body is not sufficient to experience the feeling of having a body. This somewhat staggering assumption has long been demonstrated by studies on phantom limbs (Ramachandran, 1998), distortions of body image following right brain damage (Hécaen and Ajuriaguerra, 1952), and experimentally induced body illusions (Botvinick and Cohen, 1998; Petkova and Ehrsson, 2008). A critical issue is now to understand which mechanisms underlie bodily experience (de Vignemont, 2010), a prerequisite to develop studies on tool incorporation, and neurorehabilitation. That being said, the orientation of research strongly depends on the selected epistemological options. The present work aims at discussing two epistemological options, one being representational (i.e., bodily experience relies on the activation of specific cognitive modules devoted to body representations), and the other being structuralist (i.e., bodily experience is an epiphenomenon of both multisensory integration and cognition).

#### Edited by:

*Mariella Pazzaglia, Sapienza University of Rome, Italy*

#### Reviewed by:

*Satoshi Nobusako, Kio University, Japan Jean-Paul G. Noel, New York University, United States*

\*Correspondence: *Josselin Baumard josselin.baumard@univ-rouen.fr*

#### Specialty section:

*This article was submitted to Cognitive Neuroscience, a section of the journal Frontiers in Human Neuroscience*

Received: *30 May 2019* Accepted: *26 August 2019* Published: *11 September 2019*

#### Citation:

*Baumard J and Osiurak F (2019) Is Bodily Experience an Epiphenomenon of Multisensory Integration and Cognition? Front. Hum. Neurosci. 13:316. doi: 10.3389/fnhum.2019.00316* DEFINING BODY SCHEMA AND BODY IMAGE

Classical taxonomies (Sirigu et al., 1991; Schwoebel and Coslett, 2005) have made a distinction between three body representations. First, the body schema is an immediate sensorimotor representation that specifies the relative positions of body parts in space over time (Buxbaum, 2001). Second, body semantics are of conceptual and linguistic nature, and describe the functions and categories of body parts (e.g., both the wrist and elbow are joints). Third, the body structural description is mainly of visual nature and provides individuals with knowledge on the normal structure of the body (e.g., relative positions of body parts; Goldenberg, 1995). It is a long-term body representation that may also be broken down into a general body image (e.g., knowing that all humans have two arms) and an individual body image (i.e., the stable representation of one's own body over time). The latter implies that individual experience plays a key role in body representation (i.e., the habitual body; Merleau-Ponty, 1945). Due to the conceptual ambiguity of these concepts (de Vignemont, 2010), we shall use the "bodily experience" label as a whole category encompassing body schema and body image and, more generally, the experience of having a body.

### EPISTEMOLOGICAL ISSUES

The abovementioned taxonomy admits that distinct cognitive modules are devoted to specific body representations. Nevertheless, "there are so many bodily disorders, and therefore so many possible dissociations, that one would end up with an almost infinite list of body representations"

**126**

(de Vignemont, 2010, p. 7). In this view, the virtually infinite multiplication of cognitive modules would result in unfalsifiable theories of bodily experience (see also De Vignemont, 2007). It follows that a scientific theory should rely on a limited number of cognitive modules. However, if that is so, why are there so many different bodily disorders (de Vignemont, 2010)? Perhaps one solution would be to consider theoretical options of structuralism, an epistemological account initially developed in linguistics (De Saussure, 1915), and anthropology (Lévi-Strauss, 1958), and occasionally applied to neurological patients (Sabouraud, 1995).

The main assumptions of structuralism are as follows. (1) The human mind consists of a minimum set of modular components (i.e., the structure), the number of which is limited by their universality (e.g., all humans are capable of language independently of the multiplicity of languages). This is consistent with the modularity and universality of mind assumed in cognitive psychology (Marr, 1982; Fodor, 1983). However, (2) the diversity of individual experiences does not reflect the activity of specific cognitive modules but rather is incidental and conjuncture-dependent: Different individual experiences may lead to infinite variations of individual psychological conformations (e.g., the painting), and yet the underlying structure should be the same across individuals (e.g., the canvas). (3) Components of the mind are interdependent rather than independent, sequential and hierarchical. "Heterarchy" may better reflect the complexity and non-linearity of brain activity (Fuster, 2009). (4) By contrast with strict modularity, sensations are processed by all of the components simultaneously (e.g., one familiar tool is simultaneously the object of both semantic and technical reasoning; Osiurak, 2014). This amounts to considering that components of the mind interfere with one another (i.e., the interference hypothesis) in the construction of phenomena (e.g., human written language may result from the interaction of language and technics; Gagnepain, 1990). On this account, perhaps bodily experience does not reflect specific body representations, but rather results from the interaction of all human cognitive skills (**Figure 1**).

### IMPLICATIONS FOR THEORIES OF BODILY EXPERIENCE

Even though structuralism is questionable for being too holistic, over the past few years embodied cognition experiments provided data consistent with the interference hypothesis. But before demonstrating relationships between bodily experience and any cognitive mechanism, it is necessary to delineate specific neurological mechanisms underlying bodily experience. In this regard, it is argued that multisensory integration is a prerequisite to experiment the unity and continuity of the body and self, and that interference with additional cognitive dimensions may underlie various body-related phenomena.

FIGURE 1 | Two epistemological accounts of bodily experience. (A) The cognitive account of bodily experience. Specific body representations determine bodily experience, while additional, non-specific cognitive components modulate the expression of these representations. (B) The structuralist account of bodily experience. There are no body-specific representations. Instead, all of the cognitive processes (not necessarily body-specific) interfere with multisensory integration, which results in bodily experience. Cognitive dimensions also interact with each other (small gray bidirectional arrows), resulting in various phenomena. Adopted from https:// pixabay.com/fr/service/license/.

## Multi-Sensory Integration as a Body-Specific Process?

Before the advent of cognitive architectures, bodily experience has long been viewed by neurologists as an epiphenomenon of multisensory integration (i.e., the combination of sensations arising from different modalities and brain regions; Bonnier, 1905; Head and Holmes, 1911; see also de Vignemont, 2010; Longo and Haggard, 2012a,b). The latter is at the root of body unity (e.g., one can perceive her/his hand as a unitary body part because she/he sees and feels it at the same point in space) and makes it possible to distinguish between self and non-self (e.g., objects, others) stimulations. More recent body illusion experiments have revived and extended this multisensory account of body ownership (Botvinick and Cohen, 1998; Petkova and Ehrsson, 2008).

Multisensory integration depends on the activity of multiple brain regions including the sensory, premotor, posterior parietal, temporal superior, and internal cortex (Wiener, 1996; Calvert and Thesen, 2004; Petkova et al., 2011a; Ursino et al., 2014; Yau et al., 2015). Remarkably, these regions are not specific to bodily experience for they are also involved in cognitive functions like memory, visuospatial, praxis, and social skills. It is then plausible that these brain regions are actually not body-specific but contribute to bodily experience.

### Cognition and the Body

This section enumerates, non-exhaustively, findings/hypotheses that are in line with the interference hypothesis (i.e., the interaction between cognition and bodily experience).

### Bodily Experience and Perceptual Analysis

It has long been assumed that body schema/image is independent from perceptual analysis. Nevertheless, well-known works have demonstrated that the physical characteristics of the body have an impact on visuospatial analysis (Proffitt, 2006). Likewise, motor imagery (i.e., the ability to mentally simulate movements of specific body parts) presumably involves the body schema but is sensitive to peripheral bodily conditions like chronic pain (Breckenridge et al., 2019). Therefore, body representations seem to be highly dependent on immediate bodily experience, and it is no longer possible to admit full independence between personal, and extrapersonal perception.

### Bodily Experience and Action

In the field of apraxia, categorical apprehension (i.e., the ability to select and combine parts of multipart objects into a whole configuration) deficits may account for both apraxia of tool use and visuo-imitative apraxia (Goldenberg, 2009), two disorders that have been explained by either body schema (Buxbaum, 2001), or body image deficits (Goldenberg, 1995). Remarkably, categorical apprehension may be the direct psychological expression of the particular neuronal architecture of the left parietal lobe (rather than a specific cognitive module), which is why it may apply to both body, and non-body stimuli indiscriminately (i.e., body parts and objects). In the same vein, technical reasoning (i.e., the ability to infer tool/object characteristics that are relevant to achieve a given goal; Osiurak, 2014) might condition the selection of body parts during action (e.g., one may use her/his nails to play scratch-card games because nails have the same properties as coins to achieve the goal of scratching).

### Bodily Experience and Language

Interestingly, a similar differentiation/combination function prevails in the field of linguistics (De Saussure, 1915), and presumably underlies categorization and concept formation (i.e., the ability to identify and group recurrent information across infinite experiences; Vallila-Rohter and Kiran, 2015). It is not surprising then, that correlations are frequently observed between gestures conveying meaning on the one hand, and language on the other hand (Vingerhoets et al., 2013). Actually, many works consistently demonstrated the bidirectional relationships between language and the body, especially action (Schwartz et al., 2008; Shebani and Pulvermüller, 2018), and body part localization (Mattioni and Longo, 2014). Perhaps bodily experience not only develops under the influence of language, but also varies greatly in everyday life by the mere fact of thinking and talking.

### Bodily Experience and Semantic Memory

Broadly speaking, body semantics correspond to knowledge about the body and are independent from the body schema. Nevertheless, these representations are more intermingled than expected. Conceptual knowledge on body parts grows as a function of their involvement in action (Auclair Jambaqué and Jambaqué, 2015), and children with spinal cord injury may show selective deficits of body image (Salvato et al., 2017). It follows that body semantics are not the mere result of explicit, didactic learning but also of embodied, individual experience. The fact that body image can be selectively impaired in adults can be understood as an effect of culture-dependent brain plasticity. After all, partially different brain regions may underlie English and Greek in bilingual individuals and yet, they are not the expression of completely different cognitive processes (Ekiert, 2003). This embodied account of body image predicts that actionbased tasks should be as efficient as semantic-based tasks in the rehabilitation of body image deficits.

### Bodily Experience and Social Skills

Another property of semantic memory—and hence body semantics—is that it is a shared, collective memory acquired through social interactions. The fact that similar brain regions represent the self and others (Kruse et al., 2016) supports the hypothesis of a socially grounded bodily experience. Besides, the observation of others shapes the multisensory peripersonal space (Pellencin et al., 2018). Likewise, the estimated metrics of someone else's body depends on social features like gender (Linkenauger et al., 2017). In this regard, it is likely that attitudes toward social partners influence bodily experience, especially since the emotional valence of stimuli has an influence on movement control (so, perhaps, on body schema; Esteves et al., 2016). This might imply that the quality of the relationship between a patient and its physical therapist have a direct effect on rehabilitation, in that positive attitudes toward the therapist may reconfigure peripersonal space in itself.

### Bodily Experience and Individual Experience

Studies on body-swapping illusions (Petkova and Ehrsson, 2008) have demonstrated that they are limited (de Vignemont and Farnè, 2010): The illusion does not work with objects that are not body-shaped, and the feeling of owning the new body occurs only from a 1st person perspective (Petkova et al., 2011b). Nevertheless, with regard to the plasticity of perception (Sachse et al., 2017), it is probably because we are used to experiment our body in a 1st person perspective (i.e., the habitual body). Indeed, body-related tasks are influenced by both individual habits (Isaac and Marks, 1994), and the experience of either the first or the third perspective (Edwards et al., 2019). Contrary to long-standing beliefs, there may be no limit to the plasticity of bodily experience with the possible exception of experience (i.e., the habitual body). This "habitual body" might correspond to the concept of "body model" (i.e., the implicit representation of the usual size and shape of one's own body parts), and can be understood as the phenomenological expression of the somatotopic organization of the somatosensory cortex (Longo and Haggard, 2010). It should be acknowledged that the crucial role of individual experience in stabilizing the body model is not incompatible with the existence of a basic, innate organization of the brain acquired through phylogenesis (Longo et al., 2012).

### POTENTIAL IMPLICATIONS FOR NEUROREHABILITATION

As mentioned in the introduction, theoretical options should have implications for clinical practice, especially neurorehabilitation. The structuralist account of bodily experience posits that the latter is an epiphenomenon of both multisensory integration and cognitive processes that are not body-specific. On this ground, future research on clinical syndromes may include extensive testing of both multisensory and cognitive processing. Indeed, setting up therapies implies the upstream demonstration of the level of impairment. In the absence of a consensual, unified framework for the study of bodily experience (de Vignemont, 2010), this would involve thorough testing of cognition and body representation.

Furthermore, two strategies could be tested. The first strategy focusing on multisensory integration would aim at modifying bodily experience by modulating one or several afferent sensory inputs (e.g., enrichment or impoverishment). This corresponds to most of the strategies currently tested based on the now wellestablished role of multisensory integration in bodily experience (e.g., Chokron et al., 2007; Moseley et al., 2008; Diers et al., 2013). A second, complementary strategy based on the interference hypothesis could consist in testing the influence of nonspecific cognitive processes on abnormal bodily experience. This could include, at least, perception (e.g., does modifying the environment of the body modulate bodily experience and improve symptoms?), action (e.g., does tool use action improve symptoms?), language (e.g., does talking and thinking modify bodily experience?), semantics (e.g., does mental imagery of the body improve symptoms in peripheral syndromes? Does the modulation of peripheral afferent information help asemantic patients drawing body parts?), social skills, and emotion (e.g., does the empathy or emotional state of the patient have an impact on symptoms?), and individual experience (e.g., does the intensity of symptoms vary as a function of previous individual habits?).

Furthermore, seeing the extensive list of body-related disorders (de Vignemont, 2010), it seems necessary to test which therapy is effective on which syndrome. For instance, it is unlikely that the same strategies may apply to both peripheral neurological conditions, and syndromes caused by brain lesions. In return, this could lead to categorize body-related disorders depending on which therapy is effective, and hence to better understand the either common or different underlying nature of seemingly different body-related disorders (e.g., if one and the same treatment is effective on both eating disorders and somatoparaphrenia, one might consider that these conditions share a common denominator). Ultimately, neurorehabilitation studies could lead to either confirm or invalidate the hypothesis that bodily experience is an epiphenomenon of multisensory integration and cognition.

### CONCLUSION

The now well-documented permeability of bodily experience and cognitive functioning raises a critical epistemological issue. On a cognitive account of bodily experience, one could argue that body representations exist, and may plastically change during action (Maravita et al., 2003). Nevertheless, if representations are plastic to that point, then one may also wonder what the nature of these representations is, and which specific function they subserve. An alternative structuralist account would be to consider body representations as the consequence rather than the cause of bodily experience and cognition. On this ground, it is proposed that bodily experience is an epiphenomenon of multisensory integration (likely the most body-specific process), and cognition (**Figure 1**). This is not fully in line with theories of embodied cognition because it amounts to considering that cognition shapes the body as much as the body shapes the mind, whereas embodied cognition accounts generally posit that the body shapes cognition. Demonstrating abnormal bodily experience in the context of completely normal sensory, and cognitive functioning would stand against the hypothesis defended here.

### AUTHOR CONTRIBUTIONS

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

### FUNDING

This work was performed within the framework of the LABEX CORTEX (ANR-11-LABX-0042) of Université de Lyon, within the program Investissements d'Avenir (ANR-11- IDEX-0007; FO) operated by the French National Research Agency (ANR).

### REFERENCES


Lévi-Strauss, C. (1958). Anthropologie Structurale. Paris: Plon.

Linkenauger, S. A., Kirby, L. R., McCulloch, K. C., and Longo, M. R. (2017). People watching : the perception of the relative body proportions of the self and others. Cortex 92, 1–7. doi: 10.1016/j.cortex.2017.03.004


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Baumard and Osiurak. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# How Tool-Use Shapes Body Metric Representation: Evidence From Motor Training With and Without Robotic Assistance

Valentina Bruno<sup>1</sup> , Ilaria Carpinella<sup>2</sup> , Marco Rabuffetti <sup>2</sup> , Lorenzo De Giuli <sup>3</sup> , Corrado Sinigaglia<sup>3</sup> , Francesca Garbarini <sup>1</sup> \* and Maurizio Ferrarin<sup>2</sup>

<sup>1</sup>MANIBUS Laboratory, Psychology Department, University of Turin, Turin, Italy, <sup>2</sup> IRCCS Fondazione Don Carlo Gnocchi, Biomedical Technology Department, Milan, Italy, <sup>3</sup>PHI-LAB, Department of Philosophy, University of Milan, Milan, Italy

Previous evidence has shown that tool-use can reshape one's own body schema, extending peripersonal space and modulating the representation of related body parts. Here, we investigated the role of tool action in shaping the body metric representation, by contrasting two different views. According to a first view, the shaping would rely on the mere execution of tool action, while the second view suggests that the shaping induced by tool action on body representation would primarily depend on the representation of the action goals to be accomplished. To this aim, we contrasted a condition in which participants voluntarily accomplish the movement by representing the program and goal of a tool action (i.e., active tool-use training) with a condition in which the tool-use training was produced without any prior goal representation (i.e., passive tool-use training by means of robotic assistance). If the body metric representation primarily depends on the coexistence between goal representation and bodily movements, we would expect an increase of the perceived forearm length in the post- with respect to the pre-training phase after the active training phase only. Healthy participants were asked to estimate the midpoint of their right forearm before and after 20 min of tool-use training. In the active condition, subjects performed "enfold-and-push" movements using a rake to prolong their arm. In the passive condition, subjects were asked to be completely relaxed while the movements were performed with robotic assistance. Results showed a significant increase in the perceived arm length in the post- with respect to the pre-training phase only in the active task. Interestingly, only in the post-training phase, a significant difference was found between active and passive conditions, with a higher perceived arm length in the former than in the latter. From a theoretical perspective, these findings suggest that tool-use may shape body metric representation only when action programs are motorically represented and not merely produced. From a clinical perspective, these results support the use of robots for the rehabilitation of braindamaged hemiplegic patients, provided that robot assistance during the exercises is present only "as-needed" and that patients' motor representation is actively involved.

Keywords: coexistence between goal representation and bodily movements, peripersonal space, tool-use, body metric representation, robotic assistance, passive movements

#### Edited by:

Giulia Galli, Fondazione Santa Lucia (IRCCS), Italy

#### Reviewed by:

Jean-Paul G. Noel, New York University, United States Lucilla Cardinali, Istituto Italiano di Tecnologia, Italy

> \*Correspondence: Francesca Garbarini francesca.garbarini@unito.it

### Specialty section:

This article was submitted to Sensory Neuroscience, a section of the journal Frontiers in Human Neuroscience

> Received: 21 June 2019 Accepted: 12 August 2019 Published: 12 September 2019

#### Citation:

Bruno V, Carpinella I, Rabuffetti M, De Giuli L, Sinigaglia C, Garbarini F and Ferrarin M (2019) How Tool-Use Shapes Body Metric Representation: Evidence From Motor Training With and Without Robotic Assistance. Front. Hum. Neurosci. 13:299. doi: 10.3389/fnhum.2019.00299

## INTRODUCTION

Acting with tools is a familiar aspect of everyday life. People use tools for eating cakes, moving logs, picking up leaves, and writing papers. A characterizing feature of tools is that they often make out-of-reach objects reachable and manipulable. There is a lot of evidence that using a rake-like tool exerts a deep impact on the agent's space representation, enlarging her own reaching space according to the range of tool action. It has been demonstrated that tools are treated by the nervous system as sensory extensions of the body rather than as simple distal links between the hand and the environment (Miller et al., 2018). A seminal study by Iriki et al. (1996) on non-human primates showed that a repeatedly used small rake expanded the receptive fields of parietal visuo-tactile neurons to encompass the space around both the hand and the rake. If the monkey held the rake without using it, the receptive fields shrank back to their usual extension. Analogous results have been obtained in both healthy and brain-damaged humans. For instance, studies on healthy subjects showed that tool-use might increase the impact of far visual distracters on tactile discrimination (Maravita et al., 2002; Holmes et al., 2004) as well as the sensitivity to the affording features of out-of-reach objects (Costantini et al., 2011). Similarly, studies on patients with visuotactile extinction indicated that the severity of their extinction could be modified by using tools, which extend the reach of hand actions (Farnè and Làdavas, 2000; Maravita et al., 2001; Farnè et al., 2005). In the same vein, patients with neglect only for the hemispace close to their body have been found to worsen their performance in a line bisection task in the far space when using a tool like a long stick (Berti and Frassinetti, 2000; Neppi-Mòdona et al., 2007).

Strikingly, tool-use has also been reported to affect the agent's body representation (Martel et al., 2016). For instance, it has been shown that tool-use might alter the kinematic profile of forearm movements in a reach to grasp task. Even more interestingly, tool-use has been found to modify body metric representation (Cardinali et al., 2009a). Sposito et al. (2010) took advantage of an arm bisection paradigm (Bolognini et al., 2012; Tosi et al., 2018), by asking participants to estimate the subjective midpoint of their own forearm before and after a training phase with long (60 cm) and small (20 cm) tools (Sposito et al., 2012). The results showed that participants indicated a more distal midpoint, thus exhibiting an increased representation of the length of the arm handling the tool, after long-tool-use training only. Indeed, using small tools did not alter participants' body metric representation. More recently, Romano et al. (2019) have investigated how different actions with a tool may impact the subjective metric representation of the body. They found a proximal shift in the perceived midpoint when the training phase with tool mostly involved proximal movements (e.g., shoulders), while a distal shift occurred after the training phase asking for a large use of proximal movements (wrist and fingers).

There is a mounting consensus that the representation of the body is similar to the representation of the surrounding space with respect to its being action-oriented (Maravita and Iriki, 2004). In a nutshell, this means that body representation is not only sensory but also motor in nature, and it is for this reason that actions may shape how the body is represented (Gallese and Sinigaglia, 2010). Acting with tools makes this point vivid. As the aforementioned studies indicate, tool actions can alter agents' body metric representation, with this effect being related both to which tool is used and how it is used. However, postulating a link between body and action allows two different and (partially atleast) alternative views on how tool actions may shape the way in which the body is represented.

According to a first view, the shaping would rely on the mere motor execution of tool actions. Some evidence speaks for this first view, albeit indirectly. For instance, it has been shown that tools have to be effectively used to reach far objects, since just holding them (Iriki et al., 1996; Farnè and Làdavas, 2000; Maravita et al., 2001; Serino et al., 2007) is not enough to alter space representation (Serino, 2019). It seems therefore natural to assume that something similar holds for body representation. But this assumption could be disputed by a second view, according to which the possibility for tool action to shape body representation would primarily depend on the coexistence between goal representation and bodily movements. According to this view, in order for the tool-use to shape the body representation, goals and motor programs have to be represented to intentionally accomplish tool actions. There is some evidence supporting this second view. For instance, it has been shown that imaging acting with tools is sufficient to modify one's own arm's length representation (Baccarini et al., 2014). Furthermore, Garbarini et al. (2015a) reported the case of brain-damaged hemiplegic patients who manifested a pathological embodiment of other people body parts (Fossataro et al., 2016, 2018b; Ronga et al., 2019). The patients were asked to estimate the midpoint of their paralyzed forearm before and after a training phase in which an experimenter repeatedly used a tool, being aligned or misaligned relative to patients' shoulders. When the experimenter was aligned, the patients were (delusionally) believing to perform the tool-use training with their own paralyzed arm. This induced a significant modulation of the perceived arm length. Indeed, the patients located their forearm midpoint more distally (i.e., close to the hand) in the post- than in the pre-training phase. No effect occurred when they were misaligned to the experimenter during the training phase (Garbarini et al., 2015a). Other evidence supporting the second view comes from two studies of Cardinali and colleagues in healthy subjects. In a first study (Cardinali et al., 2009b), when investigating the differential role played by the morpho-functional characteristics of a tool and the sensorimotor constraints that a tool imposes on the hand, they found that tool-use induces a rapid update of the hand representation in the brain, not only on the basis of the morpho-functional characteristics of the tool but also depending on the specific sensorimotor constraints that each tool imposes to the user's motor program. In a second study (Cardinali et al., 2012), when assessing functional against non-functional tool-use with respect to the effects on body representations, they found that the same tool, used for different tasks (i.e., a grabber to grasp object or a grabber to perform a perceptual task), differently affects arm length representation, depending on how it is used. This suggests that our perceived body metrics is differently modulated, according to the way in which specific goals and motor programs of a tool action are represented.

The main aim of the present study is to specifically investigate how tool action may shape body representation, by contrasting these two views. In doing this, we need a pair of situations that differ in that one involves the representation of the tool action goals and motor programs, whereas the other does not. To create such a pair of situations, we adapted the arm bisection paradigm used by Sposito et al. (2012) and Garbarini et al. (2015a), by contrasting a condition in which there is a coexistence between action goals to be accomplished and bodily movements (i.e., active tool-use training) with a condition in which the tool-use training was produced without representing a corresponding action goal (i.e., passive tool-use training by means of robotic assistance). The comparison between active and passive movements has been previously used to dissociate the representational component of the movement from the mere displacement of our body in space, by using different techniques such as hand-twitches induced by single-pulse transcranial magnetic stimulation (e.g., Bolognini et al., 2016; Bruno et al., 2017) and limb mobilization induced by mechanical device (e.g., Bisio et al., 2017; Fossataro et al., 2018a) and by the experimenter during ischemic nerve block (Christensen et al., 2007) or during resting condition (Garbarini et al., 2015b). Upper limb movements have been studied in healthy people and subjects with neurological conditions also by taking advantage of robotic arms, since they are able to produce different force fields aimed at enhancing the subject's residual motor control or at imposing highly controlled, reliable, and repeatable passive movements (Patton and Mussa-Ivaldi, 2004; Carpinella et al., 2009, 2012; Pan et al., 2011; Casadio et al., 2015; Cardis et al., 2018). Irrespective of the techniques employed, the common feature of the passive movement is the lack of the intentional component and, therefore, the consequent absence of motor representation. Indeed, during passive movements, subjects do not have to represent the goal of the action in order to voluntarily produce it, but their actions only depend on externally generated forces.

If tool actions may shape the body representation by virtue of their effective production (first view), no differences in the subjective metric estimation of the body after active and passive training should be expected. On the contrary, if the body metric representation primarily depends on whether, during tool-use, the action programs and goals are motorically represented rather than merely produced (second view), we would expect to find a significant increase of the perceived forearm length in the postwith respect to the pre-training phase after the active training phase only.

### MATERIALS AND METHODS

### Participants

Twelve healthy participants (five females; mean age ± SD: 24.3 ± 1.4) took part in the study. The sample size was based on our previous study exploring the modulation of the right arm body metric representation after tool-use training (i.e., N = 10; in Garbarini et al., 2015a). A similar sample (N = 11) was used in the original article of Sposito et al. (2012). Therefore, in the present study, 12 participants were recruited in order to obtain a sample of at least 10 participants showing the modulation of the right arm body metric representation after tool-use training (see details in ''Experimental Paradigm'' section). All participants were right-handed (Oldfield, 1971) and naïve to the purpose of the experiment. None of them had history or evidence of neurological, psychiatric, or other relevant medical problems. Participants gave informed written consent. The study was approved by the Ethics Committee of the Don Carlo Gnocchi Foundation IRCCS (session 2014-12-10) and conforms to the Declaration of Helsinki.

### Experimental Paradigm

The experimental paradigm is shown in **Figure 1**. Participants performed a forearm bisection task (for more details, see the next section) immediately before and after 20 min of tool-use training. The tool-use training was performed by means of the planar robot for the upper limb shown in **Figure 2** (Braccio di Ferro, Celin, Italy; Casadio et al., 2006), which was equipped with a customized handle. The handle connected the robotic arm to a tool consisting of a 120-cm wooden rod with a U-shape extremity (i.e., the rake). The opposite extremity of the tool was fixed to participants' right forearm through a bondage to prolong their arm. After preparation, participants, sitting in a comfortable position with both forearms on a table, underwent the tool-use training involving the repeated execution of ''enfold-and-push'' movements. In particular, for each repetition, one of three cubic objects (green, yellow, and red cubes with a side of 3.5 cm) was placed on the table by the operator in random order at a distance of 120 cm from anterior torso along participants' midsagittal plane. Therefore, the object had to be ''enfolded'' by the participants using the U-shaped extremity of the tool and smoothly pushed to the target area with the same color as the moved cube (see **Figure 1**). This robotic version of motor training is functionally similar to the ''grasp-and-place'' task previously employed in previous studies (Garbarini et al., 2015a; Romano et al., 2019). The three target areas were placed at a distance of 20 cm from the starting position respectively at 60◦ , 90◦ , and 120◦ from the horizontal to cover a significant part of the reaching space. Each participant performed the tool-use training in two different sessions, separated by a week: active and passive. In both sessions, participants were asked to execute the ''enfoldand-push'' task for 20 min. In the active session, the robot did not provide any force toward the target area and the subjects actively performed the movements. During the passive session, performed after a week, the robot generated an assistive force that moved the tool (and consequently the forearm) towards the target area. The assistive force was implemented ad hoc in order to impose to the robotic handle a minimum-jerk trajectory that is typical of reaching movements naturally executed by healthy subjects in real-life contexts (Flash and Hogan, 1985). In the passive session, the participants were asked to relax as much as possible and to let the robot move their arm without any active intervention. Both the active and passive training sessions

20 min of tool-use training in two different sessions, separated by a week: active session (upper part) and passive session (lower part). The tool-use training consisted of an "enfold-and-push" task. In the active session, the robot did not provide any force and the subjects actively performed the movements. In the passive session, the robot generated an assistive force that moved the tool (and consequently the forearm) towards the target area.

were performed with the eyes open. In the passive session, only participants (N = 10, 5 females; mean age ± SD: 24.4 ± 1.2, according to the sample size of the previous mentioned studies) showing the classical pattern of modulation of the perceived arm length after the active session (see details in ''Forearm Bisection Task'' section) were called back; therefore, the active session was always performed first. Previous evidence with this paradigm showed no sequence effect in the active condition if performed in two different sessions at 1 week of distance (Garbarini et al., 2015a); therefore, it makes it unlikely that any difference found between the two manipulations in the present study (active and passive) should be due to the sequence order (active first).

### Forearm Bisection Task

The experimental task consisted in a forearm bisection task already used in previous studies aiming at investigating the effectiveness of tool-use training (e.g., Sposito et al., 2012; Garbarini et al., 2015a). While blindfolded, participants were instructed to indicate, by using their left index finger, the midpoint of their right distal upper limb segment comprising the forearm and the hand, considering the elbow and the tip of the middle finger as the two extremities. During the task, in order to prevent any possible tactile feedback from the bisections, the right forearm was kept in a radial posture and placed inside a Plexiglas parallelepiped (70 × 10 × 11 cm<sup>3</sup> ). On the top of the Plexiglas screen, above the arm, a paper ruler with centimeters was attached. The 0 cm of the ruler corresponded to the elbow, in order to easily measure the position of the subjective midpoint (p). Then, in order to obtain a percentage score relative to each participant's subjective arm length, we used the following formula: [(p/arm length in cm) × 100]. During the task, corrections were not allowed. In each session (i.e., active; passive), each participant performed a total of 30 forearm bisection judgments, 15 before (pre-training) and 15 after tool-use training (post-training; Sposito et al., 2012; Garbarini et al., 2015a).

### Statistical Analysis

The mean forearm bisection value obtained for each subject in each session (i.e., active; passive) before and after the tool-use training was used as the dependent variable. These data were entered in a 2 × 2 repeated-measures ANOVA with two withinsubject factors Session (two levels: active; passive) and Time (two levels: pre-training; post-training). Post hoc comparisons were performed by means of Newman–Keuls test. The analysis was performed using Statistica software 8.0 (StatSoft, Inc., Tulsa, OK, USA). We reported mean, standard deviation, and p-value, and when a significant effect was found, the effect size (η 2 ) and power were reported as well.

### RESULTS

With respect to the mean forearm bisection values, the ANOVA found a significant main effect of Time (F(1,9) = 25.47, p = 0.0007, η <sup>2</sup> = 0.74, power = 0.99), with significantly greater values (i.e., increased arm length perception) in the post-training than in the pre-training phase. Crucially, a significant Session∗Time interaction (F(1,9) = 21.04, p = 0.001, η <sup>2</sup> = 0.7, power = 0.98) was found, suggesting that the perceived length of the forearm was modulated by the session. In particular, post hoc comparison showed a significant increase of the perceived arm length in the post- with respect to the pre-training phase in the active session (p = 0.001; **Figure 3**), while no difference emerged between the post- and the pre-training phase of the passive session (p = 0.76). It is important to note that the pre-training of both the active and passive session did not differ (p = 0.08), but interestingly, the post-training phases of both sessions were significantly different (p = 0.008), with a significant increase of the perceived arm length in the post-training of the active with respect to the post-training of

the passive session. Furthermore, the post-training phase of the active session was significantly different from all the other conditions (p always <0.01 for each comparison; percentage score relative to each participant's subjective arm length, mean ± SD: pre-training active = 46.4 ± 6.7; post-training active = 54.7 ± 7.3; pre-training passive = 47.8 ± 5.6; post-training passive = 47.4 ± 8.7).

## DISCUSSION

The present study aimed at investigating how tool-use may shape the body metric representation. We contrasted two different and (partially at least) alternative views. According to the first view, the actual execution of tool action would be enough for the shaping to occur, while the second view postulates that a coexistence between action goals to be accomplished and bodily movements is necessary; i.e., it is not enough that the bodily movements are merely executed, the action programs and goals have to be motorically represented. Body metric representation was measured by means of a forearm bisection task. In this task, participants were asked to indicate the midpoint of their right upper limb segment comprising the forearm and the hand, considering the elbow and the tip of the middle finger as the two extremities (Sposito et al., 2012). The forearm bisection task was performed before and after two different tool-use training sessions. Indeed, participants underwent a session in which they actively performed 20 min of tool-use training and a session in which the tool-use training was passively performed by means of robotic assistance. The main finding was that participants exhibited a significantly increased arm length estimation in the post- with respect to the pre-training phase after the active session only. Indeed, when the tool-use training was performed in the Passive session, in which participants were instructed to maintain a relaxed posture while the robot passively moved their arm, no modulation of the perceived arm length occurred. This suggests that the mere production of tool action is not enough for shaping agent's body representation. Specific motor programs of the tool action need instead to be voluntarily implemented and represented.

Our finding is in line with some previous studies suggesting a role of motor processes and representations in the subjective estimation of body metric. For instance, Garbarini et al. (2015a) showed that hemiplegic patients may increase the length estimation of their paralyzed forearm after a training phase in which an experimenter was aligned to them and repeatedly used a tool. Indeed, the patients showed a pathological embodiment of the experimenter's arm, thus having real intentions to move the tool as if they were actually performing the training with their own paralyzed arm. And this was enough for the perceived arm length increase to occur, or so the authors argued. In a similar vein, a very recent study on healthy subjects has demonstrated that body metric estimation can be modulated by the sense of agency (D'Angelo et al., 2018). Participants were asked to perform a forearm bisection task before and after a training phase, in which they virtually grasped objects and make precision grip by controlling a far 3D virtual hand. The training phases consisted of two conditions characterized by a different timing

condition; no difference between pre- and post-training was found in the passive condition. Error bars represent standard error of the mean. ∗∗p < 0.001.

in the visual feedback. In a synchronous condition, participants were shown virtual hand movements responding in real time to their own right-hand movements, while in an asynchronous condition, a 3-s delay was interposed between the participants' real hand and the virtual hand movements. The results showed that participants pointed to their forearm midpoint more distally after performing the training phase in the synchronous condition, where they sensed agency for the far virtual hand. According to their results, only if participants sensed agency for the virtual hand, induced by the synchronicity, and therefore experienced a sense of congruency between the intention to perform the action and the motor output coming from the movement performed did they show the classical modulation of body metrics. Similarly, the notion of congruency is ubiquitous within the body literature. We experience the rubber-hand illusion (Botvinick and Cohen, 1998) under the synchronous condition but not under the asynchronous condition, for instance. More specifically, in the context of tool-use, it has been demonstrated that the peri-personal space expands after a session of near touch and congruent visual stimuli presented far (Serino et al., 2015). This is not the case when training was incongruent. Accordingly, in our study, the shaping of the body metric representation occurs only when there is a congruency between action goals and bodily movements, as in the active training.

Taken together, these and our findings indicate that motor processes and representations, involved in planning and monitoring tool action, may also play a critical role in shaping one's own body metric representation. But how to explain this? A candidate hypothesis is that subjective estimation of body metric hinges on processes and representations which are not only sensory but also motor in nature. Planning and monitoring a tool action requires the agent to represent motorically both bodily and tool movements as if the tool was a part of the agent's body (Gallese and Sinigaglia, 2010). This would involve not only an increase of the range of action, by making reachable things otherwise unreachable, but also a functional extension of the body, with the tool being incorporated much like a prosthetic device (Serino et al., 2007). Such incorporation does not occur if tool action is passively performed with the assistance of a robotic arm. There is here no need for an agent to represent her own body and action goals because tool action execution is fully driven by the robotic arm.

This hypothesis seems to be supported by evidence coming from different domains. For instance, Anelli et al. (2015)reported a similar dissociation between active and passive tool action in the time domain. Participants were asked to perform a time bisection task, by reproducing half of the duration of visual stimuli presented in near and far space, before and after an active tool-use training phase. The results showed a clear dissociation in the perceived duration between far and near stimuli. Indeed, participants exhibited a leftward bias in the time bisection task with near stimuli and a rightward bias with far stimuli. Strikingly, this dissociation disappeared after the training phase, since the far stimuli were perceived as nearer. In line with our findings, the dissociation did not disappear if the tool actions involved in the training phase were passively executed, without any motor preparation and control.

Similar results have been found in the spatial domain. There is a huge amount of evidence that tool actions may extend the agent's space representation, with this extension occurring after short-term (Serino et al., 2007) as well as long-term (Serino et al., 2007; Bassolino et al., 2010) tooluse, even if the interpretation of the consequences of tool-use in the spatial domain is controversial (Holmes et al., 2004). Several studies took advantage of a cross-modal congruency task (Spence et al., 2004). In this task, participants speeded their performance when stimuli from different modalities (e.g., tactile and visual) are temporally and spatially congruent. Indeed, it has been shown that the detection of tactile stimuli delivered to the body is more effectively influenced by visual (Macaluso and Maravita, 2010) or auditory (Occelli et al., 2011) stimuli occurring near to, as compared to far from, the body. Interestingly, short-term tool-use has been found in healthy subjects to increase the impact of far visual distracters on tactile discrimination (Maravita et al., 2002; Holmes et al., 2004). Analogously, acting with a tool, which gets things otherwise outof-reach, has been demonstrated in brain-damaged patients to expand visuo-tactile extinction from near to far space (Farnè and Làdavas, 2000; Maravita et al., 2001; Farnè et al., 2005). As far as long-term tool-use is concerned, blind cane users provide a paradigmatic case of extensive and functionally highly relevant population that constantly perform actions by means of a tool. Serino et al. (2007) asked blind cane users and sighted subjects to respond as soon as possible to tactile stimuli on their hand, while ignoring concurrent sounds presented either near to the stimulated hand or approximately 120 cm far from it, before and after a training phase, which consisted in exploring the far space with a cane. The results showed that sighted subjects responded faster to tactile stimuli associated with far sounds after the training phase only. The effect was absent before the training phase and disappeared when the sighted subjects no longer used the cane. On the contrary, holding the cane, without actually using it, was enough for the blind subjects to result in faster reaction times to touches coupled with sounds occurring at the far space (i.e., at the tip of the cane). Things were different when the blind subjects held a short handle. As in sighted subjects before the training phase, reaction times were faster to tactile stimuli associated with near sounds only (Serino et al., 2007).

All these results point to a change in the way in which the body and the space around it are represented when tool actions are planned and monitored, suggesting that these actions may involve a short or even a long-term tool embodiment, such that the tool becomes part of the acting body (Berti and Frassinetti, 2000; Maravita et al., 2001; Farnè et al., 2005). However, although body and space representations are strictly related, this does not imply that they both rely on the same processes and mechanisms. For instance, in the Galli et al. (2015) study, healthy subjects performed a training with a very special tool (i.e., wheelchair) in Active and Passive conditions and, after that, they underwent a classical audio-tactile looming task (Canzoneri et al., 2012; Serino et al., 2018) used to evaluate the post-training effect on the peripersonal space representation. They did not find the expected results after the active condition, likely because, as proposed by the authors, the very unfamiliar tool action (such has moving a wheelchair for healthy subjects) might have prevented the occurrence of the external space remapping, by shifting the attention on the internal motor effort. Interestingly, they found a remapping of the peripersonal space after a passive training (i.e., when the wheelchair was pushed by someone else) but only when participants can see the explored environment (and not when they are blindfolded). On the same vein, Costantini et al. (2011) have systematically investigated how tool action affects space representation. They found that not only actively using a tool but also merely observing someone else using a tool may extend one's own reaching space. For the extension to occur, the observer had to do nothing more than holding a tool compatible with the goal and the spatial range of the observed action, thus sharing the same action potentialities with the observed agent. It makes sense that visual information, when present in Passive condition (Galli et al., 2015), as well as in observation condition (Costantini et al., 2011), plays a crucial role in shaping the coding of the space around the body. A different result was obtained when the effect of tool-use observation on body representation was investigated. Garbarini et al. (2015a) asked participants to perform a forearm bisection task after and before observing someone else performing tool actions. The results did not show any modulation of the perceived arm length, even when the participants held a tool compatible with the observed action. Although further research is needed, this indicates that, differently from space representation, the representation of the body is mostly sensitive to motor processes and representations typically involved in planning actions and monitoring their execution. Since here, as in the latter study, we focused on body representation (and not on space representation), it is likely that visual information, commonly available during both active and passive training, may result in a less effective shaping of the space representation, thus making unaffected our forearm bisection task.

To sum up, when there is a coexistence between action goals and bodily movements, tool-use may shape body metric representation. Otherwise said, whether people represent (or do not represent) the program and the goal of their actions, when using a tool, has important consequences on what they perceive about the length of their body parts. This can be of interest not only from a theoretical but also from a clinical point of view. First, the present findings confirmed that motor planning and control play a crucial role for the promotion of motor learning, which is responsible for the plastic changes in body representation (Classen et al., 1998; Benarroch, 2006) and is the basis of the rehabilitation in neurologically impaired subjects (Lotze et al., 2003). Indeed, if no active participation is provided, no motor learning is attained and, reasonably, no plastic modulation of body representation can occur, as found in the present study after the Passive condition. By contrast, it is well established that motor learning is promoted if the assistance is reduced to a minimum (assist-as-needed mode), allowing the subject to exert his/her residual voluntary control as much as possible during the execution of goal-directed movements (Sanguineti et al., 2009). This specific assistive mode, easily implementable in robotic devices, can therefore optimize the effect of rehabilitation through facilitation of motor learning and the promotion of neural plasticity.

### DATA AVAILABILITY

The datasets generated for this study are available on request to the corresponding author.

### ETHICS STATEMENT

The studies involving human participants were reviewed and approved by Ethics Committee of the Don Carlo Gnocchi Foundation IRCCS (session 2014-12-10). The patients/participants provided their written informed consent to participate in this study.

### AUTHOR CONTRIBUTIONS

FG and CS conceived the study. FG, MR, IC, MF, CS, and LD designed the experiment. LD recruited the volunteers. IC implemented the robotic paradigm. LD and IC carried out the data collection. LD, VB, and MR analyzed the data. All authors participated in data interpretation. IC and VB prepared the figures. VB and CS drafted the manuscript. FG, MF, MR, and

### REFERENCES


IC critically reviewed it. All authors approved the final version of the manuscript.

### FUNDING

The study has been supported by MIUR-SIR 2014 Grant (RBSI146V1D) and by the San Paolo Foundation 2016 Grant (CSTO165140) to FG. The study was also supported by the Italian Ministry of Health (IRCCS funding ''Ricerca Corrente'') to MF and MR.

### ACKNOWLEDGMENTS

We are grateful to all the volunteers who participated in the study.


it from someone else's hand: pathological embodiment receding in braindamaged patients. Cortex 104, 207–219. doi: 10.1016/j.cortex.2017.06.004


**Conflict of Interest Statement**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The handling Editor declared a shared affiliation, though no other collaboration, with several of the authors IC, MR, MF.

Copyright © 2019 Bruno, Carpinella, Rabuffetti, De Giuli, Sinigaglia, Garbarini and Ferrarin. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Embodiment and Presence in Virtual Reality After Stroke. A Comparative Study With Healthy Subjects

#### Adrián Borrego<sup>1</sup> , Jorge Latorre1,2, Mariano Alcañiz <sup>1</sup> and Roberto Llorens 1,2 \*

<sup>1</sup> Neurorehabilitation and Brain Research Group, Instituto de Investigación e Innovación en Bioingeniería, Universitat Politècnica de València, Valencia, Spain, <sup>2</sup> NEURORHB-Servicio de Neurorrehabilitación de Hospitales Vithas, Valencia, Spain

The ability of virtual reality (VR) to recreate controlled, immersive, and interactive environments that provide intensive and customized exercises has motivated its therapeutic use after stroke. Interaction and bodily presence in VR-based interventions is usually mediated through virtual selves, which synchronously represent body movements or responses to events on external input devices. Embodied self-representations in the virtual world not only provide an anchor for visuomotor tasks, but their morphologies can have behavioral implications. While research has focused on the underlying subjective mechanisms of exposure to VR on healthy individuals, the transference of these findings to individuals with stroke is not evident and remains unexplored, which could affect the experience and, ultimately, the clinical effectiveness of neurorehabilitation interventions. This study determined and compared the sense of embodiment and presence elicited by a virtual environment under different perspectives and levels of immersion in healthy subjects and individuals with stroke. Forty-six healthy subjects and 32 individuals with stroke embodied a gender-matched neutral avatar in a virtual environment that was displayed in a first-person perspective with a head-mounted display and in a third-person perspective with a screen, and the participants were asked to interact in a virtual task for 10 min under each condition in counterbalanced order, and to complete two questionnaires about the sense of embodiment and presence experienced during the interaction. The sense of body-ownership, self-location, and presence were more vividly experienced in a first-person than in a third-person perspective by both healthy subjects (p < 0.001, η 2 <sup>p</sup> <sup>=</sup> 0.212; <sup>p</sup> <sup>=</sup> 0.005, <sup>η</sup> 2 <sup>p</sup> <sup>=</sup> 0.101; <sup>p</sup> <sup>=</sup> 0.001, <sup>η</sup> 2 <sup>p</sup> = 0.401, respectively) and individuals with stroke (p = 0.019, η 2 <sup>p</sup> <sup>=</sup> 0.070; <sup>p</sup> <sup>=</sup> 0.001, <sup>η</sup> 2 <sup>p</sup> = 0.135; p = 0.014, η 2 <sup>p</sup> = 0.077, respectively). In contrast, no agency perspective-related differences were found in any group. All measures were consistently higher for healthy controls than for individuals with stroke, but differences between groups only reached statistical significance in presence under the first-person condition (p < 0.010, η 2 <sup>p</sup> = 0.084). In spite of these differences, the participants experienced a vivid sense of embodiment and presence in almost all conditions. These results provide first evidence that, although less intensively, embodiment and presence are similarly experienced by individuals who have suffered a stroke and by healthy individuals, which could support the vividness of their experience and, consequently, the effectiveness of VR-based interventions.

#### Edited by:

Yusuf Ozgur Cakmak, University of Otago, New Zealand

#### Reviewed by:

Holger Thomas Regenbrecht, University of Otago, New Zealand Iris Charlotte Brunner, Aarhus University, Denmark

> \*Correspondence: Roberto Llorens rllorens@i3b.upv.es

#### Specialty section:

This article was submitted to Neurorehabilitation, a section of the journal Frontiers in Neurology

Received: 04 June 2019 Accepted: 20 September 2019 Published: 10 October 2019

#### Citation:

Borrego A, Latorre J, Alcañiz M and Llorens R (2019) Embodiment and Presence in Virtual Reality After Stroke. A Comparative Study With Healthy Subjects. Front. Neurol. 10:1061. doi: 10.3389/fneur.2019.01061

Keywords: embodiment, body-ownership, self-location, agency, presence, stroke, virtual reality, immersion

## INTRODUCTION

Classical definitions of embodiment have resorted to the concepts of corporeal awareness (1), bodily self-consciousness (2), and the sense of one's own body (3). Embodiment, however, is a complex multi-component phenomenon that could be better described as the representation of an element within the body schema (4), the mental representation of body parts and reachable space, which effectively extends or displaces the normal area of influence of the body parts by real or artificial body parts, habitually used tools, or prostheses (5). Although its neural mechanisms are unclear, it is hypothesized that embodiment operates both via automatic bottom-up and potentially conscious top-down processes to permit the establishment of sensorimotor maps of one's body parts with respect to one's body (5). Previous research has identified different constitutive components of embodiment, including body-ownership, self-location, and agency (5). An understanding of their dissociation is, however, uncertain (6). Body-ownership can be defined as the sense that the body that one inhabits is one's own. Self-location alludes to the sense of being in the place where one's body is. Agency refers to the sense that one can move and control one's own body.

Experiments on multisensory and/or sensorimotor stimulation have allowed remarkable modulations of the perceived body to be performed, with respect to its natural configuration, which are known as body illusions (7). Body illusions have enabled the investigation of the embodiment subcomponents separately, with a particular emphasis on bodyownership. Because body-ownership should be continuous and omnipresent (8), synchronous visuotactile stimulation has been proven to be sufficient to induce this sense not only over rubber hands (9) or feet (10), but also over mannequins (11), in the absence of agency. In contrast, only experiments involving selftriggered actions, fired by efferent signals, have elicited agency (12). Virtual reality (VR) is a paradigmatic case of the former, as it provides multisensory stimulation while allowing real-time user interaction (13). Previous experimentation employing VR has successfully induced embodiment over virtual body parts (14, 15) and entire virtual bodies (16). Importantly, embodying virtual selves may not only affect the body schema, but could also modulate perception (17) and behavior, according to the physical characteristics of the incarnated avatar (18, 19).

Experience in VR is likewise strongly modulated by the sense of presence, which is the sense of being in the virtual environment (VE) (20) or, in other words, the sense of existing inside it (21). Similarly to embodiment, presence is a multi-component construct (22). Both user characteristics (either demographical, psychological, or clinical) and media characteristics (content or form, also known as immersion) contribute to the experience (22). However, the interaction between presence and immersion, the extent to which VR is capable of delivering an illusion of reality to the human senses (13), is not obvious. Nevertheless, a greater sense of presence is expected for higher levels of immersion, provided that other characteristics of the experience remain unchanged (22).

Although it seems reasonable that being in a specific environment can impact the sense of having, moving, and being in a body, and vice versa, research has focused on each construct individually and, consequently, interactions between embodiment and presence remain underexplored. A preliminary uncontrolled study using consumer HMDs attempted to find interactions between embodiment and presence by modulating the existence of a static avatar in the VE, which was supposed to be embodied by the participants (23). The experiment did not find connections between the investigated constructs because, among other possible reasons, it failed to elicit embodiment over the avatar, which was not able to reproduce the participants' movements. Analogously, a study with an augmented realitybased mirror found no connections between embodiment and presence because, in this case, it failed to promote presence in the VE (24). Another experiment by the same group, however, was successful at promoting high levels of embodiment and presence in a mixed reality environment, but interactions between constructs were not discussed (25). The only true attempt made to disentangle this interrelation suggested that perspective influences body-ownership and self-location over a virtual avatar, but not presence nor agency (26), which contradicts previous reports on presence (27–29).

The ability of VR to recreate controlled, immersive, and interactive environments that engage participants in intensive and customized exercises, has motivated its use in different neurological populations, especially stroke. VR-based exercises provide goal-directed tasks that are accomplished in the virtual world by the actions of virtual selves, which are controlled by the participants (30). Users usually experience the virtual world either from a first-person (egocentric) perspective, using a headmounted display (HMD), or from a third-person (allocentric) perspective, displayed on a screen. In motor interventions, interaction is usually facilitated by body movements, which are transferred to a virtual avatar that mimics the actions in the virtual world (31–33). Although an increasing number of studies show the potential of VR-based interventions on motor function after stroke (34, 35), with a special emphasis on balance (36) and upper limb (37), little is known about how VR experiences are mediated in this population. On the contrary, all our insights into embodiment and presence have been provided by studies involving healthy subjects, predominantly young adults (38–41). The scant existing literature suggests that the ability to sense presence after stroke may be preserved (42), but there have been no previous reports on the ability to embody virtual selves. A few reports on body illusions in the real world, involving individuals with stroke, have shown contradictory results (43, 44).

While a significant body of research has focused on the underlying subjective mechanisms of exposure to VR on healthy individuals, the transference of these findings to individuals with stroke is not evident and remains unexplored. The importance of investigating such mechanisms in individuals after stroke is that they may influence the experience and performance in the VE, which, ultimately, could affect the clinical effectiveness of neurorehabilitation interventions. Thus, the hypotheses of this study were: first, that healthy subjects can experience a vivid sense of embodiment and presence after interaction with a virtual task that would vary with the level of immersion and spatial representation, in accordance with the findings of previous studies; and second, that the elicited experience and variation

would be analogously reproduced in a sample of individuals with stroke. Hence, the objectives of this study were to determine and compare perceived embodiment and presence, under different conditions of immersion and spatial representation, in samples of healthy subjects and individuals with stroke.

### METHODS

### Participants

A convenience and representative sample of healthy subjects and individuals with stroke was recruited from the staff and outpatient unit of the neurorehabilitation service of Vithas Hospital Valencia al Mar (València, Spain).

Healthy subjects, with no know musculoskeletal or psychological impairment, and matched ages and genders to those of the stroke group, were recruited. Individuals with stroke were included in the study if they had the ability to understand and interact with a VR-based task. Specifically, the exclusion criteria applied to the stroke group included: first, severe cognitive impairment, as defined by scores below 23 in the Mini-Mental State Examination (45); second, an inability to follow instructions, as defined by scores below 45 in the receptive language index of the Mississippi Aphasia Screening Test (46); third, a risk of falling, as defined by scores below 45 in the Berg Balance Scale (47); fourth, visual or hearing impairment that did not allow for interaction; and finally, unilateral spatial neglect.

Forty-six healthy subjects (25 men and 21 women), with a mean age of 50.8 ± 10.9 years, agreed to participate in the study (**Table 1**). Thirty-two individuals with stroke (18 men and 14 women), with a mean age of 48.8 ± 11.8 years, satisfied the participation criteria and agreed to participate in the study (**Table 1**). These participants presented either ischemic (n = 25) or haemorrhagic stroke (n = 7), with a mean time since onset of 9.2 ± 3.0 months. Both groups were comparable in terms of age and gender.

Ethical approval for the study was granted by the Institutional Review Board of Vithas Hospital Valencia al Mar (TU763198CIS0/1). All participants provided written informed consent before taking part in the study.

### Instrumentation

An adaptation of an interactive VR-based stepping task, which had been previously administered to individuals with stroke for therapeutic purposes (32, 48), was used as a control task. The VE consisted of an infinite checkered floor, with a central gray circle with a diameter of 50 cm, and a gendermatched mesomorph avatar, which synchronously mimicked the participants' movements (**Figure 1**). Playdough-colored items (cubes, spheres, and cones), with a bounding box of 20×20×20 cm, appeared on the floor in front of the central circle. The objective of the task was to step on the items before they disappeared with the closest avatar foot, while keeping the other foot inside the central circle. In between stepping on the items, the foot used had to be moved back into the circle (32, 48). Specific animations and sound effects indicated when an item appeared, disappeared, and was squashed. Extrinsic feedback was TABLE 1 | Characteristics of the participants.


Sex, etiology, and lesion side are expressed as a percentage of the total number of participants. Age, time since injury, and scores in the clinical measures are expressed in terms of mean and standard deviation. NS, non-significant.

provided during the task, with information on the number of items successfully stepped on and the remaining time (**Figure 1**). At the end of the task, the percentage of items stepped on was shown.

To reproduce two of the most widely used VR configurations, the VE was represented either from the avatar's egocentric point of view (first-person perspective), and displayed with a HMD, the Oculus Rift CV1 (Oculus VR, Irvine, CA), or from an allocentric point of view (third-person perspective), displayed on a 60′′ LED Screen (LG, Seoul, South Korea), which was hung on a wall, with its center at ∼175 cm from the floor (**Figure 1**). The center of the VE was defined as being at a distance of 2 m in front of the screen, which was marked on the floor. The HMD had a resolution of 2,160 × 1,200, a refresh rate of 90 Hz, and horizontal and vertical fields of view of 94◦ and 93◦ , respectively (49). The screen had a resolution of 1,920 × 1,080, a refresh rate of 60 Hz, and had approximate effective horizontal and vertical fields of view to the participants of 37◦ and 11◦ , respectively. Auditory feedback was provided by the integrated headphones in the HMD or the integrated speakers of the TV screen, as appropriate. Interaction was facilitated by a Kinect for Windows v2 (Microsoft, Redmond, WA), which was fixed under the TV at a height of 80 cm from the floor to ensure full body-tracking (50, 51). This device provided the positions of the main joints of the participants at 30 Hz. In the third-person perspective, all joints were used to animate the avatar. By way of contrast, in the first-person perspective, head rotation and acceleration were provided by the HMD (**Figure 1**).

A high-end computer, including an 8-core Intel <sup>R</sup> CoreTM i7- 4790 @3.60 GHz, 8 GB of RAM, and a NVIDIA <sup>R</sup> Geforce <sup>R</sup> GTX

third-person condition. The figure shows: (A) a participant interacting with the virtual task in first-person perspective (up), and the virtual environment displayed by the HMD (down), and (B) the same participant interacting with the virtual task in third-person perspective (up), and the virtual environment displayed by the screen (B). In both conditions, the participant is squashing a pink playdough item located on the ground with his left foot.

Titan Xp with 12 GB of GDDR5, was used to run the VE during the experiment.

### Procedure

Two experimenters were in charge of conducting the sessions and ensuring the safety and comfort of the participants. The participants, who were blind to the purpose of the experiment, were briefly introduced to the instrumentation, procedure, and task. They were then situated on the mark on the floor, looking toward the Kinect for Windows v2, and then the experiment started. All participants interacted with the VE for 10 min, in counterbalanced order, under both conditions: first-person perspective with the HMD and third-person perspective with the TV screen. After each condition, the participants were asked to evaluate their perceived sense of embodiment and presence, using two dedicated questionnaires: an adapted version of the Embodiment of Rubber Hand Questionnaire (38) and the Slater-Usoh-Steed Questionnaire (52), respectively. The adapted version of the Embodiment of Rubber Hand Questionnaire contained the same 10 items as the original version but references to a rubber hand were replaced by equivalent references to the virtual avatar. Consequently, this questionnaire assessed the extent to which the participant: can control the avatar with their movements; feels located in the same place as the avatar; and feels the body of the avatar belonged to them. The Slater-Usoh-Steed Questionnaire is a three-item Likert-scale questionnaire that evaluates: the sense of being in a VE; the extent to which a VE feels real; and the extent to which a VE is thought of as a place visited. The scores for both questionnaires ranged from 1 (strongly disagree) to 7 (strongly agree). A speech therapist was in charge of explaining the questions of the questionnaires to the participants with stroke and solving any possible doubt about their meaning.

### Data Analysis

Subcomponents of embodiment were defined, according to the original description on the questionnaire, as the average score of the first five statements (body-ownership), of the sixth to eighth statements (localization), and of the last two statements (agency) (38). Average scores >4 were considered as denoting a meaningful reflection of the vividness of the experience (38, 43).

Mixed ANOVAs were performed to determine the differences between the conditions for the individual groups (healthy and stroke) as between-subject, and perspective as within-subject, factors. The investigators performing the data analysis were blinded. The analyses were computed using SPSS for Windows <sup>R</sup> v22 (IBM <sup>R</sup> , Armonk, NY, USA).

## RESULTS

Scores for the experienced embodiment and presence, and the statistical differences between them, are provided in **Figures 2**, **3**, respectively.

Statistically significant differences were identified between the healthy subjects under the first- and third-person conditions for body-ownership (5.41 ± 0.88 vs. 4.36 ± 1.39; F(1,76) = 20.473, p < 0.001, η 2 <sup>p</sup> = 0.212), self-location (5.43 ± 0.84 vs. 4.74 ± 1.47; F(1,76) = 8.553, p = 0.005, η 2 <sup>p</sup> = 0.101), and presence (5.49 ± 0.81 vs. 4.03 ± 1.24; F(1,76) = 50.973, p < 0.001, η 2 <sup>p</sup> = 0.401). Higher values of these variables were consistently detected for the first-person perspective. Nonetheless, both conditions seemed to successfully induce body-ownership and self-location over the virtual avatar, and presence in the VE, based on the fact that all values exceeded the meaningful threshold. The scores for the sense of presence under the third-person condition were, however, borderline. No differences between conditions were detected for agency (5.98 ± 0.75 vs. 6.04 ± 0.82), which showed the highest values of all the subconstructs of embodiment.

Similar to the healthy participants, statistically significant differences were found between the first- and third-person conditions in the stroke group for body-ownership (4.61 ± 2.02 vs. 3.95 ± 1.92; F(1,76) = 5.753, p = 0.019, η 2 <sup>p</sup> = 0.070), self-location (5.10 ± 1.53 vs. 4.13 ± 2.15; F(1,76) = 11.910, p = 0.001, η 2 <sup>p</sup> = 0.135), and presence (4.45 ± 1.80 vs. 3.83 ± 1.61; F(1,76) = 6.357, p = 0.014, η 2 <sup>p</sup> = 0.077). While these variables were rated as having been vividly experienced in the first-person condition, the third-person condition barely induced self-location over the virtual avatar, and the scores for body-ownership and presence did not (although they almost did) reach the meaningful threshold. As in the healthy group, agency did not show differences between conditions (5.59 ±

1.41 vs. 5.25 ± 1.58), and obtained the highest scores of all the embodiment subconstructs.

A comparison between the groups evidenced statistically significant differences in presence (5.49 ± 0.81 vs. 4.45 ± 1.80; F(1,76) = 6.952, p = 0.010, η 2 <sup>p</sup> = 0.084) in the first-person condition. The scores for embodiment and presence in the healthy group were consistently higher than those in the stroke group for all measures and under both conditions.

## DISCUSSION

This study investigated and compared the sense of embodiment and presence experienced by a sample of healthy subjects and individuals with stroke during exposure to an interactive VR task from different perspectives and under different levels of immersion. The results evidenced that the first-person perspective using HMD elicited a greater sense of bodyownership and self-location over a virtual avatar and a greater sense of presence in both populations. Agency, however, remained almost invariable between conditions. The participants with stroke consistently reported less vivid experiences than the healthy participants, while the differences between the groups were only statistically significant for body-ownership and presence under the first-person condition.

The greater sense of body-ownership and self-location under the first-person condition, and the comparable sense of agency under both conditions, reported by the healthy subjects are in line with previous findings (26, 53–55) and also evidence the distinct natures of the different illusions. The contradictory findings of a previous study, which reported similar vividness of bodyownership and agency, regardless of the perspective, might be explained by a lack of sensitivity in the measurement tools used, as only one question was used to assess each illusion (56). Similar to the effect of perspective on body-ownership and self-location, the greater levels of presence reported by the healthy subjects during the high-immersion condition is supported by previous research (27–29). Interestingly, the only previous experiment that simultaneously investigated embodiment and presence varying the point of view in the VE reported an invariable sense of presence between perspectives for an invariable degree of immersion (26). This could indicate that the differences in presence detected under each condition in our study might be better explained by the difference in immersion rather than perspective. Unfortunately, as both concepts were jointly modified for each condition in our study, it is not possible to identify the determining factor for presence. Despite the differences in the sensed embodiment and presence between conditions, their scores may indicate that healthy subjects are able to successfully incarnate virtual avatars and be present in a virtual world, independent of the perspective and the mediating technology.

Although they did not reach statistical significance, the lower scores for body-ownership and self-location in individuals with stroke in the first-person perspective might suggest a difficulty in these participants to experience the virtual body as their own and being located in it. Although there have been no previous studies that have investigated embodiment in VEs after stroke, experiments involving body illusions in the real world have reported contradictory results that have shown either an increased (43, 57) or decreased (44) predisposition to bodyownership. In our study, post-stroke cognitive disorders, such as a diminished capacity for abstract thinking, which is commonly identified after stroke (58), might have challenged the incarnation of the virtual avatars that, while mimicking the participants' movements, still remained neutral, non-real versions of the participants. In addition, limitations in the body motion-tracking provided by the Kinect v2 (50) and of the mobility of the avatars may have led to some pathological motor patterns in the individuals with stroke to not be exactly reproduced by their virtual selves, which may have reduced their identification of the avatar as their own body and being located in the virtual avatar. This could explain why participants with stroke reported control over the avatar movements, but did not report ownership or self-location. It is important, however, to highlight that the vividness of the embodiment subcomponents was supported by the scores for the different illusions, which exceeded the meaningful threshold under both conditions, but for bodyownership, which were slightly lower than that under the thirdperson condition. The statistical differences in the sense of presence between the groups were analogous to differences for body-ownership and self-location, with the stroke group showing lower scores. Likewise for these illusions, the cognitive condition of the individuals with stroke might have complicated their interpretations of the VE as real and, consequently, decreased their sense of existing in it; however, also similarly to bodyownership and self-location, scores for this sense support the vividness of the experience.

In summary, the highest scores for sensed body-ownership, self-location, and presence, were provided by the healthy subjects, regardless of condition, and for the first-person perspective with a HMD, regardless of the group. The sense

### REFERENCES


of agency, in contrast, received invariably high scores under all conditions and for both groups. The analogous scores for individuals with stroke and the healthy control group support that the sense of embodiment and presence were similarly experienced in both populations. This suggests that the basic mechanisms that modulate these phenomena could be preserved after a stroke, and may support the effectiveness of VR interventions in this population.

### CONCLUSIONS

A sense of embodiment and presence were effectively experienced in both healthy subjects and individuals poststroke, although less intensively in the latter. The feelings were similarly modulated by perspective and level of immersion.

### DATA AVAILABILITY STATEMENT

The datasets generated for this study will not be made publicly available confidential data.

### ETHICS STATEMENT

Ethical approval for the study was granted by the Institutional Review Board of Vithas Hospital Valencia al Mar (TU763198CIS0/1). All participants provided written informed consent before taking part in the study.

## AUTHOR CONTRIBUTIONS

AB and RL designed the study. RL defined the clinical aspects regarding individuals with stroke and JL assessed their condition. AB and JL conducted the experimental sessions. All the authors analyzed and discussed the results of the experiment.

### FUNDING

This study was funded by Ministerio de Economía y Competitividad of Spain (Project RTC-2017-6051-7 and Grant BES-2014-068218), Fundació la Marató de la TV3 (Grant 201701-10), and Universitat Politècnica de València (Grant PAID-10-18).

### ACKNOWLEDGMENTS

We acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.


**Conflict of Interest:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Borrego, Latorre, Alcañiz and Llorens. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Moderate-Intensity Aerobic Exercise Restores Appetite and Prefrontal Brain Activity to Images of Food Among Persons Dependent on Methamphetamine: A Functional Near-Infrared Spectroscopy Study

Hongbiao Wang<sup>1</sup> , Yifan Chen<sup>2</sup> , Xiawen Li <sup>2</sup> , Jiakuan Wang<sup>2</sup> , Yu Zhou<sup>2</sup> \* and Chenglin Zhou<sup>2</sup> \*

<sup>1</sup>Department of Physical Education, Shanghai University of Medicine & Health Sciences, Shanghai, China, <sup>2</sup>Department of Sport Psychology, School of Sport Science, Shanghai University of Sport, Shanghai, China

The brain prefrontal control system is critical to successful recovery from substance use disorders, and the prefrontal cortex (PFC) regulates striatal reward-related processes. Substance-dependent individuals exhibit an increased response to drug rewards and decreased response to natural, nondrug rewards. Short-term aerobic exercise can ameliorate craving and inhibitory deficits in methamphetamine users, but the effect of exercise on food reward is unknown. This study used functional near-infrared spectroscopy (fNIRS) to measure the effects of moderate- and high-intensity short-term aerobic exercise on prefrontal activity related to food images and recorded the subjective feelings of appetite in methamphetamine-dependent users. In total, 56 men who met the Diagnostic and Statistical Manual of Mental Disorders (Fifth Edition) criteria for methamphetamine dependence, with a mean (SD) body mass index of 24.7 (3.5) kg/m<sup>2</sup> and age of 30.2 (5.1) years, were randomly assigned to one of two exercise groups: moderate intensity (n = 28; 65%–75% of maximum heart rate) and high intensity (n = 28; 76%–85% of heart rate maximum). Each group also performed a resting control session for 35 min 1 week before or after the exercise, in a counterbalanced order. Mean oxygenated hemoglobin concentration changes in the PFC when viewing visual food cues were assessed by fNIRS, and subjective feelings of appetite were self-rated using visual analog scales after moderate- or high-intensity aerobic exercise and after the resting control session. A continuous-wave NIRS device was used to obtain functional data: eight sources and seven detectors were placed on the scalp covering the PFC, resulting in 20 channels per participant. We found that moderate-intensity aerobic exercise significantly increased both, the activation of the left orbitofrontal cortex (OFC) to images of high-calorie food (P = 0.02) and subjective sensations of hunger (F(1,54) = 7.16, P = 0.01). To our knowledge, this study provides the first evidence that

#### Edited by:

Yusuf Ozgur Cakmak, University of Otago, New Zealand

#### Reviewed by:

Mei Peng, University of Otago, New Zealand Edythe D. London, University of California, Los Angeles, United States

#### \*Correspondence:

Yu Zhou ulysses91zy@163.com Chenglin Zhou chenglin\_600@126.com

#### Specialty section:

This article was submitted to Health, a section of the journal Frontiers in Human Neuroscience

> Received: 31 May 2019 Accepted: 25 October 2019 Published: 13 November 2019

#### Citation:

Wang H, Chen Y, Li X, Wang J, Zhou Y and Zhou C (2019) Moderate-Intensity Aerobic Exercise Restores Appetite and Prefrontal Brain Activity to Images of Food Among Persons Dependent on Methamphetamine: A Functional Near-Infrared Spectroscopy Study. Front. Hum. Neurosci. 13:400. doi: 10.3389/fnhum.2019.00400 moderate-intensity aerobic exercise increases OFC activity associated with high-calorie food images and stimulates appetite in methamphetamine-dependent individuals. These changes suggest that exercise may reestablish the food reward pathway hijacked by drugs and restore sensitivity to natural rewards. This evidence may contribute to the development of specific exercise programs for populations with methamphetamine dependence.

Keywords: aerobic exercise, food reward, fNIRS, drug dependence, methamphetamine

### INTRODUCTION

Drug addiction is considered a chronic brain disease (Volkow and Morales, 2015). Long-term use of addictive substances leads to lasting changes in the brain structure and function of individuals, including the reward system, which is considered the basis for the development and maintenance of substance addiction (Robinson and Berridge, 1993; Koob and Volkow, 2010; Noël et al., 2013). Methamphetamine (MA) is the second most common illegally used drug in the world, and no drugs have shown efficacy in treating MA dependence (Rawson, 2013). Long-term MA use has been linked to repeated relapse episodes, possibly exacerbated by cognitive impairment during drug withdrawal (Dean et al., 2013; Bernheim et al., 2016). Despite our increased understanding of MA, the effects of repeated MA use, and the severity of the problem, no treatment has been consistently effective in alleviating the symptoms of MA addiction, especially in terms of cognitive impairment and drug-seeking (O'Brien, 2005; Brackins et al., 2011).

Therefore, measures must be taken to diminish cognitive impairment during drug withdrawal. Cognitive impairment, especially within the reward system, in users dependent on drugs continues even after withdrawal has begun. Several studies have reported a decrease in dopamine D2 receptors and dopamine release in the striatum of individuals dependent on drugs that can persist for months after detoxification (Gradin et al., 2014). This finding has been reported for various addictive drugs, including cocaine, alcohol, methamphetamine, and nicotine (Wang et al., 1997; Fehr et al., 2008; Volkow et al., 2009). These persistent neuroadaptive changes may lead to reduced sensitivity to nondrug reinforcers (Koob and Volkow, 2010) and may even impair the ability to respond adequately to rewards unrelated to the drug even during abstinence (Wrase et al., 2007). Individuals dependent on substances show a decreased response in the striatum to natural rewards and thus appear to search for alternative stimuli (drugs) to maintain their equilibrium (Garavan et al., 2000; Koob and Le Moal, 2001; Paulus et al., 2005; Lubman et al., 2009; Volkow et al., 2009, 2010). Brain regions associated with substance abuse and natural reward overlap (Garavan et al., 2000; Karama et al., 2002), for example, with the food reward region (DiLeone et al., 2012), which supports the hypothesis that drug dependence ''hijacks'' the natural reward pathways, leading to overestimation of drug-related rewards and underestimation of nondrug-related rewards (Diekhof et al., 2008; Feltenstein and See, 2008).

It has been reported that MA-dependent research participants consume large amounts of food during at least the first month of abstinence (Zorick et al., 2012), suggesting that in the absence of the ability to choose MA, appetite for food is not impaired. A preclinical drug vs. food choice procedure has been used to evaluate candidate medications for MA use disorder (Banks, 2017), indicating that intervention must be applied after abstinence to restore appetite for food. Thus, food reward, as a natural reward, has been used to evaluate the sensitivity to natural reward vs. drug reward after abstinence among individuals who use drugs. The region of the brain that responds to a natural reward should be reclaimed from responding to drugs by using interventions to increase brain activation in the appropriate region of individuals dependent on drugs in response to that natural reward stimulus, such as food.

Aerobic exercise may be a substance use disorder treatment method (Pareja-Galeano et al., 2013; Wang et al., 2014; Linke and Ussher, 2015). Moderate-intensity short-term aerobic exercise was found to reduce drug craving in MA-dependent individuals and to promote recognition of normal and drug-related inhibitory control (Wang et al., 2015, 2016). Recent functional neuroimaging findings also suggest that long-term regular exercise may alter how brain reward regions respond to visual food cues (Cornier et al., 2012; Nock et al., 2012), and short-term aerobic exercise may change the neuronal responses in food reward brain regions, regardless of whether that aerobic exercise is moderately intense (Crabtree et al., 2014) or of high intensity (Evero et al., 2012).

The prefrontal cortex (PFC) regulates striatal reward-related processes and exhibits activity that predicts treatment outcome with respect to maintaining abstinence (Garavan and Weierstall, 2012). Neuroscience research has pointed to the importance of the orbitofrontal cortex (OFC) in food-seeking because of its responsiveness to changes in the reward value of stimuli (Thorpe et al., 1983; Rolls, 1984, 1990). The use of functional near-infrared spectroscopy (fNIRS) technology has been emerging in the field of cognitive neuroscience in recent years. This technology can be used to directly reflect the hemodynamic changes in cerebral cortical areas, including the PFC. Functional magnetic resonance imaging (fMRI) and fNIRS have a common neurophysiological basis, namely, the neurovascular coupling mechanism. NIRS is an optical technique that noninvasively measures changes in hemoglobin and oxygenation in the human brain (Jöbsis, 1977).

The working principle of NIRS is that neural activity in a brain region results in increased glucose and oxygen consumption at local capillary beds. The increased cerebral blood flow carries oxygen to the active areas, and this temporarily exceeds local neuronal oxygen utilization, resulting in an overabundance of cerebral blood oxygenation in active areas. Thus, NIRS can be used as an index of neural activity. Changes in cerebral blood flow and oxygenation are closely related to neural activity. Changes in oxygen-containing hemoglobin concentrations during the task reflect neuronal activity because they are associated with induced changes in regional cerebral blood flow (Hock et al., 1995; Tanida et al., 2004; Irani et al., 2007). When neurons become active, local blood flow to related brain regions increases, and oxygenated blood replaces deoxygenated blood. Among the three NIRS parameters (oxyHb, deoxyHb, and totalHb; Hoshi et al., 2001; Strangman et al., 2002), the change in oxyHb is the most sensitive indicator of regional cerebral blood flow change; thus, the measured oxyHb concentration can be used to directly reflect hemodynamic changes. Although the spatial resolution of NIRS is lower than that of other functional neuroimaging methods, such as positron emission tomography and fMRI, NIRS has the advantages of high time resolution (<0.01 s) and that it can be performed under natural conditions (Miyai et al., 2001). Thus, NIRS is arguably the best choice for use at the drug rehabilitation bureau.

The objective of the current study was to investigate the short-term effects of moderate- and high-intensity aerobic exercise training on prefrontal activity related to images of high-calorie and low-calorie foods among individuals who were MA-dependent. Subjective sensations of hunger, fullness, and desire to eat were also self-reported after exercise to examine differences in appetite. On the basis of the current literature, we hypothesized that there would be a dose-response effect of exercise intensity leading to increased prefrontal activity to high-calorie foods images relative to that in a non-exercise control condition among MA-dependent individuals.

### MATERIALS AND METHODS

### Ethical Statement and Study Participants

This study was conducted according to the guidelines laid down in the Declaration of Helsinki and was approved by the ethics committee of Shanghai University of Sport (No. 102772019RT041). Written informed consent was obtained from all participants before enrolling them in the study.

In total, 56 men [mean and standard deviation (SD) age, 30.2 (5.1) years; mean (SD) body mass index (BMI), 24.7 (3.5) kg/m<sup>2</sup> ] who met the Diagnostic and Statistical Manual of Mental Disorders (Fifth Edition) criteria for MA dependence were recruited from the Drug Rehabilitation Bureau of Shi Liping in Zhejiang province. Participants were included if they met the following criteria: (1) no metabolic or chronic disease, no medical conditions, and not taking medication known to influence gastric emptying or appetite; (2) no eating disorder, psychiatric diagnoses, or neurological illness; (3) weight stable (<3 kg change in the last 3 months); (4) aged 18–45 years; (5) right-hand dominant; and (6) abstinent and receiving treatment for 3 months.

### Study Design

In total, 73 of 303 eligible participants were randomized to either the moderate- or high-intensity aerobic exercise group; 56 participants completed the entire trial (**Figure 1**). One participant's data were excluded from the analysis because the wrong fNIRS data acquisition sample rate was used; the standard sample rate was 7.81 Hz, whereas the excluded participant's sample rate was 3.91 Hz. There were no significant differences between the moderate-intensity (n = 28) and high-intensity (n = 28) exercise groups in demographic characteristics (age, weight, and height), fitness (BMI and resting heart rate), or drug use (duration, usage, and frequency) prior to exercise intervention (**Table 1**). Each group completed testing at baseline and after exercise and rest, with 1 week separating the exercise and rest test sessions, and in a counterbalanced order. During the preliminary session, anthropometric data were collected, and participants were asked not to engage in strenuous exercise or to drink alcohol for 24 h prior to testing. Eating or drinking of caloric or caffeinated beverages was to be avoided 2 h prior to testing.

### Visual Food Cue Paradigm

The food cue paradigm was adapted from a previous study (Killgore et al., 2003) by using high-quality, full-color photographs from a commercial stock photography website<sup>1</sup> . Participants were instructed to look carefully at each image while undergoing fNIRS. The images were presented on a computer screen positioned 30 cm from the eyes of the participant. Control images were of non-food objects from nature (such as flowers) that were similar in shape, color, and texture. Low-calorie food images were of fruits and vegetables, whole-grain foods, and the like, whereas high-calorie food images were of hamburgers, ice cream, etc. Seven alternating blocks of 10 images of food and non-food images each were shown for 3 s per image in the order presented in **Figure 2**. The blocks were delimited by a 10-s fixation cross (+). The entire scan lasted 240 s.

### fNIRS Data Acquisition

In this study, data were recorded by using a multichannel, continuous wave, fNIRS instrument (NIRScout, NIRx Medical Technologies LLC; Minneapolis, MN, USA). We acquire dual-wavelength (760 and 850 nm) near-infrared light to measure the relative concentration changes in oxyHb and deoxyHb (Maki et al., 1995; Yamashita et al., 1996) based on the modified Beer-Lambert law (Cope et al., 1988) with a sampling frequency of 7.81 Hz. For the NIRS experiment, eight sources and seven detectors (yielding 20 channels) were placed over the PFC region (see **Figure 3**). Sensors were located by aligning the bottom row of electrodes with the International 10–20 sites AF7- Fp1-Fpz-Fp2-AF8 line (Jurcak et al., 2007). The distance between the source and the detector was 3 cm. The midpoint of the source geophone distance was defined as the channel position. Motion artifacts were maintained at a minimum by asking the participants to remain still during the probe.

<sup>1</sup>https://www.58pic.com/tupian/shiwutupian.html


### fNIRS Data Processing

The fNIRS data were assessed with Homer2 software (MGH-Martinos Center for Biomedical Imaging, Boston, MA, USA) using Matlab (Mathworks, Natick, MA, USA). Motion artifacts were detected within 0.5 s when the signal changed by more than 10% of the SD. Wavelet filtering was used in Homer2 to detect and remove these artifacts (Molavi and Dumont, 2010). The fNIRS signals were preprocessed: baseline drift was removed using a high-pass filter with a cut-off frequency of 0.01 Hz, and a low-pass filter with a frequency of 0.1 Hz was used to reduce the influence on the signal of heartbeat, respiration, blood pressure, and skin blood flow. Hemoglobin concentration changes were calculated using the modified Beer-Lambert law. Block averaging was performed on the data to obtain the average response of each participant to the images at the 20 channels before and after exercise. The magnitude of change in the HbO concentration was used as the primary measure because it has a better signal-tonoise ratio than does HbR (Strangman et al., 2002).

### Subjective Appetite Sensations

Subjective appetite sensations were measured immediately after exercise or rest using visual analog scales on an electronic

appetite self-rating system (Gibbons et al., 2011). Participants rated the following three feelings using a visual analog scale that ranged from ''not at all'' to ''extremely'': their desire to eat, how full they felt, and how hungry they felt.

### Exercise Protocol

The aerobic exercise was performed using a bicycle ergometer (SH-5000U) at 50 rpm. Participants were allowed to warm-up for 5 min and to cool down for 5 min. The experimental exercise period was 25 min. For this period, the participant's heart rate was maintained either within 65%–75% or within 76%–85% of their estimated maximum heart rate (calculated as 206.9–0.67 × age; Gellish et al., 2007). Heart rate was monitored using a Suunto Smart Sensor (Suunto Oy, Vantaa, Finland). The participants in the control rest condition performed no exercise. Instead, they sat in a quiet room for 35 min and read about drug use disorder treatments as well as exercise- and fitnessrelated material.

### Statistical Analysis

Statistical analyses were performed using IBM SPSS for Windows (Chicago, Illinois, version 20). Two-way repeated-measures analysis of variance was used to determine the main effects of exercise intervention (control and exercise) and exercise intensity (moderate and high) and their interaction effects on the PFC region activity and appetite measures. Post hoc tests with Bonferroni adjustments were used to determine where significant differences existed. We corrected for multiple comparisons using the Benjamini-Hochberg false discovery rate procedure (Benjamini and Hochberg, 1995). Values are presented as means ± SDs unless otherwise stated. Differences with 2-sided P-values < 0.05 were considered statistically significant.

## RESULTS

## fNIRS

Repeated measures analysis of variance results revealed significant main effects or interactions on the HbO concentration in 3 of 20 channels while participants were viewing high-calorie or low-calorie food images (**Table 2**). After applying post hoctests and controlling for multiple comparisons using the Benjamini-Hochberg false discovery rate procedure (Benjamini and Hochberg, 1995), one channel (Ch11) remained significant, and this channel was located over the left OFC (**Table 3**, **Figure 4**). The grand averaged waveform of statistically significant HbO concentration changes in Ch11 is shown in **Figure 4**. There was a significant main effect of exercise (F(1,54) = 6.66, P = 0.01) and of intensity (F(1,54) = 4.34, P = 0.04), and a significant interaction (F(1,54) = 5.33, P = 0.03) between exercise and intensity for Ch11. Furthermore, the HbO concentration changes increased significantly after moderate-intensity exercise (P = 0.02) but not high-intensity exercise (**Figure 4**). We also conducted additional analyses incorporating the BMI data as a covariate to determine whether BMI would independently affect the results. We found that the results of the analyses were similar to that of the previous analysis in that there was also a significant main effect of intensity (F(1,54) = 4.34, P = 0.04) and a significant interaction between exercise and intensity for Ch11 (F(1,54) = 5.19, P = 0.03). The HbO concentration changes also increased significantly after moderate-intensity exercise (P = 0.02).

### Subjective Sensations of Appetite

Subjective feelings of appetite, in terms of hunger, fullness, and desire to eat after exercise and control sessions, are given in **Table 4**. There was a higher mean rating of hunger after high-intensity exercise compared with that after moderateintensity exercise (**Figure 5**), but no main effect of exercise intensity nor any significant interaction effect was found. No significant effect of exercise, intensity, or their interaction was observed for the subjective feelings of fullness or the desire to eat.

### DISCUSSION

This study used fNIRS to examine the short-term effects of moderate- and high-intensity aerobic exercise on prefrontal brain activity and visual analog scale scores to assess the


(HbO) indicates a relative concentration change in oxyhemoglobin.

TABLE 3 | Mean changes in oxyhemoglobin concentration (HbO) among individuals viewing high-calorie food following moderate-intensity exercise or resting control sessions measured in 20 prefrontal NIRS channels divided into four areas.


<sup>a</sup>One P-value was corrected for multiple comparisons using the Benjamini-Hochberg false discovery rate procedure. DLPFC indicates, dorsolateral prefrontal cortex; FPA, frontopolar cortex; NIRS, near-infrared spectroscopy; OFC, orbitofrontal cortex; VLPFC, ventrolateral prefrontal cortex; SD, standard deviation; <sup>∗</sup>P < 0.05.

effects on appetite in individuals with MA dependence. The findings showed that activation of the OFC associated with viewing images of high-calorie foods increased in these individuals following moderate-intensity aerobic exercise compared with that at rest and that the subjective feeling of hunger increased in these individuals after exercise, especially after high-intensity exercise.

Thus, consistent with our hypothesis, the present study found that short-term aerobic exercise increased OFC activation to high-calorie food cues in users of MA. Studies have found that individuals who are not drug users and are either normal weight or have obesity show different neural responses to visual food cues in the reward region; people with obesity have greater neural responses to food cues than people with normal weight, no matter whether they are hungry or full (Dimitropoulos et al., 2012). This is in contrast to individuals who are drug users because people who are dependent on drugs show lower neural responses to natural reward cues than non-drug users, including to food cues (Volkow et al., 2010). Numerous studies have shown that short-term exercise can modulate appetite regulation in peripheral and central areas in normal weight, overweight, and individuals with obesity (Killgore et al., 2013; Crabtree et al., 2014). With the use of functional MRI analyses, researchers have shown that compared with controls, participants performing short-term moderate aerobic exercise show decreased activation of the insula or OFC, which are associated with hedonic ''liking,'' and decreased activation of the putamen, which is associated with motivational ''wanting,'' in response to visual food cues mean; <sup>∗</sup>P = 0.02.

(Evero et al., 2012), which is inconsistent with our findings that exercise increased neuronal activation to high-calorie food cues in users of MA.

participant means, and vertical lines represent standard errors of the

The present study showed that activation of the OFC associated with viewing images of high-calorie vs. low-calorie foods increased in these individuals following moderate-intensity aerobic exercise. This result is consistent with the finding that brain responses were positively associated with the self-rated desire to consume high-calorie food, in particular, savory food rather than sweet food. Our results also indicated that physical exercise was associated with the responsiveness of specific brain regions, and these brain region responses have been associated with the preference and desire for high-calorie foods (Killgore et al., 2013). Thus, the food images used in the present study were classified based on calorie content, not on sugar content.

The present study using neuroimaging is an early study of food reward among patients who are MA-dependent. By contrast, other such studies of human addiction have generally used healthy participants and focused on their neural responses to drug-related cues. Such studies have shown that drug-related cues activate the brain regions that are normally stimulated by nondrug-related rewards (Sell et al., 2000; Heinz et al., 2004; Wrase et al., 2007; Diekhof et al., 2008). Fewer researchers have explored brain region responses to nondrug-related rewards in persons with substance dependence on cocaine, alcohol, nicotine, opiate, or methamphetamine, and those studies have shown decreased neural responses to nondrug-related rewards. Compared with that for drug-related cues, the response of the brain is less sensitive to films of outdoor nature scenes or to explicit sexual content among cocaine users than among persons who do not use drugs (Garavan et al., 2000). People with cocaine (Goldstein et al., 2007), alcohol (Wrase et al., 2007), opiate (Martin-Soelch et al., 2010), or nicotine (Bühler et al., 2010) misuse show deficits in brain activity to monetary rewards compared with that in the control group. Such findings suggest that a lower response to reward anticipation in the ventral striatum may be a vulnerability factor for the development of early nicotine use (Peters et al., 2011). One study finds that, compared with controls, MA users chose to view more MA-related images than pleasant images, and that the lower the dopamine D2 receptor availability is in the lateral OFC, the more they chose the MA-related images, refining the central hypothesis that dopamine-system deficits contribute to drug-biased decision-making in addiction, and showing a role for the OFC (Moeller et al., 2018). The higher neural activation in the food reward region following exercise in the present study illustrates that exercise may restore the pathway hijacked by drug reward and shows activation to natural rewards similar to that among individuals without substance dependence, which is one of the signs of reward function recovery in drug users.

Previous research suggests that the prefrontal control system may be the key to successful drug withdrawal. The frontal lobe, which has been shown to modulate processes associated with reward in the striatum, is one of the regions that predict treatment outcomes and has shown increased reward and cognitive control in patients who successfully remain abstinent (Garavan and Weierstall, 2012). Although the present experiment monitored the activation of the PFC regions only through fNIRS, the PFC has a strong relationship with both drug dependence and food reward. The present study showed that aerobic exercise increased the activation of the OFC to the presentation of images of high-calorie foods, which is consistent with studies in primates that have shown that the amygdala


The significance of bold values is 0.013.

and the ventral PFC, especially the orbitofrontal regions, are important to the visual evaluation of food stimuli (Rolls, 1984, 1990; Wilson and Rolls, 1993). In addition, decreased activity in the OFC is consistent with a decreased decision-making ability and decreased food pleasantness and palatability (Zald, 2009). Therefore, the PFC, as an important brain region in drug users, can monitor the activation of drug cues and food cues on the one hand, and the executive function of drug users on the other hand, and is thus a crucial brain region associated with the recovery of cognitive function in drug users.

Effects of exercise have been investigated in substance use disorder. Research has found that 10 min of exercise significantly reduces responses to images related to smoking in the OFC and dorsolateral PFC, as well as the subjective perception of cravings, consistent with reduced activation in food reward areas and better inhibitory control (Janse Van Rensburg et al., 2009). Several studies have shown that exercise improves cognitive function while reducing drug-related craving and relapse rates. It has been shown that structured exercise training can ameliorate striatal dopamine D2/D3 receptor deficits in MA users (Robertson et al., 2016). Moderate-intensity short-term aerobic exercise has been found to reduce drug craving in persons with MA dependence and to promote drug-related inhibitory control (Wang et al., 2015, 2016). It has also been found that MA use decreases in individuals with MA dependence after exercise (Rawson et al., 2015). Combined with the results of the present study, these findings indicate that moderate aerobic exercise both reduces drug-related cravings and improves neural activation of the food reward brain region. Future studies should examine the combination of food reward and drug craving as specific indicators of reward function repair with exercise intervention among people who are dependent on drugs.

There are a number of limitations to consider in the evaluation of this study. First, our experiment was conducted in a drug rehabilitation center, with many practical and ethical restrictions regarding patient participation in studies. Therefore, it was not possible to bring people out of the center to conduct fMRI scans to measure neural activity in other brain regions. Second, The effect of an exercise intervention on food reward may vary according to an individual's BMI (Rothemund et al., 2007; Stoeckel et al., 2008), which may have contributed to the large standard deviation in the present experimental results. Thus, future studies may be needed to confirm the present results using larger populations sizes and using BMI as a variable. Third, hunger ratings before and after the exercise/rest and for a period afterward would be stronger for the measurement of subjective appetite. Fourth, no healthy control or non-MA–dependent individuals were recruited into the trial to compare with the MA-dependent sample, and this should be addressed in future research seeking to extend these findings. Fifth, only men participated in the study; thus, the results may not be generalizable to women.

### CONCLUSION

The present study reports a novel finding, to our knowledge, that short-term moderate-intensity aerobic exercise may increase neuronal responses related to food reward in the PFC region among persons who are MA-dependent. Exercise increased the appetite response, especially following high-intensity exercise. This suggests that moderate exercise may reestablish the food reward pathway acutely hijacked by drugs, restore sensitivity to natural rewards, and further promote drug withdrawal among drug-dependent users. This evidence may contribute to the development of specific exercise programs for MA-dependent populations.

### DATA AVAILABILITY STATEMENT

The datasets generated for this study are available on request to the corresponding author.

### ETHICS STATEMENT

The studies involving human participants were reviewed and approved by the ethics committee of the Shanghai University of Sport. The patients/participants provided their written informed consent to participate in this study.

### AUTHOR CONTRIBUTIONS

YZ and CZ contributed to the conception and design of the study. JW and YC conducted experiments and analyzed data. YZ wrote the first draft of the manuscript. HW and XL contributed to manuscript revision. All authors read and approved the submitted version.

### FUNDING

This work was supported by the National Social Science Foundation of China (grant number 17ZDA330) and by the project of Shanghai University of Sport ''Overseas Visiting Program'' (grant number stfx20190116).

### REFERENCES


associated with impaired motivation and self-control in cocaine addiction? Am. J. Psychiatry 164, 43–51. doi: 10.1176/ajp.2007.164.1.43


**Conflict of Interest**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Wang, Chen, Li, Wang, Zhou and Zhou. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Action Observation and Effector Independency

#### Sonia Betti <sup>1</sup> \*, Marie Deceuninck <sup>2</sup> , Luisa Sartori <sup>1</sup> and Umberto Castiello<sup>1</sup>

<sup>1</sup>Department of General Psychology, University of Padova, Padova, Italy, <sup>2</sup>Department of Experimental Psychology, Ghent University, Ghent, Belgium

The finding of reasonably consistent spatial and temporal productions of actions across different body parts has been used to argue in favor of the existence of a high-order representation of motor programs. In these terms, a generalized motor program consists of an abstract memory structure apt to specify a class of non-specific instructions used to guide a broad range of movements (e.g., "grasp," "bite"). Although a number of studies, using a variety of tasks, have assessed the issue of effector independence in terms of action execution, little is known regarding the issue of effector independence within an action observation context. Here corticospinal excitability (CSE) of the right hand's first dorsal interosseous (FDI) and abductor digiti minimi (ADM) muscles was assessed by means of single-pulse transcranial magnetic stimulation (spTMS) during observation of a grasping action performed by the hand, the foot, the mouth, the elbow, or the knee. The results indicate that observing a grasping action performed with different body parts activates the effector typically adopted to execute that action, i.e., the hand. We contend that, as far as grasping is concerned, motor activations by action observation are evident in the muscles typically used to perform the observed action, even when the action is executed with another effector. Nevertheless, some exceptions call for a deeper analysis of motor coding.

Keywords: motor resonance, action execution-action observation, effector-independency, motor evoked potentials, transcranial magnetic stimulation, corticospinal excitability

#### Edited by:

Mariella Pazzaglia, Sapienza University of Rome, Italy

#### Reviewed by:

Giulia Bucchioni, Université de Tours, France Elisabeth Rounis, University of Oxford, United Kingdom

#### \*Correspondence:

Sonia Betti sonia.betti@studenti.unipd.it

#### Specialty section:

This article was submitted to Sensory Neuroscience, a section of the journal Frontiers in Human Neuroscience

> Received: 01 July 2019 Accepted: 08 November 2019 Published: 26 November 2019

#### Citation:

Betti S, Deceuninck M, Sartori L and Castiello U (2019) Action Observation and Effector Independency. Front. Hum. Neurosci. 13:416. doi: 10.3389/fnhum.2019.00416 INTRODUCTION

When considering the issue of effector independence, two studies are frequently cited for empirical support, Merton (1972) and Raibert (1997). Both of these studies provide samples of handwriting phrases, which were similarly executed with different muscle-joint effector systems. Many have interpreted these findings as evidence that the motor program representation is generalized (see Keele, 1981; Schmidt et al., 1988; Rosenbaum, 1990).

This observed affinity of style across different effectors suggests that the representation of handwriting may be independent of the muscular activations that guide the pen. It must be said, however, that some differences between the effectors in terms of the size of the end result were noticed. For example, writing with a pen taped to the foot results in a spatially bigger end product. Nevertheless, the individual characteristics of the writer's motor plan

**159**

(e.g., the penmanship) remain visible. It seems, then, that the writing patterns shared only the very highest and most abstract representation (Wright, 1990; Castiello and Stelmach, 1993).

Effector-independency in action execution has also been investigated for grasping actions (Castiello, 1997; Parma et al., 2011). In a seminal study by Castiello (1997), mouth and hand movements were compared in a task asking participants to grasp pieces of cheese of different sizes with either the hand or the mouth. The pattern of mouth aperture with respect to the size of the food was similar to that found for grasping the very same objects with the hand. Similarly, it has been shown that hand and lip apertures are similarly scaled according to the size of an object evoked by a flavor. Maximum hand and lip apertures were greater when the action toward a small target (e.g., strawberry) was preceded by a sip of a ''large'' (e.g., orange) than a ''small'' (e.g., almond) flavor solution. Conversely, maximum hand and lip apertures were smaller when the action toward a large visual target (e.g., apple) was preceded by the presentation of a ''small'' (e.g., strawberry) rather than a ''large'' (e.g., orange) flavor solution (Parma et al., 2011).

Altogether these findings support the evidence concerned with the presence of a unique motor plan underlying the act of grasping with-the-hand and with-the-mouth, suggesting that coordinated actions are subserved by the use of a common coordinating schema independently from the effectors involved. Similarly, it has been demonstrated that the same holds true for tool use. When performing an action (e.g., pounding a nail) with different effectors (i.e., hand, foot, elbow), spatiotemporal parameters characterizing the execution of the action are kept constant among effector (Osiurak et al., 2018). This suggests that general motor programs are applied when using tools with different body parts.

The effector-independent coding for movements is also evident at neural level (Castiello et al., 1999; Rijntjes et al., 1999; Jastorff et al., 2010; Heed et al., 2011, 2016; Lorey et al., 2014). To dissociate brain regions devoted to the implementation of movement parameters from those relevant to the chosen effector, Rijntjes et al. (1999) asked participants to write their signature with their dominant index finger and ipsilateral big toe, and determined those areas activated by both conditions using functional magnetic resonance imaging (fMRI). The results show that movement parameters for this highly trained movement are stored in secondary sensorimotor cortices of the extremity with which it is usually performed, i.e., the dominant hand, including dorsal and ventral lateral premotor cortices. These areas can be accessed by the foot and are therefore functionally independent from the primary representation of the effector.

In another study, participants were required to perform or imagine an action (grasping a sweet) with either the mouth or the hand while the brain was scanned (Castiello et al., 1999). When ''polished'' from the motor component (i.e., execution) the registered activity showed inferior parietal lobe (IPL) activations for both movements. The proposal here was that the IPL plays a pivotal role in the coding of general action patterns in humans and it is the repository for effector independent representations.

Support to this contention comes from a study in which neural activity during memory-guided eye, hand, and foot movements in human participants was measured (Heed et al., 2011). The results did not reveal any significant activation differences during the planning of hand and foot movements, except in the most anterior part of the posterior parietal cortex (PPC). This region showed a lateral-to-medial gradient for hand vs. foot movement planning. The limb-unspecific PPC regions were functionally connected with hand and foot motor regions. Thus planning-related activity across effectors considerably overlapped.

The issue of effector independency is not confined to action execution, but it extends to action observation. For instance, when volunteers were presented with video clips showing four different motor acts (dragging, dropping, grasping, and pushing) performed with different effectors (foot, hand, and mouth), the coding of observed motor acts differed between the premotor and the parietal cortex. In the premotor cortex, they clustered according to the effector used, whereas in the inferior parietal lobule (IPL), they clustered according to the type of the observed motor act, regardless of the effector. Of interest, these results also suggest that in the case of motor acts typically done with the hand, the representations of such acts are used as templates for motor acts executed with other effectors (Jastorff et al., 2010).

In line with this latter observation, Senna et al. (2014) showed that when participants viewed a typical hand action (grasping a pencil) performed by either a hand or a foot, hand motor evoked potentials (MEPs) increased not only during the observation of actions performed by the hand but also for grasping actions performed by the foot. This evidence confirms that motor activations by action observation occur in the muscles typically used to perform the observed action, even when the action is executed with another effector (see also Betti et al., 2015). This kind of ''hand'' template activation has also been shown in a study in which corticospinal excitability (CSE) of participants observing the opening and closing movements of the mouth and hand was measured (Finisguerra et al., 2015).

The current research was set up to provide further evidence regarding effector-independent processes during action observation with specific reference to the hand template. Is the hand a reference point for whatever effector taking possession of an object? The majority of studies have investigated motor acts performed with the hand or the mouth, two effectors intimately related at both neural (Matelli et al., 1985; Rizzolatti et al., 1988) and functional level (Gentilucci et al., 2001). Grasping a fork to nail a piece of food is usually followed by a mouth grasp for eating the food, consequently, the grasp command can be sent to different distal effectors to prepare a series of successive motor acts. Further the fact that effector independency occurs when hand and foot actions are observed might not be surprising given that from an evolutionary perspective certain types of grips involving the entire surface of either the hand or the foot are part of the behavioral repertoire of primates (Macfarlane and Graziano, 2009; Castiello and Dadda, 2018).

With this in mind, here we test how far effector independency—in terms of hand template—goes by asking participants to passively observe not only grasping actions performed with either the foot, the hand or the mouth, but also grasping actions performed with effectors which are ''distant'' as far as grasping is concerned, namely the elbow and the knee. Specifically we assessed MEPs of two hand muscles, the first dorsal interosseous (FDI) and Abductor Digiti Minimi (ADM), during the observation of the above mentioned grasping movements. If observing a grasping action performed by whatever effector calls for an involvement of hand grasp representation, then we should find general facilitation in hand muscles for all effectors. This would signify that the hand template comes into play whatever grasping effector is observed and would shed more definite light on the notion of effector independence for action observation. Conversely, if hand MEPs modulations are evident only for more grasp-related effectors, then we should find an increase in the MEP amplitudes only during a hand, mouth and foot grasp observation, but not for elbow and knee.

### MATERIALS AND METHODS

### Participants

A total of 29 healthy subjects (15 females, mean age: 22.8, range: 19–31 years) participated in the study. All participants had normal or corrected-to-normal vision and were right-handed. Handedness was assessed with the use of an Italian adapted version of the Edinburgh handedness inventory, a 10-item questionnaire to determine expressed hand preference (Oldfield, 1971). Subjects were screened for neurological, psychiatric or medical problems. None had a contraindication to TMS (Rossi et al., 2009). Written informed consents were given prior to the experiment and all participants were naïve to the studies' purpose. The experimental procedures were approved by the Ethical Committee of the University of Padova and conducted in accordance with the ethical standards of the Declaration of Helsinki. No discomfort was reported during TMS stimulation and MEP acquisition. A right-handed female (age 24) with a background in ballet has performed the different actions showed in the video-clips. She provided written informed consent for the recorded videos to be used in the experiment and to be published.

### Stimuli

Six video-clips were used as experimental stimuli (**Figure 1**). The videos depicted a right-handed nonprofessional actress performing a grasping action with different effectors (hand, foot, mouth, elbow, and knee). The sixth video clip showed the object without any manipulation. The model was instructed to grasp the top of the object in a natural way and with the rightsided effectors. Furthermore, when grasping the object with the hand, the actress performed a pinch grasp. The object was a 3D printed rectangular parallelepiped (13 × 200 mm, 18 g) held uprights with the use of a small separate black platform (60 × 60 × 60 mm). The different video-clips were filmed from a lateral point of view with the use of a Canon Legria HFM36 (Tokyo, Japan) mounted on a tripod. They were later edited with Adobe Premiere Pro CS 5.5 software to minimize the visibility of other effectors unrelated to the performed action. All videos included the effector at rest in front of the object before the actual action, followed by a top grasp of the object and a straight upwards lift of the stick. The model was instructed to minimize any time variations between the start and the grasp. Each stimulus presentation lasted 3,297 ms and the animation effects were obtained by presenting each frame 33.3 ms in series. Notably, the first and last frames lasted 200 ms. The grasp occurred approximately 1,665 ms after video onset. The end of each action was decided to represent a similar object height. The dimension of each stimulus was 1,024 × 768 pixels displayed on a 24-inch monitor (resolution: 1,440 × 1,080 pixels, refresh rate 120 Hz, color depth: 32 bits). Each frame was presented in the center of the screen with a black background. The experimental task was designed and run with the use of E-prime software (Psychology Software Tools, version 2.0).

### Procedure

Subjects were instructed to sit down in a slightly raised armchair upon arrival. Their right arm was positioned on a cushion and their head was placed on a fixed headrest in the most comfortable way. They were instructed to keep their hand and head still and relaxed during TMS stimulation. The experiment was presented on a monitor at eye level, located 80 cm from the participant's head. After scrubbing the skin on the points of interest on the right hand, the electromyography (EMG) was set up. TMS-induced MEPs were obtained from the FDI and ADM muscles of the participant's right hand. After acquiring an accurate signal, the coil was fixed on its optimal position and a threshold value for primary motor cortex (M1) stimulation, i.e., resting motor threshold (rMT), was defined. Participants had the task to carefully observe the video clips presented on a monitor in front of them in random order. Between each video presentation, the participant was reminded to remain attentive to the video and as relaxed as possible. The experiment consisted of 120 single-pulse TMS and lasted approximately 25 min. Stimulation was given at 120% of the rMT. A total of 30 pre- and post-experiment stimulations (2 × 15) were used to acquire each participant's baseline CSE. During baseline registration, each trial lasted 10 s and consisted of a black screen for 5 s followed by a white fixation cross (10 × 10 mm) for another 5 s. Stimulation was given during the latter. Furthermore, 90 TMS pulses (15 repetitions × 6 conditions) were given during each video clip presentation at 1,432 ms after video onset. This corresponds to seven frames before the actual contact point with the effector. As shown by Urgesi et al. (2010), higher motor facilitation can be found during the start and middle phases of a grasping action compared to the end phase. We, therefore, adopted a stimulation time that was anticipated with respect to the effector-object contact. An equal time frame was used for the object condition. To match the moment of stimulation for all grasping movements, both the start and end frames were prolonged (from 200 to 533 ms and from 200 to 800 ms, respectively). By adopting a variable duration for the first frame across conditions, effects due to anticipation of the stimulation timing are avoided. An interpulse interval of 10 s was applied to minimize possible carryover effects of the TMS pulse on the subsequent one.

## TMS and EMG

Single-pulse TMS was delivered with the use of a figure-eight coil (70 mm) connected to a Magstim BiStim<sup>2</sup> Stimulator (Magstim Co., Whitland, UK). Stimulation was given to the hand region of the left M1. The coil was positioned tangentially to the scalp of the participant, with the handle pointing laterally and caudally, 45◦ from the midsagittal axis (Brasil-Neto et al., 1992; Mills et al., 1992). The optimal scalp position (OSP) was then determined by moving the coil in approximately 0.5 cm steps around the presumed area. Visual inspection of the MEPs of the right FDI and ADM, recorded through EMG, were used as feedback. More precisely, the location which elicited a maximal amplitude for both muscles was used as a hotspot. Once M1 OSP was obtained, the coil location was marked on a tight-fitting cap placed on the participant's head. The optimal position of the coil was maintained still on the head with the use of a mechanical arm attached to a tripod. This position was checked continuously throughout the experiment. The rMT, i.e., the lowest stimulation intensity inducing peaks (≥50 µV peak-to-peak amplitude) in 50% of 10 trials in a relaxed muscle (Rossini et al., 1994), was found for each participant. rMT ranged from 32% to 50% (mean ± SD: 41.52 ± 4.39) of the maximum stimulator output for both muscles. Stimulation intensity was set at 120% of the individual's rMT during the experimental session to ensure a stable and clear MEP signal.

MEPs of the right FDI and ADM muscle were recorded through pairs of Ag-AgCl surface electrodes (9 mm in diameter) placed in a belly-tendon montage. The right wrist was used for the ground electrode. Skin impedance was considered of good quality when it was below the 5 threshold level. This was assessed prior to the experimental session when the participant was at rest. The five electrodes were connected to an isolated portable ExG input box (Professional BrainAmp ExG MR, Munich, Germany). A twin-fiber optic cable transmitted the signals from the input box to the main EMG amplifier. The raw myographic signals were sampled at a 5 kHz rate, filtered and amplified before digitalization. Filtering occurred at a bandpass of 20 Hz-1 kHz and the data were stored on a computer for offline analysis. EMG activity was monitored during the stimulation to ensure relaxation in both muscles. To check for any EMG activity before TMS stimulation, pre-stimulus activity recordings of 100 ms were obtained. Any trials with an activation higher than 50 µV before TMS onset were discarded from the data to prevent any contamination of the MEP measurements. EMG data were collected up until 200 ms after TMS pulse.

### Post-experimental Questionnaire

A short questionnaire at the end of the experimental session was included to measure participant's affinity with the actions. After presenting a picture depicting the moment of stimulation, three questions were asked. The participant had to respond to these questions on a five-point Likert scale. The order of the conditions was randomized between participants. First, the naturality of the action was inquired, followed by the probability of using this action and lastly, how many times they executed this action. The three questions were (as translated from Italian): (Q1) ''How natural is the observed action to you?''; (Q2) ''What is the probability that you would perform this action?''; and (Q3) ''How many times do you usually perform this action?''

### Data Analysis

Peak-to-peak MEP amplitudes for the FDI and ADM muscles were recorded and analyzed offline using Brain Vision Analyzer software (Brain Products GmbH; Munich, Germany). All analyses were conducted on 25 of the 29 participants. Four participants were excluded from the analyses due to technical difficulties. MEP amplitudes were then averaged over each

condition, for each participant. All deviations bigger or smaller than 2 standard deviations (SD) from the mean were removed from further analysis. A total of 10.23% of the trials were excluded as outliers; either due to pre-activation, no activation at all or because they exceeded 2 SD. For hand, foot, mouth, elbow and knee conditions, a mean (±SD) total of 12 ± 11%, 12 ± 11%, 8 ± 7%, 7 ± 5%, 11 ± 11% and 9 ± 8% of MEPs were excluded, respectively. The remaining MEP amplitudes for each subject were then normalized based on the participants' baseline MEPs. A ratio was computed by dividing the mean MEP amplitude for each condition by the mean MEP amplitude obtained during pre- and post-baseline measurements (MEPratio = MEPobtained/MEPbaseline).

First of all, a paired samples t-test between pre- and post-baseline MEPs was performed for each muscle individually. Second, a repeated-measures analysis of variance (ANOVA) was performed on the MEP ratios. Both muscle (FDI, ADM) and conditions (hand, foot, mouth, elbow, knee, object) were withinsubjects factors. Effect size estimates were obtained using partial eta-squared (η 2 p ). The sphericity of the data was verified prior to analysis. Mauchly's test of sphericity indicated a violation on the assumption of sphericity (χ 2 condition = 31.07, p = 0.006; χ 2 condition∗muscle = 27.47, p = 0.017). Greenhouse-Geisser estimates of sphericity (εcondition = 0.67; εcondition∗muscle = 0.66) are used to correct for the degrees of freedom. A one-sample t-test against 1 on the normalized data was conducted to look for modulations compared to the baseline. We tested against 1 as this value represents equal activation between the baseline and condition as it is conducted on the normalized MEPs. To analyze the questionnaire responses, a one-way ANOVA on the mean score for the three questions (Q1, Q2, Q3) was conducted with the five grasping actions (hand, foot, mouth, elbow and knee) as withinsubject factors (Norman, 2010; Sullivan and Artino, 2013). Post hoc pairwise comparisons were conducted using t-tests. A Bonferroni correction was used to counteract the problem of multiple comparisons, i.e., reducing the chance for a type-I error. Alpha levels for all statistical tests were set at 0.05.

### RESULTS

No significant difference between the mean raw pre- and post-baseline MEP measurements in both muscles was found (ADM: t(24) = 0.126, p = 0.90; FDI: t(24) = 0.427, p = 0.809). Consequently, motor excitability before and after the experiment did not differ, which let us to conclude that any modulations in the MEPs are exclusively linked to our experimental conditions.

The ANOVA on the normalized MEP amplitudes showed a main effect of condition (F(3.36,80.56) = 5.425, p = 0.001, η 2 <sup>p</sup> = 0.184), indicating that observing different effectors elicits different MEP amplitudes in both hand muscles. Furthermore, a two-way interaction effect of muscle × condition (F(3.3,79.23) = 4.708, p = 0.003, η 2 <sup>p</sup> = 0.166) was found.

When considering the difference between the FDI and ADM muscle activations, post hoc comparisons showed a significant difference during observation of grasping actions performed by the hand (p = 0.040), foot (p = 0.010) and elbow (p = 0.036). More precisely, the FDI muscle was significantly more activated compared to the ADM muscle during observation of grasping actions performed by these three effectors (**Figure 2**). For the mouth condition, the difference between FDI and ADM did not reach significance (p = 0.081). This higher FDI muscle elicitation during observation of a hand grasp was expected and suggest correct motor resonance.

Additionally, no significant difference between both muscles emerged while observing the object (p = 0.976). This is in line with the object affordance effect as the adopted object can be grasped with both a pinch grip (involving mainly the FDI) and a whole hand grasp (involving both muscles). Although not significant, an inverse activation of both muscles was instead found for the knee condition (p = 0.397). Plus, when considering the differences between conditions for the two muscles, post hoc analysis showed significant differences in the ADM between the hand and the foot (p = 0.011), and the hand and the object (p = 0.025), with lower ADM activation for the hand compared to the other conditions. For the FDI, a significant difference between the foot and the hand (p = 0.004), the foot and the mouth (p = 0.014), and the foot and the knee (p < 0.001) was found, with the foot condition having greater MEP amplitudes compared to the other effectors.

In terms of muscle facilitation with respect to the baseline condition, the FDI muscle showed an increased activation for all conditions, except the knee (p<sup>s</sup> < 0.05; **Figure 2**). The ADM muscle was also significantly more activated during observation of the foot grasping an object compared to the baseline (p = 0.03; **Figure 2**). This is represented by having a significantly higher activation to one as analyses were conducted on ratios. These results show that the FDI muscle is generally activated during action observation, independently of the effector used in said observation.

**Table 1** reports the mean scores of the post-experimental questionnaire, investigating how natural (Q1), probable (Q2) and frequent (Q3) is executing the observed grasping action with the different effectors. The ANOVA on the mean scores for the three items showed a main effect of condition (F(4,96) = 68.741, p < 0.001, η 2 <sup>p</sup> = 0.741). Post hoc comparisons showed higher scores were given to the hand action compared to all the other conditions (p<sup>s</sup> < 0.05), and lower scores were given to the knee compared to all other effectors (p<sup>s</sup> < 0.05).

### DISCUSSION

The present study was set up to address effector independency during action observation. By measuring CSE in the FDI and the ADM muscles during single-pulse transcranial magnetic stimulation (spTMS) on the primary motor cortex, we investigated if the hand, the effector typically used to perform a grasp, would present motor resonance only during observation of hand grasping or also while other body parts grasped an object, i.e., foot, mouth, elbow, and knee.

In general, the findings of this study point to an effectorindependent activation of the motor system during action observation. CSE facilitation was evident in the effector usually adopted to perform a grasping action, namely the hand, even when the observed grasping was performed with a different body part. An explanation for this is that because the hand action templates were used to comprehend the goal of motor acts carried out using other effectors. It seems that one computational module is responsible for translating the sense of the same action performed with other effectors. This mechanism might imply the mapping of the observed action goal, which, in turn, might be functional to action understanding.

In this view, actions are abstractly encoded at a higher level in terms of its goal, regardless of the effector involved to achieve it. Indeed, when testing aplasic individuals who were born without arms and hands during observation of manipulative hand actions, overlapping activations emerged for foot and mouth action execution (Gazzola et al., 2007b). Even in the absence of a corresponding effector, these findings suggest that the mirror neuron system—matching action observation with action execution—is recruited for matching the observed action goal with the effector most frequently recruited to perform the action. Similarly, professional foot painters, who not only use their feet to compensate for missing hand function, but also achieved an extremely skilled and fine-grained control of their toe, showed a correspondence to canonical hand organization in their somatotopic toe map (Dempsey-Jones et al., 2019). This


suggests that motor expertise and goal coding represent two critical aspects that can affect effector-independent motor representations.

Some authors have proposed that it is within the parietal cortex that motor acts are clustered according to the goal, irrespective of the effector used (Jastorff et al., 2010; Lorey et al., 2014). The recruitment of the muscles typically involved in the observed action found in our study, therefore, might reflect the parietal role in categorizing motor acts according to their functional meaning and in generalizing the observed actions across effectors. On the other hand, our typical sensory-motor experience, when considering grasping, implies that we typically take possession of objects with the hand.

Consistent with this interpretation are the findings that the human grasping circuit is strongly activated during the observation of grasping performed with artificial devices, even when the artificial device differs from a grasping hand in shape and kinematics (Gazzola et al., 2007a; Peeters et al., 2009). And they are also in line with previous evidence reporting generalized CSE for hand muscles during the observation of other effectors such as the hand and the foot grasping objects (Senna et al., 2014; Finisguerra et al., 2015).

A caveat of the present findings is that when observing foot grasping, the highest CSE modulations in both hand muscles was observed. A possible explanation for this result comes from action execution. As previously mentioned, writing with the foot determines an exaggeration of the writing size (e.g., Bernstein, 1967; Merton, 1972; Raibert, 1997). When we go more distal in our body schema, execution becomes more difficult and therefore less precise. As we are not skilled in writing with our foot, control over this distal body part is more difficult and requires more effort. Similarly, in action observation the mapping of the grasping action as performed by the foot might require more generalized activation to adapt the hand template to the foot grasping representation. Muscle activity may then reflect the neural parameters encoded in the motor program for actually executing or mentally performing the foot action.

Another aspect of the present results is that observing a knee grasp did not facilitate the targeted hand muscles. This lack of activation poses limits on the conclusion that it is only goal coding to determine effector independency for action observation. One of our hypotheses was that the ''hand template'' effect might be challenged by effectors which are ''distant'' as far as grasping is concerned. Indeed the knee was the most awkward to observe. This consideration is supported by data from our post-experimental questionnaire in which participants scored the knee action as the least natural, probable and frequent to execute. Therefore, not only the goal of an action but also how the observed action is feasible and it is part of our behavioral repertoire that allows accessing motor templates (e.g., Buccino et al., 2004; Gazzola et al., 2007b; Betti et al., 2015). In addition, action plausibility based on the available context may guide our processing of others' actions. Along this line, Brass et al. (2007) investigated the role of mirror and non-mirror brain areas while observing goal-directed actions performed with an unusual effector (e.g., operating a light switch with the knee) in plausible (e.g., hand occupied by heavy folders) or implausible (e.g., hands-free) contexts. Results showed that presenting goal-directed knee actions did not activate mirror areas, rather the activation of the superior temporal sulcus was modulated by action plausibility, which the authors interpreted as a reflection of an inferential processing guiding action understanding. In our study, we did not look at contextual contingencies and constrains that could have justified the use of one effector with respect to another, still we found no motor activations for the knee grasping action. Overall, the results from Brass et al. (2007) support our findings, suggesting that the knee effector is hardly associated with goal-directed actions classically performed with the hands.

As a final aspect, one could argue that the heightened activation in the hand during the observation of other effectors performing a grasp is due to attention as the observed actions are rather unusual. However, the appropriate motor resonance response for the hand condition (muscle-specific activation for a pinch grasp with FDI > ADM, e.g., Cavallo et al., 2011) suggests that we are measuring CSE responses as a result of action processing. In addition, no difference between the FDI and ADM muscle was found for the object condition. As hypothesized, we did not expect to find a difference between these muscles, as the object is prone to both a pinch grasp and a whole hand grasp, the latter relying on both muscles. As previously found, the mere observation of an object should elicit activation in the muscles used to manipulate it (Cattaneo et al., 2005).

In conclusion, the present findings suggest that during the observation of grasping actions, the motor system is activated independently of the effector used by a model to perform the action. In particular, there was a tendency to match the observed action with its prototypical effector (i.e., the hand). This might simplify the understanding of action goals based on our experience. However, as witnessed by the lack of facilitation for the knee condition, this generalization process has some limits. If the effector used cannot be associated with a particular template, then there is no prototypical activation. This result bolds out the complexity of the involved mechanisms and calls for further experimentation to determine the boundaries of motor coding.

Overall, the existence of effector-independent action representations would allow us to flexibly map actions favoring the achievement of the underlying goal rather than the means to fulfill it. This would represent an advantage also in evolutionary terms: suppose that you are hungry and you find a nut, whether you crack it using your hand or your foot is irrelevant as long as you manage to eat it. During action observation, an effector-independent coding of the observed action would permit us to understand other's goal-directed behavior, even in the presence of a non-canonical visual input. This applies, for example, for actions performed by people with motor impairments. In such circumstances the advantage is bidirectional: observers may easily understand goal-directed actions performed in an atypical way, and likewise people with motor impairments can map others' actions according to their actual motor possibilities.

### DATA AVAILABILITY STATEMENT

The datasets generated for this study are available on request to the corresponding author.

## ETHICS STATEMENT

The studies involving human participants were reviewed and approved by Ethical Committee of the University of Padova. The patients/participants provided their written informed consent to participate in this study. Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.

### AUTHOR CONTRIBUTIONS

SB, MD, UC and LS designed the research. SB and MD built the experimental set up. SB and MD analyzed the data

### REFERENCES


and performed the statistical analyses. All authors discussed the results, contributed to the writing, reviewed and edited the manuscript.

### FUNDING

The present work was carried out within the scope of the research program Dipartimenti di Eccellenza (art.1, commi 314-337 legge 232/2016), which was supported by a grant from MIUR to the Department of General Psychology, University of Padua. This work was also supported by SIR grant (Scientific Independence of Young Researchers-N, RBSI141QKX) to LS.

### ACKNOWLEDGMENTS

We would like to thank Rossella Delrio, Michele Matthew Freyhoff and Mattia Grava for their help in conducting the experiment.


Clin. Neurophysiol. Potentials Sect. 85, 17–21. doi: 10.1016/0168-5597(92) 90096-t


**Conflict of Interest**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Betti, Deceuninck, Sartori and Castiello. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# The Embodiment of Objects: Review, Analysis, and Future Directions

Aubrie Schettler1,2 \*, Vicente Raja<sup>2</sup> and Michael L. Anderson1,2,3

<sup>1</sup> Department of Philosophy, Western University Canada, London, ON, Canada, <sup>2</sup> Rotman Institute of Philosophy, Western University Canada, London, ON, Canada, <sup>3</sup> Brain and Mind Institute, Western University Canada, London, ON, Canada

Here we offer a thorough review of the empirical literature on the conditions under which an object, such as a tool or a prosthetic (whether real or virtual), can be experienced as being in some sense a part or extension of one's body. We discuss this literature both from the standpoint of the apparent malleability of our body representations, and also from within the framework of radical embodied cognition, which understands the phenomenon to result not from an alteration to a representation, but rather from the achievement of a certain kind of sensory/motor coupling. We highlight both the tensions between these frameworks, and also areas where they can productively complement one another for future research.

Keywords: embodiment, prosthethics, virtual reality, body schema, body image

#### Edited by:

Mariella Pazzaglia, Sapienza University of Rome, Italy

#### Reviewed by:

Carlotta Fossataro, University of Turin, Italy Silvia Serino, Lausanne University Hospital (CHUV), Switzerland

> \*Correspondence: Aubrie Schettler aschettl@uwo.ca

#### Specialty section:

This article was submitted to Neuroprosthetics, a section of the journal Frontiers in Neuroscience

Received: 02 August 2019 Accepted: 26 November 2019 Published: 13 December 2019

#### Citation:

Schettler A, Raja V and Anderson ML (2019) The Embodiment of Objects: Review, Analysis, and Future Directions. Front. Neurosci. 13:1332. doi: 10.3389/fnins.2019.01332

## INTRODUCTION

Is it possible to "embody" external objects, such as prosthetics or tools, that is, to treat or regard them as in some important sense actually part of our bodies? Most of the literature concludes that people can extend the borders of the physical body to temporarily incorporate different prosthetics, such as rubber hands, into their body image (i.e., their conscious beliefs regarding their bodies; see Botvinick and Cohen, 1998; Ehrsson et al., 2004, 2005, 2008; Tsakiris and Haggard, 2005; Tsakiris et al., 2006; Marasco et al., 2011; D'Alonzo and Cipriani, 2012; D'Alonzo et al., 2014; Collins et al., 2016), and certain external objects, such as tools, into their body schema (i.e., their unconscious knowledge of their bodies and its capacities; see Cardinali et al., 2009b, 2012; Sposito et al., 2012; Baccarini et al., 2014; Garbarini et al., 2015). Several studies indicate that there are surprisingly few constraints on incorporating objects into either of these two kinds of body representations.<sup>1</sup> In fact, along with using real life tools, even imagining using tools can cause them to be incorporated into the body schema (Baccarini et al., 2014).

In this review, we highlight neuropsychological research and the numerous illusion studies on neurotypical individuals that explore embodiment, including those that use the rubber hand paradigm and/or virtual reality setups. The rubber hand illusion has been a paradigm widely used in studies of the relationship between the body image and the body schema. In the rubber hand illusion (detailed further in the next sections) an experimenter simultaneously touches a suitably positioned fake hand and the participant's real hand hidden from view. The effect is participants feel the rubber hand to be their own. Virtual reality environments have also been used to investigate embodiment phenomena, and offer flexibility to address how and to what extent objects and

<sup>1</sup>As we will detail below, the number and types of body representations that exist is a matter of dispute. See Anderson (2018) for discussion. For our purposes we will be content with the standard distinction between body image and schema.

tools are experienced and incorporated into our body representations. It is flexible in the sense that different virtual environments can be created to test the incorporation of objects and tools into our body representations and virtual body parts and bodies can be manipulated differently than in other non-virtual reality experimental setups. It also allows for one's "objective body to remain in the real world, while one's phenomenal body can be projected into terminal reality" (Murray and Sixsmith, 1999). Namely, that while one is immersed in a virtual environment, at least several aspects of her phenomenal body (e.g., the visually perceived body) are part of the virtual environment: one visually perceives whatever body she has in the virtual environment and not her body in the real world.

In this paper, first we outline two competing frameworks for understanding object and tool embodiment and review the relevant literature of embodiment of objects and tools from the point of view of these two fundamental frameworks. In Sections "Embodying Objects" and "Embodying Tools," respectively, we analyze the way object and tool embodiment are understood in terms of body representations. In both cases we analyze experimental paradigms such as the rubber hand illusion and the use of virtual environments. In Section "Radically Embodied Tools and Objects," we analyze object and tool embodiment from the specific point of view of radical embodiment. In all cases we pay special attention to the deeper theoretical commitments and the experimental methodology of the different frameworks. Importantly, our aim is not to critically analyze one of these two frameworks from the point of view of the other one, but to review them and to explore their compatibilities and incompatibilities.

### EMBODIMENT

Writ large, the embodiment of objects and tools may be defined as the sense that those objects and tools has become "part of us" in a similar way that our limbs or our fingers are parts of us. More specifically, the embodiment of objects and tools includes the sense of ownership, the sense of agency, and the sense of self-location (Kilteni et al., 2012), which often are characterized as affective, motor, and spatial embodiment, respectively (de Vignemont, 2011).<sup>2</sup> We turn to them now to offer a brief introduction. The main concepts will be revisited and elaborated in later sections.

### Affective, Motor, and Spatial Embodiment

The sense of ownership or affective embodiment refers to a situation where an individual shows the same affective reactions for the object as for their own body (de Vignemont, 2011). In this sense, affective embodiment is one of the fundamental components of the embodiment of objects and tools: the first step toward such an embodiment is the actual sense that a given object or tool is part of one's body. Measures of affective embodiment involve evaluating behavioral and physiological responses in situations in which an object may be said to be embodied.

The sense of agency or motor embodiment corresponds to the situation where an object or tool "is processed in the same way as a part of one's body for motor tasks" (de Vignemont, 2011, p. 87). Measures of motor embodiment are more often used in virtual reality and tool embodiment studies. If "the motor system takes the properties of [a tool] into account as properties of the effector in planning, then [it] is embodied" (de Vignemont, 2011, p. 86). Underlying the sense of agency is a modulation of the body schema. For instance, after tool use there can be consequences on free hand movement kinematics (Cardinali et al., 2009b).

The sense of self-location or spatial embodiment involves a bodily frame, an external frame, and a peri-personal frame (de Vignemont, 2011). The bodily frame is the body space's boundaries. For instance, if an object "is taken into account by the representation of the body space, by replacing a missing body part, by adding a new body part, or by stretching an existing body part, then [it] can be said to be embodied" (de Vignemont, 2011, p. 85). Moreover, localizing "bodily sensations in [an object] shows that [it] is taken into account by the representation of the body space and that [it] is embodied" (Ibid.). For instance, amputees can sometimes feel things in contact with their prosthetic (Murray, 2004). The external frame is represented the mis-localization of a bodily object toward a non-bodily object (aka proprioceptive drift). For instance, if "the location of [an object] within the external frame is processed in the same way as the location of a part of one's body, then [it] is embodied" (de Vignemont, 2011, p. 85). Lastly, the peri-personal frame "is the frame of the space immediately surrounding a specific part of one's body (<30–50 cm)" (de Vignemont, 2011, p. 85). An object or tool is embodied if the object or tool is processed as peri-personal space (de Vignemont, 2011).

It is worth noting that the three senses of embodiment are not independent from each other. For example, brain studies and movement disorders seem to show that there is a mutual relationship between affective embodiment and the motor system (Schütz-Bosbach et al., 2006; Della Gatta et al., 2016; Burin et al., 2017; Fossataro et al., 2018). Affective embodiment requires multisensory integration within a fronto-parietal network. This network includes an important area for the interaction between affective embodiment and motor system, the ventral premotor cortex (Ehrsson et al., 2004; Makin et al., 2008b; Blanke et al., 2015). Also, spinal cord injuries have been shown to have an altered sense of body ownership (Scandola et al., 2014).

Additionally, affective embodiment and some aspects of spatial embodiment, such as proprioceptive drift, reflect different components of the multisensory integration process (Ehrsson et al., 2005; Longo et al., 2008; Makin et al., 2008a; Rohde et al., 2011; Martinaud et al., 2017). For example, fMRI data shows there is a relationship between ventral premotor activation and subjective report, which is related to affective embodiment (Ehrsson et al., 2004) and inferior parietal lobule activation

<sup>2</sup> It is worth noting that de Vignemont (2011) refers to affective, motor, and spatial embodiment as implicit measurements of embodiment. We do not use this wording in our article because we take them to be aspects of a conceptual analysis of embodiment that can actually be mapped onto the senses of ownership, agency, and self-location respectively. Also, we prefer to use the wording of "measurements" for the explicit measurements in the literature. We want to thank an reviewer for pointing out this fact and giving us the opportunity to clarify it.

during recalibration of perceived hand position, which is related to proprioceptive drift (Ehrsson et al., 2004, 2005).

Finally, it is important to note the influence of perspective for embodiment although it is still a matter of discussion. On the one hand, some studies have suggested that the firstperson perspective is important for embodiment (Blanke and Metzinger, 2009), which seems to be confirmed in rubber hand experiments (Pavani et al., 2000; Kalckert and Ehrsson, 2012; Bucchioni et al., 2016) and full body virtual reality experiments (Petkova and Ehrsson, 2008; Slater et al., 2010; Petkova et al., 2011; Maselli and Slater, 2013). Also, researchers found with the use of a questionnaire (Tieri et al., 2015b), measuring skin conductance response (Tieri et al., 2015a), and electroencephalographic response (Pavone et al., 2016), that passively observing a virtual body in first-person perspective is sufficient for embodiment (Maselli and Slater, 2013; Tieri et al., 2015b, 2017). A similar result found that "visual capture by a fake hand (without any synchronous or asynchronous tactile stimulation) affects body ownership in a group of hemiplegic patients with or without disturbed sensation of limb ownership" (Martinaud et al., 2017, p. 174). On the other hand, others do not observe significant differences between first and third person perspectives for embodiment (Pomés and Slater, 2013).

## The Cognitive Science of Embodiment: Two Approaches

When it comes to understanding how we embody objects or tools, two wide theoretical frameworks are used in contemporary research. One framework regards the embodiment of objects or tools as a matter of incorporating them into our body representations, that is the body schema, and/or image. The other framework, that will be referred as radical embodiment, takes the process of embodiment to be the constitution of a new complex system with both somatic and extrasomatic components.

The framework based on body representations posits the existence of some kind of body model or models that represent diverse features of the body and the way non-bodily objects and tools may be integrated into those representations. A way to categorize these representations is to refer to them as the body image or one's perceptions, beliefs, and attitudes toward their body, and the body schema or one's motor capacities that work without conscious appraisal (Gallagher and Meltzoff, 1996). The body image is often characterized as a cognitive appraisal, perception, or evaluation of one's own appearance and body shape, and the related or resulting affect (Levine and Smolak, 2014). The body schema, in contrast, is a non-conscious neural map of the spatial relations among the body parts and the body's motor capacities. The body schema is a plastic representation (Giummarra et al., 2008; Gallagher, 2013) and is constantly updated due to incoming sensory input (Dijkerman and de Haan, 2007; Giummarra et al., 2008).

Although the dichotomy between body image and body schema is quite standard in the literature, it is by no means uncontested. Specifically, the relationship between body image and body schema as body representations is intensely debated. Some authors defend the existence of a unitary body representation that must encompass the features usually attributed to body images and body schemas as distinct representational entities (de Vignemont and Farnè, 2010). Support for this interpretation comes from the self not being experienced as many separate body parts, but as a single whole-body representation. This is important since the brain might resolve sensory conflicts by checking the compatibility of multisensory input with a prior body representation, which determines what can and cannot be embodied (Tsakiris and Haggard, 2005; De Preester and Tsakiris, 2009; Tsakiris, 2010; Moseley and Flor, 2012). This long-term body representation is an anatomical structure of the body and is continuously updated from the sensory modalities (Botvinick and Cohen, 1998; Naito et al., 2002; Lackner and Dizio, 2005). On this view, the process of "embodiment must respect some basic anatomical constraints. Therefore, only some objects [and tools] under certain circumstances can be processed as if they were parts of one's body" (de Vignemont and Farnè, 2010, p. 205).

Alternatives to this unitary approach include the dyadic and triadic body representation models (Schwoebel and Coslett, 2005; Gallagher, 2013) or somatosensory streams (Dijkerman and de Haan, 2007). On the one hand, the dyadic model takes the body schema to be an action-based sensorimotor representation, and the body image to be a non-action-based body representation (Gallagher and Cole, 1995; Rossetti et al., 1995; Dijkerman and de Haan, 2007). This is the standard view presented before in this section. On the other hand, the triadic body representation model accepts the standard notion of body schema but dichotomizes the body image into two different representations: the body semantics and the body structural description representations (Schwoebel and Coslett, 2005). Another model has also further divided the body schema into a motor schema and a somatosensory schema so that there are at least five body representations (Anderson, 2018). Here, we will simply accept the simple dyadic model as sufficient for our purposes in framing the following discussion. As prosthetics become more sophisticated in their capacity to offer various kinds of somatic feedback, our understanding of how such objects can be embodied will have to be likewise updated.

The framework of radical embodiment does not appeal to the concept of body representations to understand the way objects and tools can be embodied. In general, radical embodiment takes cognitive systems to be a kind of complex, self-organized system in which many components interact with each other to give rise to a given cognitive ability (Van Orden et al., 2003; Chemero, 2009; Cavagna et al., 2010; Anderson et al., 2012). The different components of these cognitive systems extend through the brain, the body, and the environment; and, in this sense, object and tools of very different kinds may be parts of cognitive systems and, therefore, may be embodied in those systems just insofar as they are properly coupled to them.

Understanding object and tool embodiment in terms of the activity of a cognitive system extending beyond the brain and the body and integrating those objects and tools into that activity, is a common feature within some radical proposals in the embodied approach to cognition. The strongest forms of the hypothesis of the extended mind (Menary, 2010; Kirchhoff and Kiverstein, 2019) and those approaches based on ecological psychology, enactivism, and dynamic systems share this form of radical embodiment. In both cases, the signatures of the embodiment of objects and tools are not taken to be a matter of body representations, but a matter of coupling between the different components of the system (e.g., the body + prothesis, the body + a tool, and so on; see, e.g., Van Orden et al., 2005; Dotov et al., 2010). The nature and degree of such a coupling requires its own theoretical and methodological commitments to be fully understood.

### EMBODYING OBJECTS

fnins-13-01332 December 12, 2019 Time: 14:58 # 4

Physical objects are those things that are animate or inanimate and can persist through time, such as a car, a pen, or a prosthetic device. Objects are affected by external forces, can be threatening or non-threatening, and can be cognitively reflected on Longo (2016). Some views on the body take it to be a multisensory object that seems to obey our will, has the ability to interact with other objects, can incorporate them, and can be perceived and understood from the inside (Blanke, 2012). Having such understanding of the unique status of the body as an object, the study of the way individuals embody objects has been carried out using experimental paradigms involving fake limbs, like the rubber hand illusion, and immersive environments in virtual reality.

### The Rubber Hand Illusion

The rubber hand illusion is a well-established paradigm to study the sense of ownership in healthy individuals (Botvinick and Cohen, 1998; Ehrsson et al., 2004, 2005, 2008; Tsakiris and Haggard, 2005; Tsakiris et al., 2006; Marasco et al., 2011; D'Alonzo and Cipriani, 2012; D'Alonzo et al., 2014; Collins et al., 2016). In the original rubber hand paradigm, participants sit with their left arm resting on a table, hidden behind a screen, and are asked to fixate on an anatomically congruent rubber hand (Botvinick and Cohen, 1998). The experimenter simultaneously stroked the participant's hand and the fake hand with two paintbrushes and then quantified the effect on affective embodiment by having participants respond to a questionnaire that included nine perceptual effects on a sevenpoint Likert scale ranging from 'agree strongly' to 'disagree strongly.' Statements included items such as "It felt as if the rubber hand were my hand" and "It seemed as if I were feeling the touch of the paintbrush in the location where I saw the rubber hand touched" (Botvinick and Cohen, 1998). After 10–15 s of synchronous visuo-tactile stimulation (less than 300 ms of temporal stroking discrepancy), participants reported significantly stronger agreement with such statements as compared to their self-reports after asynchronous stimulation, confirming a vivid sense of ownership over the rubber hand (Ehrsson et al., 2004, 2005; Lloyd, 2007; Shimada et al., 2009).

In addition to eliciting agreement with ownership statements like those above, there were also changes in the feeling of hand position (proprioception), which occurred between 10 and 60 s (Ehrsson et al., 2004, 2005; Lloyd, 2007). When asked to point with their hand where their real hand's location is, participants tended to locate it as being closer to the nonbodily object or rubber hand, that is, participants exhibited the proprioceptive drift (Botvinick and Cohen, 1998; Ehrsson et al., 2004, 2005; Longo et al., 2009; Kalckert and Ehrsson, 2014). The idea is "proprioception drifts rapidly in the absence of vision, and in the [rubber hand illusion] set-up this results in overwriting the proprioceptive location information of one's own hand with the visual location information of the rubber hand" (Kammers et al., 2009b, p. 205).

Proprioceptive drift is supported by affective measurements on the real and fake hand. One study found a decrease in the real hand's skin temperature and a touch dulling effect (Moseley et al., 2008), although the result was not replicated in other studies (Rohde et al., 2013; De Haan et al., 2017). Also, the effects of somatosensory inputs to the real hand are reduced, leading to a lower intensity sensation (Folegatti et al., 2009; Zopf et al., 2011b; Kilteni and Ehrsson, 2017). Further, TMS elicited motor evoked potentials are reduced, suggesting the motor system is less activated toward the real hand's muscles (Della Gatta et al., 2016). Threatening the fake hand also elicits strong cortical startle responses (Ehrsson et al., 2007; Gentile et al., 2013).

Some studies show that proprioceptive drift can also be experienced while the participant's real hand maintains normal kinematics, that is, although the conscious sense of hand location is affected, reaching movements are not (Kammers et al., 2009a). According to these studies, even with perceptual or body image changes (i.e., a rubber hand instead of the real hand) the body schema seems to be left unaffected by the drift. This may show that there are different mechanisms for moving the body and judging bodily properties or a dissociation between the body image and the body schema. Also, it might suggest that vision of a body part does not override the elements of proprioception that maintain kinematic functions. Further, this shows a major difference between perceptual embodiment, which consists in object representation within the body image, and motor embodiment, which consists in object representation within the body schema (de Vignemont and Farnè, 2010). While the location of the rubber hand is perceptually embodied, i.e., it is similarly processed as a body part for perceptual tasks, it is not motorically embodied. However, some rubber hand experiments have found decreased motor performance with the real hand (Heed et al., 2011), which seems to entail that the motor system and perceptual judgments are both affected and leaves the question regarding the relationship between body image and body schema still open.

An interesting aspect of the study of proprioception with the experimental setting of the rubber hand illusion is the possibility of using neurophysiological evidence to enrich behavioral evidence and the subjective reports in order to address open questions. For instance, some studies have used evoked potentials (Zeller et al., 2011, 2015) or functional imaging (Limanowski and Blankenburg, 2015) to find out whether the mentioned conflict between vision and proprioception can be resolved. For example, using EEG, researchers have found an amplitude reduction of early evoked potential components over the contralateral somatosensory cortex during the rubber hand experiment (Zeller

et al., 2015) suggesting a reduced connectivity in the that cortex (Zeller et al., 2016). Also, not only do schizophrenic patients have a stronger and faster (five times than normal control subjects) onset of the rubber hand experiment, but they also have long latency evoked somatosensory evoked responses (Peled et al., 2000). In a different example, experimenters have explored the role of the premotor cortex and intraparietal sulcus in the sense of ownership also using electroencephalography (Rao and Kayser, 2017). Further, the neural mechanisms, i.e., the multisensory areas of the premotor and intraparietal areas, underlying the experiment have been observed by fMRI in both healthy participants and upper limb amputees (Ehrsson et al., 2004, 2005; Bekrater-Bodmann et al., 2012; Brozzoli et al., 2012; Schmalzl et al., 2013).

Given these results, the rubber hand illusion illustrates that a non-bodily object can be incorporated into both the body image and schema. The brain appears to have modified the body image, possibly causing the proprioceptive drift, so that the participant experiences limb ownership over the rubber hand (Tsakiris and Haggard, 2005). This directly affects the body (motor) schema in that it now displays the real hand as having acquired the position of the fake one (Botvinick and Cohen, 1998; Ehrsson et al., 2004, 2005). Due to the need for a stable body image, the real hand becomes disembodied and the non-bodily object becomes part of the body image. This results in the body (somatosensory) schema being modified since touch is often reported to be felt in the rubber hand (Botvinick and Cohen, 1998; Durgin et al., 2007). The model proposed by Botvinick and Cohen (1998) understand this phenomenon in terms of the malleability of the body representation caused by multisensory processing.

The extent to which non-bodily objects are incorporated into the body image and the body schema can be further measured (Armel and Ramachandran, 2003). For instance, in one study researchers placed a Band-Aid on the participants' real hand and a table. Researchers then stroked participant's hidden, real hand while stroking the table at the same time (Armel and Ramachandran, 2003). To quantify the effect of affective embodiment, researchers used questionnaires and skin conductance responses, an objective measure of changes to skin electrical conductance that occur from the anticipation of pain (Armel and Ramachandran, 2003).<sup>3</sup> The researchers found that participants had strong skin conductance responses when the table was threatened, i.e., the table's Band-Aid was partly pulled off, verifying the participants' questionnaire responses that they felt the table was their hand (Armel and Ramachandran, 2003). Also, participants "frequently reported that the table illusion was vivid when the touch was received through a common covering—the band-aids- but weak in its absence (Armel and Ramachandran, 2003, p. 1505). This suggests that consistency in the seen and felt touch is critical for body ownership. Neuropsychological evidence also confirms that the body image and body schema can be affected by inanimate objects. In one case, a brain-lesioned patient did not experience a sense of ownership toward their arm, hand, and wedding ring on the hand. However, when the ring was removed, the patient viewed it as their own (Aglioti et al., 1996). Such findings indicate the body image can be affected by higher cognitive processes.

In slight opposition to Botvinick and Cohen's model (1998) and Armel and Ramachandran (2003) proposed the previous results were due to Bayesian perceptual learning based on the strong likelihood that stroking the real hand has caused the multisensory stimulus and not two different events, i.e., the synchronous stroking of the real hand and a table. The brain's statistical correlations use vision over touch (Armel and Ramachandran, 2003) resulting in the viewed table becoming part of the body image. However, this proposal has been criticized due to a focus on the likelihood of sensory data, and a failure to explain why synchronous stimulation does not always lead to a sense of ownership (Tsakiris and Haggard, 2005; Tsakiris et al., 2009; Tsakiris, 2010; Zopf et al., 2010). For instance, studies have found that when a hidden real hand and visible checkerboard floor of the box were brushed synchronously, the subjective reports indicated a weak or non-existent illusion (Zopf et al., 2010). Notably, the self-reports and the skin conductance responses were much lower when a table was used, suggesting a role for top–down mechanisms in the sense of ownership (Armel and Ramachandran, 2003).

Similarly, the sense of ownership was inhibited when using a wooden block and fingers (Tsakiris et al., 2009) and when participants saw a smaller rubber hand (Pavani and Zampini, 2007). The illusion is also disrupted if the hand is seen as wooden sticks (Tsakiris and Haggard, 2005) or wooden slabs (Guterstam et al., 2013), and if they see their full body as a cuboid with no limbs (Lenggenhager et al., 2007). Additionally, proprioceptive drift might not be explained by statistical correlations since the rubber hand's position has also been found to drift toward the real hand to a location between both hands (visual drift) (Erro et al., 2018).

A different model to explain the rubber hand experiment's results is the neurocognitive model, which emphasizes a role for multisensory processing and higher cognitive functions (Tsakiris, 2010). Under this model, the rubber hand illusion results from bottom–up processing of multisensory inputs and top–down processing or stored representations of one's hands. The sense of ownership is due to three critical comparisons. First, the object's visual form and the pre-existing reference body model must match (Tsakiris and Haggard, 2005; Costantini and Haggard, 2007). The "more the viewed object matches the structural appearance of the body part's form, the stronger the experience of the body-ownership will be" (Tsakiris, 2010, p. 13). This is supported by numerous studies (Ehrsson et al., 2005; Tsakiris and Haggard, 2005; Haans et al., 2008; Tsakiris et al., 2009). The sense of ownership is also not affected by skin color and texture, suggesting the body model is different than the body image. In addition, the current body schema state and the object's anatomical, structural and postural features must match (Tsakiris, 2010). The sense of ownership is elicited if the object's posture and body part match but reduces with postural and anatomical discrepancies (Ehrsson et al., 2004; Tsakiris and Haggard, 2005; Costantini and Haggard, 2007). For instance, placing the rubber

<sup>3</sup>Other experiments studying affective embodiment have also used skin conductance responses using a similar experimental setting (Ehrsson et al., 2007; Guterstam et al., 2011).

hand 180 and 90 degrees to the real hand reduces the sense of ownership (Ehrsson et al., 2004; Tsakiris and Haggard, 2005; Kalckert and Ehrsson, 2014). Furthermore, the seen and felt touches of the brush must match for the sense of ownership to be elicited (Tsakiris, 2010). Notably, a recently discovered constraint is that exteroceptive signals must be integrated with interoceptive signals, such as from the cardiovascular system (Tsakiris, 2017). These comparisons suggest that for objects to be embodied into the body image, they need to be compatible with our unconscious body schema, such as postural congruency, but not with our current body image.

And yet, Guterstam et al. (2013) observed that participants embodied a volume of empty space, while others found they embodied a rubber hand 3 cm larger than their real hand (Pavani and Zampini, 2007) and three times as long as their real arm (Kilteni et al., 2012). Also, in one study synchronous visuo-tactile stimulation of one real and two rubber hands led participants to feel touch on and ownership toward, although less vividly, two right-handed rubber hands (Ehrsson, 2009). Other studies have found the sense of ownership toward two left hands (Newport et al., 2009) and four hands (Chen et al., 2018).

Rubber hand experiments have also elicited a sense of ownership with visuo-thermal stimulus patterns (Trojan et al., 2018), without vision (Ehrsson et al., 2005; Petkova et al., 2012), and without touch by using a laser pointer (Durgin et al., 2007). Thus, the phenomena are clearly more complex than can be accounted for by simple visuo-tactile integration, and researchers have accordingly moved beyond simple tactile stimulation.

Studies have also provided participants the ability to move the fake limb (Tsakiris et al., 2006; Dummer et al., 2009; Sanchez-Vives et al., 2010) to generate affective embodiment, while others have investigated motor embodiment through questionnaires (Kalckert and Ehrsson, 2012, 2014; Jenkinson and Preston, 2015). In one study, participants could move the rubber hand's index finger (Kalckert and Ehrsson, 2012). Researchers manipulated the synchrony between the real and fake hand's finger movements, the mode of movement (passive vs. active), and the rubber hand's positioning. They found that both affective and motor embodiment are distinct since asynchrony eliminated both, but only motor embodiment was abolished with passive movements, while affective embodiment was reduced with incongruent positioning. Another study also found support for a double dissociation between motor and affective embodiment (Marotta et al., 2017).

Because of these complexities and variety of results, researchers have also begun to explore embodiment in virtual reality, which allows for a greater variety of experimental setups than the simple rubber hand device permits. It is to a review of that literature that we now turn.

### Embodying Objects in Virtual Reality

Virtual reality technology creates an interactive environment for its users by way of a diverse number of devices such a head mounted display (HMD), head tracking, real time motion capture, tactile feedback, and audio. To study object embodiment in virtual reality, researchers have used limb illusions (Slater et al., 2008), full body illusions using avatars (Slater et al., 2010), or combined virtual and non-virtual reality conditions (Ijsselsteijn et al., 2006). In one virtual rubber hand experiment that was set up similar to Botvinick and Cohen's (1998) experiment, questionnaire responses and proprioceptive drift showed that participants had a sense of ownership toward the virtual hand (Slater et al., 2008).

In another study, spinal cord injury patients received tactile back stimulation while viewing virtual legs being touched through an HMD (Pozeg et al., 2017). Studies have found that there is cortical reorganization after spinal cord injuries and the lower back is connected with the leg representations (Wrigley et al., 2009). For this reason, the researchers manipulated the synchrony between the stroking of the virtual legs and the participant's lower or upper back. The results showed that these patients experienced weaker leg ownership than healthy control participants, with the time since injury also negatively correlated with leg ownership. This suggests that these patients less readily integrate the available visual and tactile information to experience leg ownership. There was also no difference between the lower and upper back conditions, suggesting that instead of a reorganization of the primary somatosensory cortex (S1), that maybe other leg representations are involved or there are larger receptive fields on the back in S1 so that the upper and lower back conditions were engaging similar S1 locations.

In a different study, researchers used an unmediated condition, a virtual reality condition where a video-projection of a rubber hand and its visuo-tactile stimulation was on a flat tabletop surface, and a mixed reality condition, where the fake hand was projected but had unmediated visuo-tactile stimulation (Ijsselsteijn et al., 2006). Self-reports and proprioceptive drift showed that the non-virtual reality condition had the strongest sense of ownership. There was also a stronger sense of ownership in virtual reality than mixed reality but no difference in drift. The existence of differences between the virtual reality and non-virtual reality conditions contradicts a Bayesian learning explanation (Ijsselsteijn et al., 2006). Instead, these differences seem to support the neurocognitive model in that there is a role for top–down mechanisms that specify requirements for an object, since the physical objects would be experienced as more real. Improvements in VR technology may eventually erase this effect.

Virtual reality experiments on object embodiment typically use objective measures, such as defensive motor responses to a threat to a limb (Kilteni et al., 2012), an electroencephalogram via event-related potentials (González-Franco et al., 2014), temperature changes in the hand (Hohwy and Paton, 2010) and a cross-modal congruency task (Pavani et al., 2000; Zopf et al., 2011a), discussed in Section " Tools and Body Representations: Peri-personal Space," below.

For example, in one study participants had to fixate on the virtual hand of the collocated avatar that had been placed on a desk (González-Franco et al., 2014). Event-related potentials were recorded when a knife attacked the virtual hand and when it struck the virtual table. The results showed that similar to nonvirtual reality experiments, participants experienced a sense of ownership toward the virtual hand and body, suggesting that the body image has been manipulated to include the avatar and

its virtual hand. Moreover, the strong sense of ownership found in the questionnaires correlated with larger P450 amplitudes. Also, when the virtual hand was threatened the motor cortex had Mu rhythm event-related desynchronization and there was more readiness potential (C3–C4) negativity.

Virtual reality paradigms are also used to study object embodiment for non-bodily objects. In one study, researchers synchronously tapped the participant's forearm and collocated virtual forearm (Hohwy and Paton, 2010). After 30 s, the virtual forearm was replaced by a virtual cardboard box that was then synchronously tapped with the real forearm. The results showed that some participants believed they felt touch on the virtual cardboard box. However, the cardboard box illusion was not elicited without the preceding virtual hand induction, which may explain Armel and Ramachandran's (2003) results since participants were exposed to the rubber hand illusion prior to the table touch (Hohwy and Paton, 2010).

Virtual reality paradigms can further expand on how flexible the body image and schema are and what constraints there are to embody an object. Virtual reality can easily stretch body parts past their normal size, such as a finger to double its size (Newport and Preston, 2010) or have one collocated limb that continuously grows in length and moves with the real limb (Kilteni et al., 2012). Virtual and non-virtual reality studies on object embodiment suggest that "the topographic representation in the primary somatosensory cortex (S1) reflects the perceived rather than the physical aspects of peripheral stimulation" (Schaefer et al., 2007, p. 700). In one non-virtual reality study, vision and somatic sensations were manipulated to elicit an illusion of an elongated arm (Schaefer et al., 2007). An artificial hand and arm were attached over the subject's real one, extending it by about 20 cm. They found a sense of ownership over the lengthened limb by way of a questionnaire and neuromagnetic source imaging, which "revealed a corresponding modulation of S1 to the extent of feeling the arm elongated: the more the subjects felt the arm elongated, the more the cortical distance between D1 and D5 [digit 1 and digit 5 on the left hand] decreased" (Schaefer et al., 2007, p. 703). It was speculated that due to the perception of a longer arm, there was a larger cortical arm representation and hence participants felt a longer arm. In virtual reality, however, there does not seem to be any investigations into the amount of cortical reorganization for different virtual limb lengths (Kilteni et al., 2012). Nonetheless, there seems to be a correlation between perceiving one's body size and shape and modulations in the cortex (Schaefer et al., 2007; Normand et al., 2011; Kilteni et al., 2012; Banakou et al., 2013; Won et al., 2015).

If there is a top–down perceptual body model and virtual reality experiments give us some insight into the embodiment of objects, then it seems the specific bodily features that it encodes are unclear. Researchers have found that the participants can embody a virtual arm that is triple in length (Kilteni et al., 2012), an avatar of a different race (Peck et al., 2013), and an avatar with a different shaped body (Normand et al., 2011; Won et al., 2015). Adult participants have also embodied a child body (Banakou et al., 2013), and feel identified with realistic androids (Cooney et al., 2012) and unrealistic humanoid robots (Aymerich-Franch et al., 2015). Such virtual reality studies appear to contradict other studies that had used a wooden stick (Tsakiris and Haggard, 2005), a rubber sheet (Haans et al., 2008), and a wooden slab (Guterstam et al., 2013) and found a reduced or eliminated sense of ownership. While a smaller sized hand was shown to inhibit the embodiment (Pavani and Zampini, 2007), other studies showed the embodiment of 30 to 400 cm virtual dolls (Van der Hoort et al., 2011) and full-body mannequins (Petkova and Ehrsson, 2008; Petkova et al., 2011).

To explain these results, functionality may be the key feature of the body model (Aymerich-Franch and Ganesh, 2016). The brain perceives an object as one's body part or whole-body if the object's physical properties are sufficient to allow certain actions (Aymerich-Franch and Ganesh, 2016). For instance, the rubber hand and full body illusions are possible even when rubber hands are of different color (Longo et al., 2009) or texture (Haans et al., 2008), and when the body is a different in size, gender, or age since these features do not affect proper functions (Petkova and Ehrsson, 2008; Van der Hoort et al., 2011; Maister et al., 2015). Multiple rubber hands (Ehrsson, 2009), longer arms (Kilteni et al., 2012) and larger rubber hands (Pavani and Zampini, 2007) maintain functionality, while a wooden stick (Tsakiris and Haggard, 2005), a wooden slab (Guterstam et al., 2013), and smaller rubber hands (Pavani and Zampini, 2007) cannot adequately perform tasks.

The variety of embodiment effects, as summarized by this review, seems to suggest that there is not one physical body representation. Instead, it could be the case that these observed embodiment effects are due to the interaction of multiple body representations. Indeed, there is no consensus yet on how many different bodily representations there are, what their characteristics are, and whether and to what degree they can be truly dissociated.

### EMBODYING TOOLS

Objects are those things that are nameable, identifiable, stable, and can persist through time, such as pencils and cars. Tools are a specific kind of object employed to alter or interact with other objects. Three categories of tools are physical-interaction tools, which interact with the environment; pointing tools, such as a laser; and detached tools, such as a computer mouse (tool) that interacts with objects via an interface (Holmes and Spence, 2005). Tools are different from other objects since they are associated with specific manipulation-actions, such as grasping a hammer and manipulating a nail, used to reach for objects, and used to sense one's surroundings. This suggests a major role for the sensorimotor system in tool embodiment. They are also different from other objects since individuals typically lack a sense of ownership toward a tool. For Botvinick (2004), "the feeling of ownership that we have for our bodies does not extend to, for example, the fork we use at dinner" (p. 783). As a result, tool embodiment studies do not typically use affective measures, such as using a threat to measure physiological responses, since tool use can involve dangerous situations, such as stirring hot water.

Tool embodiment research focuses on the spatial modulation of multisensory integration following tool use, which involves

spatial embodiment, and the plasticity of body representation following tool use, which involves motor embodiment The incorporation of a tool into the body and the expansion of peripersonal space are similar in that they seem to be due changes in the body schema (Iriki et al., 1996; Maravita and Iriki, 2004). But they are also separate in that the body schema requires somatic sensation, while the multisensory peri-personal space is based on vision and audition (Cardinali et al., 2009b).

### Tools and Body Representations: Peri-Personal Space

Peri-personal space is located directly around the body and involves multisensory integration in the frontal and parietal lobes (Làdavas and Serino, 2008; Serino, 2019). Evidence suggests that peri-personal space representations are multimodal since they respond to visual information (Longo and Lourenco, 2006) and visuo-tactile information (Noel et al., 2015). Also, bodily dimensions such as body part size changes the size of peripersonal space. For instance, individuals with longer arms tend to have more peri-personal space (Longo and Lourenco, 2007).

Peri-personal space has been divided into near and far space or the space within and beyond reach (Berti and Frassinetti, 2000; Witt et al., 2005). Here, reachability may be a perceptual metric such that everything that is reachable is perceived to be in action space (Witt et al., 2005). Tool use also seems to alter the perception of the environment bringing objects as perceived as farther away as closer and into the action space (Witt et al., 2005), although others have found that near peri-personal space appears to grade off from the body in that it does not abruptly end at arm's reach (Longo and Lourenco, 2006, 2007). Studies with monkeys have shown that near space and far space are coded differently in their brain (Colby et al., 1996). Also, one PET study showed that during tool use the monkey's brain showed an increase in the activity and reorganization of the reach representation in the intraparietal region, basal ganglia, premotor cortex, and cerebellum (Obayashi et al., 2001). These results are further supported by human behavioral studies (Berti and Frassinetti, 2000; Maravita et al., 2003; Farnè et al., 2007), PET studies (Weiss et al., 2000) and TMS studies (Lane et al., 2011). Similarly, a PET study on humans showed that during tool use the ipsilaterial posterior parietal cortex was activated (Inoue et al., 2001).

To understand what happens to the peri-personal space during and after tool use with a non-threatening object, researchers trained macaque monkeys to control a rake via a computer screen to reach food placed in far space (Iriki et al., 1996). Reachability was used when macaque monkeys were taught to reach with a rake, which extended their reach (Iriki et al., 1996). This suggests that some visual neurons code reach and adapt to changes in reachability due to tool use. They found that by recording premotor or posterior parietal cortices, tool use expanded the receptive fields of bimodal neurons to include the tool's entire length. The enlargement of peri-personal space has been interpreted as a remapping of farther away objects as nearer ones (Di Pellegrino et al., 1997; Farnè and Làdavas, 2001). Additionally, when the macaque monkeys passively held the rake, the peri-personal space shrank back to pre-tool size. These results suggest that deliberative action is needed to expand peri-personal space (Iriki et al., 1996; Farnè et al., 2005).

In the neuropsychological literature, peri-personal space is understood as the changes in patients' perceptual performance as stimuli are moved further away. These changes are grounded in the activity of multisensory neurons, which "decreases as the distance between visual stimuli and tactile stimuli increases" (de Vignemont, 2011, p. 85). Brain injured patients typically experience extinction, which results in contralesional events being "perceived in isolation, yet are missed when presented concurrently with competing events on the ipsilesional side" (Kennett et al., 2010, p. 15). Researchers have used crossmodal extinction, where extinction arises cross-modally, to study these patients' peri-personal space. Results have shown that the location of visual stimuli, such as when it is close to a body part, can interfere with patients' tactile performance (Farnè and Làdavas, 2000; Maravita et al., 2001, 2002; Farnè et al., 2005; Bonifazi et al., 2007). When these patients see a visual stimulus by the ipsilesional hand, it can extinguish a touch delivered at the same time on the contralesional hand (Di Pellegrino et al., 1997; Costantini et al., 2007). Typically, as the visual stimulus moves away from the hand or vice versa, tactile detection improves on the other hand (Làdavas, 2002).

Researchers have studied the effects of patients with tactile extinction using a tool on the peri-personal space (Farnè and Làdavas, 2000). They found that short periods of rake use with the ipsilesional hand to reach for objects (red fish) in far space increases extinction of tactile contralesional stimuli (hand touch) after visual stimuli (light flash) were placed close to the tool tip. This suggests peri-personal space expanded to include the rake's entire axis and that the rake is now part of the body. Also, similar to Iriki et al. (1996), the peri-personal space returned to pretool use size after 5–10 min of passively holding the rake. Further, free hand pointing movements toward the fish had results comparable to the pretool condition. This suggests that mere motor activity is not sufficient to expand peri-hand space (Farnè and Làdavas, 2000).

In a follow-up study, Farnè et al. (2005) used the same task to research whether the absolute of functional tool length was important to modulate the peri-personal space. They used a long (60 cm) wooden rake, a short (30 cm) wooden rake, and a hybrid long (60 cm) wooden rake where the functional part was attached at 30 cm (Farnè et al., 2005). They found that after using the hybrid (60 cm) tool, the crossmodal extinction was compatible with the 30 cm short tool. This suggests that the tool's functional part is directly related to the expansion of peripersonal space. Indeed, "the main advantage provided by the extension of the peri-hand area, whereby vision and touch are integrated, seems to be that of allocating multisensory processing where the goal of the action is" (Làdavas and Serino, 2008, pp 1106–1107). Similarly, Tomasino et al. (2012) found an increase in fMRI activity in the extrastriata body area when a joystick was used in a more compatible environment in near space than when a less appropriate tool (extended pliers) was used in a less congruent environment in near space. This finding suggests that the body's neural representation is adapted in a functional manner, depending on tool compatibility.

Another method to assess peri-personal space, the crossmodal congruency task, tests "how strongly visual stimuli affect the processing of simultaneously presented tactile stimuli" (Holmes, 2012, p. 273). Similar to cross-modal extinction, the effect is larger when spatially incongruent distractors are near the tactually simulated hand and reduces when presented close to the opposite hand. For example, in one study participants judged whether tactile vibration was delivered to the thumb or index finger on either hand while holding two golf clubs. Two visual distractor lights were at each far end of both golf clubs (Maravita et al., 2002). Each trial consisted in a vibration from one of four possible locations (finger or thumb on either hand) accompanied with one distractor light. In the uncrossed condition, visual distractors interfered with the tactile discrimination task on the same side of the tool. When tools were crossed the visual distractors had stronger interference on the hand placed in the opposite space. Like in other studies, passively holding the tools did not have this effect.

The studies reviewed suggest that tool-use shows characteristics of tool embodiment, such that peri-personal space extension is dependent on active (Farnè and Làdavas, 2000), goal-directed (Farnè and Làdavas, 2000), and functionally effective (Farnè et al., 2005) tool interaction. Other studies also support that during and after tool use, the peri-personal space is enlarged. For example, researchers have found that using a computer mouse enlarges peri-personal hand space, which may also include the screen monitor (Bassolino et al., 2010). In another study, peri-personal space was expanded when blind individuals merely held the cane without using it Serino et al. (2007). This suggests that "blind people, who continuously use the cane to integrate auditory and tactile information in far space, in order to compensate for the lack of visual information, developed a new, extended representation of auditory peri-hand space, which is selectively activated when holding the cane" (Serino et al., 2007, p. 1108). Long term regular use with a tool then may create a durable extension of peri-personal space. Further, other studies have found that tool use imagery triggers similar effects on the peri-personal space (Baccarini et al., 2014). Notably, the peri-personal space can also contract when weights impair arm movement (Lourenco and Longo, 2009) and after limb loss (Canzoneri et al., 2013).

However, an alternative explanation for some of these results is that "tools act as spatial attentional and motor cues" (Holmes et al., 2007, p. 466). To investigate this, Holmes et al. (2004) modified the cross-modal congruency task by placing the visual distractors near the hand, the tools' middle, and far from the hand or the tool's tip. Results showed that the effects of tool use were mostly at the tool's tips and weakly at the tool's middle. They also conducted a fMRI study that showed that there was a shift in participants' spatial attention to the functional part of the tool. They asked participants to ignore visual distractors but pay attention and respond to vibrotactile targets that were at the tool's tip and felt in the hand. The results were indicative of a shift in spatial attention since visual distractors at the tool's tip enhanced the BOLD response in retinotopic portions of the occipital visual cortex and decreased the BOLD response in the ipsilateral visual field (Holmes et al., 2008).

Another important research area explores the idea that tool use can make changes to the body schema. Some researchers have indicated that the peri-personal space expansion occurs because the tool is incorporated into body schema, i.e., into the sensory-motor map of the limb with the tool (Iriki et al., 1996; Farnè and Làdavas, 2000; Maravita and Iriki, 2004; Serino et al., 2007). Additionally, some studies have also shown that the tactile signals felt in the hand that occur when a held tool comes into contact with an object are referred directly to the tip of the tool (Yamamoto and Kitazawa, 2001; Maravita et al., 2002; Yamamoto et al., 2005; Collins et al., 2008). This result suggests that somatosensory integration is not required to use a tool. Hammering a nail with a hammer tends to result in the feeling of touch on the part of the hammer touching the nail, and not at the hand holding the tool. This result seems to suggest that the body (somatosensory) schema has been changed and now includes the tool's functional part. Tool use then not only seems to have effects on peri-personal space coding but also seems to change other aspects of body representations, a topic to which we now turn.

### Tool and Body Representations: Body Schema

The peri-personal space and body schema are separate entities that are highly flexible and seem to be connected to the motor system.<sup>4</sup> Some studies have found that the modulation of the body schema during tool use involves increasing the length of the arm's representation and requires goals and motor programs (Cardinali et al., 2009b, 2012; Sposito et al., 2012; Baccarini et al., 2014; Garbarini et al., 2015). For example, Cardinali et al. (2011) found that tool use involving a 40 cm long mechanical grabber resulted in a free hand kinematic pattern compatible with having a longer arm, while there were no changes in the hand-related or grip component. This suggests that for a time after tool use the body schema was still modified as if participants were still using the mechanical grabber and had a longer arm.

In the same study, Cardinali et al. (2011) investigated whether tool use can elicit a functional update of the body schema without affecting the body image by way of a motor and perceptual task (Cardinali et al., 2011). For the motor task, blindfolded participants were asked to point with their left index fingertip to a specific location on their right arm that had been touched by the experimenter, i.e., the finger, wrist, or elbow. For the perceptual task, participants verbally reported where the experimenter had touched them. They found that after tool use participants perceived a longer arm since they localized touches to their elbow and middle fingertip as farther apart. In contrast, there was no observed change when localizing named body parts, suggesting tool use may only affect the body schema.

Other studies have used a direct behavioral task, e.g., an arm bisection task, to investigate whether tool use might alter the metric representation of limbs (Sposito et al., 2012). Researchers asked healthy participants to estimate the subjective midpoint of their own forearm before and after a training

<sup>4</sup> It is worth noting that, despite this classical view regarding the differences between peri-personal space and body schema, some researchers have suggested they are just two labels for the same concept (Cardinali et al., 2009a).

phase with long functionally relevant (60 cm) tools and small functionally irrelevant (20 cm) tools. They found that participants indicated a more distal midpoint, thus exhibiting an increased representation of the arm's length after training with only the long functionally relevant tool.

Similarly, other studies used this task on brain-damaged hemiplegic patients with pathological embodiment (Garbarini et al., 2015). The participants were asked to estimate the midpoint of their paralyzed forearm before and after a training phase in which an experimenter repeatedly used a tool and were aligned or misaligned relative to patients' shoulders. In the aligned condition, patients thought that they were using the tool with their paralyzed arm. In effect, there was significant modulation of the perceived arm length in that they located their forearm midpoint closer to the hand, indicating an increased length, in the post- training phase. No effect occurred when they were misaligned to the experimenter during the training phase (Garbarini et al., 2015). These "findings show the existence of a tight link between spatial, motor and bodily representations and provide strong evidence that a pathological sense of body ownership can extend to intentional motor processes and modulate the sensory map of action-related body parts" (Garbarini et al., 2015, p. 402).

Cardinali et al. (2016) also found that using tools (sticks and pliers) that elongate the fingers results in an update of the brain's hand representation due to the tools' morpho-functional characteristics and its specific sensorimotor constraints. They found that the kinematics of the grasping hand component were affected, but the arm representation and reaching kinematics were not (Cardinali et al., 2016). Similarly, Miller et al. (2014), used a perceptual task to find that the hand representation increased in size when using a tool morphologically similar to the hand. They also found that the use of an arm-like tool affects its representation by an increase in arm length (Miller et al., 2014). Therefore, the plasticity of the limb's representation is constrained by the resemblance of the tool's morphology.

Studies have also found that the peri-personal space and the body schema can be separated. One study found that after immobilizing participants' right limb for 10 h, that the limb had a reduction of peri-personal space but no effect on the body schema since there were no consequences on the perceived limb's length. They also found the left overused limb's peri-personal space did not extend while the perceived length increased (Bassolino et al., 2015). These results suggest that both the peri-personal space and body schema depend "on different mechanisms; while [peri-personal space] representation is shaped as a function of the dimension of the acting space, metric characteristics of [body representation] are forged on a complex interplay between visual and sensorimotor information related to the body" (Bassolino et al., 2015, p. 385).

Further, these tool embodiment studies show a role for both somatic sensations and vision in the incorporation of a tool into the body schema. Somatic sensations seem to be necessary and sufficient for tool incorporation into the body schema. In one study using blindfolded participants, Martel et al. (2019) concluded that participants assessed arm length representation at an implicit level by comparing movement kinematics before and after tool use. The results showed that after tool use participants had modified movement kinematics that suggested an increased arm length representation. Martel et al. (2019) also found that "explicit arm representation seems immune to tool-use when only somatosensation is available. When participants were asked to explicitly estimate their arm length, we observed no effect of tool use" (p. 10). Neuropsychological evidence also supports this in the case where a deafferented patient had no incorporation of a tool due to a lack of proprioception (Cardinali et al., 2016).

### Tools and Virtual Reality

As in the case of embodying objects, virtual reality provides a good medium to test different measures and to investigate how interacting with virtual tools might compare to their physical counterparts. Studying the expansion of peri-personal space and body schema changes in virtual reality can provide important insights into how these environments effect our movement planning and execution.

The cross-modal congruency task has also been used in virtual reality to test whether peri-personal space can change with virtual robotic tools (Sengül et al., 2012, 2013). Like in Maravita et al.'s (2002) experiment with physical golf clubs, the task was to cross or uncross the virtual golf clubs (in one experiment) and to just hold the tool interface (in the other experiment), where the experimenter did the crossing and uncrossing (Sengül et al., 2012). The latter experiment aimed to explore whether active use of the tool was important to spatial modulation of the crossmodal congruency effect. Subsequently, participants made discriminations of vibrotactile stimuli delivered to the thumb and index finger using a foot pedal while ignoring visual distractors at the virtual tool's tip. Results showed that virtualrobotic tools can influence multisensory integration in peripersonal space. Moreover, "there was an interaction of vision and touch as reflected in the cross-modal congruency effect (CCE) for virtual robotic tools. Second, it was found that actively crossing the tool resulted in a remapping of peri-personal space, as reflected in a stronger CCE when visual stimuli appeared at a different side than the tactile vibration, at the tip of the tool that was held in the stimulated hand. Third, it was found that this remapping of peripersonal space did not depend on active tool use, as passive crossing of the tools resulted in a change in the CCE side effect" (Sengül et al., 2012, e49473). Like in other non-virtual reality studies then, active crossing of tools results in remapping the peri-personal space at the tool's tip. However, unlike some non-virtual reality studies, they found that active tool use was not required for the remapping of peripersonal space.

In the context of virtual reality, another study demonstrated that body metric estimation can be modulated by motor embodiment (D'Angelo et al., 2018). Participants were asked to perform a forearm bisection task before and after a training phase, in which they virtually grasped objects on a PC screen with a virtual hand. Researchers used a leap motion controller, a virtual reality device used for hand tracking, to synchronize the virtual hand to the participant's real hand. In a synchronous condition, participants had collocated virtual hand movements to their own right-hand movements, while in an asynchronous

condition had a 3 s delay between the participants' real hand and the virtual hand movements.

The results seem to confirm that the plastic changes of peripersonal space and the body schema "depend on the experience of controlling the course of events in space through one's own actions, i.e., the sense of agency" (D'Angelo et al., 2018, p. 1). Participants pointed to their forearm midpoint more distally, consistent with an increased length of the arm representation, after performing the training phase in the synchronous condition only. This suggests that having a different body metric depends on the intention to perform the action being congruent with the movement's motor output. These findings suggest "that body schema and peri-personal space are affected by the dynamic mapping between intentional body movements and expected consequences in space" (D'Angelo et al., 2018, p. 1).

These studies may be taken as preliminary evidence for virtual tool embodiment. Similar to object embodiment, virtual reality paradigms have the potential to further expand on how flexible the body schema is and what constraints there are to embody a tool. Virtual reality can easily stretch a tool past its original size to see the effects on peri-personal space and the body schema, while also discovering the tool's breaking point of no longer being a functioning tool for a task and the associated brain changes.

### RADICALLY EMBODIED TOOLS AND OBJECTS

So far, we have reviewed different ways in which the embodiment of tools and objects is understood as affecting body representations. Both in the case of objects and in the case of tools, embodiment occurs when they are incorporated into either the body image (consciously) or into the body schema (somewhat unconsciously). Moreover, we have seen research based on paradigms such as the rubber hand illusion and virtual reality to assess embodiment of different elements and of in different forms. In this section, we will sketch the issue from the point of view of radical embodiment. As a caveat, it is important to note that the embodiment of objects and tools from the point of view of radical embodiment is not essentially different from the one already reviewed. At the end of the day, the radical embodiment of objects and tools also consists of taking extra-somatic parts of our environment to be proper parts of our body. The differences with the framework based on body representations come from the very characterization of such an embodiment in non-representational terms and from some radical consequences that follow from this move. We will turn to these topics but, first, we succinctly introduce radical embodiment.

### A Primer on Radical Embodiment

We use the umbrella term of radical embodiment to refer to a group of theories of perception, action, and cognition inspired in phenomenology (Gallagher and Zahavi, 2007; Käufer and Chemero, 2015), ecological psychology (Gibson, 1966, 1979; Michaels and Carello, 1981; Richardson et al., 2008; Chemero, 2009; Turvey, 2019), and enactivism (Varela et al., 1991; Hutto and Myin, 2013; Di Paolo et al., 2017), as well as to some radical proposals that follow from the hypothesis of the extended mind (Menary, 2010; Kirchhoff and Kiverstein, 2019). Although we are well aware that there are more or less important differences between all these approaches (Walter, 2010a,b), we are not going to take issue with them as, for the sake of our study, their similarities overrule their differences.

The features of radical embodiment relevant for our discussion are its commitment to anti-representationalism and its characterization of cognitive systems as complex, self-organized dynamical systems. Cognitive systems are not taken to be in the business of constructing internal representations of the external environment to be able to deal with the latter. On the contrary, cognitive systems are already embodied, embedded systems which are in the business of maintaining their internal dynamics in harmony with their dynamical interaction with the environment. According to radical embodiment, such a fundamental relation between cognitive systems and their environments makes the need for internal representations of the latter either superfluous (Brooks, 1990; Chemero, 2009) or unintelligible in most of the cases (Hutto and Myin, 2013). Mental representations are supposed to play an informative role in cognition. They are the elements cognitive systems use to make epistemic contact with their surroundings. Once that contact is granted through embodiment and embedment and explained in terms of dynamical interactions, the need for mental representations disappear.

For the topic of this article, the consequences of radical embodiment are pretty straight forward: the embodiment of objects and tools is not a matter of incorporating them in body representations. Although nobody denies that the brain carries information about the states of our bodies and environment (and thus in this minimal way "represents"), for radical embodiment this fact is a trivial background condition for the possibility of adaptive behavior and is not in and of itself explanatory. Thus, the embodiment of objects and tools must be explained in some other terms. The question is which ones. In the following we try to sketch them.

### Tools of Radical Embodiment

The philosophical works of Martin Heidegger and Maurice Merleau-Ponty are useful to get a sense of the notion of tool and object embodiment from the coordinates of radical embodiment. In a famous passage of Being and Time, Heidegger (1962/2001) considers the foundational aspects of the use of tools. According to him, tools are never meaningless chunks of matter that must be incorporated into our meaningful interactions with the environment by some kind of reflective effort. Rather, we typically encounter tools within our purposive engagements with the world. In this sense, tools are typically about us and about many things in many interrelated ways.

First, a tool—or equipment in the Heideggerian jargon—is always something about what it is for and about the kind of practice or task in which it is used. For this reason, a tool is always about the capacities and interests of the user—the Dasein—and about other tools and objects that belong to the same kind of practices (see, e.g., Heidegger, 1962/2001, p. 97). A pan is for cooking, but a specific kind of cooking, one that requires oil and

fire rather than, say, citrus juices, ají, and salt as in the case of ceviche. As such, a pan is the kind of tool it is by virtue of its role in the specific practice of cooking in which it is used; that is, a pan only make sense as a pan when it is on a fire, with oil, with some kind of food within it, etc.

As important as fire, oil, and food for the pan to be a pan are the abilities and interests of the users of pans. Heidegger claims: "The work produced [by tools] refers not only to the 'towardwhich' of its usability and the 'whereof' of which it consists: under simple craft conditions it also has an assignment to the person who is to use it or wear it" (Heidegger, 1962/2001, p. 100). In other words, a tool is by itself always about the user: there is always a meaningful relationship between tools and users just by virtue of being tools and users. A tool is about users in that it is for them to do something with it that is "for-the-sake-of " and determined by the "totality of [their] involvements" (Heidegger, 1962/2001, p. 116). Such a referential aspect of the involvement of users and tools inspires a non-representational understanding of tool use and embodiment. In a famous locution, tools are ready-to-hand for the Dasein (Heidegger, 1962/2001, p. 91 & ff.); namely, users engage with tools and tools are incorporated into the users' dynamics without requiring any conceptual work from them. Tools are ready to be used and, when that happens, they become tacitly integrated into the user. Such is the Heideggerian way to refer to something equivalent to tool embodiment.

The key aspect of Heidegger's point regarding tools is that they can become embodied without the need for conscious reflection. Prima facie, such an approach is compatible with a strong form of anti-representationalism: no need for conscious assessment of tool embodiment means no need for a conscious mental representation into which the tool must be integrated. In this sense, Heidegger seems to avoid the need for a body image to understand tool embodiment. However, his proposal still seems to be compatible with some notion of body schema, as it is unconscious by definition (Gallagher, 2000). Moreover, the concept of body schema itself has been used within the phenomenological (and Heideggerian) tradition. For example, Merleau-Ponty (1945/2012) uses the notion of schema corporel, which he also calls the "lived" or "habitual" body, to refer to the relevant understanding to of the body to capture human behavior and engagement with the different elements of the environment.

Given the theoretical compatibility between the phenomenological tradition that is at the roots of radical embodiment and the notion of body schema usually entertained in the representational framework of object and tool embodiment, there must be something else to distinguish between the two frameworks. As the notion of body schema is ambiguous regarding its representational character—i.e., it can be understood in terms of representations, like in the representational framework of object and tool embodiment, and in non-representational terms, like in the case of Heidegger or Merleau-Ponty—the non-representational character of radical embodiment must be manifest in other considerations. If this were not the case, the distinction between the two frameworks would be a matter of linguistic preference if not directly trivial. So, what are these considerations that make radical embodiment actually radical and anti-representational?

The relevant considerations are methodological. The differences between the representational framework of object and tool embodiment and the framework based on radical embodiment do not (only) lie on theoretical grounds. It is true that there are some theoretical issues and incompatibilities between them, but it is also true that notions as body schema may be a point of connection if the debate were strictly held in theoretical terms. However, the differences between both frameworks become more salient when the methods to assess tool embodiment are studied. As we have seen in previous sections, when it comes to understanding the embodiment of tools in terms of body image and body schema the methods used have to do, for instance, with the illusions provoked by the rubber hand paradigm, with reaction times, with personal reports, etc. In the case of radical embodiment, the measurements of the embodiment of object and tools are completely different and have to do with the tools provided by complexity science (Riley and Van Orden, 2005; Holden et al., 2013).

As we have noted in the previous section, from the point of view of radical embodiment, cognitive systems are complex, self-organized dynamical systems. This amounts to saying that cognitive systems are systems that impose their own order (aka organization) on themselves by virtue of the ongoing interactions between their components and of the ongoing interactions of the whole system with its environment. Given this, it is commonly acknowledged that this kind of system exhibits some specific patterns of complexity by virtue of their self-organization and the foundation on multiple interactions at multiple scales such as organization around critical states and fractal features (Juarrero, 1999; Riley and Turvey, 2002; Van Orden et al., 2003; Carello and Moreno, 2005; Stephen and Dixon, 2009; Kuznetsov et al., 2013; Lamb and Chemero, 2013).

The features of complex, self-organized dynamical systems have been studied both to determine the kinds of activities cognitive systems are involved (Holden et al., 2009) and to characterize and individuate cognitive systems themselves (Van Orden et al., 2005). In the latter context, these features are relevant to understand object and tool embodiment. The underlying idea is that, when an object or a tool is embodied in a cognitive system, the tool becomes a proper part of the cognitive system and, therefore, the whole system (e.g., human body + tool) must exhibit the expected signatures of self-organization and complexity, such as a fractal organization of its different scales. In this sense, the studies of object and tool embodiment from the standpoint of radical embodiment do not really look for reaction times simpliciter or for participants' reports of ownership. Those studies look for the features of complex, self-organized dynamical systems at the scale of the cognitive systems and the object/tool as a whole, unitary system. If those features are present, the object/tool may be said to be embodied in the cognitive system. If not, the object/tool is disembodied.

The work developed by Dotov and colleagues is a paradigmatic example of this kind of study (Dotov et al., 2010, 2017). In their studies, Dotov and colleagues study tool embodiment by explicitly addressing the Heideggerian notions of readiness-to-hand, roughly equivalent to embodiment, and presence-at-hand, roughly equivalent to disembodiment. In

other words, when a tool is embodied in a cognitive system, it is ready-to-hand and, when a tool is not embodied in a cognitive system, it is present-at-hand. For example, when a person rides a bike and it works perfectly, the bike is ready-to-hand, meaning that the bike becomes embodied and the whole person-bike system exhibits the fractal features of complex, self-organized dynamical systems. When the bike stops working for some mechanical reason, for example, it becomes present-at-hand, meaning that the bike becomes disembodied and the wholeperson bike system is not a unitary system anymore and does not exhibit the fractal features of complex, self-organized dynamical systems. The work of Dotov and colleagues shows that these transitions and their fractal signatures are common in our engagement with tools and take them to be examples of tool embodiment. In sum, the traditional view takes embodiment to indicate an alteration in one or more body representations; radical embodiment takes it to indicate the achievement of a particular kind of dynamic coupling.

What has not to our knowledge been studied, but should be, is the relationship between these kinds of signatures of synergistic coupling between bodies and tools, and the feeling of ownership and embodiment of, for instance, a prosthetic limb. Such work could inform prosthetic design, as it may be that the ease with which a prosthetic can be coupled to the brain-body-environment system is a better predictor of the likelihood of prosthetic embodiment [an important aspect of positive experiences with prosthetics (Anderson, 2018)] than are aesthetic design dimensions. Such work could be carried out in VR environments, which would give fine control over the dynamics of the prosthetic, allowing exploration of the different ways a prosthetic might ease or disrupt coupling, and the effect of such manipulations on experience.

### CONCLUSION

The study of object and tool embodiment is already a mature field in which different frameworks, methodologies, and experimental paradigms work in parallel, and sometimes together, to understand the ways and conditions under which we take objects and tools to be part of our bodies. In this paper, we have seen that such study is pursued both from a representational and from a radical embodied point of view. We have also seen that, despite the apparent incompatibilities between the two points of view, notions such as body schema may offer an important connection between them. In this sense, the difference between the representational and the radical embodied notions of object and tool embodiment is mostly methodological. While the representational point of view uses methods like personal reports, skin conductance measurements, etc., the radial embodied point of view uses the tools provided by complexity science to see whether there are tacit, non-conscious engagements with object or tools that can be labeled as embodied.

The theoretical compatibility between both points of view opens an interesting path of research in which semantic debates could be set aside (e.g., the debate on whether the body schema is a proper representation or not) and experimental work from both frameworks could inform each other to achieve a better understanding of the phenomenon and could be combined to improve future research. For example, in both tool and rubber hand studies, it is unclear whether the multisensory effects of tool embodiment can be ascribed to a change in the body schema, in the processing of the peri-personal space, and/or a shift in attention.

It has been shown that using a tool quickly modifies the perception of the peri-personal space, the kinematics of subsequent bodily movements, and the perceived size of the limb. However, this might suggest a faster mechanism, such as shift or projection of spatial attention toward the tool tip rather than a likely slower mechanism, such as tool embodiment (Holmes et al., 2007). The methods used in the radical embodied approach to embodying object and tools (e.g., Dotov et al., 2010) could help to further clarify this issue as they provide a neat way to better understand whether the body schema is involved or not in a given task—by discriminating between readiness and unreadiness-to-hand, for example.

Another field for future research has to do with the ability to use tools and have them become a part of one's own body while interacting with them in the virtual space. It is difficult to know when a virtual tool has become embodied. Some researchers suggest that instead of questionnaires that we evaluate these technologies by using change in attention as a measure of how people interact with virtual environments (Dotov et al., 2010). Again, the methodological approach of radical embodiment could be used to gain better understanding of this issue.

There are also some non-virtual reality tasks that could be used in virtual reality to study the flexibility of the body representations and the somatosensory cortex, such as with the elongation of a limb (Kilteni et al., 2012), and the brain areas responsible for object and tool embodiment. Further work could be done on the amount of cortical reorganization under various lengths of the virtual limbs for object embodiment and tools for tool embodiment. More research could be on how fast or gradual the peri-personal space reduces after tool use and how fast the cortical reorganization and its reduction is. Future studies could also expand on our understanding of clinical populations and, for instance, their amputated arms or legs by way of objective measures, such as fMRI, in both non-virtual reality and virtual reality rubber hand and full body paradigms. Also, future studies are needed to better understand the use of a robotic touch interface (Marasco et al., 2011) and its effects on the sense of ownership over a prosthetic limb in both non-virtual reality and virtual reality paradigms.

Understanding the underlying brain mechanics is crucial for a full understanding of how object and tool embodiment can work with body representations. Further work on studying experienced and inexperienced tool users with participants in ecologically valid non-virtual and virtual settings will expand the tool use literature. Additionally, further methods to study the body image during tool use should be explored. Continuing to research on tool and object embodiment may allow for the development of more effective prosthetics for missing limbs. In any case, all these are open questions that can be addressed from the framework based on body representations or from a different one.

### AUTHOR CONTRIBUTIONS

fnins-13-01332 December 12, 2019 Time: 14:58 # 14

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

### REFERENCES


### FUNDING

This research was supported in part by a Canada Research Chair award to MA, grant number SSHRC 950-231929.

excitability modulation during pain observation. J. Cogn. Neurosci. 28, 1760– 1771. doi: 10.1162/jocn\_a\_01003


IEEE Trans. Neural. Syst. Rehabil. Eng. 23, 450–457. doi: 10.1109/tnsre.2014. 2337952


congruency task. Exp. Brain Res. 183, 465–476. doi: 10.1007/s00221-007- 1060-7


spatial attention. Hum. Brain Mapp. 34, 356–366. doi: 10.1002/hbm. 21433


Menary, R. (2010). The Extended Mind. Cambridge, MA: The MIT Press.



**Conflict of Interest:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Schettler, Raja and Anderson. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Tool-Use Training Induces Changes of the Body Schema in the Limb Without Using Tool

### Yu Sun and Rixin Tang\*

Department of Psychology, School of Social and Behavioral Sciences, Nanjing University, Nanjing, China

#### Edited by:

Mariella Pazzaglia, Sapienza University of Rome, Italy

#### Reviewed by:

Luke Edward Miller, INSERM U1028 Centre de Recherche en Neurosciences de Lyon, France Marco D'Alonzo, Campus Bio-Medico University, Italy Jared Medina, University of Delaware, United States

> \*Correspondence: Rixin Tang trxtrx518@nju.edu.cn

#### Specialty section:

This article was submitted to Sensory Neuroscience, a section of the journal Frontiers in Human Neuroscience

> Received: 29 June 2019 Accepted: 09 December 2019 Published: 20 December 2019

#### Citation:

Sun Y and Tang R (2019) Tool-Use Training Induces Changes of the Body Schema in the Limb Without Using Tool. Front. Hum. Neurosci. 13:454. doi: 10.3389/fnhum.2019.00454 Previous studies have shown that tool use affects the plasticity of the body schema. In other words, people will perceive the tool as a part of their body, and thus feel like they have "longer limbs" after using tools. However, it is unclear whether tool embodiment could spread to a limb that is not using the tool, and whether other limbs could utilize the proprioception of a limb. In Experiment 1, blindfolded participants were asked to search with a cane (Condition 1) or to walk with a cane (Condition 2). The results in Condition 1 illustrated that the tactile distance perception on the forearm was lengthened after tool use, while other body parts did not significantly change. In Condition 2, the tactile distance perception on the hand and forearm extended significantly after using tools. Additionally, tool-use training even induced an increased perception of the calf that was not using the tool. Possible interference from the difference between walking and standing was excluded in Experiment 2. These results demonstrate that the proprioception information of one limb could be exploited by another limb to extend the body schema even though that limb was not using a tool. It was also observed that the effect of direction was task-dependent in the tactile perception task.

Keywords: tool use, body schema, tool embodiment, limb-specific hypothesis, proprioception, plasticity

## INTRODUCTION

Tool use contributes to human survival by allowing humans to reach inaccessible spaces and protect their bodies from harm. Many studies have revealed that participants tend to perceive tools as part of their own bodies after tool use, their perception of their body parts is extended, and their body schema is changed, which is also known as the phenomenon of tool embodiment (e.g., Iriki et al., 1996; Cardinali et al., 2009, 2011; Sposito et al., 2012). Humans use tools more efficiently and accurately when it is incorporated into the body schema, allowing the brain to control it just like other body parts (Cardinali et al., 2016a). Perhaps tool embodiment is more crucial for individuals who are more dependent on the tools. For example, amputees could use prostheses to perform movements with greater flexibility, and blind people could use canes to explore the spaces around them more efficiently.

Previous studies emphasized training a specific limb with the tool then testing the body schema of the same limb. Nonetheless, it is still unclear whether the phenomenon of tool embodiment is based on the specific limb or the general procedure for all limbs. Some studies have discovered that using tools with the hand only changed the body schema of the forearm, but did not change the body schema of the foot (Jovanov et al., 2015) or the cheek (Miller et al., 2017a). Additionally, Miller et al. (2014) found that only the limb that is similar to the morphology of a tool could be modified by tool use. However, in previous studies, the participants used the tool with their hands, and the tool accordingly gave functional benefits to the hands by expanding the space that the hands can reach. Therefore, it was reasonable that the body schema of other limbs failed to change. It is still unclear whether the body schema of limbs that do not use a tool would change when tools give functional benefits to that limb. If the body schema of the limb not using a tool changed, then the tool embodiment is general to all limbs. Otherwise, tool embodiment is limb-specific.

Sensory input from multiple modalities–such as vision, proprioception, and tactile sensation—played an important role in the incorporation of tools into the body schema (Miller et al., 2015; Cardinali et al., 2016b; Martel et al., 2019). The proprioception provides information on perceived position and movement of limbs and the body without visual feedback (Gilman, 2002). It was considered that the proprioception was necessary (Cardinali et al., 2016b) and sufficient (de Vignemont et al., 2005; Martel et al., 2019) to induce the changes of the body schema. Furthermore, the proprioceptive input may be dominant to the body representation even compared to vision (Shenton et al., 2004). Several neurophysiology and lesion studies have also suggested that the body schema is predominantly based on proprioception (Head and Holmes, 1911; Paillard, 1999; Gallagher, 2005). It is clear that visual information could be used to control all limbs, but proprioception is fixed to the specific limb. Blindfolded and blind participants could adapt to the new force environment on the basis of proprioception (DiZio and Lackner, 2000). The proprioceptive information from one limb arrives at the primary somatosensory cortex (SI) from the thalamus (Kaas et al., 1979). It was then possible for the proprioceptive information of one limb stored in the SI to be used by other limbs, so as to drive dynamic adaptation when walking blindly. It should be noted that the visual feedback was usually available in most of the tool studies previously conducted, in which the proprioception information did not make a significant difference in the tasks. However, when the visual feedback was absent, the limb not using the tool became dependent upon the proprioceptive information of the limb using the tool to program its movement. It is interesting to study whether the proprioception of the limb using the tool could be exploited by other limbs not using the tool, as well as whether the proprioceptive information related to tool could induce changes in the body schema of the limb not using the tool.

In this study, two experiments were conducted to investigate whether tool embodiment could spread to the limb not using the tool and whether proprioception is general to all limbs or specific to one limb. In Experiment 1, blindfolded participants were instructed to search for the target object with a cane (Condition 1) or to walk with a cane (Condition 2). They performed a tactile distance perception task before and after the training to investigate if the body schema of their hands, forearms, feet, or calves had changed. It was expected that limbs that experienced a change in body schema would be different between Condition 1 and Condition 2, even though the same limb used the same tools in both experiments. Possible interference from the difference between walking and standing was studied in Experiment 2.

### EXPERIMENT 1

### Participants

Fifty-six participants (31 females; mean age ± SD: 21.71 ± 2.10; ranging from 18 to 28 years of age) took part in Condition 1 and Condition 2. All participants were right-handed and had normal or corrected normal vision. They all received monetary compensation for their participation. All participants gave written and informed consent to participate in the study, which was approved by the Ethical Committee of the Department of Psychology, Nanjing University. These participants also attended Experiment 2 on different days.

### Apparatus and Procedures

As shown in **Figure 1A**, all participants were required to perform two experimental conditions and both experimental conditions were composed of three phases: a pre- and post-tool-use session (18 trials × 4 blocks each), during which participants were instructed to perform a tactile distance perception task, separated by a tool-use training session. The two experimental conditions differed only in the tool-use training session.

The specific experimental procedures were as follows. At the beginning of the experiment, participants would be instructed to put on a blindfold, and this condition of having no vision would last the entire experiment. Tactile distance perception task would then be performed on the hands, forearms, feet, and calves of the participants. During this task, participants comfortably sat on a chair and placed their right hand on the table, while also placing their right leg on the other chair. Every time before starting the tactile distance perception task, participants would practice at least five times to ensure that they could complete the task. This was interspersed with a tool-use training task between two tactile distance perception tasks. During the training task, participants were asked to search for the object with the cane (Condition 1) and walk with the cane (Condition 2) on two different days.

### Tactile Distance Perception Task

The tactile distance perception task is an implicit and sensitive task for measuring the plasticity of the body schema (Taylor-Clarke et al., 2004; de Vignemont et al., 2005; Longo and Haggard, 2011; Tajadura-Jiménez et al., 2012). In this task from previous studies, participants were instructed to verbally report which body part (target body part, e.g., forearm; reference body part, e.g., forehead) that was touched was perceived longer (Canzoneri et al., 2013; Miller et al., 2014). In the version of

the tactile distance perception task, participants made verbal estimations of the distance between two tactile points manually applied to the target body part (Longo and Sadibolova, 2013; Miller et al., 2017a). In the present study, the tactile distance perception task was adapted from Longo and Sadibolova (2013) and Miller et al. (2017a). Tactile points were administered manually and longitudinally (from wrist to knuckles) to four target body parts (the dorsal surface of the right hand, right forearm, right foot, and right calf) with a stainless steel digital caliper in separate blocks. There were three types of tactile distances (separated by 20, 30, or 40 mm), and every tactile distance was administered six times for a total of 18 trials in every block. The order was random. The tactile stimuli administered to the body parts of the participants lasted for approximately 1 s, and then the participants verbally reported the estimated distance in millimeters. Participants were blindfolded throughout the procedure. There was no limit to the time that participants made their verbal reports.

### Tool-Use Training

In the condition of searching with a cane (see **Figure 1B**), the tool-use training was adapted from Serino et al. (2007) and Canzoneri et al. (2013). The tool was a 120 cm aluminum alloy cane with a diameter of 13 mm. The blindfolded participants were required to find a 4 × 4 × 8 cm wooden target object randomly placed in one of 30 different locations on the table, and to knock the front, top left, and right sides of the target with the cane. There were three possible longitudinal distances from the body (100 cm, 130 cm, and 160 cm), and 10 transversal positions covering a space ranging from 0 to 90 cm. All participants stood at a fixed starting position and used the cane with their right hands. At the beginning of the training, the experimenter placed the target object randomly in one of 30 possible locations on the table, avoiding making any sounds that could give the participants a hint about the exact location of the object. Participants were then instructed to explore the space in front of them with the cane, imitating the movements that blind people use with a cane until they found the object. Once the participants found the target, they knocked it over. Then the experimenter removed the first object and placed another one on the table. There were no time constraints for how quickly the participants had to find the target object.

In the condition of walking with the cane (see **Figure 1C**), the tool-use training required blindfolded participants to walk with the cane in a similar way that a blind person would navigate himself or herself. In the 20 min of training, participants needed to explore the space around them and find their way with the help of the tool. The tool was the same as that used in Condition 1. In this experimental condition, the experimenter would accompany participants in case of accidents but avoided making any sounds that could interfere with the participants. To reduce auditory interference, rubber was attached to the bottom of the cane to lessen the sound. Additionally, participants were asked to use the cane in a similar way in the two tasks to reduce the effect of the sound caused by different usages of the tool.

### Results

The normality of data was checked and most of the data conformed to normal distribution. For the data that deviated from normality, either a repeated ANOVA was conducted after replacing the outliers with means or a non-parametric test was done when the outliers did not cause the deviation from normality. As for the data satisfying the normal distribution, three 2 (Condition: searching with cane, walking with cane) × 2 (Phase: pre, post) × 3 (Tactile distance: 20 mm, 30 mm, 40 mm) repeated measure ANOVAs were separately conducted for the verbal estimated tactile distances of three limbs. The current study concerned the different modulations under different tool-use training tasks, where the interactions between factors Condition and Phase was crucial to the goals of the present study.

For the analysis of hands (see **Figure 2A**), two non-parametric tests between pre- and post-tool use were conducted because Shapiro–Wilk tests signaled that the data deviated from normality. The non-parametric test of walking with cane condition showed that there was a significantly increased perception after using tools (p = 0.014). Meanwhile, a marginally significant increase between pre- and post-tool use in the condition of searching with a cane (p = 0.066) was found. Additionally, significant differences among three tactile distances in both conditions were found (pmax < 0.001), demonstrating that participants indeed increased their estimations as the actual distance increased.

The repeated ANOVA of arms (see **Figure 2B**) showed significant main effects of Phase (F(1,55) = 14.135, p < 0.001, η<sup>p</sup> = 0.204) and Tactile distance (F(2,110) = 500.829, p < 0.001, η<sup>p</sup> = 0.901). The main effect of Phase showed an increased perception after tool use in both conditions. No other main effects or interactions were found (F(1,55)max = 2.518, pmin = 0.118, η<sup>p</sup> = 0.044).

For the repeated ANOVA of feet (see **Figure 2C**), a significant main effect of Tactile distance (F(2,110) = 970.033, p < 0.001, η<sup>p</sup> = 0.946) was found. There were no other main effects or interactions (F(1,55)max = 2.308, pmin = 0.134, η<sup>p</sup> = 0.040).

Finally, for the analysis of calves, a repeated ANOVA was conducted. The results (see **Figure 2D**) showed that there was a significant interaction between factors Condition and Phase (F(1,55) = 7.223, p = 0.010, η<sup>p</sup> = 0.116). Two significant main effects were found for Tactile distance (F(2,110) = 459.361, p < 0.001, η<sup>p</sup> = 0.893), and Phase (F(1,55) = 13.729, p < 0.001, η<sup>p</sup> = 0.200). Simple effect on the calf showed that perceived tactile

FIGURE 2 | Modulations of perceived distance on different limbs in Condition 1 and Condition 2. (A) The perceived distance of hand extended significantly in the task of walking with a cane, but it remained unaltered in the searching with cane condition. (B) The estimated tactile distance of the forearm significantly increased after tool use training of searching with cane and walking with cane. (C) The perceived distance of the foot in both conditions. (D) The perception of the calf extended significantly after walking with a cane, but it remained unchanged in the task of searching with cane. <sup>∗</sup>p < 0.05, ∗∗p < 0.01.

distance of post tool use was longer than that of pre tool use in the condition of walking with cane (p < 0.001), but it did not change after tool use in the condition of searching with cane (p = 0.468). No other main effects or interactions were observed (F(1,55)max = 2.849, pmin = 0.097, η<sup>p</sup> = 0.049).

### EXPERIMENT 2

A previous study has suggested that the peripersonal space expands in the case of walking rather than standing still (Noel et al., 2015). The body schema expansion of calf found in Experiment 1 might result from the walking movement. In Experiment 2, whether the walking movement would affect the body schema of the calf was particularly emphasized.

### Methods

The participants were the same as that of Experiment 1. The apparatus and procedures were consistent with Condition 2 except for the walking phase. In the walking phase of Experiment 2, participants were instructed to walk without the cane.

### Results

The tests of normality showed that all data including difference scores was normally distributed after replacing an outlier with mean. Two 2 (Phase: pre, post) × 3 (Tactile distance: 20 mm, 30 mm, 40 mm) repeated measure ANOVAs were separately performed with verbal distance estimations on the foot and calf (see **Figure 3**). The repeated ANOVA of foot showed a significant main effect of Tactile distance (F(2,110) = 493.170, p < 0.001, η<sup>p</sup> = 0.900). No other main effect (F(1,55) = 3.289, p = 0.075, η<sup>p</sup> = 0.056) or interaction (F(2,110) = 0.571, p = 0.567, η<sup>p</sup> = 0.010) was found. The repeated ANOVA of calf implied that there were significant main effects of Phase (F(1,55) = 9.023, p = 0.004, η<sup>p</sup> = 0.141) and Tactile distance (F(2,110) = 269.746, p < 0.001, η<sup>p</sup> = 0.831). The main effect of Phase showed a decreased perception after walking without cane. There were no interaction (F(2,110) = 0.180, p = 0.836, η<sup>p</sup> = 0.003).

The effects of tool use in three different conditions on the calf and foot were calculated with difference scores (the estimated distance of post tool use—the estimated distance of pre tool use). Two 3 (Condition: searching with a cane, walking with cane, walking without a cane) × 3 (Tactile distance: 20 mm, 30 mm, 40 mm) repeated measures ANOVAs were separately conducted for the foot and calf (see **Figure 4**). The repeated ANOVA of the foot suggested that there was a significant main effect of Condition (F(2,110) = 4.440, p = 0.014, η<sup>p</sup> = 0.075). The post hoc test showed that the estimated distance of walking with cane condition was significantly longer than that of walking without cane (p = 0.003), but there were no significant differences between searching with cane and walking with cane (p = 0.185), or between searching with cane and walking without cane (p = 0.119). There were no other main effect (F(2,110) = 1.190, p = 0.308, η<sup>p</sup> = 0.021) or interaction (F(4,220) = 0.860, pmin = 0.489, η<sup>p</sup> = 0.015) on the foot. Additionally, a significant main effect of Condition was observed on the calf (F(2,110) = 16.658, p < 0.001, η<sup>p</sup> = 0.232). Specifically, the post hoc test showed that there were significant differences between walking with a cane and searching with a cane (p = 0.004), significant differences between walking with cane and walking without cane (p < 0.001), and significant differences between searching with cane and walking without cane (p = 0.009). No other significant main effect (F(2,110) = 0.584, p = 0.559, η<sup>p</sup> = 0.011) or interaction (F(4,220) = 1.151, p = 0.331, η<sup>p</sup> = 0.021) on the calf were found.

### DISCUSSION

In this study, the change in body schema of the limb not using the tool that received functional benefits from the tool was

investigated, and whether proprioception was general to all limbs or specific to one limb was also analyzed. Two experiments were conducted to answer these questions. Blindfolded participants were instructed to search for the target object with a cane (Condition 1) or walk with a cane (Condition 2). In Condition 1, it was revealed that the perceived tactile distance applied to the forearm was significantly extended after tool use, and the perception of the hand showed a marginally significant increase after tool use, while the body schema of other limbs remained unaltered. In Condition 2, the results showed that the tactile distance perception on the hand and forearm extended significantly after using tools. Additionally, tool-use training even induced an increased perception of the calf that was not using the tool. Furthermore, in Experiment 2, the potential interference of walking was excluded by instructing participants to walk blindfolded without the cane.

As was predicted, the body schema of the forearm changed after using the cane to touch the target object and to walk, which was consistent with previous studies (e.g., Iriki et al., 1996; Cardinali et al., 2009, 2011; Sposito et al., 2012). Even though the visual information was unavailable, it was still revealed that participants exhibited extended forearms after tool use in Experiment 1, indicating that only proprioception could change body schema (de Vignemont et al., 2005; Martel et al., 2019). Additionally, the body schema of the hand showed a significantly increased perception in Condition 2 and a marginally significant increased perception in Condition 1, which was inconsistent with the studies conducted by Miller et al. (2014). This was perhaps due to differences in the experimental tasks. In the task of Miller et al. (2014), participants squeezed a vertical handle to control pincers at the tooltip. The movement of the hand either curling up or squeezing potentially caused the shrinkage of perception on the hand. It may also cause the separation of the hand and forearm due to their different methods of movement. In the present study, it is likely that the dorsal of the hand and forearm were a whole because of the same movement used between them. Moreover, the morphology of the dorsal of the hand was similar to that of the forearm and tool to some extent. Therefore, the perception of the hands produced the same changes as that of the forearms.

In Condition 2, when the cane was used by the hand to assist with walking, the body schema of the hand and forearm changed. Importantly, the body schema of the calf that was not using the tool experienced a change. The different modulations on the calves between Condition 1 and Condition 2 suggest that different functions of tool use caused different changes in the body schema of limbs. This result is consistent with previous studies emphasizing the functionality of tools. For example, the functional length of the tool was more important to body representation than physical length (Farnè et al., 2005; Sposito et al., 2012). Additionally, Reed et al. (2010) found that the acceleration of target stimulus recognition only occurred at the functional side of the tool and that there was no effect on the other side.

In the present study, the functionality of tools even altered the body schema of the limb not using the tool. One potential explanation for this is that blindfolded participants in the walking with cane condition paid more attention to the lower limbs than that of searching with cane condition. The tactile perception then became larger after training. However, it is unlikely to happen because the visual attention is usually prioritized for the space near the distal of limbs (Makin et al., 2007; Brozzoli et al., 2011; Gentile et al., 2011) or the tip of tools (Maravita et al., 2001; Farnè et al., 2005; Kao and Goodale, 2009; Reed et al., 2010), causing the faster detection of visual targets. Therefore, it should have been determined that the body schema of the foot in Condition 2 was affected after tool use. It should be noted that the body schema of the foot was not affected in the present study. A possible reason for this is that the activation of leg muscles is fundamental to the control of human walking (Franz and Kram, 2012) rather than the foot muscles. Moreover, the vertical positional relationship between foot and calf while walking perhaps induced that the morphology of cane was more similar to the calf. Thus, the calf-shaped cane only changed the body schema of the calf, but not that of the foot (Miller et al., 2014; Cardinali et al., 2016a).

Another possibility is that the sensorimotor representation of the calf was activated when participants used the proprioceptive information obtained from the tooltip to program its movement (see Miller et al., 2017b). The triangle formed among the cane, the calf, and the ground appeared to be an extension of the reachable area of the calf. In contrast, blindfolded participants would not dare to walk without the cane due to the fear of falling, thus showing a slower walking speed (Hallemans et al., 2010). The mental state of not daring to walk possibly induced a perception that the reachable area of the calf was shrinking, further leading to a decreased perception of tactile distance applied to the calf. The proprioception information obtained from the tooltip includes the perceived length of the tools (Solomon et al., 1989), the size of object that came into contact with the tool (Turvey et al., 1998), as well as the relation between self and the surrounding environment (Harrison and Turvey, 2010; Turvey and Carello, 2011). In the present study, proprioception information of tool use was exploited to manage walking blindly, which is similar with the study conducted by Harrison and Turvey (2010), in which the blindfolded participants could have place learning by walking, stepping, and cane probing. Moreover, motor programming might play a role in the plasticity of body schema since the motor imagery (Baccarini et al., 2014) and visual illusion (Miller et al., 2017b) of tool use can change the tactile perception on the stationary arms. This also suggests that the proprioception fixed to one limb can be used by another limb and change their body schema, reflecting the common representation of proprioception in the brain.

It should also be noted that an increased perception in the tactile distance within the current study was longitudinal and not transversal. The increased perception in the longitudinal orientation was interpreted as an increase in the represented size of the body part (de Vignemont et al., 2005; Tajadura-Jiménez et al., 2012). However, this effect of direction was different from the studies conducted by Canzoneri et al. (2013) and Miller et al. (2014), which were transversal. Romano et al. (2019) suggested that shoulder- or wrist- training induced different changes in body representation. The training of previous studies involved arm retraction (Canzoneri et al., 2013) or bending (Miller et al., 2014), which depended more on the proximal part rather than the distal part. In the present study, the arms of the participants stretched to use the cane to search or navigate in the tasks, which were more dependent on the distal part. This indicates the current effect is task-dependent.

Overall, the current study shows that different goals of tool use caused the different changes of the body schema. Importantly, tool use could induce the body schema changes of the limb

### REFERENCES


even though that limb was not using a tool. Furthermore, the direct effect of tactile distance perception tasks might be taskdependent. Finally, the present study also tested and verified that tool use could cause the changes of body schema on the sole basis of proprioception.

### DATA AVAILABILITY STATEMENT

The datasets generated for this study are available on request to the corresponding author.

### ETHICS STATEMENT

This study was carried out in accordance with the recommendations of the Ethical Committee, Nanjing University. All participants gave written and informed consent to participate in the study. The protocol was approved by the Ethical Committee of the Department of Psychology, Nanjing University.

### AUTHOR CONTRIBUTIONS

RT and YS designed the experiment, discussed the results and wrote the article. YS conducted the experiment and analyzed the data under the supervision of RT.

### FUNDING

This study was supported by grants from the National Natural Science Foundation of China (Grant #31571131) to RT.


**Conflict of Interest**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Sun and Tang. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# A Role for the Action Observation Network in Apraxia After Stroke

Gloria Pizzamiglio1,2† , Zuo Zhang<sup>3</sup>† , James Kolasinski1,4, Jane M. Riddoch<sup>5</sup> , Richard E. Passingham<sup>5</sup> , Dante Mantini6,7 and Elisabeth Rounis<sup>1</sup> \*

<sup>1</sup> Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, United Kingdom, <sup>2</sup> Wellcome Centre for Human Neuroimaging, Institute of Neurology, University College London, London, United Kingdom, <sup>3</sup> Social, Genetic and Developmental Psychiatry Centre, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, United Kingdom, <sup>4</sup> Cardiff University Brain Research Imaging Centre, Cardiff University, Cardiff, United Kingdom, <sup>5</sup> Department of Experimental Psychology, University of Oxford, Oxford, United Kingdom, <sup>6</sup> Research Centre for Motor Control and Neuroplasticity, KU Leuven, Leuven, Belgium, <sup>7</sup> Brain Imaging and Neural Dynamics Research Group, IRCCS San Camillo Hospital, Venice, Italy

#### Edited by:

Mariella Pazzaglia, Sapienza University of Rome, Italy

#### Reviewed by:

Jennifer Randerath, Universität Konstanz, Germany François Osiurak, Lumière University Lyon 2, France

#### \*Correspondence:

Elisabeth Rounis elisabeth.rounis@ndcn.ox.ac.uk †These authors have contributed equally to this work

#### Specialty section:

This article was submitted to Motor Neuroscience, a section of the journal Frontiers in Human Neuroscience

Received: 29 July 2019 Accepted: 14 November 2019 Published: 20 December 2019

#### Citation:

Pizzamiglio G, Zhang Z, Kolasinski J, Riddoch JM, Passingham RE, Mantini D and Rounis E (2019) A Role for the Action Observation Network in Apraxia After Stroke. Front. Hum. Neurosci. 13:422. doi: 10.3389/fnhum.2019.00422 Limb apraxia is a syndrome often observed after stroke that affects the ability to perform skilled actions despite intact elementary motor and sensory systems. In a large cohort of unselected stroke patients with lesions to the left, right, and bilateral hemispheres, we used voxel-based lesion-symptom mapping (VLSM) on clinical CT head images to identify the neuroanatomical correlates of the impairment of performance in three tasks investigating praxis skills in patient populations. These included a meaningless gesture imitation task, a gesture production task involving pantomiming transitive and intransitive gestures, and a gesture recognition task involving recognition of these same categories of gestures. Neocortical lesions associated with poor performance in these tasks were all in the left hemisphere. They involved the pre-striate and medial temporal cortices, the superior temporal sulcus, inferior parietal area PGi, the superior longitudinal fasciculus underlying the primary motor cortex, and the uncinate fasciculus, subserving connections between temporal and frontal regions. No significant lesions were identified when language deficits, as indicated via a picture naming task, were controlled for. The implication of the superior temporal sulcus and the anatomically connected prestriate and inferior parietal regions challenges traditional models of the disorder. The network identified has been implicated in studies of action observation, which might share cognitive functions sub-serving praxis and language skills.

Keywords: apraxia, voxel-based lesion-symptom mapping, gesture production, gesture recognition, meaningless gesture imitation, superior temporal sulcus, action observation

### INTRODUCTION

Limb apraxia refers to a range of deficits in skilled action that are not consequences of motor weakness, sensory impairment, or lack of comprehension or coordination (Heilman and Rothi, 2003). Patients with the disorder have difficulties performing skilled actions, such as shaving or making a cup of tea. In stroke patients, limb apraxia can be demonstrated by impairments both when they use the affected and the unaffected hand. The syndrome is increasingly recognized as a predictor of poor functional recovery after a stroke that affects patients' activities of daily living, with greater rates of patients with this disorder being dependent or ending up in nursing homes

(Donkervoort et al., 2006; Bickerton et al., 2012). In addition to the motor impairments caused by this disorder, apraxia may worsen other cognitive impairments, such as aphasia, by compromising patients' ability to communicate through gestures.

Traditional theories of the disorder have categorized praxis deficits according to errors made by patients in tasks involving (1) Imitation of both meaningless and meaningful gestures (e.g., asking a patient to copy meaningless hand or finger gestures or else to copy a familiar gesture, such as saluting), (2) Pantomiming of meaningful gestures or tool use (either intransitive, e.g., "show me how you stop traffic" or transitive gestures, e.g., "show me how you would brush your teeth, using a toothbrush in your hand"), and (3) Actual tool use (e.g., asking the patient to demonstrate the use of a torch) or in the performance of complex sequences of actions (e.g., asking the patient to make tea) (Leiguarda and Marsden, 2000; Donkervoort et al., 2006; Dovern et al., 2011). Whereas pantomime and object-use tasks pertain to deficits implicating conceptual (semantic) planning for meaningful gestures, imitation of meaningless gestures tests the implementation or production systems (Cubelli et al., 2000; Leiguarda and Marsden, 2000; Heilman and Rothi, 2003).

Most screening batteries for apraxia involve the use of pantomiming and imitation of meaningless hand gestures, because these tasks are particularly sensitive for detecting praxis deficits (Niessen et al., 2014; Buchmann and Randerath, 2017). This has formed the basis for their inclusion for testing praxis in the Birmingham Cognitive Screen (BCoS) (Bickerton et al., 2012; Humphreys et al., 2012).

Lesion-mapping studies investigating limb apraxia agree that left hemisphere damage plays a role in this disorder, implicating the fronto-temporo-parietal network (Mengotti et al., 2013; Buxbaum et al., 2014; Hoeren et al., 2014; Goldenberg and Randerath, 2015). They report a significant role for the inferior parietal lobe in tool-use pantomime and in imitation of meaningless gestures (Buxbaum et al., 2014; Hoeren et al., 2014; Dressing et al., 2018). However, there is no clear dichotomy between the two, as the neural correlates of pantomime are widespread (Daprati and Sirigu, 2006; Goldenberg et al., 2007; Price et al., 2010; Manuel et al., 2013; Goldenberg and Randerath, 2015).

Several factors could account for these findings. Lesionmapping studies of apraxia have been limited by methodological issues, notably in the analysis methods used, and variability in the tasks used to study the disorder. There have been inconsistencies in the screening tools used to assess various subtypes of the disorder (Goldenberg, 2017). The lesion-symptom mapping methods employed have included the use of manual delineation of abnormal brain tissue, which can produce inconsistencies across operators (Gillebert et al., 2014). The use of dichotomized data, categorizing apraxia as being present or absent instead of including continuous scores, had meant that initial studies incorporated small numbers of patients.

The use of voxel-based lesion-symptom mapping (VLSM) has enabled the inclusion of much larger and unselected cohorts of patients in more recent studies (such as in Manuel et al., 2013; Buxbaum et al., 2014; Hoeren et al., 2014). The use of continuous, rather than dichotomized, apraxia scores has also allowed for a more fine-grained description of the neural correlates of praxis deficits by improving power in these analyses (Cohen, 1983). The variability caused by the inclusion of patients at various stages of recovery after stroke – from early subacute to chronic stages – is being mitigated by studying more homogenous cohorts of patients (Hoeren et al., 2014; Weiss et al., 2016).

An important factor that has been overlooked in several lesion-mapping studies of the disorder in the past has been the relationship of apraxia with other cognitive disorders, in particular, aphasia (Goldenberg and Randerath, 2015). Several studies report the co-occurrence of the two disorders, with little evidence of the presence of apraxia with no aphasia in right-handed patients (Selnes et al., 1991; Papagno et al., 1993; Weiss et al., 2016). This has become increasingly relevant in light of recent studies that indicate that pantomime of tool use, which is widely used in diagnosing this disorder (Buchmann and Randerath, 2017), might have a communicative role (Dressing et al., 2018; Finkel et al., 2018). A lesion-mapping study investigating apraxia and aphasia in left-hemisphere stroke patients distinguished between a network involving frontal, insular, inferior parietal, and superior temporal areas supporting language functions and lesions involving the sensorimotor, premotor, and parietal cortices associated with praxis tasks, with the inferior premotor area (BA44) co-localizing for both (Weiss et al., 2016). Another lesion-mapping study by Finkel et al. (2018) identified two putative networks sub-serving communication and motor functions when stroke patients pantomimed tooluse actions.

In this study, we make use of a large database of stroke patients that included both neuropsychological measures of praxis and imaging data, available from the patients' clinical CT scans on admission. Previous lesion-symptom mapping studies of the disorder have used MR imaging because of the wide availability of analytic methods for lesion delineation in this imaging modality (Seghier et al., 2008), which can then be used to identify correlations between lesion and behavioral deficits (VLSM – Sperber and Karnath, 2018). Most of these studies investigated the disorder at the chronic stage (Manuel et al., 2013; Buxbaum et al., 2014), and some in the earlier stages after stroke (Hoeren et al., 2014). The advantage of investigating patients in the acute and subacute stages is that lesions directly relating to the stroke can be identified before changes such as atrophy (caused by post-stroke degeneration) take place (Lindberg et al., 2007). This is important when using automated lesion delineation techniques, as atrophy may affect the delineation of lesions.

Our aim was to investigate the neural correlates of deficits in praxis in a large cohort of subacute stroke patients who took part in the BCoS (Humphreys et al., 2012). The validity of the screening tasks for apraxia administered in the BCoS has been confirmed previously (Bickerton et al., 2012).

We used an automated CT processing toolbox, developed in our laboratory (fully described in Gillebert et al., 2014), which enabled lesion delineation for voxel-based lesion-mapping analyses to be performed. We conducted large-scale retrospective VLSM analyses (Bates et al., 2003) on a group of subacute unselected brain-damaged patients using continuous rather than descriptive cognitive scores of praxis from the BCoS (Humphreys et al., 2012).

### MATERIALS AND METHODS

fnhum-13-00422 December 18, 2019 Time: 16:0 # 3

### Patients

The patients were recruited into the Birmingham Cognitive Screen project (BCoS), a multi-center clinical study investigating cognitive impairments after subacute stroke (patients were recruited from several stroke units across the West Midlands area of the United Kingdom). This study was approved by the National Research Ethics Service (NRES): Essex 1 Research Ethics Committee (REC) and local NHS trusts. Patients were included in the study if: (1) they were within 3 months of a confirmed first stroke and medically stable; (2) they were judged by the clinical team to be able to concentrate for at least 30 min to enable cognitive testing; (3) they had sufficient command of English to follow instructions; and (4) they were able to provide written informed consent to participate in the study (Bickerton et al., 2012). Hence, all the patients in this study had provided informed consent for the use of their neuropsychological and imaging data in the research.

The BCoS comprises the assessments of apraxia detailed below. Additionally, we included assessments of other cognitive domains, namely: attention, memory, language, and number processing. These data were supplemented by a CT head scan and demographic information, which was obtained from the patients' clinical files.

Patients were excluded if they had no lesion visible on CT scan or had scans that were not adequate for further analyses (e.g., those not fulfilling the imaging criteria set out below). They were also excluded if they had ventricular enlargement documented in the report.

From an initial cohort of 484 patients who had taken part in the BCoS screening and had imaging available, a final sample of 387 sub-acute stroke patients who had both adequate imaging and a full set of praxis testing was included in this study. Patients with a first stroke in either their left, right, or both hemispheres were included to form an unselected, unbiased group of patients at the acute and subacute stages after stroke. **Table 1** provides complete demographic information on the patient cohort. The group included 353 right-handed patients and 34 left-handed patients. Of the patients who were left-handed, four patients had right-hemisphere lesions, two patients had bilateral lesions, and 28 patients had left-hemisphere lesions. A total of 349 patients from the cohort had had an ischemic stroke, and 38 patients had had a hemorrhagic stroke.

### Neuropsychological Assessments – Praxis Tasks

The cognitive assessment of the patients took place in hospital settings in the acute and sub-acute stage (≤3 months) post-stroke. The average time between stroke onset and test administration was 24.3 days (minimum = 1 day, maximum = 93 days), with 264 patients tested within 1 month TABLE 1 | Patient demographics and imaging details (SD = standard deviation).


after stroke. Neuropsychological testing was conducted using the BCoS (Humphreys et al., 2012).

The praxis tasks in the BCoS are aimed at assessing the cognitive processes subserving praxis, namely: (1) the input of visually conveyed gestures; (2) the coding of body part and position; (3) access to stored knowledge about the meaning of gestures; and (4) access to motor output transforming spatiotemporal concepts of gestures into motor commands (see **Figure 1** in Bickerton et al., 2012; Humphreys et al., 2012).

In the current study, we used three of the BCoS praxis tests to assess the presence of apraxia: Gesture Production, Gesture Recognition, and Meaningless Gesture Imitation. The BCoS also includes an assessment of orientation in time and space, providing a brief measure of orientation in time, person, and place and of overall comprehension, which was used in our imaging analyses as a covariate of no interest to remove deficits in basic cognitive ability (which could be caused by other clinical conditions at early stages after stroke, such as delirium) as potential confounds.

A previous study examined the validity and reliability of the praxis tasks in the BCoS against existing screens and included the patient cohort reported here (Bickerton et al., 2012). The interrater reliability for praxis in this particular cohort of patients has been reported and published before, in Chapters 6 and 7 of the BCoS manual (Zwinkels et al., 2004; Humphreys et al., 2012).

According to criteria published previously (Humphreys et al., 2012), patients were considered apraxic if they scored below the previously published set cut-off score in at least one of these three praxis tasks. **Table 2** lists the cut-off scores. However, for the purposes of the current study, the patients' praxis scores were entered as a continuous variable for the imaging analyses. Each of the three praxis tasks is detailed below. Two of the tasks (Gesture Production and Gesture Imitation) required emptyhanded execution of gestures to test conceptual and production deficits, respectively, according to traditional models of the disorder, without the confound of having the object-at-hand (Goldenberg, 2013a). Patients used their dominant hand or, if they had hemiparesis, their unaffected hand. A total of 266 patients used their left hand, and 121 patients used their right hand for the performance of all praxis tasks reported in this study.

TABLE 2 | Age adjusted cut-off scores for praxis tasks used in this study.


### Gesture Production

The Gesture Production task involved pantomiming a total of six gestures (three transitive, three intransitive) upon verbal command. The test included body-centered (salute, using a glass), non-body-centered (stop, using a salt cellar), repetitive (hitch-hiking, using a hammer), and non-repetitive (stop, using a glass) actions. All actions can be carried out as a single-step sequence. Patients were allowed a maximum of 15 s per item to respond and were asked to execute the action once. Two points were given for a correct and accurate gesture; 1 point for a recognizable but inaccurate gesture (e.g., including spatial and/or movement errors); 0 points were given for either no response after 15 s, an unrecognizable response or perseveration from previous gestures. The final sum score (maximum = 12) was used in the analyses.

### Gesture Recognition

In the Gesture Recognition Task, the examiner produced six actions, which patients had to recognize: three transitive (using a cup, using a key, using a lighter) and three intransitive (come over, good, goodbye) actions. As the examiner showed each gesture, the patients had to select the action being performed from a multiple-choice list, which included four alternative responses for each action, in writing. The four alternatives for each action corresponded to: (1) the correct action (e.g., using a lighter); (2) a semantically related action (using a match); (3) a visually related action (using a gun); and (4) an unrelated action (using a torch). The patients were allowed a maximum of 15 s per item to respond by pointing to their chosen statement, and they were given one point for each correct response. The final sum score (maximum = 6) was used in the analyses.

The data from both transitive and intransitive gestures in these tasks were entered together as a composite measure. Hence this study does not report differences between the two.

### Meaningless Gesture Imitation

The patients were asked to copy four meaningless gestures presented by the examiner. Two of these gestures involved a sequence of two hand positions in relation to the head, and the other two involved a single finger position. This task contrasted the indirect route to action production (i.e., imitating meaningless gestures) with "lexical" action recognition and production to name (see Bickerton et al., 2012). Three points were given for a gesture that was correctly and precisely imitated after the 1st presentation; two points if the gesture was correct and precise after the 2nd presentation; 1 point if patients made only one error after the 2nd presentation (e.g., incomplete movement sequence, incorrect spatial relationship between hand and head, or incorrect finger/hand position); 0 points if patients made more than one error, gave no response or showed perseveration from previous item(s) after the 2nd presentation. The final sum score (maximum = 12) was used in the analyses.

**Table 2** gives the praxis tasks cut-off scores based on the 5th percentile across age groups (from Humphreys et al., 2012). We report the rates of praxis deficits according to these cut-off scores in the section "Results."

### Picture Naming

fnhum-13-00422 December 18, 2019 Time: 16:0 # 5

The Picture-Naming task was used to control for language deficits in our study. The task involves asking patients to name objects that the examiner shows them a picture of. There were 14 objects that patients had to recognize and name. These were: bell, peas, grape, umbrella, raspberry, colander, leak, stopwatch, bat, pineapple, chisel, tiger, hook, and spanner. Patients scored one point for each correct naming, with a potential total score of 14.

### Imaging and Lesion Analysis CT Data Acquisition

CT scans were acquired as part of the patients' clinical assessment during their hospital admission. For the 387 patients included in this study, the average time between the stroke and CT scan acquisition was 4.4 days (Minimum = 0 days, Maximum = 64 days; Standard Deviation of 11 days, with more than 80% of cases scanned within 1 week).

The study used standardized CT imaging protocols, as follows. The scanners used were a Siemens Sensation 16 and a GE Medical System LightSpeed 16 and LightSpeed Plus. The images covered the whole brain, with slices aligned along the AC-PC plane and an in-plane resolution of 0.5 × 0.5 mm<sup>1</sup> and a slice thickness varying between 4 and 5 mm. A CT database of more than 500 patients with acute/subacute stroke was available, together with their clinical and demographic data, as well as a completed battery of neuropsychological tests from the BCoS (Humphreys et al., 2012). Patients with inappropriate CT scans were excluded from the study: these were patients with a CT scan in which a shunt was visible or patients in whom the field of view did not encompass the head (n = 127) (Gillebert et al., 2014).

### Automated Lesion Delineation Method

We implemented an automated toolbox for pre-processing and lesion mapping of CT brain scans (Gillebert et al., 2014). This procedure, fully described in Gillebert et al. (2014), involved the normalization of CT images from stroke patients to template space (Rorden et al., 2012a). Areas of hypo- or hyper-intensity, corresponding to ischemic or hemorrhagic stroke, respectively, were defined by voxel-wise comparisons with a group of control CT images. The validation and effectiveness of this approach were demonstrated both by visual inspection using CT images in sub-samples of stroke patients from the same dataset as in this study (CT image database collected for the Birmingham University Cognitive Screen, see text footnote 1) and by using simulated lesions. Both checks are reported in a previous study (Gillebert et al., 2014).

According to this method, CT scans were pre-processed using SPM8 (The Wellcome Trust Centre for Neuroimaging, London, United Kingdom), and lesion delineation was performed using in-house software written in Matlab (The MathWorks, Natick, MA, United States). Firstly, threshold-based clustering at 0.1% maximum intensity was implemented to remove irrelevant signals (Batenburg and Sijbers, 2009). The resulting CT images were spatially aligned to a template using the co-registration

<sup>1</sup>https://www.mccauslandcenter.sc.edu/mricrogl

tool in SPM8. CT image intensity was then transformed using an invertible formula to emphasize the contrast between cerebrospinal fluid and parenchyma (Rorden et al., 2012a).

The converted CT images were then warped to MNI space using a CT template (Rorden et al., 2012b). Firstly, a normalization function was used to calculate and apply a 12 parameter affine transformation that maximized the alignment to the template. The distribution of all image intensities was then calculated to create masks of the brain and the ventricles that were applied to generate skull-stripped images. These were then normalized and resliced at a 1-mm isotropic resolution using a large bounding box that included both the cortex and the cerebellum. The normalized CT images were smoothed with a 4 mm FWHM Gaussian filter (Salmond et al., 2002; Stamatakis and Tyler, 2005) according to the assumption of random field theory used in the statistical analysis (Worsley, 2003).

The lesion of each stroke patient was automatically identified using a voxel-based outlier detection procedure based on the Crawford-Howell parametric t-test for case-control comparisons (Crawford and Howell, 1998; Crawford et al., 2009). An outlier t-score map was generated using this test that coded the degree of abnormality of each voxel intensity based on a comparison to the normal range from control CT scans. These t-score maps were thresholded to generate binary lesion maps in MNI space (Gillebert et al., 2014) that were used to perform VLSM analyses.

### Voxel-Based Lesion-Symptom Mapping

The lesion maps obtained from the aforementioned procedure underwent VLSM analyses to identify the neural underpinnings of praxis deficits after stroke, based on the analysis toolbox provided by Bates et al. (2003)<sup>4</sup> . The behavioral results for each of the praxis tasks available from 387 patients were entered into separate VLSM analyses as the variable of interest, with additional covariates of age, handedness, total lesion volume, and assessment of orientation in time and space to control for these confounding factors (Gillebert et al., 2014; Chechlacz et al., 2015). We added an assessment of orientation in time and space based on correlations of deficits in this generic cognitive domain with praxis.

A linear model was fitted at each voxel, relating the unique score for each praxis task to lesion intensity (0 for no lesion; 1 for lesion). Tests were confined to those voxels in which at least 10 patients had a lesion. Only voxels that reached the false discovery rate (FDR) threshold of p < 0.05 were considered significant.

The use of CT imaging did not allow a clear segmentation of gray and white matter as is usually performed in VLSM analyses of MRI data. However, this has been used in CT in previous publications on neglect and attention (Gillebert et al., 2014; Chechlacz et al., 2015).

The anatomical localization for significant regions (FDRcorrected at p < 0.05) was identified based on the multimodal parcellation of human cerebral cortex provided by the Human Connectome Project (HCP) (Andreas, 2016; Glasser et al., 2016). The anatomical localization of regions located within white matter tracts was based on the Catani Atlas of Human Brain Connections (Thiebaut de Schotten et al., 2011). The interpretation of our results was supported by the expertise of an anatomist (Prof. R. E. Passingham). **Figures 1**, **2** were created using the template at MRICroGL<sup>2</sup> .

### RESULTS

### Behavioral Results

fnhum-13-00422 December 18, 2019 Time: 16:0 # 6

The behavioral results from individual praxis tasks and comparisons with other cognitive functions were used to identify the prevalence of deficits in each subtask in this cohort of patients. Note that this analysis was not used to inform the lesion-mapping analyses reported below. Instead, the behavioral data for each task were entered as a continuous variable. The reason for reporting the behavioral results below was to provide an indication of the number of patients who were deemed to be performing below the cut-off for praxis on these screening tasks. This was not used to inform our imaging analyses.

In the behavioral analyses, cut-off scores for normal performance (two standard deviations below the mean of agematched healthy controls) were 11.5 on Gesture Production, 5.8 Gesture Recognition, and 11.5 in Imitation, based on normative data published previously (see **Table 2** and Chapter 6 and 7 of Humphreys et al., 2012). Based on these criteria, 204 out of 387 patients performed abnormally on Gesture Production, 248 out of 387 on Gesture Recognition, and 252 out of 387 Imitation (on average, 235 patients out of 387 scored below range for apraxia). Of the left-handed patients, 12 out of 28 patients with left-hemisphere damage had no praxis deficits, whereas 16 patients with left-hemisphere damage scored below the cut-off in at least two of the praxis tasks, indicating they were most likely left-hemisphere dominant (Goldenberg, 2013b).

```
2http://aphasialab.org/vlsm
```
The average patient results on the three praxis tasks are outlined in **Table 3**.

In addition to praxis scores, we computed patients' general orientation in time and space and aphasia (using a picturenaming task from the BCoS). A total of 63 out of 387 (16%) of patients performed below the cut-off score for the orientation task, and 212 patients out of 387 (55%) performed below the cutoff score for picture naming, indicating language deficits adjusted for age.

We ran correlation analyses to identify whether our covariates of no interest were significantly correlated with a composite measure of apraxia, incorporating the scores of each of the three praxis tasks. Orientation in time and space correlated significantly with the composite Apraxia score (r<sup>387</sup> = 0.396, p < 0.0001), as did lesion size (r<sup>387</sup> = −0.120, p = 0.018) and age (r<sup>387</sup> = −0.200, p < 0.0001).

### Imaging Results

Lesion overlap is shown in **Figure 1**. **Figure 2** shows the lesionsymptom maps for each of the three tasks, FDR-corrected at p < 0.05, in axial and rendered views; **Table 4** provides the coordinates for each area and each task.

Deficits in the Gesture Production task were associated with lesions in a network of areas involving the left superior temporal sulcus (x = −50, y = −36, z = −12; t = 3.99), the left uncinate fasciculus (x = −28, y = −4, z = −16; t = 4.41) (which connects the temporal lobe with the inferior frontal cortex including Broca's area), and the white matter beneath the left primary motor cortex, within the superior longitudinal fasciculus (x = −34, y = −25, z = 31; t = 4.05). The lesions identified disconnections between the temporal and parietal lobes with the frontal lobe, leading to impairment in converting gestures into motor commands.


TABLE 4 | Coordinates of lesion-symptom mapping results, FDR-corrected at p < 0.05, based on HCP (Andreas, 2016; Glasser et al., 2016) and Catani white matter (Thiebaut de Schotten et al., 2011) atlases.


Deficits in the Gesture Recognition task revealed significant associations with lesions in the left superior temporal sulcus (x = −54, y = −44, z = −6; t = 4.24).

Finally, regions significantly associated with the meaningless gesture imitation task comprised the left visual striate and prestriate cortices (x = −29, y = −88, z = −8; t = 5.74), PGi parietal area (x = −42, y = −56, z = 13; t = 4.97), and parahippocampal area (x = −32, y = −39, z = −15; t = 5.13). We report the results for all patients combined in **Table 4**.

Subgroup analyses were performed to identify lesions pertaining to right- versus left-handed patients with rightversus left-hemisphere lesions, separately. Only the analyses pertaining to right-handed patients with both left- and righthemisphere lesions combined revealed significant results (FDRcorrected at p < 0.05). No significant results were identified in the other subgroups. Nevertheless, we identified the lesion locations at p < 0.005 uncorrected for left-, followed by righthemisphere lesions in right-handed patients, which are reported in section "Subgroup VLSM Analyses" of the **Supplementary Material**. Of note, unlike other reports (Goldenberg, 2013b), in our data, there were no significant differences in performance of the BCOS praxis tasks between subgroups of patients, as reported in this dataset previously (Bickerton et al., 2012; Humphreys et al., 2012).

A follow-up analysis was performed to identify lesionsymptom mapping that isolated praxis deficits from screened in the BCoS from language (picture naming). This was done by re-running the VLSM analyses outlined above with scores from the Picture Naming task in the BCOS (Humphreys et al., 2012) as an additional covariate. No significant results were found in this analysis.

We explored this result further by correlating the separate praxis with the picture-naming task. Picture Naming significantly correlated with Gesture Production (r<sup>387</sup> = 0.501, p < 0.0001), Gesture Recognition (r<sup>387</sup> = 0.368, p < 0.0001), and Meaningless Gesture Imitation (r<sup>387</sup> = 0.407, p < 0.0001). Moreover, we implemented VLSM analyses for Picture Naming. The results identified the superior temporal gyrus and are provided in the **Supplementary Material** (section "VLSM Results of Picture Naming Task").

### DISCUSSION

We conducted VLSM analyses for apraxia based on a large cohort of acute and subacute stroke patients. A validated battery of cognitive tasks for praxis (BCoS) was used (Bickerton et al., 2012) and analyzed alongside clinical CT images in which stroke lesions were automatically delineated. Our findings relate specifically to the early stages after stroke. Left, right, and bilateral hemisphere lesions were included in a VLSM analysis, in which the patients' scores in three praxis tasks from the BCoS were entered as continuous variables, creating an unbiased data sample.

Our results confirmed that deficits leading to apraxia result from left-hemisphere lesions (Goldenberg, 2013a). The lesion locations identified involved a network of areas comprising extrastriate visual areas, superior and medial temporal gyri, inferior parietal and inferior frontal areas, and white matter connections between the latter. As in recent VLSM studies of apraxia, our findings challenge traditional theories, which describe a prominent role of the parietal lobe in the disorder (Goldenberg, 2014). We identified instead ventral stream regions that pertain to the action-observation network. We discuss our results in relation to previous studies of apraxia, drawing parallels with the literature on language disorders after stroke. The last section highlights the implications of using clinical CT imaging in lesion-symptom mapping of apraxia.

### Neural Correlates of Apraxia Identified in Our Study

Our results identified an association of left-hemisphere lesions affecting the superior and medial temporal areas with all praxis tasks, namely gesture production, recognition, and meaningless gesture imitation. In addition, damage to the underlying white matter connections between the temporal cortex and the inferior frontal gyrus (the uncinate fascicle), as well as the superior longitudinal fasciculus underlying the primary motor cortex (Pandya et al., 2015), were associated with deficits in gesture production. Damage in the inferior parietal region PGi (Andreas, 2016; Glasser et al., 2016), prestriate, and parahippocampal area 2 (as identified in the HCP atlas; fusiform area, in other atlases) were associated with deficits in meaningless gesture imitation.

TABLE 3 | Patients' average results in the three praxis tasks.

Voxel-based lesion-symptom mapping studies of apraxia report wide networks of brain regions in the disorder, parallelling ours. These include inferior frontal (Pazzaglia et al., 2008), parietal, and temporal (Buxbaum et al., 2014; Hoeren et al., 2014) and also subcortical areas (Pramstaller and Marsden, 1996; Haaland et al., 2000; Leiguarda and Marsden, 2000). An important factor determining the outcome of patient studies relates to the tasks used to elicit conceptual and production deficits in apraxia, as well as the imaging modalities used to study these brain functions. We discuss the impact of these in the sections below.

### Traditional Brain Networks Identified in Apraxia and the Role of Tasks Used in Understanding the Neural Correlates of the Disorder

Lesions of the parietal lobe, particularly affecting the dominant hemisphere, have traditionally dominated neuropsychological models of apraxia (Liepmann, 1908, 1920). Much of our understanding of the role of parietal areas in action has come from anatomical and physiological studies of nonhuman primates. A dorsal visual stream has been subdivided into dorso-dorsal and ventro-dorsal streams, subserving motor representations allowing the implementation of reach and grasp actions, respectively (Rizzolatti and Matelli, 2003; Daprati and Sirigu, 2006). The ventral stream, which was originally proposed to mediate perceptual information (Goodale and Milner, 1992), has also been shown to play a role in the selection of actions (Milner and Goodale, 2008; Weiller et al., 2009; Rijntjes et al., 2012). Recent literature suggests there are connections between the two, supporting a role for ventral stream structures in both action observation and object use (Borra et al., 2010; Ramayya et al., 2010; Passingham et al., 2014; van Polanen and Davare, 2015).

### Parietal Cortex Contribution to Apraxia

The role of the inferior parietal cortex in limb apraxia has been reported in studies that used both real objectuse tasks (Goldenberg and Hagmann, 1998; Osiurak et al., 2008; Goldenberg and Spatt, 2009) and pantomime of object use (Buxbaum et al., 2014; Hoeren et al., 2014). Functional neuroimaging studies report a prominent role for the left inferior parietal cortex in the actual use of objects (Lewis, 2006; Osiurak and Badets, 2016; Reynaud et al., 2016). Our study did not involve the use of functional neuroimaging, and the tasks used for screening for apraxia involved pantomime of both transitive (with objects) and intransitive (with no objects) gestures. In particular, it did not include the use of real objects.

The lack of significant lesions in inferior parietal areas in our pantomime tasks could be due to task-related factors (reported below) and the imaging modality used (namely lesion-symptom mapping rather than fMRI, reported in greater detail in sections "Materials and Methods," "Settings Used for Our Voxel Based Lesion Symptom Mapping" of the **Supplementary Material**, and "Interpretation of Our Imaging Results Based on CT Imaging").

In relation to the former, it is noteworthy that there are anatomical connections between the superior temporal areas identified in our study and the inferior parietal areas reported in non-human primates (Rozzi et al., 2006). One possibility is that an effect of a lesion in the superior temporal sulcus could be to disconnect flow of information relating to biological motion (see below) from the inferior parietal cortex. This could elicit behavioral deficits in tool use. Lesion-mapping studies are descriptive. Unlike functional neuroimaging studies, they do not give an appreciation of how lesions in one area might impact activation or function in another area connected to it.

### Role of the Temporal Cortex in Apraxia

There is an increasing amount of evidence for a communicative component to pantomiming gestures, even those that pertain to object-use. A lesion-symptom mapping study involving pantomiming of object-use identified two networks implicated in the task: a "posterior" network of brain regions, comprising inferior parietal and dorsal stream areas, representing the motor aspects of object use and an "anterior" network of brain regions, comprising inferior frontal and temporal areas, relating to the communicative components of the task (Finkel et al., 2018). The Gesture Recognition task in the BCOS requires comprehension of gestures and what they represent when choosing among a multiple-choice set of options in writing. What is more, the scores we obtained from Gesture Recognition and Gesture Production tasks combined both transitive and intransitive gestures, possibly emphasizing a role for communication as in Finkel et al.'s (2018) study. The lesions identified in our tasks were located predominantly in superior temporal rather than parietal areas, corresponding to the "anterior" network, which was attributed to communication in Finkel et al's., 2018 study.

Nevertheless, our results on the Meaningless Gesture Imitation task, which did not require any verbal comprehension, also implicated both the superior and infero-temporal cortex, as well as inferior parietal area PGi. The study by Buxbaum et al. (2014) also identified lesions in the posterior temporal lobe and temporo-occipital areas as significant both in gesture representations of tools and in abstract movement representations when tested with meaningless gesture imitation. Both our results and theirs challenge the traditional model of apraxia in which the parietal lobe plays a central role, revealing the involvement of a wider network that comprises the left temporal lobe in the disorder (Goldenberg, 2009).

In the sub-sections below, we argue for a possible role of the temporal cortex in understanding action intentions, either through comprehension or through action observation.

### **A role for the temporal cortex in praxis and comprehension**

In our study, we found no voxels pertaining to apraxia alone when covarying for language deficits measured using a Picture Naming task. Moreover, the two deficits co-existed in approximately 50% of our patient cohort. Lesions involving superior temporal areas, identified in Gesture Production, were also present in Picture Naming. Taken together our findings suggest that the two deficits might overlap (Goldenberg and Randerath, 2015; Finkel et al., 2018). In another study by Weiss et al. (2016), praxis and language were differentiable.

One reason for the discrepancy between our and Weiss et al.'s (2016) results could relate to the behavioral tasks used in this study. The tasks used in our study were part of a cognitive screening program developed to test stroke patients (BCOS, Humphreys et al., 2012) in which language is tested using Picture Naming. This task involves the naming of a large number of graspable objects (Bub et al., 2018). Previous studies using fMRI have identified a role for dorsal stream structures in identifying manipulable objects (Chao and Martin, 2000; Creem and Proffitt, 2001). There is evidence that naming manipulable objects influences actions (Bub et al., 2018; Masson, 2018). One possible explanation of our inability to differentiate between these two disorders in our data might relate to the fact that ventral stream networks to "name" and "use" objects may overlap (Mahon et al., 2007). Another possibility relates to the fact that both Gesture Recognition and Production tasks in the BCOS involve comprehension and that this may overlap with language functions (Goldenberg and Randerath, 2015). Our measure of these praxis tasks combined transitive and intransitive gestures, which have been shown to test for communication (Johnen et al., 2016; Dressing et al., 2018; Finkel et al., 2018).

Nevertheless, these factors still fail to explain the fact we identified the superior temporal gyrus in a meaningless gesture imitation task that involved no communication. We outline below a possible explanation for this latter result.

### **Regions identified in our task that form part of the action observation network**

The involvement of the superior temporal area in the gesture imitation task in our study, which did not involve any verbal or semantic interpretations, parallels the roles described for these areas in action observation, which have been identified in nonhuman primates.

Studies have demonstrated the presence of cells in the superior temporal sulcus that code for action observation and are sensitive to biological motion stimuli (Jellema and Perrett, 2003; Barraclough et al., 2009). This region is anatomically connected to inferior parietal regions, which in turn connect to central premotor areas (Rozzi et al., 2006; Borra et al., 2008). The latter network of areas has been described as the "mirror neuron" network (Bonini et al., 2011), which is involved in understanding actions. Similar areas have been described in human fMRI studies, with evidence that the inferior parietal region is activated when healthy subjects are required to understand the meaning of gestures (Passingham et al., 2014) or when experts are asked to observe skillful actions that are familiar (Calvo-Merino et al., 2005). In our study, both the prestriate and inferior temporal cortices were involved in the imitation of meaningless gestures. This could relate to the role of the inferior temporal cortex in the discrimination between shapes (Huxlin et al., 2000). It may be that patients have to understand the shape of the hand that has to be copied.

A patient study by Achilles et al. (2016) provides some support for the above. Left-hemisphere stroke patients with and without apraxia were asked to rate the familiarity of meaningless gestures, which they imitated. Patients with apraxia were found to have better performance when copying meaningless gestures that were judged as being familiar by the whole patient cohort, suggesting that they were able to recognize familiarity in meaningless gestures.

Our results support a role for temporal lobe and prestriate areas in understanding the meaning of actions in meaningless gesture imitation tasks, even when language functions are not implicated (Buxbaum et al., 2014; Passingham et al., 2014). This might provide a non-verbal network sub-serving both the understanding of action intentions and communication.

### **"Domain-general" and "domain-specific" deficits after stroke and interpretation of our lesion-mapping results**

The presence of similar areas sub-serving functions such as praxis and language skills might indicate that their involvement in these could be generic to both tasks (Geranmayeh et al., 2014). This has been demonstrated in the case of parietal lobe involvement, which is implicated in a large range of cognitive functions (Humphreys and Lambon Ralph, 2015). In the language literature, the parietal cortex has been shown to influence both "domain-general" and "domain-specific" deficits. An example of the former is the "Multiple Demand" system, which exerts top-down control on a wide range of tasks and involves processes such as cognitive flexibility, behavioral inhibition, and attentional control (Duncan, 2010; Hampshire et al., 2012).

The same is likely to be true for the role of the temporal lobe in praxis. Based on the literature, the role of the temporal cortex in praxis may be "domain specific", in providing knowledge of tool function (Campanella et al., 2010; Buxbaum et al., 2014; Hoeren et al., 2014), or "domain general", in understanding action meaning and "theory of mind" (Allison et al., 2000; Saygin, 2007). The former system may be used for naming and using tools (Mahon et al., 2007), whilst the latter system would be used for understanding others' intentions through actions and non-verbal communication cues (Allison et al., 2000; Finkel et al., 2018).

Our study, like others, highlights a relationship between language and apraxia (Goldenberg and Randerath, 2015). However, we cannot draw conclusive evidence of the influence of one on the other. Some authors have tried to achieve such a differentiation with novel imaging analyses in lesionsymptom mapping, allowing the subtraction of one effect from the other (Dressing et al., 2018). However, to formally differentiate the relative contribution of the temporal lobe between the two cognitive domains, a systematic comparison between language and praxis skills would require more dedicated tasks, which would include tasks for biological motion targeted at differentiating between speech and hand gestures. This would need to be supplemented with converging evidence from fMRI and lesion-mapping techniques (Mahon et al., 2007).

### Interpretation of Our Imaging Results Based on Clinical CT Imaging

This study is one of a few to have implemented lesionsymptom mapping techniques on the clinical CT scans of a retrospective cohort of stroke patients (Rorden et al., 2012b; Gillebert et al., 2014; de Haan and Karnath, 2018). Clinical CT is

the imaging method of choice in patients admitted to hyperacute stroke units in the United Kingdom.

Recent advances (Ripolles et al., 2012; Rorden et al., 2012b) have made the identification of both ischemic and hemorrhagic lesions possible on the same CT scan (Chawla et al., 2009; Gillebert et al., 2014). The lesion delineation technique we used compares CT image intensity from a single patient with a group of images from control participants to identify outlier voxels (Crawford et al., 2009; Gillebert et al., 2014). In effect, this approach resembles the analysis of MR images (Stamatakis and Tyler, 2005). The use of standardized preprocessing techniques for CT (Rorden et al., 2012a) allowed us to obtain comparable results, in terms of lesion localization, to those reported in MRI studies (Buxbaum et al., 2014; Hoeren et al., 2014). Nevertheless, our lesion sizes and the number of patients required to obtain these results did differ significantly from lesion-mapping techniques that have used MRI (Manuel et al., 2013; Buxbaum et al., 2014; Weiss et al., 2016). This may have occurred due to the following methodological caveats. (1) The use of automated lesion delineation in our study may have underestimated lesion sizes, particularly for ischemic strokes, which are often difficult to detect on CT. The technique might benefit from more refined information that could be provided with complementary perfusion CT (Wing and Markus, 2019), which was not available at the time of data collection. (2) A study investigating the impact of sample size on the reproducibility of lesion-symptom mapping results (Lorca-Puls et al., 2018) reported striking differences in terms of either under- or overestimated effect sizes. An additional shortcoming of lesionsymptom mapping techniques called "the partial injury problem" (Rorden et al., 2009) is that they may fail to consider the contribution of anatomically distributed areas in producing a behavioral deficit. This is because patients may present with different lesions in a distributed network, for which mass univariate analyses may miss the critical regions involved, due once again, to low statistical power (Herbet et al., 2015; Gajardo-Vidal et al., 2018). Some authors have proposed ways of mitigating the biological constraints of lesion distributions with the use of multivariate pattern analysis techniques (Smith et al., 2013; Mah et al., 2014). (3) Patient selection: although we tried to obtain an unbiased data-sample, the majority of our patients had strokes affecting the middle cerebral artery, with lesions located in the convexity of the hemisphere. This led to low numbers of patients with more superior lesions, probably reducing the statistical power to detect effects in these cortical regions (Kroliczak and Frey, 2009; Agnew et al., 2012; Buxbaum et al., 2014). (4) Lesion localization: the use of CT imaging had the caveat of requiring different anatomical atlases for gray and white matter localization. The review by de Haan and Karnath (2018) outlines significant differences in the interpretation of lesion mapping results based on which atlas is used for anatomical localization. Atlases such as the AAL (Tzourio-Mazoyer et al., 2002) and Harvard-Oxford atlases (Desikan et al., 2006), which are widely available in statistical analysis packages, underrepresent the number of cortical areas (Van Essen et al., 2012). To avoid the mislabeling of areas (Passingham and Rowe, 2015), the anatomical localization of significant regions in this study were identified using separate atlases for white and gray matter regions (see section "Materials and Methods" and **Table 4**, above). For localization of gray matter areas, we selected to use a more detailed atlas, namely the HCP atlas (Andreas, 2016; Glasser et al., 2016).

## CONCLUSION

We have conducted a lesion mapping study on praxis deficits with the largest cohort studied to date. The patients were in the early stages after a stroke (Bernhardt et al., 2017). Our results suggest an important role for temporal lobe structures in the disorder. This area was not only implicated in the knowledge of tool functions when testing patients on pantomime tasks but was also present in the imitation of meaningless gestures. This finding concurs with other VLSM studies of the disorder in stroke (Buxbaum et al., 2014; Hoeren et al., 2014) as well as with previous literature involving praxis deficits in neurodegenerative disorders (Crutch et al., 2007; Johnen et al., 2016).

The implication of ventral stream areas in praxis, even when no object recognition is required, such as in the meaningless gesture imitation task, has been largely overlooked (Goldenberg, 2014). It is likely that the network implicated in apraxia evolved to sub-serve parallel functions for praxis and language in humans (Badets and Osiurak, 2017). New tasks are being developed that provide evidence that skillful tool use may support linguistic abilities (Brozzoli et al., 2019). Our results support recent studies designed to use action observation tasks for the rehabilitation of this devastating disorder (Pazzaglia and Galli, 2019). Further work is required to identify the granularity of the contributions of the temporal lobe and its connections in praxis and language deficits in patients with stroke and neurodegenerative conditions.

The adoption of analysis techniques borrowed from MRI (Seghier et al., 2008) that help the automated normalization into standard space and, therefore, inter-individual comparisons of CT images provides a window of opportunity for lesion-symptom mapping in larger patient cohorts (Gillebert et al., 2014). This will pave the way for a better understanding of cognitive deficits after stroke, such as apraxia.

### DATA AVAILABILITY STATEMENT

The datasets generated for this study are available on request to the corresponding author and contingent on the approval of sharing this dataset by the BCOS team and local ethics committee.

### ETHICS STATEMENT

The studies involving human participants were reviewed and approved by the National Research Ethics Service (NRES): Essex 1 Research Ethics Committee (REC). The patients/participants provided their written informed consent to participate in this study.

### AUTHOR CONTRIBUTIONS

fnhum-13-00422 December 18, 2019 Time: 16:0 # 11

ER conceptualized the study with DM. DM provided the data analysis techniques, which GP and ZZ implemented. JR was part of the original BCOS team who collected the data and together with the late Professor Humphreys provided access to it to complete this study. ZZ re-analyzed data with covariates of no interest (of aphasia and neglect) and created the **Figure 1**. JK and ZZ created the supplementary figures. JK and RP provided the anatomy and atlas support. ER and RP wrote up the manuscript. All authors reviewed and edited the manuscript.

### FUNDING

This work was supported by the Stroke Association UK (grant to Prof. G. W. Humphreys), and an Oxfordshire Health Services

### REFERENCES


Research Committee research grant to ER (Ref. 1227). JK holds a Wellcome Trust Sir Henry Wellcome Postdoctoral Fellowship (204696/Z/16/Z).

### ACKNOWLEDGMENTS

We would like to thank Prof. Glyn Humphreys' team and the clinical teams in Birmingham, who were involved in the data collection for the Birmingham Cognitive Screening Program.

### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnhum. 2019.00422/full#supplementary-material




**Conflict of Interest:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Pizzamiglio, Zhang, Kolasinski, Riddoch, Passingham, Mantini and Rounis. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# "Fake it till You Make it"! Contaminating Rubber Hands ("Multisensory Stimulation Therapy") to Treat Obsessive-Compulsive Disorder

Baland Jalal 1,2\*, Richard J. McNally <sup>2</sup> , Jason A. Elias 3,4 , Sriramya Potluri 3,4 and Vilayanur S. Ramachandran<sup>5</sup>

<sup>1</sup>Department of Psychiatry, Behavioural and Clinical Neuroscience Institute, University of Cambridge School of Clinical Medicine, Cambridge, United Kingdom, <sup>2</sup>Department of Psychology, Harvard University, Cambridge, MA, United States, <sup>3</sup>Obsessive-Compulsive Disorder Institute, McLean Hospital, Belmont, MA, United States, <sup>4</sup>Department of Psychiatry, Harvard Medical School, Boston, MA, United States, <sup>5</sup>Center for Brain and Cognition, University of California, San Diego, San Diego, CA, United States

### Edited by:

Mariella Pazzaglia, Sapienza University of Rome, Italy

#### Reviewed by:

Daniela Rabellino, University of Western Ontario, Canada Jane Elizabeth Aspell, Anglia Ruskin University, United Kingdom

> \*Correspondence: Baland Jalal bj272@cam.ac.uk

#### Specialty section:

This article was submitted to Sensory Neuroscience, a section of the journal Frontiers in Human Neuroscience

> Received: 04 June 2019 Accepted: 07 November 2019 Published: 09 January 2020

#### Citation:

Jalal B, McNally RJ, Elias JA, Potluri S and Ramachandran VS (2020) "Fake it till You Make it"! Contaminating Rubber Hands ("Multisensory Stimulation Therapy") to Treat Obsessive-Compulsive Disorder. Front. Hum. Neurosci. 13:414. doi: 10.3389/fnhum.2019.00414 Obsessive-compulsive disorder (OCD) is a deeply enigmatic psychiatric condition associated with immense suffering worldwide. Efficacious therapies for OCD, like exposure and response prevention (ERP), are sometimes poorly tolerated by patients. As many as 25% of patients refuse to initiate ERP mainly because they are too anxious to follow exposure procedures. Accordingly, we proposed a simple and tolerable (immersive yet indirect) low-cost technique for treating OCD that we call "multisensory stimulation therapy." This method involves contaminating a rubber hand during the so-called "rubber hand illusion" (RHI) in which tactile sensations may be perceived as arising from a fake hand. Notably, Jalal et al. (2015) showed that such fake hand contamination during the RHI provokes powerful disgust reactions in healthy volunteers. In the current study, we explored the therapeutic potential of this novel approach. OCD patients (n = 29) watched as their hidden real hand was being stroked together with a visible fake hand; either synchronously (inducing the RHI; i.e., the experimental condition; n = 16) or asynchronously (i.e., the control condition; n = 13). After 5 min of tactile stimulation, the rubber hand was contaminated with fake feces, simulating conventional exposure therapy. Intriguingly, results suggested sensory assimilation of contamination sensations into the body image via the RHI: patients undergoing synchronous stimulation did not report greater contamination sensations when the fake hand was initially contaminated relative to asynchronous stroking. But contrary to expectations, they did so after the rubber hand had been contaminated for 5 min, as assessed via disgust facial expressions (a secondary outcome) and in vivo exposure (upon discontinuing the illusion). Further, to our surprise, synchronous and asynchronous stroking induced an equally vivid and fast-emerging illusion, which helps explain why both conditions initially (5 min after initiating tactile stimulation) provoked contamination reactions of equal magnitude. This study is the first to suggest heightened malleability of body image in OCD. Importantly, it may pave the way for a tolerable technique for the treatment of OCD—highly suitable for poorly resourced and emergency settings, including low-income and developing countries with minimal access to high-tech solutions like virtual reality.

Keywords: obsessive-compulsive disorder (OCD), rubber hand illusion, therapy, contamination fears, exposure and response prevention (ERP), multisensory integration

### INTRODUCTION

Obsessive-compulsive disorder (OCD) is a deeply enigmatic psychiatric condition that afflicts 2% to 3% of the general population (Robins et al., 1984; Ruscio et al., 2010). One variant of OCD is characterized by severe contamination fears and excessive cleansing rituals (Rachman, 2004; Markarian et al., 2010). These patients may feel anxious even after incidents of slight ''contamination'' (e.g., touching a door knob) and might spend hours painstakingly washing and scrubbing their hands—sometimes until they bleed. The primary treatment for OCD is called exposure and response prevention (ERP; Meyer, 1966). During ERP, the patient is first ''contaminated'' (e.g., touches a toilet bowl), which can trigger an acute spike in anxiety, and then prevented from performing the compulsive ritual (e.g., washing hands). This procedure may help the patient experience a subsequent decrease in anxiety, resulting in habituation (Abramowitz et al., 2009). But unsurprisingly, many OCD patients do not benefit from ERP (Kozak, 1999); the notion of being contaminated in this crude fashion is simply too unbearable. Alarmingly, 50% of patients who start ERP do not improve, 20% drop out prematurely, and 25% refuse to initiate therapy (Kozak, 1999; Schruers et al., 2005; Abramowitz, 2006), mainly due to fear of treatment (Maltby and Tolin, 2005). As such, developing gentler (less distressing) interventions for OCD represents an unmet need.

To overcome challenges of existing exposure therapies, we recently proposed a simple and tolerable (immersive yet indirect) low-cost technique for the treatment of OCD (Jalal et al., 2015) that we call ''multisensory stimulation therapy.'' Healthy volunteers watched as their occluded real hand was being stroked together with a visible fake hand in precise synchrony, producing the so-called ''rubber hand illusion'' (RHI; Botvinick and Cohen, 1998). After 5 min of such tactile stimulation, we contaminated the dummy with fake feces, in effect, mimicking traditional exposure therapy. To our astonishment, participants reported disgust sensations—as if arising from the rubber hand! This finding with potential clinical utility (discussed in more detail below) has since been replicated in a large Japanese sample, suggesting the effect is both robust and cross-culturally reliable (Nitta et al., 2018).

One interpretation for the emergence of the RHI evokes the ''Bayesian logic'' of perceptual systems (e.g., Armel and Ramachandran, 2003; Ramachandran et al., 2011; Jalal et al., 2015). The brain's sensory system is hardwired to detect statistical correlations that provide the basis for making predictions and, ultimately, visual representations of the external world, including one's body (see also Corlett et al., 2011). The brain considers it highly unlikely that the random stroking seen on the fake hand and felt on the real hand is due simply to chance; it infers therefore that the sensations must be arising from the rubber hand, however absurd. As such, the illusion is driven by bottom-up mechanisms (i.e., statistical correlations between senses) and any object in theory could become part of one's body image including a table (Armel and Ramachandran, 2003). Consistent with this account, the RHI does not occur (or is greatly diminished) following asynchronous stimulation of the real and rubber hand. This ''gold standard'' control procedure shows the importance of spatial and temporal congruence of the tactile and visual inputs in driving the illusion (e.g., Shimada et al., 2009).

To date, research has explored various measures and versions of the RHI (e.g., Armel and Ramachandran, 2003; Costantini and Haggard, 2007; Ehrsson et al., 2007; Capelari et al., 2009; Kammers et al., 2009; Ramachandran et al., 2011). The basic effect emerges fairly quickly, in most healthy volunteers usually around 10–30 s after the synchronized stroking begins (Ehrsson, 2012). In our own studies, we have found that the illusion is reliably induced in healthy individuals within 2.5–5 min of tactile stimulation (e.g., in approximately 73% of subjects across two separate experiments; see Jalal et al., 2015; see also Armel and Ramachandran, 2003). The illusion is most commonly assessed with a subjective measure of limb ownership and an objective test of proprioceptive drift, where participants after the illusion onset close their eyes and point to the direction of their real hand. Botvinick and Cohen (1998) showed that after RHI induction, participants point to the artificial hand instead of their real hand unlike in the asynchronous control condition, and that the degree of this displacement is associated with the prevalence of the RHI over time (i.e., as measured within a 30-min stimulation period). In line with this, Tsakiris and Haggard (2005) demonstrated that continuous tactile stimulation during the RHI gradually increases such proprioceptive drift, suggesting a gradual intensifying of the illusion over time. This proprioceptive drift test correlates with the subjective vividness of the illusion (e.g., Longo et al., 2008).

The RHI has also been examined in psychiatric groups: for example, one study found a stronger illusion and faster onset in schizophrenia, suggesting a malleable self-representation in this population (Peled et al., 2000). Comparable results were reported in patients with eating disorders, who likewise have a pronounced RHI compared to healthy volunteers (Eshkevari et al., 2012). Other studies have revealed a more complex picture vis-à-vis body-related processing in psychopathology. For instance, although patients with posttraumatic stress disorder (PTSD; i.e., with dissociative symptoms) initially have a more intense illusion than do healthy controls, after three consecutive trials (over the course of 2 weeks), a comparable intensity to that of healthy subjects was reported (Lev-Ari and Hirschmann, 2016). Kaplan et al. (2014) did not find the intensity of the RHI to differ in patients with body dysmorphic disorder (BDD) and healthy controls, yet surprisingly, the BDD group displayed proprioceptive drift towards the rubber hand in both the synchronous and asynchronous control condition, unlike healthy individuals, who only did so in the RHI condition as expected. Finally, children with autism spectrum disorders (ASD) have a delayed susceptibility to the illusion (i.e., exhibit a later illusion onset compared to non-autistic children). Notably, children with ASD who have lower levels of empathy are less likely to experience the RHI (Cascio et al., 2012). Taken together, these studies suggest that some forms of psychopathology are associated with aberrant self-referential processing as assessed on the RHI.

To date, no studies have examined the RHI in OCD. The illusion may be particularly pertinent to OCD given the role of dopamine in the pathophysiology of the disorder (e.g., Denys et al., 2004; Koo et al., 2010). Although the function of dopamine in OCD is multifaceted (e.g., Fineberg et al., 2007), research has shown that dopamine antagonists [as an adjunct to selective serotonin reuptake inhibitor (SSRI) drugs] can reduce OCD symptoms (i.e., augment the effects of SSRIs; Vulink et al., 2009). In contrast, dopamine agonists can generate OCD-like behaviors in animals (Szechtman et al., 1998) and humans (Borcherding et al., 1990), providing clues about the functional role of dopamine in OCD.

Interestingly, research suggests that dopamine is a key modulator of multisensory integration as assessed via the RHI. For instance, the dopamine releaser drugs ketamine and dexamphetamine (with potential to trigger schizophrenia-like symptoms; Angrist and Gershon, 1970; Pomarol-Clotet et al., 2006) augment the illusion during regular synchronous stroking, but curiously also, in the (illusion-attenuating) asynchronous control condition (Albrecht et al., 2011; Morgan et al., 2011). Analogously, patients with Parkinson's disease (receiving dopaminergic drugs) fail to reject the RHI in the asynchronous condition as strongly as healthy control participants do, according to the authors, possibly due to dopamine dysregulation (Ding et al., 2017). Collectively, this research is in keeping with findings that schizophrenia (a disorder of dopamine abnormality; e.g., Howes et al., 2015) results in heightened illusory effects, and points to the pervasive role of dopamine in self-referential processing.

Research should disclose whether OCD is associated with multisensory processing abnormalities. By beginning to probe the corporeal self in OCD, one may eventually clarify how the processes that produce a sense of body ownership differ in this disorder vs. other psychiatric conditions. Indeed, if research reveals aberrant somatosensory integration in OCD, efforts to establish specificity could elucidate OCD etiology and differentially inform novel treatments (e.g., drug and behavioral interventions) aiming at restoring aspects of self-referential processing (also see Eshkevari et al., 2012).

The illusion may be of special interest to contaminationrelated OCD, i.e., provide an experimental probe for exploring pathological disgust and novel therapeutic techniques. As noted, we have shown that contaminating the fake hand during the RHI provokes OCD-like disgust reactions in healthy volunteers (Jalal et al., 2015): in this study, 81% of participants reported greater disgust during synchronous stroking vs. the asynchronous control condition, and, on overage, those undergoing the RHI reported significantly higher levels of disgust. In a ''direct replication study,'' Nitta et al. (2018) likewise showed that such ''exposure'' during the RHI triggered greater disgust reactions than asynchronous stroking in healthy individuals from Japan.

Notably, disgust plays a key role in OCD and is a strong predictor of contamination fears (e.g., Olatunji et al., 2005; see also Deacon and Olatunji, 2007; Olatunji et al., 2007; for reviews, see Ludvik et al., 2015; Knowles et al., 2018). Although disgust and contamination aversion overlap, they are indeed distinct concepts. Disgust is a basic emotion that induces a unique response (e.g., a facial expression; Rozin and Fallon, 1987), whereas contamination fears arise from post hoc interpretive processes, e.g., triggered by disgust or related emotions like anxiety (Rachman, 2004; also see Ludvik et al., 2015). Like disgust, anxiety is an independent driver of contamination fears (but may interact with disgust to trigger contamination concerns; Cisler et al., 2007). Interestingly, although traditional ERP triggers and degrades anxiety and washing urges (Rachman, 2004; Cougle et al., 2007), research suggests that disgust is also amenable to exposure therapy in OCD (McKay, 2006).

Although the results of Jalal et al. (2015) comport with the literature on ERP (i.e., disgust induced by ''fake hand exposure'' mirrors the effects of in vivo exposure; e.g., McKay, 2006), several issues remain vis-à-vis the clinical utility of this RHI contamination procedure. First, research should extend this work to a clinical population to assess the therapeutic use of the RHI; i.e., it is important to establish the presence of this basic ''RHI contamination effect'' in OCD patients. Second, to the extent that such rubber hand exposure evokes clinically relevant contamination reactions in OCD, research should examine whether this eventually leads to habituation.

Such research may have important treatment implications: if contaminating a fake hand during the RHI provokes contamination reactions (akin to ERP) via an immersive multisensory mechanism, this may pave the way for a novel (tolerable) intervention. As noted, such dummy contamination may eventually (after an extended period and/or repeated trials) lead to habituation, i.e., overall global reduction in contamination fears, analogous to conventional ERP. Another possibility is that contaminating a fake hand during the RHI, minimally, is useful during the initial stages of ERP (e.g., in an ''exposure hierarchy''; Wolpe, 1958; see also Abramowitz et al., 2003). This technique might sufficiently desensitize patients such that they are willing to undertake ERP, providing a convenient ''transitional link'' (Jalal et al., 2015).

### Primary Study Aims

In the current study, the key aim was to explore the therapeutic potential of the RHI for OCD. We examined whether ''contaminating'' the rubber hand during the illusion would result in greater contamination sensations as compared to the asynchronous control condition. We also tested whether such dummy contamination eventually resulted in habituation, assessed both during the illusion and during an in vivo exposure procedure immediately upon discontinuing the illusion (i.e., ceasing the stimulation of the real and rubber hand).

### Hypotheses

If contaminating the fake hand during the RHI (5 min after initiating stroking) provokes greater disgust than asynchronous stroking in healthy individuals (Jalal et al., 2015; Nitta et al., 2018)—given the role of disgust in OCD—this should also hold for patients with contamination obsessions. Moreover, considering that ERP targets both anxiety and washing urges (Rachman, 2004), RHI exposure should likewise evoke such contamination sensations overall (i.e., in addition to disgust). Finally, given that OCD patients dependably experience habituation following prolonged exposure to ''contaminants'' during ERP (on habituation see, e.g., Foa et al., 1983; Rachman, 2004; Abramowitz, 2006), RHI exposure should after an extended period lead to habituation [This latter hypothesis is partly grounded in research showing that the RHI emerges quickly and does not wane with time (e.g., Tsakiris and Haggard, 2005; Ehrsson, 2012), preserving the realistic nature of the exposure procedure].

Assuming that: (1) contaminating the fake hand during the RHI results in greater contamination sensations than does asynchronous stroking in OCD; and that (2) such exposure over time leads to habituation, we advanced the following hypotheses.

RHI contamination: OCD patients in the RHI condition would report greater contamination sensations (disgust, anxiety, and handwashing urges), and be more likely to exhibit a disgust facial expression, when the fake hand is contaminated (i.e., 5 min upon initiating the real and rubber hand stroking), compared to those in the asynchronous control condition.

RHI habituation: OCD patients in the RHI condition would report lower contamination sensations (disgust, anxiety, and handwashing urges), and be less likely to exhibit a disgust facial expression, 5 min after contaminating the dummy (i.e., 10 min upon initiating the real and rubber hand stroking), compared to those in the asynchronous condition.

In vivo exposure (habituation assessment): OCD patients in the RHI condition would report lower contamination sensations (disgust, anxiety, and handwashing urges) when their real hand is contaminated (i.e., immediately upon ceasing the stimulation of the real and rubber hand) compared to those in the asynchronous condition.

### Secondary (Exploratory) Aims

A secondary aim was to broadly explore multisensory processing in OCD. In view of research: (1) indicating that dopamine, implicated in OCD (e.g., Denys et al., 2004; Koo et al., 2010), is a modulator of multisensory processing (e.g., Albrecht et al., 2011; Morgan et al., 2011); and (2) suggesting aberrant somatosensory integration in psychiatric disorders more generally (see above), we tentatively hypothesized that OCD would be associated with atypical multisensory processing. For example, OCD patients would show high susceptibility to the illusion (indexed by illusion onset and intensity measures) compared to healthy populations (e.g., as reported in our own studies; Jalal et al., 2015). Given the exploratory (open-ended) nature of this inquiry, no directional hypothesis was made a priori.

### MATERIALS AND METHODS

### Participant Selection and Clinical Characteristics

Study participants included 29 OCD patients recruited from the McLean Hospital Obsessive-Compulsive Disorder Institute (OCDI), an intensive residential treatment (IRT) program affiliated with Harvard Medical School. At the OCDI, patients receive intensive (2–4 h daily) cognitive–behavioral therapy and psychopharmacological management, i.e., by a team of behavioral and family therapists, psychiatrists, etc. Medications are used on a case-to-case basis (i.e., determined during weekly psychiatric assessment) and often include SSRIs (e.g., venlafaxine and clomipramine) and antipsychotics (i.e., as an adjunct to SSRIs). Although treatment duration is based on individual need, patients on average remain at the OCDI for 45 days, with 25% of patients for at least 12 weeks (Athey et al., 2015). Inclusion criteria for admission to the OCDI include major OCD-related functional impairment and lack of response to treatment in other settings. The program does not have official exclusion criteria, but patients are not admitted if they have a condition that would interfere with treatment; e.g., severe intellectual disability (mental retardation or neurodevelopmental disorders etc.), current substance abuse and active psychosis (for details on McLean Hospital's IRT program, see also Stewart et al., 2005).

In the current study, all participants were diagnosed with OCD by an expert clinician on staff as part of standard clinical procedures based on DSM-IV or DSM-5 criteria and had disgust- and/or contamination-related obsessions. The presence of disgust- and contamination-related symptoms were defined by elevated scores on the Disgust Propensity and Sensitivity Scale-Revised (DPSS-R; van Overveld et al., 2006) and endorsement of contamination obsessions on the Dimensional Obsessive–Compulsive Scale (DOCS; Abramowitz et al., 2010; completed as part of an admission's battery of questionnaires). This clinical assessment was not based on a specific cutoff score but whether such symptoms were present [i.e., akin to the Yale-Brown Obsessive-Compulsive Scale (Y-BOCS) symptom checklist; Goodman et al., 1989]. As the main aim of this study was to explore a novel clinical approach, no strict selection criteria were applied (aside from the general OCDI selection criteria, noted above), ensuring that our sample was representative of this patient population. As such, medicated patients were not excluded. Given all patients were undergoing IRT, they were only selected for participation insofar that it would not interfere with their treatment.

Information regarding comorbid psychiatric diagnoses was available for 27 patients [i.e., out of 29; two patients did not complete an elaborate semi-structured diagnostic interview and/or a clinician administered intake interview to determine co-occurring conditions, due to logistic reasons (e.g., unavailability of clinical staff to conduct such interviews) or interference of OCD symptoms etc.]. Of these 27 patients, 92.6% (25/27) had OCD as a primary diagnosis and 3.7% (1/27) had OCD as a secondary diagnosis (data regarding whether OCD or a related mood disorder was primary was unavailable for one patient). Individuals who did not have a primary diagnosis of OCD were diagnosed with an obsessive-compulsive-related disorder (e.g., BDD: 3.7%; 1/27) or a related mood disorder (e.g., bipolar disorder I: 3.7%; 1/27).

Moreover, 74.1% (20/27) of participants had at least one comorbid axis I diagnosis. Frequencies of most co-occurring disorders were major depressive disorder (29.6%; 8/27), dysthymic disorder/persistent depressive disorder (18.5%; 5/27), post-traumatic stress disorder (18.5%; 5/27), and generalized anxiety disorder (14.8%; 4/27), followed by eating disorder NOS/other specified feeding or eating disorder (11.1%; 3/27), specific phobia (11.1%; 3/27), excoriation/skin-picking disorder (7.4%; 2/27), panic disorder (7.4%; 2/27), hoarding disorder (7.4%/ 2/27), bulimia nervosa (3.7%; 1/27), illness anxiety disorder (3.7%; 1/27), BDD (3.7%; 1/27), depressive disorder NOS (3.7%; 1/27), and trichotillomania (3.7%; 1/27). Participants' past diagnoses (i.e., prior to attending the OCDI), included (but were not restricted to) alcohol abuse, eating disorder NOS, major depressive disorder, specific phobia, anorexia nervosa, excoriation/skin-picking disorder, stimulant use disorder, etc. Finally, for these 27 patients for which comorbidity information was available, no patient endorsed autism spectrum disorder (i.e., on a self-reported diagnosis checklist).

Participation was restricted to those aged between 18 and 65 years old (M = 26.93, SD = 6.74, range = 18–43), and who were proficient in English. Seventy-six percent (22/29) of participants were female and 21% (6/29) were male (one participant did not provide consent for their demographic data to be shown).

### Procedure

Harvard University's Committee on the Use of Human Subjects approved the study protocol and McLean Hospital's Institutional Review Board formally ceded review to Harvard's committee. Participants gave written informed consent prior to initiation of any study procedure and received monetary compensation (\$20) for their time.

The participant sat behind a desk with both hands resting on it. A vertical cardboard barrier (sagittal partition) was placed on the table, just to the left of the participant's right hand, occluding his view of his right hand. A rubber hand was placed on the left side of the cardboard. A sheet of cloth was wrapped around the wrist of the dummy extending up to the shoulder of the right arm. This arrangement prevented the participant from viewing his right hand, giving the illusion that the fake hand was his real right hand. The rubber hand was positioned in parallel to (i.e., mirrored) the real left hand. The palm of the left hand was facing down, and the left arm was positioned in an approximately 90◦ angle along the body with the elbow near the torso and the forearm resting on the table. The participant's right arm was slightly extended, with the elbow slightly away from the torso and shoulder raised, allowing for the proper placement of the partition, i.e., extending from the right collarbone onto the desk. The real right forearm and hand (palm down) likewise rested on the desk, as noted, completely out of sight during the entire stimulation period.

Next, the participant was instructed to indicate orally when he or she experienced touch sensations coming from the rubber hand (this onset rating was only reported if the participant felt the illusion; the participant was not further asked about the illusion onset). The experimenter then began to stroke the participant's right hand (i.e., dorsum, with slight fluctuation in speed and directionality) with a paintbrush while simultaneously and synchronously stroking the rubber hand with another paintbrush continuously for 10 min (i.e., without interruption to sustain the illusion). The simultaneous stroking of the rubber hand and the real right hand produces the illusion (to the participant) that the rubber one feels like his own right hand. After 5 min of such stroking, the experimenter asked the participant to rate how much the rubber hand felt like his own hand on a 20-point Likert scale (this was the only time point at which the illusion intensity was assessed). Next, the experimenter used a tissue to smear the disgust stimulus (fake feces) on the rubber hand while simultaneously dabbing a damp paper towel from a nearby water bowl on the participant's real right hand. The damp towel placed on the occluded right hand served the purpose of mimicking the sensation of having the contaminant smeared on the participant's real hand (see also Jalal et al., 2015). Immediately thereafter, the participant was asked to provide subjective contamination ratings (i.e., disgust, anxiety, and handwashing urge levels), and the experimenter rated the participant's facial expression of disgust (either present or not). The tissue that had been used to ''contaminate'' the rubber hand and the clean paper towel was then removed from the fake and real hand; the fake feces remained on the rubber hand. The rubber hand and the participant's real hand continued to be stroked for an additional 5 min, after which the participant again provided contamination ratings and the experimenter rated his facial expression. The stroking of the rubber hand and real hand then stopped (i.e., 10 min of uninterrupted rubber hand and real hand stimulation had elapsed). Immediately thereafter, the experimenter told the participant that he would place the disgust stimulus (referred to as the ''object'') on his right hand and, accordingly, took a piece of the disgust stimulus and put it on the participant's real right hand. At this point, the participant provided a final set of contamination ratings.

A second group of patients underwent the same procedure except that the stimulation of the rubber hand and real right hand was asynchronous (i.e., the stroking was temporally and spatially incongruent), thereby either greatly diminishing or preventing the illusion from developing (The setup of the experiment is shown in **Figure 1**).

### Materials and Measures

### Yale-Brown Obsessive-Compulsive Scale (Y-BOCS)

The Y-BOCS (Goodman et al., 1989) is widely considered the ''gold standard'' measure for assessing OCD symptomatology in clinical research. The Y-BOCS indexes severity of obsessions and

compulsions in the past week. Scores are generated from a total of 10 items, each rated on a five-point Likert scale, and scores range from 0 to 40. In the present study, patients completed the self-report version of the Y-BOCS (Steketee et al., 1996).

### Disgust Stimulus

The disgust stimulus visually resembled and smelled of genuine feces. It consisted of food items (a mixture of chocolate and peanut butter) and was sprayed with a joke-shop odor, and placed in a bedpan. Participants were told before the study began that the stimulus was not genuine feces (**Figure 2**).

#### Multisensory Integration

RHI onset and intensity: the time onset of the RHI (i.e., how soon after the stroking was initiated participants felt the presence of the illusion, if at all) constituted a measure of multisensory integration. Participants were asked to indicate verbally if and when they experienced touch sensations coming from the rubber hand.

The perceived intensity of the illusion provided another measure of multisensory processing (i.e., limb ownership). Participants were asked to rate how much the rubber hand felt like their own hand (5 min after initiating the stroking), on a 20-point Likert scale, ranging from 1 (''not at all'') to 20 (''exactly like my own hand''). A more rapid onset (measured in seconds) and higher intensity rating indicated greater susceptibility to the illusion.

FIGURE 2 | Disgust stimulus.

### RHI Contamination

Participants were asked to provide ratings of contamination sensations (i.e., their level of disgust, anxiety, and handwashing urges), when the rubber hand was first contaminated (i.e., 5 min after initiating the stroking), on a 10-point Likert scale ranging from 1 (''not at all'') to 10 (''extremely''). Higher ratings indicated greater assimilation of contamination sensations into their body image via the RHI.

### RHI Habituation

Participants were asked to provide contamination ratings (i.e., disgust, anxiety, and handwashing urge levels), 5 min after the dummy contamination procedure (i.e., 10 min after initiating the stroking), on a 10-point Likert scale ranging from 1 (''not at all'') to 10 (''extremely''). Lower ratings indicated greater habituation.

Disgust facial expressions: to further gauge participants' disgust reactions, we observed and noted whether their facial expression indicated disgust (or not) when: (1) the rubber hand was initially contaminated; and (2) when RHI habituation assessment took place (i.e., 5 min after the dummy contamination).

### In vivo Exposure Habituation

Participants were asked to provide contamination ratings (i.e., disgust, anxiety and handwashing urge levels), when the experimenter contaminated the participant's real hand (i.e., immediately after RHI habituation ratings were obtained), on a 10-point Likert scale ranging from 1 (''not at all'') to 10

(''extremely''). Lower ratings indicated greater habituation, an overview of the experimental procedures is shown in **Figure 3**.

### Statistical Analyses

The study included a quantitative between-subject crosssectional design comparing two conditions (experimental vs. control) on the following measures: RHI contamination sensations, RHI habituation, in vivo exposure habituation, and multisensory integration, focusing on the betweensubject effects. The study targeted the following primary outcome variables: self-reported ratings of disgust, anxiety, and handwashing urges (assessment of RHI contamination sensations and habituation effects), and RHI onset and intensity (assessment of multisensory integration). Participants' facial expression of disgust (i.e., present or non-present; rated by the experimenter) constituted a secondary outcome measure of RHI contamination sensations and habituation.

RHI onset and intensity-dependent variables were analyzed via one-way ANOVA. Disgust, anxiety, and handwashing urge rating dependent variables were analyzed using a one-way MANOVA test, followed up with ANOVA post hoc tests. A chi-squared test was used to analyze disgust facial expression dependent variables.

For all analyses testing a priori hypotheses, we applied the Benjamini and Hochberg (1995) false discovery rate (FDR; e.g., McDonald, 2014) to control for potential Type I errors. Congruent with related studies (e.g., Skandali et al., 2018) and general guidelines (e.g., Genovese et al., 2002), the FDR was set at q < 0.15. In the current study, the Benjamini–Hochberg corrected significance level was 0.06. P-values shown in the text are uncorrected (i.e., raw; e.g., McDonald, 2014). Exploratory analyses and post hoc tests (i.e., following a significant omnibus MANOVA) were not adjusted for multiple comparisons. Multiplicity correction is not required when analyses are labeled exploratory (Bender and Lange, 2001).

For all dependent variables, the distribution of residuals was checked with Q–Q plots and the Shapiro–Wilk test; residuals were often found to depart from normality. Such variables were transformed with a log10(x + 1) and a square-root transformation to test whether these improved matters (Myers and Well, 2003). As the F-test is robust to minor normality departures (Blanca et al., 2017), we report untransformed data (except when otherwise specified in the text; on all figures, error bars denote standard error of the mean).

## RESULTS

Twenty-nine OCD patients completed the study. Of these, 16 were assigned to the experimental condition (i.e., to undergo the RHI) and 13 were assigned to the control (i.e., to undergo asynchronous stroking of the real and rubber hand). One OCD patient failed to provide consent for their demographic and Y-BOCS data to be used; these were thus excluded. The final sample sizes were experimental condition n = 16 and control condition n = 13.

Additional data were missing for a few measures. Three participants did not provide an illusion time onset. One participant's data were excluded from the ''RHI contamination and habituation'' analyses due to an experimental error. Likewise, a participant was excluded from these analyses for not exhibiting an adequate contamination fear response throughout the experiment [e.g., with average contamination ratings as low as 1.3 out of 10 in intensity when directly exposed to the disgust stimulus during in vivo exposure; for a third participant, the tissues used to stimulate the real hand and contaminate the dummy were not removed after this experimental procedure. As this protocol deviation was trivial (i.e., unlikely to impact contamination sensations), the data were not excluded. As a precaution, the data were also analyzed while excluding this participant; the results remained unaltered]. For demographic and clinical characteristics of participants, see **Table 1**.

### Multisensory Integration in OCD

RHI survival rate: all participants in the experimental condition (n = 16) reported a robust RHI effect; except one participant who did not provide an illusion onset, but rated the illusion as 5 out of 20 in intensity, which suggested he had a diminished RHI (based on our previous cutoff where an intensity rating of less than 3 out of 20 indicates no illusion; see Jalal et al., 2015). Surprisingly, all patients in the control condition (n = 13) also reported the



<sup>a</sup>M, mean; SD, standard deviation; n, sample size; F, F statistic; χ 2 , chi-square statistic; df, degrees of freedom; p, p-value; NS, non-significant; Y-BOCS, Yale-Brown Obsessive-Compulsive Scale; (†) one participant did not provide consent for their demographic and Y-BOCS data to be shown. <sup>∗</sup>Log10(x + 1) transformed Y-BOCS scores. These analyses were also conducted without the two participants excluded from the RHI contamination and habituation analyses (described above); the results remained unaltered: Age (F(1,24) < 1, NS), Y-BOCS (F(1,24) = 1.32, p = 0.26), and Sex (χ 2 <sup>1</sup> < 1, NS).

RHI, except one who scored 2 out of 20 in intensity (another participant had a borderline illusion with an intensity rating of 5). Thus, the presence of the RHI did not differ in the two conditions (χ 2 <sup>1</sup> = 1.27, p = 0.26).

Illusion onset: on average, participants in the experimental condition reported experiencing the illusion after 65.50 s (SD = 68.16) vs. 57.42 s (SD = 51.16) in the control condition (experimental n = 14, control n = 12). A one-way ANOVA was conducted on the illusion onset dependent variable [i.e., log10(x + 1) transformed scores] to compare ratings in the experimental condition and control condition. The onset of the illusion did not differ in the two conditions (F(1,24) < 1, NS; see **Figure 4**).

Illusion intensity: a one-way ANOVA was conducted on the illusion intensity-dependent variable to compare ratings in the experimental condition and control condition (experimental n = 16, control n = 13). The intensity of the illusion did not differ in the two conditions (F(1,27) < 1, NS; see **Figure 5**).

OCD symptoms and RHI onset and intensity: an exploratory Pearson's Correlation Test showed that OCD symptom severity was not associated with how soon participants experienced the RHI [i.e., log10(x + 1) transformed Y-BOCS and onset scores; r<sup>11</sup> = −0.16, p = 0.61, two-tailed], in the experimental condition; similarly, such symptom severity was not associated with the strength of the illusion (r<sup>13</sup> = 0.12, p = 0.67, two-tailed). However, in the control condition, while OCD symptom severity was not associated with the illusion onset (r<sup>10</sup> = 0.15, p = 0.64, two-

tailed), Y-BOCS scores inversely correlated with the intensity of the illusion (r<sup>11</sup> = −0.73, p = 0.004, two-tailed).

### RHI Contamination ("Fake Hand Exposure")

To examine contamination sensations when the fake hand was contaminated, we conducted a one-way MANOVA on the dependent variables (experimental n = 14, control n = 13). Contamination sensations (disgust, anxiety, and handwashing urges) did not differ in the two conditions when the fake hand was contaminated (F(3,23) < 1, NS, Benjamini–Hochberg corrected; see **Figure 6**). The proportion of participants in the experimental condition and control condition who exhibited a facial expression of disgust when the fake hand was contaminated did not differ (experimental n = 14, control n = 13; χ 2 <sup>1</sup> < 1, NS, Benjamini–Hochberg corrected).

### RHI Habituation

To examine habituation 5 min after the fake hand was contaminated, we conducted a one-way MANOVA (experimental n = 14, control n = 13) that revealed that contamination sensations (disgust, anxiety, and handwashing urges) did not differ in the two conditions (F(3,23) = 1.22, p = 0.32, Benjamini–Hochberg corrected; see **Figure 7**). The proportion of participants who exhibited a facial expression of disgust was higher in the experimental condition vs. the control condition

(experimental n = 13, control n = 13; 64.7% vs. 35.3%; χ 2 <sup>1</sup> = 4.25, p = 0.04, Benjamini–Hochberg corrected).

### In vivo Exposure Habituation

To examine in vivo exposure habituation immediately upon discontinuing the stimulation of the real and rubber hand, we conducted a one-way MANOVA (experimental n = 14, control n = 13), showing that participants in the experimental condition reported higher overall contamination sensations (disgust, anxiety, and handwashing urges) compared to those in the control condition (F(3,23) = 3.12, p = 0.046, Benjamini–Hochberg corrected; see **Figure 8**). The MANOVA was followed up with a discriminant function analysis that revealed one discriminant function, which significantly differentiated the experimental and control condition (Wilks' lambda λ = 0.71, χ 2 <sup>3</sup> = 8.02, p = 0.046). A canonical correlation of 0.54 showed that the model explained 29.2% of the variation in the condition variable. The discriminant function analysis revealed that disgust ratings had the highest standardized canonical discriminant function coefficient (β = 2.40) indicating the greatest contribution to the model (i.e., the best discriminator between the two conditions), followed by anxiety (β = −1.80) and then washing urge ratings (β = −0.04).

### Dummy Exposure vs. in vivo Exposure

In an exploratory analysis, to compare contamination sensations during dummy exposure vs. in vivo exposure, we conducted two repeated measures one-way MANOVAs (experimental n = 14, control n = 13), showing that while in vivo exposure provoked more intense responses than dummy exposure in the experimental condition (F(3,11) = 3.92, p = 0.04), this was not the case in the control condition (F(3,10) < 1, NS; residuals showed moderate deviation from normality but were not improved with a log or square-root transformation and were thus analyzed with those caveats). Follow-up one-way ANOVAs showed that in the experimental condition in vivo contamination triggered marginally significantly greater disgust (F(1,13) = 3.84, p = 0.07) and significantly greater anxiety (F(1,13) = 7.60, p = 0.02) and handwashing urges (F(1,13) = 8.81, p = 0.01) than dummy exposure.

### DISCUSSION

This study yields important new findings with clinical implications. Intriguingly, our results suggest sensory assimilation of contamination sensations into the body image via the RHI—that such feelings were curiously referred to an alien hand in patients with OCD. Patients undergoing synchronous stimulation did not report greater contamination sensations when the fake hand was initially contaminated relative to asynchronous stroking. But contrary to expectations, they did so after the dummy had been contaminated for 5 min, as assessed via disgust facial expressions (a secondary outcome) and in vivo exposure (upon discontinuing the illusion). We also found that patients failed to reject the illusion during the ''gold standard'' control condition. To our surprise, synchronous and asynchronous stroking induced an equally vivid and

fast-emerging illusion, which helps explain why both conditions initially (5 min after initiating tactile stimulation) provoked contamination reactions of equal magnitude. This study is the first to suggest heightened malleability of body image in OCD. Collectively, these results argue against a sharply localized (''hierarchical'') approach to brain function and illustrate dynamic intersensory interactions and plasticity of brain modules (''holistic mediation'').

Our findings stress the importance of the temporal dimensions of the RHI, and crucially, how these can be perturbed by psychopathology. As noted, our chosen duration of tactile stimulation (i.e., 5 min) prior to dummy contamination was insufficient to initially differentiate the synchronous and asynchronous conditions in patients with severe OCD. By comparison, we have previously shown that 5 min of tactile stimulation differentiates the RHI and the control condition in healthy individuals (Jalal et al., 2015). In the current study, indeed, as both methods of stroking triggered an equally intense illusion at this time point, one would expect them to provoke comparable contamination reactions. But over time, these results suggest that synchronous stimulation more effectively assimilated the visibly contaminated rubber hand into the body image (than asynchronous stroking)—accounting for the relative rise in contamination sensations. Although we did not explicitly assess illusion intensity at a later stage, this provides a viable explanation for why synchronous stroking differentially impacted contamination reactions 10 min after initiating stimulation on two separate measures. As mentioned, research suggests that the RHI becomes more intense with time (i.e., duration of stimulation), as indexed on a key measure of the illusion (i.e., perceiving one's real hand drifting towards the fake one; Tsakiris and Haggard, 2005; on the prevalence of the RHI over time and degree of proprioceptive drift, see also Botvinick and Cohen, 1998).

The formulation of the initial hypothesis that contaminating the fake hand during the RHI results in greater contamination sensations than does asynchronous stroking in OCD, specifically 5 min after beginning the stroking, was based on prior work in healthy volunteers (Jalal et al., 2015; see also Nitta et al., 2018). Evidently, in this study, as the RHI triggered greater contamination reactions than did the control procedure, not 5 min but instead 10 min after stroking began (consistent with the overall hypothesis, but not the timeline in which the two conditions were differentiated), our study design was unable to capture any habituation effects. Nevertheless, given the literature on ERP (e.g., Foa et al., 1983; Rachman, 2004; Abramowitz, 2006; McKay, 2006; i.e., the basis for the second hypothesis), we can safely assume that such fake hand exposure would eventually lead to habituation (i.e., causes a gradual decrease in these sensations as extinction occurs). As our exposure method proved highly potent at evoking contamination reactions (surprisingly, irrespective of stroking approach), it may be that akin to ERP, at least 30–45 min of continuous exposure is needed for habituation to occur, bearing in mind

that patients vary in the rate of habituation (e.g., Simpson et al., 2010). Future research should further disentangle such habituation timeline.

That a higher proportion of patients exhibited a disgust facial expression during the RHI relative to the control condition (65% vs. 35%; i.e., 5 min after the fake hand was contaminated, 10 min after stroking began) is consistent with the key role of disgust in OCD (Ludvik et al., 2015), e.g., as a strong predictor of contamination fears (e.g., Olatunji et al., 2005; see also Deacon and Olatunji, 2007; Olatunji et al., 2007). This measure provides an objective assessment of disgust.

The results of the exploratory analysis are noteworthy. They emphasize the overall finding that synchronous stroking over time exerts selective sensitizing effects (i.e., vis-à-vis contamination reactions). But more strikingly, they imply that ''fake hand exposure'' during asynchronous stroking provokes contamination sensations as effectively as actual real hand exposure. This finding is highly counterintuitive. It dovetails with our related studies showing that both college students with OCD symptoms (Jalal and Ramachandran, 2017) and severe OCD patients (Jalal et al., under review) report indistinguishable levels of disgust when merely watching an experimenter contaminating his own hand and when their hand is contaminated. This research illustrates the cognitive impenetrability of contamination sensations (i.e., how such gut reactions can override logic and break down ''self-other'' barriers). Intriguingly, they also suggest that direct skin contamination may be unnecessary to gain the beneficial effects of exposure therapy. Contaminating proxy stimuli such as alien limbs (synthetic or biological) can potentially trigger clinically relevant contamination reactions (see also, Jalal et al., 2018).

In this study, we found an overall amplified RHI. For instance, all patients reported the illusion during synchronous stroking. In contrast, around 85% of healthy volunteers experience the effect (Jalal et al., 2015). But the finding that patients failed to reject the RHI during asynchronous stroking is more notable. It mirrors research showing that both Parkinson's disease and schizophrenia patients exhibit heightened illusory effects during asynchronous stroking compared to healthy volunteers (Peled et al., 2000; Ding et al., 2017), and that dopamine releaser drugs ketamine and dexamphetamine enhance the RHI during both synchronous and asynchronous stimulation (Albrecht et al., 2011; Morgan et al., 2011). Taken together, these data indicate that dopamine dysregulation may boost a sense of embodiment. As noted, although the role of dopamine in OCD is admittedly complex (Fineberg et al., 2007), research has shown that dopamine antagonists can be useful in reducing OCD symptoms (as an adjunct to SSRIs; Vulink et al., 2009) and that dopamine agonists can generate OCD-like behaviors [Borcherding et al., 1990; Szechtman et al., 1998; of interest, ketamine per se shows affinity for dopamine D<sup>2</sup> in addition to serotonin 5-HT<sup>2</sup> receptors (Kapur and Seeman, 2002) both blocked by quetiapine, an antipsychotic sometimes used in the treatment of refractory OCD (Gefvert et al., 2001)].

Notably, dopamine has been linked to learning (e.g., Centonze et al., 2001; Castner and Williams, 2007) and is found in brain areas underlying the RHI (Ehrsson et al., 2004; on dopaminergic projections to the prefrontal cortex, see Goldman-Rakic et al., 1990). It could, therefore, contribute to perceptual learning processes mediating corporeal awareness and possibly account for an amplified illusion in OCD. But how does dopamine induce the RHI in the face of contradictory input (i.e., asynchronous stimulation)? One explanation is that dopamine overactivity underlies salience attribution: ascribing causal importance to salient events (e.g., Howes et al., 2015). In the asynchronous condition, the patient focuses his attention on a dummy that resembles the patient's hand and it appears in its expected location. This attention-grabbing input violates expectations, rendering the event highly salient. As such, learning (''dopamine-encoding'') might ensue, i.e., driving the illusion of ownership (''the fake hand on the table must be mine'') even when incoming sensory information is incongruous, effectively overriding internally constructed models of reality (Albrecht et al., 2011; on Bayesian prediction error, see Fletcher and Frith, 2009). Together, these findings stress how a unified sense of self may rest on a delicate balance between top-down regulation and bottom-up processes.

Another noteworthy factor to consider is the idiosyncratic perceptual style in OCD, possibly exacerbating such dopamine-driven top-down visual processing and salience misattribution. As early as the 1960s, Shapiro described the obsessive-compulsive attentional style: a painstaking focus on minor details in a rigid manner, at the expense of the big picture—effectively ''missing the forest for the trees'' (Shapiro, 1965; see also Yovel et al., 2005). Research has since shown that patients with OCD indeed focus on local aspects of visual stimuli instead of holistic, organizational features (Savage et al., 1999); i.e., in line with neurocognitive models implicating frontal–striatal abnormalities in OCD (e.g., mediating cognitive inflexibility circuits; Vaghi et al., 2017; for reviews, see Menzies et al., 2008; Nakao et al., 2014) [Similar tendencies occur in students with OCD symptoms (Soref et al., 2008) and individuals with obsessive-compulsive personality disorder (Yovel et al., 2005)]. Accordingly, OCD patients in the asynchronous condition when asked to focus on the fake hand did so in an intensely focused and inflexible manner, conceivably causing them to ignore the overall conflicting sensory information, i.e., leading to global degradation in multisensory integration and overreliance on the salient visual input. This explanation dovetails with the finding that patients with the etiologically related OCD spectrum (''fronto-stratial'') disorder BDD (Grace et al., 2017) display proprioceptive drift bias towards the fake hand during both synchronous and asynchronous stimulation. Unsurprisingly, patients with BDD, like those with OCD, focus on perceptual details at the cost of global, holistic processing (Deckersbach et al., 2000), fittingly evoked as an explanation for such unusual proprioceptive drift bias in BDD (Kaplan et al., 2014).

Counterintuitively, Y-BOCS scores inversely correlated with the intensity of the illusion but only during asynchronous stimulation. One explanation for this is that top-down attention, possibly driving the illusion during asynchronous stroking (via salience misattribution), was perturbed by anxiety states in the most severe patients. Indeed, anxiety decreases attentional control (Eysenck et al., 2007) and is unsurprisingly associated with OCD symptoms (e.g., Foa et al., 1998). Anxiety overall may, therefore, have interfered with perceptual learning effects of dopamine (caused ''general blunting''), which might explain why OCD severity (irrespective of condition) did not intensify the illusion.

The primary aim of this study was to explore the therapeutic potential of the RHI. Our findings may pave the way for a novel therapeutic technique for OCD (see also Jalal et al., 2015). Practically (e.g., based on the current results), such an approach might entail 10 min of tactile stimulation, coupled with at least 5 min of continuous dummy contamination (as outlined in the ''Materials and Methods'' section). The procedure should be repeated (e.g., 3–4 times) until habituation occurs; for severe patients, possibly starting with asynchronous stroking followed by synchronous for a more immersive experience [Analogously, a session of ERP typically lasts around 90 min (van der Heiden et al., 2016)].

This method we have introduced may offer a tolerable alternative to ERP, with potential to trigger clinically relevant contamination reactions. Crucially, unlike ERP, it does not require patients to touch highly aversive ''contaminants.'' As such, it is conceivable that patients who are reluctant to engage in ERP due to fear of direct skin exposure (i.e., too frightened to confront contaminants head-on) would be more accepting of this approach. Also, as noted, it might be useful during the initial stages of exposure to help desensitize patients such that they are willing to eventually undertake ERP.

Because the RHI itself is engaging—fittingly labeled a ''mindblowing party trick'' (Lawton, 2009)—our method might appeal to a younger audience. During pilot work, volunteers often express astonishment (sometimes even slight giggling) at the uncanny sensation of touch arising from an obvious fake hand. This element of amusement (positive affect) could establish a frame for a less fearful outlook on exposure, i.e., create nonthreatening re-association to bodily contamination. All in all, this simple, immersive, and cost-effective intervention might result in higher treatment uptake and lower dropout and facilitate early intervention. It is eminently suitable for poorly resourced and emergency settings, including low-income and developing countries with minimal access to high-tech solutions like virtual reality.

Although this is the first investigation to explore the RHI in OCD, our assessment of multisensory integration per se (a secondary study-aim) was limited in several ways. For instance, we did not take into account the impact of comorbid psychiatric conditions that may have affected these results. Indeed, as noted, psychiatric disorders have been shown to differentially influence self-referential processing. Ideally, future studies should explore corporeal awareness in OCD using large samples of unmedicated patients without comorbidities (albeit severe OCD patients without comorbidities are rare). This is particularly important because of the role of dopamine as a modulator of multisensory integration, with dopaminergic agents sometimes used as an adjunct to SSRIs in the treatment of OCD.

In this study, we assessed the RHI with a subjective intensity measure in addition to the onset rating. Although a single-item intensity measure is limited compared to embodiment questionnaires (Botvinick and Cohen, 1998; Peled et al., 2000), it has proven reliable in studies by us (Jalal et al., 2015; see also Armel and Ramachandran, 2003) and others (Lev-Ari et al., 2015; Lev-Ari and Hirschmann, 2016; Nitta et al., 2018). Future research examining multisensory integration in OCD should include additional measures such as questionnaires and the objective ''proprioceptive drift'' test (e.g., Botvinick and Cohen, 1998; Tsakiris and Haggard, 2005; Longo et al., 2008; Marotta et al., 2016). Although we have provided evidence visà-vis multisensory processing in healthy volunteers from our previous research (i.e., serving as a comparison to the current findings; Jalal et al., 2015), future studies should include a healthy control group; that we did not include one constitutes a limitation.

Consistent with Botvinick and Cohen (1998) seminal investigation, in the current study, the asynchronous control condition was a between-subject factor. Using the same sample across conditions would have been optimal for assessing self-referential processing (i.e., due to reduced variance arising from individual differences, e.g., sensory suggestibility, see Marotta et al., 2016). However, the key aim of this study was to explore the clinical potential of the RHI in OCD (specifically in severe patients undergoing IRT, often refractory to treatment in other settings). As such, our design ensured that patients were not subjected to high stress by being exposed to aversive contaminants twice (while present at our treatment center for a limited period), and, importantly also, prevented carry-over effects from the exposure procedures (e.g., habituation). Indeed, with our main clinical objective in mind, our sample was suitable for the following reasons: (1) comorbid and secondary diagnoses are common in OCD patients, who often tend to be medicated. Thus, our sample was typical of this patient population; (2) severe OCD patients may be the most fearful of ERP (i.e., entailing direct contamination) and thus generally the most in need of gentler, more tolerable treatments.

### REFERENCES


Future double-blind placebo-controlled trials should directly compare our proposed ''dummy contamination'' procedure to ERP. Finally, ''multisensory stimulation therapy'' lends itself to other applications in psychiatry (Jalal et al., 2015)—like treating ''needle phobia.'' Conducting realistic exposures in this population is challenging: repeated needle injections into a real arm could result in punctured veins. Using a fake hand during the RHI, instead, may provide a clever and convenient alternative.

### DATA AVAILABILITY STATEMENT

The raw data will be made available by the authors to any qualified researcher upon request.

### ETHICS STATEMENT

The studies involving human participants were reviewed and approved by Harvard University. The patients/participants provided their written informed consent to participate in this study.

### AUTHOR CONTRIBUTIONS

BJ, RM, JE, and VR designed the study. BJ performed the data analysis and wrote the manuscript. RM contributed to data interpretation and edited the manuscript. JE and SP recruited and tested patients, and edited the manuscript. VR edited the manuscript. All authors read and approved the submitted version.

### FUNDING

The study was supported by discretionary funds awarded to VR. BJ was supported by Trinity College Cambridge, the Cambridge School of Clinical Medicine, and the Psychology Postgraduate Affairs Group.

### ACKNOWLEDGMENTS

We thank research assistants at Harvard University and McLean Hospital's OCDI for their remarkable assistance with recruitment and data collection, including Lauryn Garner, Kara Kelley, Eric Tifft, Mikaela Ingram, Zahra Dabirzadeh, and Arthur Grayson III.


improvement in contamination/washing symptoms in obsessive-compulsive disorder. J. Obsessive Compuls. Relat. Disord. 4, 20–24. doi: 10.1016/j.jocrd. 2014.11.001


**Conflict of Interest**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2020 Jalal, McNally, Elias, Potluri and Ramachandran. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Exploring Self-Paced Embodiable Neurofeedback for Post-stroke Motor Rehabilitation

Nadine Spychala<sup>1</sup> , Stefan Debener <sup>1</sup> , Edith Bongartz <sup>1</sup> , Helge H. O. Müller <sup>2</sup> , Jeremy D. Thorne<sup>1</sup> , Alexandra Philipsen<sup>2</sup> and Niclas Braun1,2 \*

<sup>1</sup>Neuropsychology Lab, Department of Psychology, University of Oldenburg, Oldenburg, Germany, <sup>2</sup>Department of Psychiatry and Psychotherapy, University of Bonn, Bonn, Germany

Neurofeedback-guided motor-imagery training (NF-MIT) has been proposed as a promising intervention following upper limb motor impairment. In this intervention, paretic stroke patients receive online feedback about their brain activity while conducting a motor-imagery (MI) task with the paretic limb. Typically, the feedback provided in NF-MIT protocols is an abstract visual signal based on a fixed trial. Here we developed a self-paced NF-MIT paradigm with an embodiable feedback signal (EFS), which was designed to resemble the content of the mental act as closely as possible. To this end, the feedback was delivered via an embodiable, anthropomorphic robotic hand (RH), which was integrated into a closed-looped EEG-based brain-computer interface (BCI). Whenever the BCI identified a new instance of a hand-flexion or hand-extension imagination by the participant, the RH carried out the corresponding movement with minimum delay. Nine stroke patients and nine healthy participants were instructed to control RH movements as accurately as possible, using mental activity alone. We evaluated the general feasibility of our paradigm on electrophysiological, subjective and performance levels. Regarding electrophysiological measures, individuals showed the predicted event-related desynchronization (ERD) patterns over sensorimotor brain areas. On the subjective level, we found that most individuals integrated the RH into their body scheme. With respect to RH control, none of our participants achieved a high level of control, but most managed to control the RH actions to some degree. Importantly, patients and controls achieved similar performance levels. The results support the view that self-paced embodiable NF-MIT is feasible for stroke patients and can complement classical NF-MIT.

Keywords: neurofeedback, motor imagery, brain computer interface, sense of ownership, sense of agency, stroke, rubber hand illusion

## INTRODUCTION

Motor impairments in the upper limbs are among the most prevalent symptoms following stroke (Grefkes and Ward, 2014). Neurofeedback-guided motor imagery training (NF-MIT) has been proposed as a promising intervention for treating upper limb motor impairments (Pichiorri et al., 2015; Zich et al., 2017). In this intervention, paretic stroke patients conduct a motor imagery (MI)

#### Edited by:

Mariella Pazzaglia, Sapienza University of Rome, Italy

#### Reviewed by:

J. Ignacio Serrano, Spanish National Research Council (CSIC), Spain Giulia Galli, Santa Lucia Foundation (IRCCS), Italy Yongtian He, University of Houston, United States

> \*Correspondence: Niclas Braun niclas.braun@uni-oldenburg.de

#### Specialty section:

This article was submitted to Brain-Computer Interfaces, a section of the journal Frontiers in Human Neuroscience

> Received: 08 March 2019 Accepted: 16 December 2019 Published: 20 January 2020

#### Citation:

Spychala N, Debener S, Bongartz E, Müller HHO, Thorne JD, Philipsen A and Braun N (2020) Exploring Self-Paced Embodiable Neurofeedback for Post-stroke Motor Rehabilitation. Front. Hum. Neurosci. 13:461. doi: 10.3389/fnhum.2019.00461

**223**

task during which they receive online neurofeedback about their brain activity. The therapeutic idea behind NF-MIT is to provide feedback to the patients on how well they are performing, by showing a beneficial neuronal activation pattern thought to support motor recovery (Sitaram et al., 2017). Typically, the feedback is a rather abstract signal, such as a moving bar or ball presented on a computer screen. In the context of an embodied cognition view (Wilson, 2002; Foglia and Wilson, 2013; Wilson and Golonka, 2013), it can be argued that an embodiable feedback signal (EFS) is more natural and intuitive for the patient. A feedback signal that closely resembles the MI act performed, in both time and space, may be potentially better accepted by patients, in particular if they suffer from cognitive impairments, and may, eventually, lead to better performance.

A few studies have developed an EFS (Perez-Marcos et al., 2009; Alimardani et al., 2013; Ono et al., 2013; Pichiorri et al., 2015; Braun et al., 2016). Most of these studies were inspired by the active rubber hand illusion (aRHI; Kalckert and Ehrsson, 2012, 2014; Braun et al., 2014) or its VR-based derivatives (Slater et al., 2008, 2009; Sanchez-Vives et al., 2010; Kilteni et al., 2012; Ma and Hommel, 2013; Pichiorri et al., 2015; Ma et al., 2017). The aRHI is a special variant of the classical rubber hand illusion, in which a movable artificial robotic hand (RH), rather than a static rubber hand, is placed visibly, and in an anatomically-plausible position, in front of the individual, while the individual's own hand is hidden from view. If the RH is repeatedly moved in synchrony with the individual's real or imagined hand movements, an illusory sense of ownership (SoO) and sense of agency (SoA) for the artificial hand can typically be induced (Kalckert and Ehrsson, 2012, 2014; Braun et al., 2014, 2016). That is, individuals may then experience the RH as part of their own body (SoO) and its movements as under their voluntary movement control (SoA). In order to provide real-time feedback for the MI within an aRHI paradigm, imagined hand movements are decoded from electrical brain activity and the corresponding commands are issued to the RH. Ideally, the RH then executes the imagined movements with little temporal delay (Braun et al., 2016).

Several studies have indicated beneficial effects of EFS (Perez-Marcos et al., 2009; Slater et al., 2009; Alimardani et al., 2013, 2014). Ono et al. (2013), for instance, compared different feedback signals presented on a computer screen and found evidence for a more robust event-related desynchronization (ERD) pattern for natural as compared to abstract feedback signals. Also, in a previous aRHI study (Braun et al., 2016), we used a RH and investigated the role of feedback-signal embodiment. Individuals experienced RH movements as more self-related and self-caused if the RH was placed in a congruent position such that it could be embodied. Individuals were also able to induce RH movements more quickly. These findings suggest that natural feedback signals may help to improve the quality of NF-MIT. It is not clear, however, to what extent stroke patients in particular benefit from an EFS, since most paradigms have been tested only with healthy individuals. A recent meta-analysis conducted by Cervera et al. (2018) showed that brain-computer interface (BCI)-based neurorehabilitation on upper-limb motor function can lead to more improvement in motor performance than other conventional therapies, supporting the general effectiveness of classical NF-MIT in stroke patients. The specific effect of EFS, however, has not yet been studied.

Most NF-MIT paradigms employ cue-based designs, in which the timing and content of mental tasks are predefined by visual or auditory cues (Scherer et al., 2008). Such rigid training regimes are easy to control experimentally, but they suffer from poor ecological validity, in particular when voluntary actions are studied. Spontaneous, self-paced designs may be needed to allow individuals to develop self-control and increased acceptance (Lotte et al., 2013).

In the present study, we investigated whether a self-paced embodiable NF-MIT, in which the individuals can freely explore the consequences of their different MI acts on a RH, is feasible. We tested this RH neurofeedback paradigm with stroke patients (n = 9), since this is the major target group for which NF-MIT is ultimately intended, as well as with healthy participants matched in age and gender. We wanted to know whether stroke patients, in particular, are able to learn to control the RH. To evaluate neurofeedback performances, participants had to perform various tasks such as conducting as many RH movements as possible in some time periods and withholding any RH movements in others. Behavioral, subjective and electrophysiological measures were collected and analyzed.

### MATERIALS AND METHODS

### Participants

Nine chronic stroke patients (one female) aged 55–75 and nine healthy controls matched in age and sex were recruited for the study (see **Table 1**, for demographic and clinical data). All participants were required to have a normal or corrected-tonormal vision and no known history of a mental disorder. All patients suffered from a first-time monolateral stroke. Months since stroke ranged from 15 to 72 months, M = 41.44, SD = 22.6. Inclusion criterion was a moderate to severe right-hand paresis due to the stroke as assessed by the Fugl-Meyer test (see ''Stroke Patient Assessment'' section). Patients were required to have no dementia, no epileptic seizures, and no severe neglect or severe aphasia that would impair their ability to follow task instructions. All participants were compensated for their participation (8e per hour), gave written informed consent and were naive to the purpose of the study. The study was approved by the University of Oldenburg ethics committee.

TABLE 1 | Demographic data of stroke patients and healthy controls.


### Overview

The study was carried out over three sessions for stroke patients and two sessions for healthy controls. The additional session for the stroke patients served to conduct the cognitive, motor and sensory assessments. The other two sessions, in which the actual NF-MIT was conducted were identical for both groups. Here, we report NF-MIT data from the second of these two sessions. In the first session, participants had to kinesthetically imagine flexion/extension movements with both of their hands in spatiotemporal synchrony with the RHs flexion/extension movements, while in the second session, they imagined these movements with only their right hand. Thus, the first session implemented a different experimental task and will be reported elsewhere.

### Stroke Patient Assessment

Cognitive assessment was conducted using the current version of the Montreal cognitive assessment (MoCA; Nasreddine et al., 2005). This test covers different cognitive domains and is an established fast screening tool for detecting mild cognitive impairments. The MoCA score ranges from 0 to 30 and Germanspeaking participants with normal cognitive ability are expected to exceed a threshold between 18 and 24 (depending on age and gender; Thomann et al., 2018).

Motor assessment was carried out using an adapted version of the Fugl-Meyer test (Sanford et al., 1993). While the original Fugl-Meyer test assesses both upper and lower limb movements, we only focused on 29 upper-limb tasks in the present study. All movement tasks were first executed with the non-paretic and then with the paretic arm. For each task, the achieved motor performance scores of the paretic and the non-paretic arm were compared and their difference assessed on a 3-point Likert scale. The scale ranged from zero (clearly lower performance on the paretic side) to two (identical performance on the paretic and non-paretic side). A summation score was then calculated by adding up all 29 individual task scores. The maximum score achievable was thus 58. The cut-off criterion indicating mild to severe right-hand paresis was set to scores lower than, or equal to, 45.

The sensory assessment was based on a testing procedure adapted from the Nottingham Sensory Assessment (Lincoln et al., 1998). We tested six different sensory modalities (pressure, light touch, pain, temperature, proprioception and vibration) on three different upper limb locations (upper arm, forearm, back of the hand), while the patients' eyes were closed. For each modality and each position, sensibility performances of the paretic and the non-paretic body side were compared and assessed on a 3-point Likert scale, ranging from zero (no detection of respective sensory stimulus), to two (identical sensory detection performance on the paretic and the non-paretic side). A summation score was then calculated by adding up all 18 individual task scores. The maximum score achievable was thus 36.

### Apparatus

The NF-MIT took place in a sound-attenuated and dimly lit experiment room. The experimental setting was adapted from our previous study (Braun et al., 2016) and is depicted in **Figure 1A**. The participant sat in front of a rectangular table (50 × 60 cm) and placed the right hand into a black box. This box was upholstered on the inside, so as to allow for a comfortable placement, and covered both hand and lower arm, hiding them from view. The anthropomorphic RH was placed in an anatomically-congruent position next to the black box, such that it was positioned medially aside the hidden real right hand. The horizontal distance between the participant's real right hand and RH was around 7.5 cm. A green LED was placed in the middle of the table. Both the RH and the participant's right (unseen) hand were covered with a thin-gauge garden glove. A blanket covered shoulders and arm and the space between the RH and the participant's body. The aim of this was to facilitate the visual impression that the RH could be the participant's own hand. The RH closely resembled a typical large human hand in terms of shape and size, and could be controlled with Matlab R2012a (Mathworks, Natick, MA, USA) via a microcontroller (Arduino mega 2560). The RH realistically mimicked hand flexion and hand extension movements (for more details, see Braun et al., 2016). The time delay between a Matlab control command sent to the microcontroller and an actual RH movement onset was less than 200 ms. The NF-MIT itself consisted of two blocks, a training block, and a feedback block.

## Neurofeedback Motor-Imagery Training

### Training Block

The training block served to acquaint the participants with the MI task and to calibrate the classifiers used in the ensemble classification algorithm for the ensuing feedback block (see ''EEG Data Recording'' and ''EEG Data Analysis and Classifier Training'' sections for more detail). To enable a successful aRHI induction in the feedback block, RH movements were already included in the training. The training block lasted 12 min and was based on a fixed trial structure, and consisted of around 50 runs, each run lasting around 14 s and beginning with a 5 s rest period. During the initial rest period, the RH was in the open state and participants were instructed to relax and not to move. After that, a small LED indicated to the participants to prepare for the subsequent MI flexion trial, which began 300 ms after LED onset and lasted for 1.5 s. During this period, the RH flexed its fingers and the participants were required to concomitantly imagine the same movement with their right hand. The instruction was thus to kinesthetically imagine the same flexion movement, in spatiotemporal synchrony with the RH movement. The flexion period was finished by the offset of the LED and the participants were instructed to relax again for another 5 s. Then, the LED switched on again, preparing the participants for the extension trial, which began 300 ms after LED onset. The extension trial also lasted 1.5 s, during which the RH extended its fingers and participants were required to concomitantly imagine this extension movement with their right hand. The extension trial was finished by the offset of the LED and the next run began.

### Neurofeedback Block

During the neurofeedback block, the RH was connected to a BCI that was trained on the training block data (see ''EEG

FIGURE 1 | Study design. (A) Experimental apparatus. Participants placed their right hand into a black box, whereas the robotic hand (RH) was placed directly alongside in front of the participant. During the training block, participants kinesthetically imagined flexion and extension movements in spatio-temporal synchrony to the flexion and extension movements of the RH. (B) Brain-Computer Interface (BCI). For neurofeedback provision, the RH was connected to a BCI that was trained on the training block data and moved whenever the BCI detected the imagination of either a participant's flexion or extension. The BCI's classification algorithm was based on signal features of the 8–30 Hz sensorimotor rhythm (SMR). (C) Neurofeedback tasks. During the feedback block, participants attempted to control the RH's movements as accurately as possible, using their motor-imagery (MI) thoughts alone. To evaluate their achieved controllability over the RH, participants had to perform different "BCI Parcours" in which they attempted to carry out various computer-given, experimenter-given or self-given commands. Data privacy remark: The person shown on the figure agreed with the publication of this figure.

Data Recording'' section), and moved whenever the BCI's classification algorithm assumed an instance of flexion or extension imagination. The neurofeedback block consisted of an acquaintance phase and three neurofeedback tasks (**Figure 1C**), each of which lasted 4 min.

During the acquaintance phase, the participants had the opportunity to familiarize themselves with self-paced NF-MIT. That is, they were allowed to freely try different MI acts and to observe how these acts influenced the RH's movement behavior. The overall aim thereby was to gain as much control as possible over the RH's flexion, extension, and resting states, using MI thoughts alone. To help the participants to accomplish this aim, we suggested they try different mental strategies (e.g., ''If you want to flex the RH, try to imagine grasping something or recall the flexion imagination from the training phase'' or ''If you want to rest the RH, mentally count numbers''). Moreover, the participants were asked whether some threshold modifications within the classification algorithm needed to be done; for instance, whether they remained trapped within one RH state. The classifier thresholds of the logistic regression used in the ensemble classification algorithm were then adjusted according to which thresholds best enabled participants to control the RH using MI. That is, if for instance the RH opened more often than intended by the participant, we set the threshold of the respective classifier to a higher value, e.g., from 0.7 to 0.8 (see ''EEG Data Analysis and Classifier Training'' and ''Ensemble Classification Algorithm and Online Data Flow'' sections).

The first neurofeedback task was a ''rest vs. move task,'' during which the participants attempted to move the RH as often as possible during 30 s move phases and as little as possible during 30 s rest phases. The respective phases alternated eight times and were indicated by a small LED that switched on during move phases and switched off during rest phases.

The second neurofeedback task was a ''follow commands task,'' during which the participants attempted to carry out movement commands given by the experimenter. To this end, the experimenter sat next to the participants, equipped with a laptop, and time-marked each given command. The experimenter chose one from four different commands, three explicit commands and one implicit commands. The explicit commands were ''open,'' ''close'' and ''grasp,'' where grasp meant to open (extension) and immediately close (flexion) the RH. The implicit command was to keep the RH in its momentary resting state (i.e., either ''remaining opened'' or ''remaining closed''), as long as the experimenter gave no new explicit command.

The third neurofeedback task, the ''announce commands task,'' was identical to the follow commands tasks, with the exception that this time the participants themselves announced their next intended RH movement. Participants orally communicated the commands to the experimenter and thereafter aimed to initiate the announced movement. Commands were again time-marked by the experimenter. During the times in which the participant gave no command it was assumed that the participant currently did not intend to move the RH. Further, during the times in which the participant announced a command, but had not yet achieved the announced movement, it was assumed that the participant was still attempting to carry out the announced movement.

### EEG Data Recording

EEG data acquisition was done with a mobile, 24-channel EEG system (mBrainTrain GmbH, Belgrad, Serbia) using an elastic cap (EASYCAP, Herrsching, Germany). The cap's electrode montage was a subset of the 10–20 system. It included the following positions: FP1, FP2, F7, F8, FZ, FC1, FC2, T7, C3, CZ, C4, T8, TP9, CP5, CP1, CPz, CP2, CP6, TP10, P3, PZ, P4, O1, and O2. AFz served as ground (DRL) and FCz as reference (CMS). The continuous EEG signal was digitized via Lab Streaming Layer (LSL<sup>1</sup> ) with a sampling rate of 500 Hz and 24-bit resolution.

### EEG Data Analysis and Classifier Training

To prepare the neurofeedback, EEG training data were analyzed on-site with EEGLAB (Delorme and Makeig, 2004) and BCILAB (Kothe and Makeig, 2013). More specifically, three probabilistic classifiers were derived and used as the basis for an ensemble-like classification algorithm (see details below). A first classifier was calibrated for discriminating flexion trials from extension trials, a second for discriminating flexion trials from rest trials, and a third for discriminating extension trials from rest trials. That is, the first classifier output referred to the probability of flexion as opposed to the extension, the second to the probability of flexion as opposed to remaining open, and the third to the probability of extension as opposed to remaining closed. The three classifiers were trained on different time segments but otherwise underwent the same signal processing steps. To derive each classifier, the EEG training data were first band-pass filtered from 8 to 28 Hz and then epoched into 1.5 s segments relative to the onsets of the extension, flexion, and rest trials. In total there were thus twice as many rest trials as extension or flexion trials. Next, the segments that included obvious non-stereotyped artifacts were identified and rejected using built-in EEGLAB functions (Delorme et al., 2007). The remaining segments were submitted to an adaptation of BCILAB's pre-built ParadigmCSP class (Kothe and Makeig, 2013). This paradigm detects classspecific changes in the sensorimotor rhythm (SMR) by means of common spatial pattern (CSP) analysis. Briefly described, given two time windows of a multivariate signal, CSP finds spatial filters that minimize the variance for one class and simultaneously maximize the variance for the other class (for reviews, see Ramoser et al., 2000; Blankertz et al., 2008). Using ParadigmCSP, 24 CSP filters were derived for each classifier. From these 24 spatial filters, the first four and last four filters (i.e., those filters who promised the highest classdiscriminability) were inspected with respect to their spatial topography and associated time course. All CSP-filters showing physiologically-plausible sensorimotor cortex activity were kept, and feature values were calculated by multiplying each EEG segment with each CSP-filter and then taking the log-variance from the CSP segments. For each classifier, the feature space was equal to the number of CSP filters used. To obtain probabilistic class estimates, a regularized logistic regression model as implemented in BCILAB was then trained on each of the three derived training sets (feature matrices). Given a training set of observations whose class membership is known, this machine learning algorithm learns to make probability estimates on the class membership of new observations (Dreiseitl and Ohno-Machado, 2002). These probability estimates served as the basis for our ensemble-like classification algorithm described below.

### Ensemble Classification Algorithm and Online Data Flow

To deliver the feedback, Matlab, LSL and BCILAB were used. The data processing and feature extraction followed the same procedure as during classifier training (**Figure 1B**). The classifiers always operated on the most current 1.5 s. EEG segment and were constantly updated every 50 ms. To control the RH's movements and resting states, an ensemble-like classification algorithm was implemented, in which the ultimate classifier output depended on the weighted estimates of the three individual classifiers. The usage of the three classifiers' estimates was determined by the RH's current state, being one of six different states: ''remaining opened,'' ''currently flexing,'' ''just closed,'' ''remaining closed,'' ''currently extending'' and ''just opened'' (**Figure 2**). For an illustration of one cycle of the RH's state transitions, first, assume that the RH is in the ''remaining opened'' state. In that state only the open vs. flexion

<sup>1</sup>https://github.com/sccn/labstreaminglayer

classifier is active and the RH remains in this state until the constantly-updated classifier output reaches a certain threshold (e.g., p = 0.7 for the flexion class), individually adjusted for the participant during the acquaintance phase. Once a threshold is reached, the RH starts flexing, i.e., it switches into the ''currently flexing'' state. During the ''currently flexing'' state, no classifier is active and the RH remains in this state for as long as the flexion movement is being completed (2 s). Next, the RH switches into the ''just closed'' state, during which the closed vs. extension classifier and the extension vs. flexion classifier are active. The RH remains in this state either until both classifier outputs reach their respective thresholds, or until 2 s have passed. In the first case, the classification algorithm assumes an extension has been imagined and thus immediately induces an extension movement, that is, it switches into the ''currently extending'' state. In the latter case, the RH switches into the ''remaining closed'' state, during which only the closed vs. extension classifier is active. The RH remains in this state until the closed vs. extension classifier output reaches its defined threshold and then it starts extending, that is, it switches into the ''currently extending'' state. During the ''currently extending'' state, no classifier is active and the RH remains in this state until the extension movement is completed (2 s). Next, the RH switches into the ''just opened'' state, during which the open vs. flexion classifier and the extension vs. flexion classifier are active. The RH remains in this state either until both classifier outputs meet their defined thresholds, or until 2 s have

passed. In the first case, the classification algorithm assumes a flexion has been imagined and thus immediately induces a flexion movement, that is, it switches into the ''currently flexing'' state. In the latter case, the RH switches into the ''remaining open'' state.

### EEG Offline Analysis

### Time-Frequency Analysis

MI-induced changes in the SMR served as the physiological basis for our classification algorithm. To further explore these oscillatory changes, an offline EEG time-frequency (TF) analysis was carried out on the training block data. EEG artifacts were attenuated using extended infomax independent component analysis (ICA; Bell and Sejnowski, 1995; Delorme and Makeig, 2004). Artifactual independent components were identified by visual inspection and excluded from the back projection. Next, the ICA-corrected continuous data were segmented from −2 to 3 s, relative to the onsets of the MI flexion and MI extension trials. EEG segments containing unique, non-stereotyped artifacts were identified by built-in EEGLAB functions and rejected. From each remaining EEG segment, a corresponding CSP segment was calculated by multiplying the data with a chosen CSPfilter. To obtain a unique CSP-component for each participant, all 24 potential CSP components were first derived from each segment and then the physiologically most plausible CSP component was kept. A TF analysis was carried out on the derived CSP segments by means of a continuous Morlet wavelet transform (Debener et al., 2005; Thorne et al., 2011). The hereby obtained frequency bins ranged from 5 to 50 Hz in 1 Hz frequency steps. The TF analysis was conducted for the time interval from −0.8 to 2.3 s.

To avoid edge artifacts, TF data were only analyzed from −0.5 s to 2 s, relative to the beginning of MI. Percent power change relative to baseline power was calculated. That is, for each frequency bin, the corresponding time series was first squared, then scaled to decibels (10 × log10) and finally its change in power (relative to the first 0.5 s mean baseline power) was calculated. For the statistical analysis, SMR power changes were separately extracted for flexion and extension trials. To this end, across trials, mean percent log power changes between 10 and 25 Hz were calculated for the 1 s time interval beginning 0.5 s after MI onset.

### ERD Latency Analysis

Based on the TF data, an ERD latency value was calculated, reflecting the time intervals between MI period onsets and ERD onsets. To this end, a threshold defined as a power reduction of at least 30% as a corresponding desynchronization was set. To counterbalance outliers, the 20th percentile of all trials exceeding the determined threshold was taken to calculate a mean ERD latency across trials for each participant.

### CSP Filter Analysis

To further evaluate the neurophysiological basis of our NF-MIT, we investigated the quality of the CSP filters that were calculated for the online analysis. For this purpose, six criteria were extracted from the heuristics of appropriate filter selection (see **Supplementary Material S1**). These criteria were, first, whether the signal in the CSP pattern appeared to originate from the sensorimotor areas; second, whether the signal in the CSP filter appeared to originate from the sensorimotor areas; third, whether there was a recognizable discriminability between the compared trial classes in the power value distributions; fourth, whether there was recognizable discriminability between trial classes on single-trial time course visualizations; fifth, whether a left-sided ERD and/or a rightsided ERD or ERS occurred; and finally, whether the power value distributions for each trial class were normally distributed. Two CSP filters (one for flexion, one for extension) for each participant were evaluated on these criteria. Each criterion could be either fulfilled or not. A sum value across the two CSP filters was calculated, leading to a maximum score of 12 points. We defined that scores ≥10 indicated plausible CSP filters.

### Questionnaire Data

A 15-item questionnaire (see **Table 2**), adapted from previous studies (Kalckert and Ehrsson, 2012; Braun et al., 2014, 2016), was used to assess the participant's subjective experiences. At the end of each block, the experimenter read each item to the participant, and the participant had to rate his or her level of agreement on a 7-point Likert scale. The scale ranged from −3 (''totally disagree'') to +3 (''totally agree''). SoO, SoA and two other phenomenal target properties—experiential realness (ER) and MI-action binding (MIAB)—were operationalized. SoA was defined as the amount of experienced authorship over the RH's movement behavior and SoO as the experienced level of ''mineness'' towards the RH. MIAB indicated the extent to which the self-induced MI percept and the RH motion percept felt bound together and ER the extent to which the MI act was felt as real and vivid. Three items were used for each phenomenal target property and later averaged to obtain a single value for each block. The remaining three items were control items, one relating to SoA, and two relating to SoO. These items entailed illusion-related statements but did not specifically capture the phenomenal experience of limb ownership or SoA. Hence, in the case of a successful SoO and SoA induction, items related to these two phenomenal constructs should have high affirmative ratings, whereas the control items should not be specifically affected by the experimental manipulation. As in former studies (Kalckert and Ehrsson, 2012; Braun et al., 2014, 2016), the illusion threshold to confirm

TABLE 2 | Questionnaire for assessment of subjective experiences.


Note. Three statements addressed each phenomenal target property. In addition, control statements were included that included illusion-related statements but which did not capture the phenomenal experience of agency or ownership. Statements were read in counterbalanced to the participants, and the participants had to rate their level of agreement on a 7-point Likert scale.

a successful SoO and SoA induction across participants was set to ≥ 1.

### Neurofeedback Performance Evaluation (Classification Accuracies)

### Training Accuracies

Online training accuracies were calculated for all three classifiers with a five-fold block-wise cross-validation procedure. The calculation relied on the training block, using the same time segments as during classifier calibration. To statistically test whether the training accuracies were above chance level, a binomial statistic with p = 0.05 was used (Combrisson and Jerbi, 2015).

### Feedback Accuracies

To evaluate performance during the feedback block, feedback accuracies were analyzed in each task. Additionally, the number of movements during rest phases was compared to the number of movements during move phases in the rest vs. move task. Since no fixed trial structure was given for the follow commands and announce commands task, we post hoc reconstructed a trial structure in order to calculate feedback accuracies (**Figure 3**). To this end, we segmented the data into 5 s intervals, relative to each command. We then defined two different trial types, intended movement trials and intended rest trials, and assigned each segment to one of the two types. A series of intended movement trials began as soon as a movement command was given by the experimenter or participant and lasted until the intended movement was finally carried out. A series of intended rest trials, in turn, started as soon as a movement was carried out and no further command was yet given. The intended movement trials in which a movement occurred were defined as true positive (TP) outcomes whereas those intended movement trials without any movement occurrence were defined as false negative (FN) outcomes. Likewise, the intended rest trials without any movement occurrences were defined as true negative (TN) outcomes whereas those intended rest trials with a movement occurrence were defined as false positive (FP) outcomes. To calculate overall feedback accuracies, the sum of all TP and TN trials was then divided by the total number of trials and multiplied by a hundred. The possible values thus ranged from zero, indicating that all trials failed, to 100, indicating only successful trials, that is, a perfect match of the participants' intentions and the RH behavior. Regarding the feedback accuracies in the rest vs. move tasks, data were equally segmented into 5 s intervals, with move phases comprising only intended movement trials, and rest phases comprising only intended rest trials. To statistically test whether feedback accuracies in the follow commands and announce commands task were above chance level, an established statistical procedure relying on the classification accuracies'' confidence intervals (CI) was used, with p = 0.05 (Billinger et al., 2012). Regarding the rest vs. move task, the same statistic as for the training accuracies was taken (Combrisson and Jerbi, 2015).

### Statistical Analyses

All statistical analyses were performed using bootstrapping procedures, as sample distributions, except for the ratings of the controls for SoA, ER and MIAB in the training block, and SoA and MIAB in the feedback block were non-normally distributed. Bootstrapping approaches have generally been shown to be a useful data analysis paradigm, particularly when assumptions underlying traditional statistical methods are violated, or when the sampling distributions of the test statistics are unknown. In these cases, an empirical sampling distribution for the statistic of interest is derived by repeatedly resampling (with replacement) from the sample at hand. In the context of null hypothesis significance testing, bootstrapping allows a data-driven approximate distribution of the test statistic, given the null hypothesis, to be obtained (instead of assuming a theoretical distribution; see e.g., Efron and Tibshirani, 1994, on using bootstrapping in null hypothesis significance testing).

To evaluate group and block differences in the subjective experiences, a two-way mixed ANOVA with bootstrapping, with the between-subject factor group (stroke patients vs. control participants) and within-subject factor block (training vs. feedback block), was run for each phenomenal target property (SoO, SoA, ER, MIAB).

FIGURE 3 | Derived trial structure in the neurofeedback block. Data were subdivided into trials of 5 s length between each pair of commands, e.g., "close" or "open." Trials were defined as either intended movement trials, expecting a movement of the RH (e.g., a flexion), or intended rest trials, expecting the RH to maintain the current state. Accordingly, trials were classified as either success [true positive (TP) and true negative (TN)] or failure [false positive (FP) and false negative (FN)].

To create the null condition, we first performed the centering of the within-subject factor. Next, we randomly resampled (with replacement) 18 cases of the centered data, and randomized the between-subjects factor. This yielded a bootstrapped data set, from which we computed the F-value of a two-way mixed ANOVA. We then repeated this process 3,000 times to create an empirical sampling distribution of F-values. Finally, the F-value of the original sample was placed within the corresponding empirical sampling F-distribution to determine the p-value. We report a significant result if the proportion of F-values larger than the observed one was below 5% (see e.g., Berkovits et al., 2000 on implementing bootstrapping in designs with repeated measures). All following two-way mixed ANOVA calculations were carried out using the same bootstrapping procedure.

Regarding the number of movements in the rest vs. move task, a two-way mixed ANOVA with bootstrapping, with the betweensubjects factor group and the within-subjects factor phase (move phase vs. rest phase), was carried out. To evaluate group and task effects on feedback accuracies, a two-way ANOVA with bootstrapping, with the between-subjects factor group and within-subjects factor task (rest vs. move task, follow commands task, announce commands task), was conducted.

We conducted two-sided independent samples t-tests with bootstrapping to evaluate group differences in ERD latencies and CSP filter qualities. Two-sided independent samples t-tests with bootstrapping were also used as follow-up t-tests for significant effects resulting from the ANOVAs. To obtain bootstrapped samples for the t-tests, we pooled data of the stroke and control group, randomly generated two samples (with replacement), and estimated the t-value in a two-sided t-test. We repeated this procedure 3,000 times in order to estimate the t-distribution under the null hypothesis. We then calculated the probability of the t-value of our original data given this distribution and reported a significant result if it was below 5% (two-sided). Here, we report both p-values and 95% CI of the t-value.

Pearson correlation coefficients were calculated for pooled groups and blocks between all four phenomenal target properties. In addition, correlation coefficients were calculated for pooled groups between training accuracies and phenomenal target properties during the training block as well as between feedback accuracies and phenomenal target properties during the feedback block. All correlations were tested for significance again using bootstrapping. To create the null condition, we shuffled one variable while keeping the values of the other as in the original dataset. The subsequent steps were as described for the t-tests. All significance tests were Bonferroni-Holm-corrected for multiple comparisons. Statistical analyses were performed with Matlab 2017 and R 3.4.4 (R Core Team, 2018).

### RESULTS

### Electrophysiological Results

To determine whether our classification algorithm operated on a physiologically-plausible EEG signal, and to check for ERD-related group differences, we conducted an offline TF analysis on the CSP-filtered EEG training block data as well as CSP filter quality check. More specifically, we assessed SMR power changes, ERD latencies and CSP filter qualities.

### Power Changes in the SMR

The TF plots across subjects for the chosen CSP channels are shown in **Figure 4**. For each plot, mean percent log power changes across trials are depicted for each frequency bin. As can be seen, both stroke patients (upper plots) and healthy controls (lower plots) showed a power reduction within the SMR's 10–25 Hz frequency range during MI flexion trials (left plots) and MI extension trials (right plots). In the stroke patients, this power decrease amounted to 26.31% (SD = 19.94%) in the flexion trials and to 33.94% (SD = 20.01%) in the extension trials, whereas in the healthy participants' SMR power decreased by 28.09% (SD = 24.58%) in the flexion trials and by 30.73% (SD = 14.97%) in the extension trials. Hence, the expected ERD pattern was clearly inducible for both groups and both MI types. To evaluate potential differences between groups or periods, a mixed two-way ANOVA with bootstrapping was conducted on the mean percent log power change between 10–25 Hz, which neither revealed a main effect of group, F(1,16) = 0.01, p = 0.931, nor a main effect of period, F(1,16) = 1.1, p = 0.327, and also no interaction, F(1,16) = 0.26, p = 0.633.

### ERD Latencies

ERD latencies were defined as the time interval between MI period onset and ERD onset, whereby an ERD onset was defined as the time-point at which the 10–25 Hz SMR power was reduced by at least 30%. Average ERD latencies across flexion and extension trials amounted to M = 631.58 ms (SD = 141.21) in the stroke patients and M = 506.56 ms (SD = 126.59) in the control participants. For evaluating potential group differences, a two-sided independent samples t-test with bootstrapping was conducted, which, revealed a non-significant trend for longer ERD latencies in stroke patients as compared to control participants, t(16) = −1.98 [95% CI (−2.19, 2.17)], p = 0.069.

### CSP Filter Plausibility

Based on our criteria for CSP filter evaluation (see **Supplementary Material S1**), we derived a CSP filter quality score for each participant, whereby values ≥10 were deemed to indicate a high CSP filter quality. Results reveal that three, stroke patients and four control participants reached a value of 10 or above, while on average, the CSP filter quality amounted to M = 8.63 (SD = 2.77) in the stroke patients and M = 8.22 (SD = 2.73) in the healthy participants. A two-sided independent samples t-test with bootstrapping revealed no differences in the quality of the selected CSP filters between stroke patients and control participants t(16) = −0.3 [95% CI (−2.26, 2.06)], p = 0.761.

### Questionnaire Results


FIGURE 4 | Event-related desynchronization (ERD) during MI. Time-frequency (TF) plots show the percentage change in power from baseline (i.e. from −0.5 s to 0 s) for MI flexion trials (left panels) and MI extension trials (right panels). MI started at time point zero and was performed for 1.5 s. Vertical lines indicate the chosen time interval for the statistical analysis (i.e. from 0.5 s to 1.5 s). The solid blue line on the bottom reflects MI-related power changes within the 10–25 Hz SMR frequency range. Topoplots above the TF plots show the standard deviations across the chosen common spatial pattern (CSP)-filters for each channel.

target property was considered as successfully induced with an average value of ≥ 1 on a group level.

### Sense of Ownership

The SoO-induction criterion was reached in both groups (patients and controls) and both blocks (training and feedback),

stroke patients and healthy controls. The black dashed line indicates successful RHI induction on group level.

while all SoO control questions showed values around zero or less, thus refuting any suggestion of response bias. The highest SoO level was reported by the stroke patients during the neurofeedback block (M = 1.26, SD = 1.66), whereas the lowest SoO level was reported by the healthy controls during the training phase (M = 1.00, SD = 1.62). A mixed two-way ANOVA with bootstrapping revealed no main effect of group, F(1,16) = 0.00, p = 0.964, no main effect of phase, F(1,16) = 0.81, p = 0.402, and no interaction, F(1,16) = 0.32, p = 0.595.

#### Sense of Agency

The SoA-induction criterion was met for the stroke group in the training phase (M = 1.33; SD = 1.29) and feedback phase (M = 1.59; SD = 0.52). By contrast, for the control group, the criterion was not reached either in the training (M = 0.33; SD = 2.2) or in the feedback (M = 0.37; SD = 2.0) phase. Mean values of the SoA control items were around zero in both groups (stroke group: M = 0.28, SD = 1.56; control group: M = −0.83, SD = 1.8), thereby refuting any suggestion of response bias. The mixed two-way ANOVA with bootstrapping revealed neither a main group effect, F(1,16) = 2.41, p = 0.157, nor a main effect of phase, F(1,16) = 0.25, p = 0.647, nor an interaction between group and phase, F(1,16) = 0.14, p = 0.718.

#### Experiential Realness

The ER-induction criterion was met for both groups in both the training phase (stroke group: M = 1.33, SD = 1.37; control group: M = 1.15, SD = 1.25) and the feedback phase (stroke group: M = 1.63, SD = 0.66; control group: M = 1.11, SD = 0.88). A two-way mixed ANOVA with bootstrapping revealed neither a main effect of group, F(1,16) = 0.58, p = 0.473, nor of phase, F(1,16) = 0.38, p = 0.552, nor an interaction, F(1,16) = 0.62, p = 0.463.

#### MI-Action Binding

Whereas the patient group met the MIAB-induction criterion in both phases (training: M = 1.41, SD = 1.2; feedback: M = 1.19, SD = 0.71), the control group did so only in the feedback phase (training: M = 0.6; SD = 1.89; feedback: M = −0.07; SD = 2.3). A mixed two-way ANOVA with bootstrapping revealed neither a main effect of group, F(1,16) = 2.46, p = 0.153, nor of phase, F(1,16) = 1.2, p = 0.327, nor an interaction, F(1,16) = 0.3, p = 0.602.

### Performance Results

#### Training Accuracies

Training accuracies were calculated for all three classifiers with a five-fold block-wise cross-validation procedure using the EEG data from the training block. Accuracies for both groups and all

three classifiers are shown in the upper panel of **Figure 6**. In the stroke group, overall training accuracies were significantly above chance level (α = 0.05, Combrisson and Jerbi, 2015) in 24 of the 27 derived accuracies (three classifiers times nine stroke patients), giving a proportion of 88.89%. In the control group, 25 of the 27 derived accuracies were above chance-level, corresponding to a proportion of 92.59%. In order to evaluate differences in groups and classifiers, a mixed two-way ANOVA with bootstrapping with the within-subject factor classifier and the between-subject factor group was conducted, revealing a main effect for the classifiers, F(1,16) = 12.98, p = 0.003, but no main effect for group, F(1,16) = 1.24, p = 0.285. Post hoc two-sided independent-samples t-tests with bootstrapping revealed a difference between the training accuracies of the opened vs. flexion classifier and the extension vs. flexion classifier, t(16) = −3.26 [95% CI (−2.07, 2.01)], p = 0.003, d = 0.77, as well as between the closed vs. extension classifier and extension vs. flexion classifier, t(16) = 3.22 [95% CI (−2.01, 2.02)], p = 0.005, d = 0.76 (all values passing the Bonferroni adjustment of alpha = 0.025).

### Feedback Accuracies

In order to assess the performance during the feedback block, feedback accuracies were analyzed for each task. Feedback accuracies are depicted for both groups and all three tasks in the lower panel of **Figure 6**. The average feedback accuracy across tasks was M = 57.95% (SD = 12.74) in stroke patients and M = 54.31% (SD = 15.16) in control participants. Differentiating between the tasks, feedback accuracies varied highly among participants in both groups, with values from 20% to 82,93% in stroke patients and from 0% to 75% in control participants. In both the control and the patient group, accuracies were significantly above chance level in 18 of the 27 derived accuracies, giving a portion of 66.67%. In order to evaluate differences between groups and tasks, a two-way mixed ANOVA with bootstrapping was conducted, revealing no main effect of task, F(1,16) = 5.13, p = 0.075, no main effect for group, F(2,15) = 0.32, p = 0.587, and no interaction effect between group and task, F(3,15) < 0.001, p = 0.978.

#### Robotic Hand Movements in Rest vs. Move Task

In addition to evaluating the feedback accuracies, performance has been further assessed by comparing the number of movements during rest phases to the number of movements during move phases in the rest vs. move task. **Figure 7** displays the mean number of movements during rest and movement phases of the rest vs. move task. A mixed two-way ANOVA with bootstrapping revealed an effect of phase (F(1,16) = 13.84, p = 0.003), in that participants conducted fewer movements during the rest phases (M = 1.76, SD = 0.86) than movement phases (M = 3.50, SD = 2.12). In contrast, no main effect of group, F(1,16) = 1.52, p = 0.252, nor any interaction effect were found, F(1,16) = 0.16, p = 0.697.

### Correlation Analysis

In order to assess the relationships between our various experimental variables, Pearson correlation coefficients were

calculated for pooled groups and blocks between, first, all four phenomenal target properties, second, training accuracies and phenomenal target properties during the training block, and third, feedback accuracies and phenomenal target properties during the feedback block. Significant positive correlations across groups and blocks were found between all four phenomenological constructs, all of which depict large effectsizes (see **Table 3**). Apart from that, no other significant correlations were found.

### DISCUSSION

The aim of the present study was to investigate the feasibility of a self-paced and embodiable NF-MIT. To this end, an anthropomorphic RH was integrated into an EEG-based BCI and used for neurofeedback provision. Within three different neurofeedback tasks, stroke patients and healthy controls freely attempted to control the RH's movement behavior in a self-paced manner, using MI.

### General Feasibility

The general feasibility of our NF-MIT was investigated on the electrophysiological, phenomenological, and performance


Note. Table includes the Pearson correlation coefficients (r), the R-squared, the t-value (t) with 16◦ of freedom, the 95% confidence interval (95% CI) of the t-value, and the p-value (p).

level. On the electrophysiological level, across trials the expected ERD pattern of a right-handed MI task was evident in most of our participants. That is, for most of our participants, the TF analysis revealed a typical MI-related decrease in 8–30 Hz oscillatory brain activity over the sensorimotor areas. Results from the CSP filter analysis were more mixed: While only 38% of our participants reached our threshold for full CSP-filter plausibility (≥10), most remaining participants were close to our (rather restrictive) cut-off value (see ''Discussion'' section below). Moreover, the criteria for CSP-filter plausibility pertain to the discriminability between compared mental states rather than to whether or not the filters are anatomically plausible in the first place. In other words: while our chosen CSP-filters were rather poor in discriminating between the different mental states, their spectrotemporal filter characteristics were as expected. Taking these findings together, we thus conclude that the classification algorithm operated on noisy, but electrophysiologically-plausible, MI-related brain signals and not merely on some technical artifacts.

Regarding phenomenology, our questionnaire results revealed that the induction criterion was reached on the group level in both blocks and both groups for the target properties SoO and ER. With respect to MIAB, the illusion criterion was met in both groups during the training block, but only in the stroke group during the feedback block. Regarding SoA, the illusion criterion was met during both blocks for the stroke patients, but in neither block for the healthy controls (see ''Discussion'' section below). These findings demonstrate that the RH could be embodied in the participants' phenomenal body scheme to a reasonable degree. That is, most participants of both groups experienced the RH as part of their own body and their MI acts as close to real. Moreover, the majority experienced the MI act as perceptually fused with the RH motion percept. Regarding the subjective ratings derived from the training block, these results replicate our former study (Braun et al., 2016). Here, we similarly showed that within a fixed trial structure, illusory SoO and SoA, as well as ER and MIAB, may be achieved over an external device by the approximate synchrony of imagined limb movements and observed RH movements. As pertains to the subjective ratings derived from the neurofeedback block, we find a similarly successful induction of the four phenomenal target properties, but this time in a scenario without a fixed trial structure. This shows that even in the presence of significant temporal mismatches between the imagined limb movements and RH motion behavior, the RH was still phenomenally embodiable. This, in turn, suggests that either the proportion of mismatches is small enough for SoO, SoA, ER and MIAB not to be disrupted, or these four phenomenal constructs are sufficiently robust to violations of expected RH actions.

Regarding the performance level, we investigated classification accuracies from both the training block (training accuracies) and feedback block (feedback accuracies). For the training block, most of our participants, regardless of whether they belonged to the control or stroke group, achieved training accuracies that exceeded the statistical chance level. This finding is in line with the typical MI classification outcomes of traditional NF-MIT paradigms (see e.g., Yong and Menon, 2015) and also with the findings of our former aRHI study (Braun et al., 2016). The finding indicates that the MI-induced ERD pattern was not only evident across trials as seen in the ERD findings, but was reliable enough to be also detectable at the single-trial level. Moreover, the finding shows that our classification algorithm was not only suitable for healthy controls but worked equally well with stroke patients. The difference in classifier training accuracy between extension vs. flexion and the other two classifiers is probably because it is harder to distinguish two types of movement in EEG than it is to distinguish movement from non-movement.

For the feedback block, we found that most participants were able to achieve at least some level of control over the RH's actions. That is, although mean feedback accuracies across tasks were rather poor (∼58% in stroke patients, ∼54% in healthy controls), almost all participants (nine of nine stroke patients and eight of nine healthy controls) performed at least one neurofeedback task above chance level. More generally, the majority of all accuracies (∼67% in both groups) obtained in the three tasks—rest vs. move, announce commands and follow commands—exceeded that threshold. Furthermore, in the rest vs. move task, both groups showed significantly fewer RH movements in the rest phases than in the movement phases. Taking these findings together, this shows that, in principle, it is possible to use a self-paced paradigm for RH control. To our knowledge, this is the first study to have demonstrated this.

It should be pointed out that we cannot claim any superiority of an embodiable over an abstract feedback signal under self-paced NF-MIT. This was not, however, the principal research objective in the present study. Rather, we intended to investigate whether an EFS is, in principle, feasible in a self-paced neurofeedback paradigm. Future studies are necessary to compare abstract and EFSs, and to investigate under which circumstances the different feedback types are more suitable.

### Group Differences

We found differences between stroke patients and control participants on the electrophysiological and phenomenological levels, but not on the performance level.

On the electrophysiological level, we found a non-significant trend for a delayed ERD onset in stroke patients as compared to healthy controls. While distinct lateralized ERD patterns in control participants as compared to stroke patients have been widely addressed in previous studies (Feydy et al., 2002; Scherer et al., 2007; Braun et al., 2017), few did so with respect to distinct ERD latencies in the context of NF-MIT. This might be due to highly varying study designs and a few direct comparisons of stroke patients with matched control participants. Schaechter (2004), for instance, reported a prolonged latency of motorevoked potentials after the infarct. Moreover, findings by Crone et al. (1998) showed a longer ERD latency for ipsilateral activation patterns as compared to contralateral activation. It has been suggested that delays in ERD onset result from slower information processing in damaged brain structures (Leocani and Comi, 2006). Although we remain cautious in interpreting the trend we found here for a delayed ERD onset, it is thus, in principle, compatible with the existing evidence.

As regards the phenomenological level, we found a difference between groups for SoA. That is, whereas the illusion criterion was met during both blocks for the stroke patients, it was not met in either block for the healthy controls. A speculative explanation might be that stroke patients generally have a vaguer percept of the affected hand, leading them to more quickly experience SoA. As functional reliability in neural tissues may be damaged in stroke patients, these patients may be accustomed to spending more effort to control their own affected hand.

### Correlation Analysis

In line with our former study (Braun et al., 2016), we observed high correlations between SoO, SoA, ER and MIAB. This suggests that these subjective measures do not relate to separate, but rather to overlapping and interacting aspects of phenomenal experience (for a discussion, see Braun et al., 2016). However, we did not find significant correlations between any subjective measure and the classification accuracies. Contradicting our argumentation for an EFS, a direct relationship between the participants' perceived level of RH embodiment and neurofeedback performances can thus not be demonstrated on a statistical level. Given the low sample sizes, these null findings may, however, be due to a lack of statistical power or due to a suboptimal EFS implementation (see ''Study Limitations'' section). Taking into account existing evidence supporting the beneficial effects of an EFS (see ''Introduction'' section), we believe that the lack of correlation does not render EFS useless, as we argue in the following.

### Arguments for an Embodiable Feedback Signal

One benefit of an EFS could be that it facilitates causal inference (Shams and Beierholm, 2010). If the provided neurofeedback signal closely enough resembles the MI act performed, in both time and space, the MI percept and the visual percept induced by the neurofeedback signal could possibly be better fused. Consequently, the brain could infer a common cause for both percepts.

A second advantage could be that the perceptual fusion just described opens up the possibility for inducing SoO, which, in turn, might provoke SoA (Kalckert and Ehrsson, 2012, 2014; Braun et al., 2014, 2016). As has been repeatedly demonstrated (Perez-Marcos et al., 2009; Sanchez-Vives et al., 2010; Braun et al., 2016), SoO is inducible by merely imagining limb movements in approximate temporal synchrony to observed hand movements. A neurofeedback signal that is inherently linked to the BCI-user's own body and its voluntary movements should increase the compliance with the NF-MIT and help to convince the BCI-user about the effectiveness of the training, thereby improving his or her motivation.

A third benefit might be that an EFS potentially reduces the patient's cognitive workload since the cognitively-demanding task of mentally rehearsing limb movements could be bottom-up facilitated by the EFS (as with mirror visual feedback; Ramachandran and Altschuler, 2009). To be clear, for healthy participants such cognitive offloading might be unnecessary or perhaps even a hindrance. The major target group of NF-MIT is, however, stroke patients whose MI abilities are often impaired, and here much more uncertainty exists as to what these BCI-users actually imagine.

Fourth, an EFS may require less abstract thinking by the BCI-user than an abstract neurofeedback signal. Whereas with an abstract neurofeedback signal the BCI-user has to understand that the signal shown on the computer screen relates to his or her own MI-act, this act of abstraction would not be needed with an EFS.

Many arguments may thus be found in favor of an embodiable rather than an abstract neurofeedback signal. We acknowledge, however, that these arguments assume an idealized scenario, in which the EFS sufficiently matches the mental acts performed. In what way such an EFS can be realized in practice, awaits further empirical validation.

### Study Limitations

Our phenomenological results support the hypothesis that the RH as a feedback signal was embodiable. Yet, an RH adjusted to user-specific sizes might have improved the embodiment—we recruited all genders, but used a larger RH for all participants, originally modeled from a male hand.

Regarding the rather poor performance accuracies in the neurofeedback block, several reasons might account for the deviation between intended and observed BCI output. First, our participants could just have had rather poor MI abilities. That is, they were not sufficiently able to vividly imagine the required limb movements, and as a consequence, were over-challenged with the given MI task. For the present study, we, cannot follow up on this possibility, given that we did not include an MI ability or severity questionnaire. It should, however, be noted that former studies found no (Rimbert et al., 2019) or rather moderate (Vuckovic and Osuagwu, 2013; Marchesotti et al., 2016) relations between subjective MI ability measures and BCI literacy.

Second, the rather poor neurofeedback performances might have been caused by shortcomings of the participants' ability to modulate their SMRs. Here, an indirect measure of SMR control is given by the derived brain activity patterns and ERD analysis. As expected, MI-related activation patterns over sensorimotor areas could be observed in both groups, with moderate and mostly above-chance level mean training accuracies across classifiers.

Third, there may have been deficits in the interpretation and translation of brain activity into control signals (Mason et al., 2006). Our classification algorithm was based on the derivation of CSP filters. We found electrophysiologically plausible, yet highly varying, filters in both groups. In order to achieve a better classification, enhanced reliability for obtaining high-quality CSPs is required (Ang et al., 2008). Overall, we consider advanced machine learning algorithms a key for future research in NF-MIT and, ultimately, for advancing NF-MIT clinical application. Without substantial improvements in EEG-based SMR signal extraction, this may remain a challenging goal.

Fourth, it should be noted that during the training block, the RH always moved in synchrony with the imagination of the participant's own limb movement. From a phenomenological perspective, we consider this design aspect appropriate, in order to bottom-up facilitate the patient's MI process and for keeping the training and neurofeedback block as similar as possible. From a signal processing perspective, however, this design aspect has a caveat, in that limb movement observation and limb movement imagination may both induce an ERD. As a consequence, it remains unknown as to how far the ERD pattern observed is driven by the limb movement imagination, and not exclusively by the limb movement observation. If only the latter were the case, this would cast problems on the BCI, since it requires a brain signal that can be mentally self-induced, independent from external sensory stimulation. For the present experiment, the above-chance level feedback accuracies during the NF block, however, indicate that the observed ERDs were at least partly driven by limb movement imaginations.

The method we adopted for CSP filter quality assessment was rather unstandardized. While we consider our six suggested CSP filter quality criteria valid, we are aware that many other criteria could be used instead, or in addition. Likewise, we are aware that our defined threshold of 10 is rather arbitrarily set and that other cut-off values could be defined. Despite these limitations, we nevertheless, consider our CSP-filter assessment procedure as an important step towards a physiologicallyplausible CSP filter selection, given that most existing CSP-based NF-MIT paradigms just select CSP filters based on class-discriminability.

Interestingly, performance accuracies greatly varied, not only among participants but similar between the different neurofeedback tasks. This might be due to a lack of experience in NF-MIT, and therefore training phases prior to the actual NF-MIT might be beneficial to reduce exercise-dependent intraindividual variability in performance. Differences between subjects, on the other hand, suggest that the same NF-MIT paradigm may not be suitable for all participants, and prior training phases might be needed to assess an individuals' potential for NF-MIT. Taking into account the participant's suitability for NF-MIT based on a training paradigm might be another possibility for achieving less variability and superior feedback accuracies (de Vries and Mulder, 2007; Ono et al., 2013). Yet, it would also limit the generalizability of the application.

One potential drawback in our experimental design concerns differences in the task instructions between our training and feedback blocks. While in the training block, participants were instructed to just relax during the rest trials, in the feedback block, they were explicitly asked to employ mental strategies, such as counting numbers. Hence, with respect to phenomenal content, the resting states during the training block and feedback block differed, which might have reduced the classification algorithm's reliability. The reason for giving different task instructions for the training and feedback resting periods was that, for the fixed and timely regular trial structure during the training block, we considered the refraining from MI to be much easier than during the self-paced feedback phase, when the rest periods were not of a fixed 5-s duration, but rather were of arbitrary length. In addition, we wanted to provide our participants with some cognitive strategies for potential benefit to their neurofeedback performances.

Finally, it should be recalled that MI instructions differed between the first (unreported here) and the second NF-MIT session. In spite of this difference in the MI tasks, general training effects might have occurred in the second session due to the first, as participants were more familiar with MI as well as the neurofeedback procedure. However, training effects of session one on session two do not in principle limit the feasibility of the second session, provided that all participants underwent both NF-MIT sessions in the same order, which was the case here.

Despite these drawbacks, we conclude that this study successfully demonstrates that healthy participants and stroke patients embodied an RH into their body scheme, and were able, to some modest degree, to control the RH in a self-paced setting. We hope this will motivate further research exploring the idea that an embodiable self-paced NF-MIT is beneficial in stroke motor rehabilitation.

### DATA AVAILABILITY STATEMENT

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation, to any qualified researcher. Scripts used for

### REFERENCES


the analysis are provided at https://github.com/nadinespy/ SelfPacedEmbodiableNeurofeedback.

### ETHICS STATEMENT

This study was carried out in accordance with the recommendations of the University of Oldenburg ethics committee with written informed consent from all subjects.

### AUTHOR CONTRIBUTIONS

NS and NB designed the experiment under the supervision of SD and JT. EB collected the data. NS and EB analyzed the data under the supervision of NB and SD. NS and NB wrote major parts of the manuscript. SD, HM, AP, EB, and JT contributed to, reviewed, and edited the manuscript.

### ACKNOWLEDGMENTS

We acknowledge the support of Cornelia Kranczioch and Joost Meekes.

### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnhum. 2019.00461/full#supplementary-material.


**Conflict of Interest**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2020 Spychala, Debener, Bongartz, Müller, Thorne, Philipsen and Braun. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Factors Influencing Manipulation of a Familiar Object in Patients With Limb Apraxia After Stroke

Gloria Pizzamiglio1,2, Zuo Zhang<sup>3</sup> , Mihaela Duta<sup>4</sup> and Elisabeth Rounis<sup>2</sup> \*

<sup>1</sup> Wellcome Centre for Human Neuroimaging, Institute of Neurology, University College London, London, United Kingdom, <sup>2</sup> Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, United Kingdom, <sup>3</sup> Social, Genetic and Developmental Psychiatry Centre, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, United Kingdom, <sup>4</sup> Department of Experimental Psychology, University of Oxford, Oxford, United Kingdom

Previous studies have shown that hand actions to visual objects are affected both by perceptual factors and by action goals. Our aim was to study how these processes affected hand actions in chronic stroke patients, based on whether they had limb apraxia. Twenty-two left hemisphere, chronic stroke patients were measured on neuropsychological tasks of limb apraxia, which was identified in a subgroup of 10 patients. All patients underwent testing on a separate task of making simple reach and grasp actions to a cup. Their performance was compared to a group of 18 healthy age-matched volunteers. Participants were instructed to grasp the top or bottom of a cup to either lift or turn it over so as to end with a hand position that was either comfortable or uncomfortable. This task tested the influence of the compatibility of hand–cup orientation, as well as goals driven by the end-state comfort of the hand, on action selection for object manipulation. Participants' performance was measured in terms of error rates, and speed of initiation and reaching (movement time) to the object. The patients' performance was significantly delayed, and error rates increased when reaching to grasp a cup under conditions of poor compatibility and end-state comfort. The subgroup of patients with apraxia showed a decreased influence of compatibility of hand interaction with the cup, with increased error rates and delayed response times, compared to patients with no apraxia and healthy volunteers. This is despite the fact they did not display significant deficits on neuropsychological tasks of real object use. The study shows that patients with apraxia have difficulties in selecting elements of object-directed actions, pertaining to both habitual and goal-directed factors.

Keywords: apraxia, goal-directed actions, habitual actions, affordances, object manipulation

## INTRODUCTION

A large number of movements can be used to achieve a goal, such as grasping to move an object. However, studies have demonstrated that skilled actions, such as object manipulation, are often stereotyped (Keele, 1968; Harris and Wolpert, 1998).

Two main factors driving object manipulation include features relating to the object's and the environment's properties [such as its shape, position, and size (Jeannerod, 1994)], as well as what one intends to do with the object, namely, action "goals" (Marteniuk et al., 1987;

#### Edited by:

Giulia Galli, Santa Lucia Foundation (IRCCS), Italy

#### Reviewed by:

Anna Maria Berti, University of Turin, Italy Marta Bienkiewicz, ´ Euromov, Université de Montpellier, France

> \*Correspondence: Elisabeth Rounis Elisabeth.rounis@ndcn.ox.ac.uk

#### Specialty section:

This article was submitted to Motor Neuroscience, a section of the journal Frontiers in Human Neuroscience

Received: 24 September 2019 Accepted: 19 December 2019 Published: 11 February 2020

#### Citation:

Pizzamiglio G, Zhang Z, Duta M and Rounis E (2020) Factors Influencing Manipulation of a Familiar Object in Patients With Limb Apraxia After Stroke. Front. Hum. Neurosci. 13:465. doi: 10.3389/fnhum.2019.00465

Rosenbaum et al., 2006). The latter rely on an evaluation of expected outcomes (Rosenbaum et al., 2006; Grafton and Hamilton, 2007).

Most studies examining the perceptual effects of object properties on hand actions have identified stimulus–response compatibility effects. Gibson (1979) introduced the concept of "affordance." This described how visual properties of objects, or the environment, can give rise to action representations, depending on contextual demands of the task. "Affordances" link graspable features of an object and an independent action elicited in a task. In a seminal study, participants responded faster if the orientation of the handle on an object was compatible with the hand used to respond, in a task in which they had to make right- or left-finger presses according to whether objects in pictures were depicted as upright or inverted (Tucker and Ellis, 1998). This is despite the fact that they were not required to make a judgment about the handle orientation. Other studies have replicated this effect (Ellis and Tucker, 2000; Bub and Masson, 2010).

The effect of action outcomes on object manipulations has been described using a phenomenon named "end-state comfort" effect (Rosenbaum et al., 1990, 1992). This reflects the preference for participants to select uncomfortable grip postures at the start of an action toward an object, in order to end in a comfortable posture. This effect most likely reflects a bias on the choice of one action in the face of an overwhelming number of possible others, when reaching to grasp an object (Wolpert and Miall, 1996).

Although previously these two factors were studied in isolation, recent investigations have looked at both combined (Herbort and Butz, 2011, 2015; Herbort et al., 2017; Rounis et al., 2017). Herbort and Butz (2011) and Rounis et al. (2017) investigated the effect of turning a cup either with the thumb positioned toward its top (lip) or with a thumb positioned in the opposite direction. Both studies identified an 'End-state comfort' effect indicating a preference for turns that started with an inverted (or pronated) grasp to end comfortably (in a supinated) grasp. However, this effect was mitigated by an "affordance" effect measured either in the choice of grasping the cup from its lip, even though it would result in an uncomfortable end state, for example, when the cup was upright in the Herbort and Butz (2011) study, or in mitigating the end-state comfort effect measured using response and movement times (MTs) in "afforded" actions (Rounis et al., 2017). Further studies have demonstrated the effect of affordances trumping the end-state comfort effects, suggesting that habitual actions provided by the former exert a separate influence on goal-directed planning (Herbort et al., 2017).

Limb apraxia is a disorder of skilled action that does not result from motor weakness, incoordination, incomprehension, or sensory impairment, following an acquired brain lesion, such as a stroke. Traditional theories of the disorder distinguish between "ideational" and "ideomotor" apraxia. In the former, patients lose the ability to represent an action conceptually and display difficulties in knowing how to use an object, despite being able to name it and knowing its function. In the latter, more common condition, actions are conceptually correct but implemented poorly. Patients with ideomotor apraxia are unable to imitate meaningless gestures and show spatiotemporal errors in pantomiming or using objects (Sirigu et al., 1995; Buxbaum et al., 2003).

Previous investigations of patients with ideomotor apraxia have hypothesized deficits in selecting among competing actions relating to "affordances" (Buxbaum et al., 2003; Jax and Buxbaum, 2013; Watson and Buxbaum, 2014, 2015; Rounis and Humphreys, 2015). This is further supported by neuropsychological findings in which patients often overrely on affordances at the expense of goal-directed behavior (Riddoch et al., 1989; Osiurak et al., 2008a,b).

We developed a new experimental procedure to assess the interplay between the compatibility of object features and goaldirected influence in the form of hand posture preferences at the end of the action, during motor planning using the task described in Rounis et al. (2017). We varied the compatibility of the initial start posture of the hand with the physical properties of a target object and the preferred end posture for the action (whether it was comfortable or uncomfortable). A group of patients with and without apraxia, and a group of healthy age-matched participants were instructed to either lift or turn a cup presented in front of them by grasping it from a specified position (namely, its lip, which corresponded to the open end of the object that could be filled, or its bottom, which corresponded to the closed end). The cup itself was placed in an upright orientation in one half of the trials or upside down in the other half. This led to four possible actions: lift with a supinated grasp or lift with a pronated grasp and turn with a supinated grasp, ending in a pronated (uncomfortable) posture, or turn with a pronated grasp, ending with a supinated (comfortable) posture.

We hypothesized that there would be dissociable effects of (i) an initial grasp preference, or "compatibility," and (ii) an end-state comfort effect, related to whether the posture of the hand at the end of the task was comfortable or not, in healthy volunteers and patients with and without apraxia. This would be demonstrated if patients with and without apraxia displayed differences in the compatibility and end-state comfort effects compared to healthy volunteers. Previous literature would suggest that patients with apraxia would be impaired in actions that were not compatible (Osiurak et al., 2008b; Lee et al., 2014).

## MATERIALS AND METHODS

### Participants

Twenty-two, left hemisphere chronic (>1 year) stroke patients (16 males, 6 females) aged between 25 and 79 years (mean age, 56.6 years) took part in the study. They had suffered their first ever stroke more than 1 year ago (mean time since stroke: 20.4 months). Details of the patient demographics are provided in **Table 1**.

An additional 18 healthy volunteers (10 males, 8 females) aged between 24 and 77 years (mean age, 58.9 years) with no history of any neurological and psychiatric illnesses took part in the study.

Written informed consent was obtained from all participants taking part in the study, which was approved by the Health Research Authority, South Central – Berkshire Ethics Committee.

#### TABLE 1 | Patient demographics.

fnhum-13-00465 February 8, 2020 Time: 12:30 # 3


Patients with ideomotor apraxia have been highlighted in bold. R = Right hand, L = Left hand, and All scores were normalized out of 100.

Participants attended the Cognitive Neuropsychology Centre at the Department of Psychology, University of Oxford, for two separate sessions. In the first session, they underwent detailed neuropsychological testing of apraxia and cognitive function. In the second session, they undertook a behavioral task, involving cup manipulation.

All participants were right handed (Oldfield, 1971). Patients used their dominant hand or, if they had hemiparesis, their unaffected hand, to complete the task. A total of 11 out of 22 patients used their left hand due to hemiparesis. The remaining patients and healthy volunteers used their right hand for the performance of all tasks in this study. The degree of impairment caused by stroke was formally assessed using the Action Research Arm Test (ARAT) scale (Lyle, 1981).

### Neuropsychological Testing

Participants taking part in the study underwent cognitive screening and assessment of praxis deficits. We report the neuropsychological data of praxis deficits from screening using three tasks, which were derived from the Birmingham Cognitive screen. These comprised a meaningless gesture imitation task, a gesture production, and a gesture recognition task.

#### Meaningless Gesture Imitation

The meaningless gesture imitation task was derived from the apraxia testing section of the Birmingham Cognitive Screening tool, version 2 (Humphreys et al., 2012). The task required imitation of 10 non-symbolic gestures using the subject's least paretic hand (see above), all of which required holding a static position after demonstration by the experimenter. The test consisted of four gestures involving whole-hand movements and six involving independent finger movements. They were performed slowly by the experimenter in front of the participants for them to reproduce immediately afterward. If an item was not reproduced flawlessly on the first presentation, a second trial was given. The participants' performance was video-recorded and assessed by two separate assessors.

Patients scored 2, 1, and 0 depending on whether their imitation was correct on the first or second presentations or never succeeded. This led to a scoring out of 8 for the hand gestures and out of 12 for the finger gestures, with the total score being out of 20.

#### Gesture Production

The gesture production task involved pantomime of a total of six gestures (three transitive, involving pantomime of object use, e.g., "show me how you would brush your teeth, using a toothbrush in your hand"; three intransitive, involving pantomime of a familiar gesture to verbal command that does not require object use, e.g., "show me how you stop traffic"). The test included bodycentered (salute, using a glass), non-body-centered (stop, using a salt cellar), repetitive (hitch hiking, using a hammer), and nonrepetitive (stop, using a glass) actions. All actions can be carried out as a single step sequence. Patients were allowed a maximum of 15 s per item to respond and were asked to execute the action once. Two points were given for a correct and accurate gesture; 1 point was given for a recognizable but inaccurate gesture (e.g., including spatial and/or movement errors); and 0 points were given for no response after 15 s, an unrecognizable response, or perseveration from previous gestures. The final sum score (maximum, 12) was used in the analyses.

#### Gesture Recognition

In the gesture recognition task, the examiner produced six actions that patients had to recognize: three transitive (using a cup, using a key, using a lighter) and three intransitive (come over, good, goodbye) actions. The examiner showed each gesture while the patients had to select the action being performed from a multiplechoice list, which included four alternative responses for each action. The four alternatives for each action corresponded to: (1) the correct action (e.g., using a lighter), (2) a semantically related action (using a match), (3) a visually related action (using a gun), and (4) an unrelated action (using a torch). The patients were allowed a maximum of 15 s per item to respond. and they were given 1 point for each correct response. The final sum score (maximum, 6) was used in the analyses.

The data from both transitive and intransitive gestures in these tasks were entered together as a composite measure.

#### Single-Object Use

The single-object use task was aimed at identifying patients with ideational apraxia. Patients were presented with one of six objects individually, one at a time (a torch, a straw, a comb, a nail clipper, a screwdriver, and matches). They were asked to demonstrate the use of each of these with the object at hand. The patients were allowed a maximum of 15 s per item to respond. Two

points were given for a correct and accurate gesture; 1 point was given for a recognizable but inaccurate gesture (e.g., including spatial and/or movement errors); and 0 points were given for no response after 15 s, an unrecognizable response, or perseveration from previous gestures. The final sum score (maximum, 12) was used in the analyses. Of note as there were no normative data to establish cutoff scores, we compared the performances of patients and healthy volunteers on this task using a paired-sample t-test (assuming unequal variance).

**Table 2** shows the praxis task cutoff scores for tasks of ideomotor apraxia from the Birmingham Cognitive Screening program, based on fifth percentile across age groups in percent estimates of the score (from Humphreys et al., 2012). Patients were characterized as having ideomotor apraxia if they scored below any of the cutoff scores of gesture production and the imitation of meaningless gesture tasks (Buxbaum, 2001; Buxbaum et al., 2003). Patients' hand gestures were videotaped and saved anonymously. They were scored offline by two independent assessors (ER and GP). None of the stroke patients had a formal diagnosis of ideomotor apraxia known to the assessors.

### Other Comparisons

We performed additional comparisons by correlating apraxic deficits with motor impairment measures on the ARAT score of the paretic hand (see **Table 1**), as well as with constructional apraxia using the Rey-Osterrieth figure copy test (Rey, 1941; Osterrieth, 1944) looking for any presence of visuospatial deficits in patients. Correlation analyses used Spearmann's rho and Bonferroni correction for multiple comparisons with a corrected p < 0.05 considered as significant.

### Behavioral Study: Object Manipulation Task

To assess the interplay between the compatibility of object features and hand posture preferences during motor planning, we used a new experimental procedure. This has been described in a previous study (Rounis et al., 2017).

This task involves the manipulation of a familiar object, namely, a cup, in which we varied the compatibility of the initial start posture of the hand with the orientation of the object. The former manipulation was indicated by specifying the position to be grasped on the cup and determined the preferred end posture for the action (whether it was comfortable, usually when ending in a supinated position, or uncomfortable, when ending in a pronated position). The latter manipulation determined the


physical properties of our target object by presenting the cup in its upright position, which is favorable for object use as the open end of the cup was "up," or upside-down position. The goal of the action, provided with a verbal instruction, was to either lift or turn the cup, which resulted in different action outcomes, based on the end-state comfort effect.

### Materials

A cup with no handle ("bodum" cup; **Figure 1**) was used as the target object in this experiment. The cup dimensions were as follows: 50 mm wide at its base, 98 mm wide at its top, and 118 mm in height, and it weighed 270 g. The center of mass of the cup was located 9.8 cm from its base. It was placed at the center of a wooden platform measuring 20 × 10 cm that sat on top of a cedrus response box 30 cm in front of the participants. On 50% of the trials, the cup was upright, and in the remaining 50%, it was oriented upside down.

### Task

Participants used their dominant (right) hand to perform this task. In the case of hemiparesis, patients used their non-dominant hand. A total of 11 patients used their left hand to complete the task. Their hand rested on a keyboard at baseline, between trials. A trial started with the opening of liquid crystal "PLATO" spectacles (Translucent Technologies, Toronto, ON, Canada). This allowed for the timely visualization of the object on the pad. A simultaneous verbal auditory instruction triggered from the computer, lasting 1 s indicated the action to be performed on the cup. The action was either to "lift" or to "turn" the cup (50% of the trials were allocated for each instruction, respectively).

A green horizontal line on the cup specified the initial hand posture to be used. If the line was at the top, participants had to grasp the cup using a supinated wrist posture, with their thumb facing "up." If the line was placed at the bottom, participants were instructed to grasp the cup using a pronated wrist posture, with their thumb facing "down." In each case, participants were asked to align their thumb and forefinger with the line. These grasps were made independent of the cup orientation. Hence, in 50% of the trials, the initial wrist posture was congruent with the cup orientation (if the line was on the same side as the open end of the cup), and this was the case whether the cup was in its upright position (in which case, the line was on the side of the open end of the cup and the grip was supinated) or upside down (in which case, the grip was pronated). We carefully designed our cup selection task so that stimuli were presented vertically rather than horizontally (we used a cup with no handles). This was to prevent a possible confound of visuospatial attention. Previous studies have used central stimuli to distinguish motor attention from visuospatial attention (Rounis et al., 2007). The two are separable both in terms of their anatomical localization (motor attention is lateralized to the left and centered in the supramarginal gyrus) and in terms of its function in orienting attention in a limbcentered representation of space (Rushworth et al., 2001; Rounis et al., 2007; Rinne et al., 2018).

The action, either to lift or turn the cup, determined the final end posture, which was again either pronated or supinated depending on the initial grip instruction and the action.

When the action was completed, participants returned their hand to the resting position, which led to the closure of the PLATO spectacles. Performance of this task was under full direct vision, and the spectacles only closed after completion of the action and upon return to the resting position. They therefore remained open for the action for an average of 4 s (±0.5 s). Participants were asked to complete their action as quickly and as accurately as possible.

The experiment was programmed on Matlab 2014b using Psychtoolbox version 3.0, triggered from a Windows PC. The experimenter recorded errors or adjustments in the grasp position and changed the cup condition for the next trial. The movements performed on each individual trial were videorecorded and reviewed offline, to assess the movement accuracy. Post hoc analyses of the movements revealed that the cups were lifted on average 10.5 cm from the platform for "lift" actions and 10.7 cm from the platform for "turn" actions.

All trial types (grasp top or bottom of the cup, lift or turn) were presented in a pseudorandom order in a total of 17 miniblocks, to ensure that all trial conditions were repeated the same number of times, for averaging. Each miniblock consisted of one trial from each of the eight trial conditions. The first set of miniblocks was always eliminated from the analysis. The total number of trials per session was 136, of which the first 8 were discarded, so only 128 were analyzed for each participant.

This arrangement allowed us to measure response times at two time points, reflecting movement preparation for the action performed in this task. The initiation time measured the time between the stimulus onset (corresponding to the opening of "PLATO" goggles allowing viewing of the cup and verbal instruction) and the time at which the participants lifted their hand from the resting position on the spacebar to initiate an action toward the cup. The movement duration was the time between the release of the spacebar and the lift of the object from the platform, measured by the trigger of buttons from the cedrus box on which the platform was positioned (**Figure 1**, bottom panel).

The choice of these timings was based on results from our previous study in Rounis et al. (2017), in which we observed different effects of affordance and end-state comfort. The initiation times represented planning during which participants likely make a decision about which action to perform and how to implement it (Welford, 1968; Wong et al., 2015). The MT measured the reach to grasp the object, reflecting effector-based movement implementation (Cisek, 2005; Bub and Masson, 2010).

### Data Analysis

fnhum-13-00465 February 8, 2020 Time: 12:30 # 6

Any errors were recorded by the experimenter using a graphical user interface. In addition, correct response latencies lower than 200 ms or longer than 3500 ms for initiation times were excluded as outliers. The lower and upper bounds chosen for these time points were set so that no more than 0.5% of correct responses were excluded either due to being classified as a false start or as an unusually prolonged response [see Ulrich and Miller (1994); Bub and Masson (2010) for a similar approach].

We hypothesized that actions with the initial grasp oriented toward its open end (or its lip) would be faster than a grasp to the closed end of the cup, as observed in our previous study (Rounis et al., 2017). For an upright cup, this would correspond to a supinated grasp; and for a cup oriented down, a pronated grasp (**Figure 1**). This concurs with previous studies that have shown that participants are more likely to choose a hand orientation appropriate for the object's use (Creem and Proffitt, 2001; Herbort and Butz, 2011).

The correct responses for errors and for each time point (initiation and movement durations) were submitted to separate analyses of variance (ANOVAs) using IBM SPSS Statistic 22 for Windows software (SPSS Inc., Chicago, IL, United States). The type I error rate was set at 0.05 for the analyses reported here. Greenhouse–Geisser correction for degrees of freedom was used when assumption of sphericity was not met. Interaction effects were evaluated with paired t-tests (p < 0.05 or p = 0.05), assuming equal variance within group and unequal variance for between-group comparisons.

A mixed-model, nested ANOVA was used to identify betweensubject effects of GROUP (with three levels: 10 left hemisphere patients with, 12 without apraxia, and 18 healthy volunteers), investigating the three within-subject factors (with two levels each), as reported in our previous study (Rounis et al., 2017). These included COMPATIBILITY (previously referred to as an initial grasp preference (or "affordance") for positioning the thumb toward the cup's open end), ACTION (which determined whether the task was to "lift" or to "turn" the cup), and the END-STATE COMFORT of the hand after the action is completed [the end state being comfortable ("thumb up") or not ("thumb down")]. **Figure 1** shows the task conditions and experimental setup. **Table 3** outlines the behavioral results on this study for each patient category and task conditions, summarized in terms of compatibility and end state comfort effects.

### RESULTS

### Neuropsychological Tasks

A total of 10 out of 22 left hemisphere stroke patients performed below the cutoff scores on assessments of gesture production and meaningless gesture imitation, indicating ideomotor apraxia. There was a complete agreement between the raters in defining those patients who had ideomotor apraxia (Cohen's kappa = 1). The patient results are reported in **Table 1**.

Performance on the single-object use task revealed no significant differences between patients (average score = 11.64, SEM = 0.16) and healthy volunteers (average score = 11.94, SEM = 0.05, t = -1.8, p = 0.08).

We found that the level of stroke impairment on the hemiparetic hand measured with ARAT significantly correlated with gesture production (rho<sup>22</sup> = 0.520, p = 0.004), meaningless gesture imitation (rho<sup>22</sup> = 0.556, p = 0.002), and the gesture recognition task (rho<sup>22</sup> = 0.416, p = 0.018).

Comparisons between results in the Rey-Osterrieth figure copy task (subdivided in total score as well as subscores for left-, middle-, and right-sided copies) and each of the praxis tasks revealed no significant correlations between the two (p > 0.1).

### Behavioral Task

#### Error Rates

Healthy volunteers made a total of 3.03% errors, and stroke patients made a total of 17.86% errors. Error trials were discarded from reaction time and MT analyses (70 out of 2304 trials in healthy volunteers, 503 out of 2816 in stroke patients). Of note, a review of error types identified that most errors related to incorrect or adjustments in grasp [with <0.04% of errors relating to other factors such as incorrect initiation or action (lifting instead of turning)]. Due to insufficient numbers in categories other than grasp errors, errors were combined and analyzed according to trial condition rather than error type.

Error rates, and initiation time and MT were submitted into three separate mixed-model nested ANOVA, with a betweensubject factor of GROUP (three levels: healthy volunteers and left hemisphere stroke patients WITH and WITHOUT apraxia) and within-subject factors of COMPATIBILITY (two levels: with the initial hand posture matching the object orientation, with the thumb orientation toward the lip of the cup), ACTION (two levels: lift versus turn), and END-STATE COMFORT (two levels: starting with a pronated to end in a supinated, comfortable grasp, or starting with a supinated grasp to end in a pronated, uncomfortable one).

The analysis on error rates revealed the main effects of COMPATIBILITY [F(1,37) = 17.2, h<sup>2</sup> = 0.32, p < 0.0001], with on average 2.75 more errors on incompatible than on compatible trials. There was also a significant effect of END-STATE COMFORT [F(1,37) = 26.1, h<sup>2</sup> = 0.41, p < 0.0001], with on average 7.35 more errors on trials that ended uncomfortably than those ending comfortably. The main effect of ACTION was not significant [F(1,37) = 0.15, h<sup>2</sup> = 0.004, p = 0.7]. These significant main effects interacted with the between-subject factor of GROUP, leading to significant GROUP by COMPATIBILITY [F(1,2) = 4.35, h<sup>2</sup> = 0.19, p = 0.02] and GROUP by END-STATE COMFORT [F(1,2) = 4.28, h<sup>2</sup> = 0.19, p = 0.02]) effects. The results of post hoc analyses describing these effects are outlined in the **Supplementary Material** and in **Figure 2**.

### Initiation Times

The analyses on initiation times revealed the significant main effects of COMPATIBILITY [F(1,37) = 7.03, h<sup>2</sup> = 0.16, p = 0.01], indicating that compatible trials were initiated 30.5 ms (SEM = 2.9 ms) more rapidly than incompatible trials. The main effects of ACTION [F(1,37) = 10.12, h<sup>2</sup> = 0.21,

TABLE 3 | Results for error rates, and initiation and movement times, in compatible and incompatible trials, and in trials that ended comfortably versus the ones that did not.


Ms, milliseconds, SEM, standard error of the mean.

p = 0.003] and END-STATE COMFORT [F(1,37) = 19.02, h <sup>2</sup> = 0.34, p < 0.0001] were also significant. Lift actions were initiated faster by an average of 64.5 ms (SEM = 2.5 ms) than turn actions, and initiation times for actions that ended comfortably were 63.25 ms (SEM = 10.35 ms) shorter than for actions that ended uncomfortably. There were two-way interactions of GROUP BY END-STATE COMFORT [F(2,37) = 3.99, h<sup>2</sup> = 0.18, p = 0.03] and ACTION BY END-STATE COMFORT [F(1,37) = 7.04, h <sup>2</sup> = 0.32, p = 0.01].

The results from our post hoc analyses appear in the **Supplementary Material** and are shown in **Figure 3**.

#### Movement Times

This analysis revealed the main effects of COMPATIBILITY [F(1,37) = 8.00, h<sup>2</sup> = 0.18, p = 0.008], indicating that MTs for trials in which the hand and cup orientation were compatible (hand grasping the cup from its lip) were significantly shorter (72 ms; SEM = 14.8 ms) than the ones in which participants grasped the closed end of the cup (t<sup>39</sup> = -2.25, p = 0.03). There were significant main effects of ACTION [F(1,37) = 23.89, h<sup>2</sup> = 0.39, p < 0.0001] as MTs for lift actions were on average 143 ms (SEM = 24.5 ms) shorter than those for turn actions and END-STATE COMFORT [F(1,37) = 35.96, h<sup>2</sup> = 0.49, p < 0.0001], indicating that MTs for actions that ended comfortably were on average 99.55 ms (SEM = 16.9 ms) shorted than those for actions that ended uncomfortably.

There were significant two-way interactions of COMPATIBILTY by GROUP [F(1,37) = 3.39, h<sup>2</sup> = 0.15, p = 0.045] and END-STATE COMFORT by GROUP [F(1,37) = 4.44, h<sup>2</sup> = 0.19, p = 0.019]. The ACTION by GROUP interaction was not significant [F(1,37) = 2.12, h <sup>2</sup> = 0.18, p = 0.35]. Post hoc analyses are shown in the **Supplementary Material**.

lift actions, when there was no significant difference in initiation times for turn actions whether they ended comfortably or not. <sup>∗</sup>p < 0.05, ∗∗p < 0.001.

There were significant two-way interactions of ACTION by END-STATE COMFORT [F(1,37) = 80.41, h<sup>2</sup> = 0.685, p < 0.0001] and COMPATIBILITY by ACTION [F(1,37) = 6.49, h <sup>2</sup> = 0.15, p = 0.015].

The former interaction reflected the difference in end-state comfort advantage on MTs between lift and turn actions. Post hoc t-tests revealed that in lift actions, MTs were on average 229 ms (SEM = 12.5 ms) shorter if they ended comfortably (t<sup>39</sup> = -9.8, p < 0.0001); whereas in turn actions, there was no significant END-STATE COMFORT effect on MTs (t<sup>39</sup> = 0.8, p > 0.1).

The COMPATIBILITY by ACTION interaction arose because there was no effect of compatibility on MTs for lift actions (t<sup>39</sup> = -1.5, p > 0.1), whereas there was a significant effect of compatibility on turn actions (t<sup>39</sup> = -3.04, p = 0.004).

There were also significant three-way COMPATIBILTY by ACTION by END-STATE COMFORT [F(1,37) = 14.18, h <sup>2</sup> = 0.28, p = 0.001] and four-way interactions of GROUP by COMPATIBILTY by ACTION by END-STATE COMFORT [F(2,37) = 6.42, h<sup>2</sup> = 0.26, p = 0.004].

The post hoc analyses for these are reported in the **Supplementary Material** and are shown in **Figure 4**.

### DISCUSSION

This study investigated factors influencing motor planning and performance when grasping to manipulate a familiar object in healthy volunteers and left hemisphere stroke patients with and without apraxia. We identified that both factors relating to perceptual processing of the object ("compatibility") and the ones that related to action outcomes, in this case determined by the "end-state comfort" effect, were differentially modulated during task performance in patients with apraxia following a stroke. We identified these differences using measures of motor preparation and performance.

In the sections below, we outline our main findings and discuss these in relation to traditional models of apraxia and brain mechanisms underpinning the disorder.

### Deficits in Integrating Perceptual Knowledge During Goal-Directed Actions in Apraxia

Our study identified differences in performance, measured using error rates and speed of movement, between healthy volunteers and stroke patients with and without apraxia. Error rates and initiation times were more elevated in stroke patients compared to healthy volunteers.

A study by Belanger et al. (1996) demonstrated that, when testing patients on neuropsychological tasks, apraxic deficits related not to the categories of praxis tasks but to task difficulty. This effect was independent of movement type or methods of movement elicitation. Our study supports these findings in that we identified that praxis deficits in our patient cohort significantly correlated with severity of impairment following their stroke, measured on the "ARAT" scale.

However, our results go further than that. We observed that patients with apraxia showed deficits that were specific when they manipulated a common object. This is despite the fact that they showed near-normal performance when grasping objects on apraxia screening tasks.

Stroke patients with and without apraxia made significantly more errors and were slower at initiating actions for trials that ended uncomfortably compared to the ones that ended comfortably, compared to healthy volunteers. This would support a generic deficit in completing actions for which the outcome was difficult or unrewarding, such as when ending in an uncomfortable hand position.

In addition, patients with apraxia made more errors on incompatible trials. There was a significant modulation of the degree of sensitivity of response-outcome (end-state comfort)

effects measured during MTs, by compatibility of the initial hand position with the object, and action type (lift or turn). In compatible trials, there were no significant differences in MTs between turn actions that ended comfortably and uncomfortably in healthy volunteers and patients with no apraxia. However, in patients with apraxia, MTs were paradoxically shorter for turns that ended uncomfortably compared to the ones that ended comfortably. This was because they displayed an increased cost in inverting their grasp to reach the cup, despite the action leading to a preferred action outcome. This result suggests that in patients with apraxia, the action goal was trumped by the biomechanical requirement of the turn action, which was easier, in the uncomfortable end-state condition compared to the comfortable one. Moreover, patients with apraxia showed delayed MTs for actions that were not compatible even when they ended comfortably, suggesting an overreliance on compatibility at the expense of the end-state comfort effect in these patients.

### Comparisons of Performance in This Task With Traditional Theories of the Disorder

Traditional theories of apraxia distinguish patients based on deficits they show in neuropsychological tasks. "Ideational" apraxia can be tested by demonstrating deficits in single-object use, or in sequencing errors in multiobject use (Poeck, 1986). "Ideomotor" apraxia can be elicited by asking patients to imitate gestures and by demonstrating spatiotemporal errors (rather than errors of content) when patients are asked to pantomime or use objects (Wheaton and Hallett, 2007). Cognitive models of the disorder conjecture separable underlying mechanisms for these types of deficits: one requires semantic processing of gestures based on knowledge, and the other requires implementation of gestures based on structural–mechanical problem solving. Nevertheless, these subdivisions have failed to fully explain the disorder, with some authors arguing that these represent one and the same problem, leading to the use of inconsistent terminology (Buxbaum, 2001; Hanna-Pladdy and Rothi, 2001).

As in other studies, we did not find deficits that would be predicted based on this dichotomy. We discuss below how deficits in ideational or ideomotor apraxia would be anticipated to influence our results, and reasons for identifying both types in our task, which we attribute to deficits in incorporating affordances in a "hierarchy of goals" framework (Bekkering et al., 2005).

### Comparisons of Our Results in Relation to Deficits Pertaining to Ideational Apraxia

Patients with ideational deficits, such as patients with semantic dementia, demonstrate a preserved ability to

elicit how an object can be grasped from its structure (Hodges et al., 1999), while having deficits in identifying or eliciting appropriate actions relating to their use (Hodges et al., 2000; Bozeat et al., 2002). These deficits can be viewed as problems in identifying and implementing action "goals." In multiobject use tasks, these problems could be confounded by an inability to sequence and task difficulty (Belanger et al., 1996; Hanna-Pladdy and Rothi, 2001).

In our task, we would predict that if our patients had ideational deficits, they would show impairments in processing the end-state comfort effect. One possibility would be that patients with a deficit in this task condition might show an inability to differentiate comfortable from uncomfortable end postures, leading to prolonged error rates or movement preparation times in task conditions ending comfortably. Another possibility, which was indeed observed in study, was that stroke patients (with and without apraxia) would overrely on the end-state comfort effect such that patients showed a greater deficit in task conditions that ended uncomfortably than did healthy volunteers. Errors and MTs in these conditions were more elevated in stroke patients with apraxia compared to healthy volunteers. It is interesting that we did observe this deficit in our patient cohort, despite the fact that they did not appear to have ideational deficits on neuropsychological testing (noting that we had not tested them on multiobject use). Deficits relating to ideational apraxia vary in their definition, creating confusion on whether the disorder is truly separable from ideomotor deficits (Hanna-Pladdy and Rothi, 2001). We would argue that our task suggests an element of ideation deficit in our patients, based on original descriptions of this deficit being related to sequencing and complexity (Poeck, 1986).

### Comparisons of Our Results in Relation to Deficits Pertaining to Ideomotor Apraxia

Ideomotor deficits have been described in the apraxia literature as corresponding to problems in "mechanical problem solving" (Osiurak et al., 2008a; Goldenberg and Spatt, 2009). These lead to deficits in accurately grasping or manipulating an object despite knowing how to use it (Daprati and Sirigu, 2006). Recent studies have demonstrated that patients with ideomotor deficits have problems inferring function from object structure, relating to "affordances" (Barde et al., 2007). Affordances represent a mechanism that triggers actions based on a stimulus–response association, which is dependent on task demands (Balleine and Dickinson, 1998; Bub and Masson, 2010). Affordances can elicit actions that represent movement preference to an object. In the case of a cup, as used in our experiment, this would be related to a compatibility effect between cup and hand orientation when it is grasped. In our study, both healthy volunteers and patients preferred to grasp the cup from its open end.

Studies in patients with apraxia have shown an overreliance of these patients in compatible (or "afforded") trial conditions. A series of experiments have identified that patients with apraxia show increased deficits when manipulating "conflict" objects, which can elicit different grasp and hand movements, for moving or using them (Jax and Buxbaum, 2013; Watson and Buxbaum, 2014, 2015). Lee et al. (2014) identified that patients with ideomotor apraxia were unable to select actions elicited by object affordances in the presence of distractors. In a similar vein, studies on patients with alien-limb syndrome have shown deficits in performing tasks in which conflicting movements may be elicited by affordances (Riddoch et al., 1998; McBride et al., 2012).

### Summary of the Conditions Causing Deficits in This Task

Our results were able to identify specific task conditions in which deficits were more likely to occur in apraxic patients compared to patients with no apraxia and healthy volunteers. These involved both deficits in processing compatibility and in end-state comfort. When tested on traditional neuropsychological tasks, our patients exhibited predominantly ideomotor deficits. However, the deficits identified in performance of this task were more complicated to interpret and suggest a combination of more generic (complexity-related) and more specific (affordance- and end-state comfort-related) deficits.

Patients with apraxia had greater difficulty in completing actions that ended uncomfortably. This could possibly suggest a generic deficit in complex task performance (Belanger et al., 1996) or in identifying action goals, both of which have traditionally been attributed to ideational deficits (Hanna-Pladdy and Rothi, 2001).

Conversely, deficits and delays in processing incompatible trial conditions would reflect deficits that have previously been attributed in "mechanical problem solving." These deficits may reflect an overreliance of apraxic patients on afforded trials, at the expense of other trial conditions (Riddoch et al., 1998; McBride et al., 2012; Lee et al., 2014). This could result from an inability to reprogram an action in relation to its goal instead of its affordance. The other possibility would be that these deficits arise due to a failure to incorporate perceptual features of objects into sequential action goals.

In a study by Bekkering et al. (2005), patients with ideomotor apraxia made imitation errors due to deficits in implementing action goals, hierarchically. Their deficits were present both when targeting body parts and when targeting objects, which the authors described as demonstrating deficits in a "hierarchy of goals" framework (Grafton and Hamilton, 2007). In a similar vein, our patients with ideomotor apraxia were compromised when carrying actions that ended in an uncomfortable end state, compared to other groups. However, they were particularly compromised when performing actions that were incompatible. Habitual actions associate stimuli with responses that were previously rewarding (Herbort and Butz, 2011; Voon et al., 2015). There is evidence that the two (goal-directed and habitual) systems parallel each other (Balleine and Dickinson, 1998; Herbort and Butz, 2011; Dolan and Dayan, 2013). Our

results suggest an interference of one system on the other (Chainay and Humphreys, 2002).

Taken together, our results have demonstrated that patients with ideomotor apraxia may have deficits in selecting among competing actions elicited by parallel action systems: a perceptual/habitual and a goal-directed system. These deficits are important in terms of identifying how patients with apraxia may fail in activities of daily living (Bickerton et al., 2012). One method traditionally used for rehabilitation of this disorder (West et al., 2008), namely, errorless learning, could be helpful both in recreating a habitual, stimulus–response, movement repertoire and in enabling this to be more readily incorporated during action sequences.

### CONCLUSION

In this study, we investigated the performance of patients with ideomotor apraxia on a task that involved manipulating a familiar object, namely, a cup. We compared their performance with patients who did not have apraxia and healthy, age-matched, volunteers. The task required participants to either lift or turn the cup, which led to comfortable or uncomfortable end states. We observed that patients with apraxia were impaired both when actions were incompatible with the familiar way of grasping the object (its "affordance") and in conditions that ended with an uncomfortable end state, suggesting a deficit in goal-directed actions. Both suggest an increased reliance of patients with ideomotor apraxia on these systems compared to stroke patients without apraxia and healthy volunteers.

Rather than an impairment in representing "affordances," this study suggests that praxis deficits may affect sensitivity to actions elicited by affordances and goal-directed actions in a dynamic fashion. These system are known to operate in parallel and may be competing at specific times of action implementation (Chainay and Humphreys, 2002; Voon et al., 2015). Further studies investigating how these interact dynamically, using decisionmaking tasks, would help elucidate the exact deficits observed in these patients.

### DATA AVAILABILITY STATEMENT

The datasets generated for this study are available on request to the corresponding author.

### REFERENCES


### ETHICS STATEMENT

Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article. The patients and participants provided their written informed consent to participate in this study, which was approved by the National Research Ethics Service (NRES) South Central Berkshire Health Research Authority (14/SC/0074).

### AUTHOR CONTRIBUTIONS

ER conceptualized the task and study. MD programmed the task on MATLAB. GP carried out the data collection on all participants including patients and healthy volunteers. ER and ZZ analyzed the data and edited the manuscript. ER wrote the manuscript.

### FUNDING

This study was supported by the following grants, made to ER: the British Medical Association (H. Lawson grant) for patient testing, the Oxford University Clinical Academic Graduate School funding for Clinical Lecturers for research assistant salary support (GP), and The Oxford University Hospitals Charitable Trust for publication of this article.

### ACKNOWLEDGMENTS

We would like to thank the patient and healthy volunteer participants for taking part in this study. We are grateful to the late Professor Glyn Humpreys, with whom this study was initially conceptualized.

### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnhum. 2019.00465/full#supplementary-material



Welford, A. T. (1968). Fundamentals of Skill. London: Methuen.

fnhum-13-00465 February 8, 2020 Time: 12:30 # 13


**Conflict of Interest:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2020 Pizzamiglio, Zhang, Duta and Rounis. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Effect of External Force on Agency in Physical Human-Machine Interaction

#### Satoshi Endo<sup>1</sup> \*, Jakob Fröhner <sup>1</sup> , Selma Music´ 1 , Sandra Hirche<sup>1</sup> and Philipp Beckerle2,3

<sup>1</sup> Chair of Information-Oriented Control, Department of Electrical and Computer Engineering, Technical University of Munich, Munich, Germany, <sup>2</sup> Elastic Lightweight Robotics Group, Department of Electrical Engineering and Information Technology, Robotics Research Institute, Technische Universität Dortmund, Dortmund, Germany, <sup>3</sup> Institute for Mechatronic Systems, Mechanical Engineering, Technische Universität Darmstadt, Darmstadt, Germany

In the advent of intelligent robotic tools for physically assisting humans, user experience, and intuitiveness in particular have become important features for control designs. However, existing works predominantly focus on performance-related measures for evaluating control systems as the subjective experience of a user by large cannot be directly observed. In this study, we therefore focus on agency-related interactions between control and embodiment in the context of physical human-machine interaction. By applying an intentional binding paradigm in a virtual, machine-assisted reaching task, we evaluate how the sense of agency of able-bodied humans is modulated by assistive force characteristics of a physically coupled device. In addition to measuring how assistive force profiles influence the sense of agency with intentional binding, we analyzed the sense of agency using a questionnaire. Remarkably, our participants reported to experience stronger agency when being appropriately assisted, although they contributed less to the control task. This is substantiated by the overall consistency of intentional binding results and the self-reported sense of agency. Our results confirm the fundamental feasibility of the sense of agency to objectively evaluate the quality of human-in-the-loop control for assistive technologies. While the underlying mechanisms causing the perceptual bias observed in the intentional binding paradigm are still to be understood, we believe that this study distinctly contributes to demonstrating how the sense of agency characterizes intuitiveness of assistance in physical human-machine interaction.

#### Edited by:

Mariella Pazzaglia, Sapienza University of Rome, Italy

#### Reviewed by:

Serena Ivaldi, Institut National de Recherche en Informatique et en Automatique (INRIA), France Hidenobu Sumioka, Advanced Telecommunications Research Institute International (ATR), Japan Ryuji Yamazaki, Osaka University, Japan

> \*Correspondence: Satoshi Endo s.endo@tum.de

#### Specialty section:

This article was submitted to Cognitive Neuroscience, a section of the journal Frontiers in Human Neuroscience

> Received: 28 June 2019 Accepted: 12 March 2020 Published: 08 May 2020

#### Citation:

Endo S, Fröhner J, Music S, Hirche S ´ and Beckerle P (2020) Effect of External Force on Agency in Physical Human-Machine Interaction. Front. Hum. Neurosci. 14:114. doi: 10.3389/fnhum.2020.00114 Keywords: human-centered control, shared control, human-robot interaction, autonomy, agency, haptics

### 1. INTRODUCTION

In the face of growing elderly population, automated assistive technologies such as powered exoskeleton and rehabilitation devices are expected to play a crucial role for meeting societal demands (Beckerle et al., 2017). A large portion of such assistive technologies involve physical human-machine interactions (HMI) in which a robot is physically coupled with a user to (semi-)autonomously guide the motion (e.g., Marchal-Crespo and Reinkensmeyer, 2009). In order to meet user-dependent requirements in guiding behavior, usually explicit control objectives are minimized. Typically, these cost functions are task-oriented and use a set of performance indices such as the muscular effort (Hamaya et al., 2017) or the task completion time (Erdogan and Argall, 2017) to define the utility of the autonomous behavior of a system. As these control schemes commonly focus on the explicit performance of the user, user experience is notecessarily considered, which might have a strong impact on the user acceptance and long-term usage. However, user experience is of high relevance for designing assistive technologies in which the human is at the center of a control loop (Limerick et al., 2014; Beckerle et al., 2017) and methodologies for objective monitoring of user experience will be valuable for advancements of HMI systems.

To this end, concepts from psychological research may provide means of validating human-centered control designs from the users' perspectives. Previous research has indicated the experienced incorporation of an intelligent tool, i.e., embodiment, appears to be a promising quality measure for a semi-autonomously controlled system (Fröhner et al., 2019). As a subcomponent of embodiment (Longo et al., 2008), in particular, the sense of agency (SoA) refers to an inference about authorship of a sensory event and a believe about whether the sensory outcome was caused by the action of oneself. Accordingly, SoA over an intelligent tool seems to be distinctly relevant when assessing shared-autonomy tasks, which we assume to relate directly to intuitiveness. Previous research on SoA suggests that the central nervous system continuously monitors a discrepancy between the intended movement and corresponding sensory feedback (Wolpert et al., 1995) and evaluates whether the observed sensory event was self-induced (Blakemore et al., 2002). As a result, SoA is reduced, for example, when contiguity between the movement and its sensory outcome is temporally or spatially perturbed, even when their action had resulted in causing the event (Sato and Yasuda, 2005; Farrer et al., 2008). In reverse, people may experience agency over an action produced by another individual when the actions of individuals are sufficiently assimilated with own desired outcome (Dewey and Knoblich, 2014). This is an interesting observation as, in shared-autonomy settings, some external autonomous system might be able to form a collective agency when the assisting force approximates the desired state of the user. Considering this, SoA may become a good holistic connotation for quality of a control system. In order to demonstrate the relationship between the SoA and a physically assistive device, therefore, the present study investigates whether SOA is modulated by the quality of external assistive force, by means of adherence to the human task goal.

In previous studies, questionnaires have been used to reflect users' opinions regarding control designs, (e.g., Lopez-Samaniego and Garcia-Zapirain, 2016) as the internal states of the user such as the intuitiveness and usability of the devices cannot directly be measured. However, using questionnaires may be impractical for tuning control parameters, as an explicit survey of subjective opinions can easily be modulated by a variety of factors, and reliable measurement would require impractically many samples. Furthermore, the physical HMI task and administration of the questionnaire has to take place separately, which appears not suitable for online adaptation of control parameters in a fine temporal resolution. One of the most promising methodologies may be the intentional binding effect (IBE) which has been used to empirically study perceptual changes associated with SoA. IBE is a psychological phenomenon in which the temporal perception of two consecutive events being reported as shorter than the real time-lapse if the events are triggered by the participant herself/himself (Haggard et al., 2002). It is considered that coupling of the self-induced motion and its outcome in the conscience experience attracts the temporal experience of the two sensory events, resulting in perception of a selfinduced action outcome being perceived as earlier than it physically is. This subjective contraction of time is considered to be an implicit measure of SoA as it reflects the internal representation of self-produced motion. Previously, the IBE paradigm was used to evaluate the attribution of agency in the presence of autonomic assistance over a series of discrete subtasks (Berberian et al., 2012). In their task, the computer and the participants shared subtasks in different degrees to control an emulated aircraft, and IBE was used to show that the degree of task sharing (or autonomy) modulated SoA. Thus, the autonomy delegated subtasks between the human and the machine, but the action performed by the users were always intact and not perturbed by the autonomy. In applications of physical HMI technologies, on the other hand, the external force from an assistive system continuously and directly influences the motion of the user and the autonomy relates not only to the level of involvement, but also to the adherence to the desired outcome of a user. As mentioned above, there are indications that embodiment can serve as a measure in such situations (Fröhner et al., 2019). Yet, another IBE-based study found that SoA over a robotic hand does not necessarily depend on the embodiment of the artificial limb (Caspar et al., 2015). Thus, we need to understand to what extent SoA is influenced by the effect of an external force on own motion.

In the present study, we varied the relevance of the force to the task-at-hand by providing the guiding force that can help or perturb the task performance of the user, and applied an IBE paradigm adapted for a physical HMI task to measure the perceptual bias resulted from the presence of the external guiding force. Specifically, the participants performed a reaching task to a target location using a force-feedback device and delayed visual feedback of a virtual cursor, a scenario that is likely to occur in teleoperation applications (Chopra et al., 2003). SoA was investigated using an adapting IBE paradigm in which the participants reported perceived delay of the own motion in the presence of the additional guiding force. We hypothesized IBE is sustained when guiding force adheres to a desired motion of the participants, while IBE is diminished when the force results in undesirable outcome.

## 2. METHODS

### 2.1. Participants

Twenty-two participants took part in this study. The participants were healthy young adults (age = 25.0 ± 3.0 years old). Three were female, and all performed the task with their right hand. The study was conducted according to the Declaration of Helsinki, and all participants gave written informed consent before the participation. The study was approved by the research ethics committee of the Technical University of Munich (project no. 205/14).

### 2.2. Stimuli and Apparatus

During the experiment, the participants held a manipulandum with their right hand to control a cursor displayed on a computer screen. The visual position of the cursor and the targets were presented with a mirrored PC monitor. A 35 × 25 mm surface mirror glass (Screen-Tech, Germany) was horizontally placed 20 cm above the center of the manipulandum (**Figure 1A**). The PC monitor was placed face-down another 20 cm above the mirror. The location of the mirrored visual cursor was manually calibrated so it was aligned with the center of the handle position of the manipulandum as the participant's forehead was rested on the padded bar. The kinesthetic rendering is actualized by a Thrusttube Module (Copley Controls, USA). The actuation device consists of two sets of a single rail stage and a linear servo motor driven cart stacked perpendicularly to create a planar workspace. The device was positioned so the x-axis and y-axis of the workspace were respectively aligned with the mediolateral and sagittal axes of the participants. A vertical handle (1 cm radius) is mounted on the cart with a JR3-67M25 6-axis force/torque sensor (JR3 Inc., USA) between them in order to measure the force applied by a participant. The different stimuli were prepared for the reaching, delay estimation, and questionnaire phases as follows.

### 2.2.1. Reaching Phase

The participants viewed a display showing a cursor, one starting platform, and three target platforms on a gray background (**Figure 1B**). The cursor was a black colored disk with 0.25 cm radius, and its motion was controlled by the participant using the manipulandum. The visualization of the cursor was delayed at one of three predefined latencies (see section 2.4) from the start of a trial until it reached the target, but otherwise the cursor position was aligned with the current manipulandum position, i.e., when returning to the starting platform. The starting platform was a 1 cm black radius circle placed midline at approximately 5 cm from the base of the screen. The cursor and the starting platform had the same color on purpose to prevent the participants from noticing when the visual cursor delay was introduced. The target platforms were a 3 × 6 cm gray colored rectangle with a black frame. One of them was placed on the midline so the front surface is 15 cm away from the starting platform and the remaining two platforms were rotated by ± 30 degrees at the same distance. The target for the reaching task was displayed by turning the color of one platform to white. When the manipulandum and the delayed cursor reached the target, the color changed into black and white, respectively. This allowed the participants only to focus on the period the target platform was colored black for the subsequent delay estimation phase.

### 2.2.2. Delay Estimation and Questionnaire Phases

The delay estimation phase was used for registering the perceptual experience of the visual cursor delay similar to previous studies (Haggard et al., 2002; Caspar et al., 2015), and the questionnaire phase was used for obtaining self-reported SoA as a comparison to the objective IBE-based SoA. The manipulandum was locked during the entire phase and it was not movable. For both phases, a visual analog scale with a width

and guiding force (fg) in a single trial. (D) Scales for delay estimation and questionnaire.

of 15 cm was displayed at the center of the screen. For delay estimation, the continuous scale ranged from 0 to 1,000 ms without any intermediate points. For SoA questionnaire, the participants rated on the continuous scale of "strongly disagree" to "strongly agree." For analyzing the self-reported SoA using the questionnaire, the agency items of the questionnaire from Caspar et al. (2015) were reformulated to fit our control-oriented task in the virtual environment. All items were displayed above the visual analog scale once at a time in a fixed order as shown in **Table 1**. The lateral force measurement of the manipulandum navigated the circle nozzle (0.5 radius), initially presented on the middle of the scale, to indicate the response by the participant. A forward push to the device above 10 N indicated a registration of the response.

### 2.3. Control Strategy of the Manipulandum

The human operator interacts with the planar actuation device by grasping the handle to move the cursor in virtual reality, as depicted in **Figure 1**. The manipulandum is controlled to have a simulated mass of 5 kg. During the experiment, a guiding force was applied to the manipulandum, with the force profile of a normal distribution curve with 10 N peak force spanned over 300 ms pushing to the target platform surface after 200 ms from the movement onset. Exemplary velocities of the device and forces applied by the human, as well as the guidance forces are depicted in **Figure 2** for the correct, incorrect, and no guidance, respectively. For more details on control of the manipulandum, see **Appendix**.


### 2.4. Design

We used a 3 × 3 within-subject design, consisting of directional correctness of the guiding force and delay of the visual cursor motion as the independent variables. The first independent variable was the Guiding force to the manipulandum either in the correct or incorrect directions, or being absent entirely, i.e., "no force." In the correct force condition, the guiding force was always directed to the illuminated target. For the incorrect force condition, the force was pseudo-randomly directed to one of the two non-target directions. In the no force condition, no guiding force was applied. Each force condition was presented as a block of trials in random order across the participants. The target location as well as the non-target force direction were varied every trial so the probability of each event was balanced. The second independent variable was the Delay of the visual feedback for which the visual cursor was delayed by either [300, 500, 700] ms. The participants were informed about the delay prior to the experiment, but were told the latency would have been randomly chosen between 0 and 1,000 ms. The participants completed 135 trials for three blocks, totaling 405 trial for the whole experiment. When the participants failed to reach the correct target, the same trial was repeated at the end of the block in the order of failed trails. The total error rate was 2.7%.

In order to investigate the change in IBE according to the Guiding force, the deviation of the perceived delay from the real delay was calculated, and this perceptual bias was used as a dependent variable for statistical analysis using a 3 (Guiding force) × 3 (Delay) repeated-measures ANOVA. In order to supplement the objective SoA results in terms of IBE, the questionnaire administered at the end of each block addressed the subjective SoA with regard to the force feedback type using

FIGURE 2 | Exemplary velocities of the manipulandum and forces in a single trial. All examples are when the participants reached to the left target, and the incorrect guidance was directed toward the right target. The cursor fully left the starting circle at t = 0 which represented a start of a trial. The increase of f<sup>h</sup> magnitude at around 800 ms is due to the contact with the virtual wall of the target platform.

the analog scale. Thus, the self-reported SoA refers only to the Guiding force and the experience over the Delay conditions was pooled together. The agreements to SoA from the four items were normalized between 0 (strongly disagree) − 1 (strongly agree), and averaged to derive the subjective SoA for statistical analysis using a one-way repeated-measures ANOVA. Post-hoc analyses were carried out with the Bonferroni correction. The alpha value was set to 0.05 for statistical significance. In order to explore the relationship between the self-reported SoA and the time perception bias, the Pearson's correlation was calculated between the questionnaire scores and the perceptual bias. In the experiment, the questionnaire was only administered at the end of each guiding force block, and it reflected experiences over all three visual delays. Thus, the means of the perceptual bias in each guiding force type were calculated to make the variable comparable to the questionnaire scores. The individual differences in the two variables were minimized by performing the z-transformation for data from each participant, resulting in 3 (guiding forces) × 22 (participants) samples. Furthermore, in order to evaluate how the different guiding force types influenced the reaching performance, we analyzed the trial duration, peak interaction force and the manipulation share. As defined in Donner et al. (2018), the interaction force is calculated as

$$f\_{int} = \frac{1}{2} \operatorname{sgn}(f\_h)(|f\_h| + |f\_\mathcal{g}| - |f\_h + f\_\mathcal{g}|),\tag{1}$$

wherein f<sup>h</sup> and f<sup>g</sup> are the forces applied by the human and the guiding force, respectively. Namely, the interaction force represents the force that is compensated by the two agents and has no contribution to the motion of the handle. Thus, excessively high interaction force suggests inefficiency in coordination between the agents, although the interaction force may become a source of communication between agents (Donner et al., 2018). The largest interaction force during a trial was then sampled for statistical analysis as an index of (in)efficiency in interaction. The manipulation force was the non-compensated force which resulted in motion of the cursor,

$$f\_{mnp} = f\_h - f\_{int}.\tag{2}$$

The manipulation share is calculated by normalizing fmnp to the total manipulation force working on the manipulandum, which gives percentage of the participants' force resulted in cursor motion, or dominance in task execution. As fint and fmnp respectively result in 0 N and 100 % in the no guiding force condition, 2 (Guiding force) × 3 (Delay) ANOVAs were used for statistical analysis without the no guiding force condition. Alpha-level of 0.05 was used for statistical significance of all tests.

### 2.5. Procedure

Participants stood in front of the manipulandum to perform the task with the right arm. The chair height was adjusted so the manipulandum handle came slightly above the waist. At the beginning of each trial, the participants held the handle and moved the cursor to the start position by looking into the mirrored display. At this stage, the cursor was not delayed. One second after the cursor was placed on the starting platform, the trial commenced by coloring one of the target platforms. The participants were instructed to complete the task by moving the handle to the white-colored target at a comfortable speed. When the handle arrived at the target, the color changed to black. The participant was then asked to hold the cursor at the target until the delayed cursor arrived to the target, and saw the target color changed back to white. The participants were told to focus on the period the target platform was colored black for the subsequent delay estimation test. After 1 s lapse, the handle was locked to prevent from further motion and a visual display on the screen asked the participant to rate the cursor delay on the visual analog scale. The participant then moved the cursor on the scale by laterally applying force to the handle in order to report their opinion. A forward push by the participant indicated the completion of the rating. When the response was registered, the handle lock was removed and the participant freely moved the handle to the start platform for the next trial. After completion of each experimental block, participants filled in the agency questionnaire with the same procedure as the delay estimation for the four questionnaire items.

Our pilot study indicated that participants found challenging to estimate the time-lapse in less than 1,000 ms time-frame. Thus, a familiarization block was prepared in which all target boxes flashed simultaneously from white to black, held it for a reference time of [200, 400, 600, 800] ms, and then to white again. The procedure was repeated 10 times for each reference time with an inter-stimulus interval of 2 s. The presentation order was counterbalanced between the participants in an increasing or a decreasing order. The reference time was also displayed at the center of the screen to help the participants to learn and calibrate visual time perception. Furthermore, the participants were allowed to practice the task with a random visual cursor delay between 0 and 1,000 ms without external force until they felt comfortable with proceeding with the main experiment after the familiarization block.

### 3. RESULTS

### 3.1. Intentional Binding Effect

The analysis revealed the participants overall perceived the delay to be rather shorter than the actual visual delay (−60.2 ± 88.9 ms, see **Figure 3**). This tendency was stronger with increasing visual delays, as the perceptual delay was the smallest when estimating the 300 ms delay (−24.3 ± 85.8 ms) and the largest with 700 ms delay (−103.4 ± 105.6 ms). The 3 x 3 repeated-measures ANOVA confirmed a main effect of the Delay in the biased delay perception, F(2, 42) = 11.12, p < 0.0005, partial η <sup>2</sup> = 0.35. The post-hoc analysis confirmed the significant differences between 300 vs. 700 ms (p < 0.01), as well as 500 vs. 700 ms (p < 0.001), but not 300 vs. 500 ms (p = 0.31). Although there was no main effect of the Guiding force (p = 0.115), there was an interaction effect between the Guiding force and Delay, F(4, 84) = 2.71, p < 0.05, partial η <sup>2</sup> = 0.11. When the Guiding force was in the correct direction or no force feedback was given, the time-lapse between the two events was perceived as shorter (−77.6 ± 105.8 ms and −56.0 ± 94.9 ms, respectively) than for the force acting in the

FIGURE 3 | (A) The perceptual bias in the cursor delay estimation. In general, the participants estimated the delay shorter than the actual. (B) The qualitative SoA measured from the four-item questionnaire. The error bars represent one standard error.

incorrect direction (−46.9 ± 90.8 ms). This IBE-like observation was evident for estimating 300 and 500 ms delays, but not for 700 ms delay estimation.

### 3.2. Self-Reported SoA and Correlation to the Perceptual Bias

In accordance with the results of the delay perception bias, the self-reported SoA was lower with the incorrect guiding force (0.56 ± 0.26) than with the correct guiding force (0.68 ± 0.19) and with no guiding force (0.76 ± 0.13). The repeated-measures ANOVA on the four questionnaire items revealed a statistically significant difference in delay estimation due to the Guiding force, F(2, 42) = 9.62, p < 0.0005, partial η <sup>2</sup> = 0.31. The posthoc analysis revealed the difference between the incorrect and no guiding forces was significant (p < 0.005), and there was a trend of difference between the correct and incorrect forces (p < 0.07). Furthermore, mild correlation of the self-reported SoA and the bias in time perception was found, r = − 0.41, p < 0.001, supporting the negative relationship between self-reported SoA and the perception of the visual event delay (**Figure 4**).

### 3.3. Performance Measures

On average, the participants spent 543.0±175.4 ms to complete a single trial and our experimental manipulation did not influence the trial duration, as indicated by the ANOVA (ps > 0.24). Furthermore, the observed peak interaction force during the trial was (5.17 ± 1.85 N, see **Figure 5**), and the 2 × 3 ANOVA indicated neither the Guiding force (p = 0.31) nor the Delay (p = 0.10) influenced the interaction force. The analysis on the manipulation share revealed that the participants were responsible for manipulating the cursor by 77.4 ± 6.2 % with the correct guiding force, while the manipulation share was higher with the incorrect guiding force (80.3 ± 4.6 %) due to a lack of assistance. The ANOVA confirmed the main effect of the Guiding force on the manipulation share [F(1, 21) = 5.06, p < 0.04, partial η <sup>2</sup> = 0.19]. The post-hoc analysis confirmed the manipulation share difference between the correct and incorrect

guiding force (p < 0.001). On the other hand no main effect of Delay (p = 0.33) or the interaction effect (p = 0.68) was found.

### 4. DISCUSSIONS

The present experiment investigated whether SoA about voluntary movement is modulated in the presence of external, independent forces effecting the task execution in physical HMI. Given the observation that people experience SoA collectively when a set of actions performed by more than one individual is

assimilated (Dewey and Knoblich, 2014), we hypothesized that the external force may be attributed as own when the guiding force lead to a desired outcome. Our results confirmed this hypothesis; the perceptual bias in delay estimation was least when the guiding force led to a wrong target, indicating the less SoA compared to when the guiding force was assistive or when the entire motion was performed by the participants, i.e., in the condition without providing a guiding force. The observed IBE was around 50−80 ms which is in accordance with previous studies (e.g., Caspar et al., 2015). Furthermore, the results are consistent with the questionnaire-based selfreported SoA score which also indicated lower but sustained SoA when the assistive force was supporting rather than perturbing. One exception was that our SoA scores seems to have been influenced by the magnitude of the event delay , and IBE was not observable for the delay of 700 ms. Remarkably, the participants reported stronger agency when being correctly assisted, while their actual contribution to the action, i.e., their manipulation share, was actually lower than with the perturbing controller. It is, however, important to note our questionnaire consisted of only positive worded statements in the ascending Likert-scale due to experimental time constraints. Thus, the results may have been subject tothe extreme and acquiescence bias (Nunnally and Bernstein, 1994) that could have exaggerated the overall agreement magnitude in their responses. In addition, the correlational analysis between the self-reported SoA and the delay perception bias only revealed a mild relationship. This may be partly due to the fact that our questionnaire scores were affected by the response bias. The supplementary analysis showed the correct guiding force lowered the amount of force required by the participants to move the manipulandum to the target. Thus, the participants contributed less to the task execution in this condition. In contrast, the guiding force did not influence the performance of the task in terms of the trial duration or the interaction force. Thus, lowered physical work load with the assistive controller appears not to reduce SoA, but the characteristics of the external force may have influenced our SoA scores.

Literature suggests that comparator models (Wolpert et al., 1995; Miall and Wolpert, 1996) play an important role in identifying the discrepancy between the intended movements and the sensory outcome (Spengler et al., 2009). Consistently, SoA is reduced when the discrepancy such as spatial misalignment or temporal delay is introduced. For instance, past studies (Sato and Yasuda, 2005; Farrer et al., 2008) observed agency can be misattributed to an external source when the outcome of an action is incongruent with own predicted sensory outcome, and attributing the motion to an external source. In reverse, the central nervous system can distinguish self-action and external sources through learning (Synofzik et al., 2006; Novembre et al., 2012; Pesquita et al., 2017). The monitoring of the sensory-motor error in regard with expected outcome may thus be linked to SoA (Bellebaum et al., 2010). However, a question remains how and to what extent assisting/perturbing external force influences SoA. In order to ensure that the guiding force can be rated as assistive or perturbing, we employed a simple decision-making task in the present study. Although the direction of the incorrect force was varied, there were only three possibilities and the predictability was relatively high. Nevertheless, the perceptual delay bias was not affected and it seems that the mere predictability does not play a role in increasing SoA. A similar observation was reported by the work of Desantis et al. (2012), which concluded prior causal beliefs about the action rather than the predictability of it was an important factor for SoA. In our study, therefore, the high compatibility of the guiding force and the desired action outcomes may have yielded (mis-)attribution of the observed motion as own action. However, we do not know how SoA correlates with undesired external force as our study employed a simple decision-making task. Thus, we need to explore how the action error is processed in the central nervous system and the self/other action attribution is performed.

Our results indicate that the desirable guidance force helped the participants to establish an experience of collective agency with an external device in our task. However, the experimental power was found to be considerably small, and was absent in longer inter-stimulus-intervals. We believe that the main reason for this is that it had been overwritten by other sources of sensory-motor biases. At larger time-intervals, for example, uncertainty about the sensory-motor events arises and may shadow other effects. In support of this claim" research has shown reduced sensitivity to a stimulus duration of longer interstimulus-intervals in various perceptual (Plomp et al., 2012) and motor coordination tasks (Wing and Kristofferson, 1973). In addition, the correlational analysis indicated the relationship between the perceptual bias and the self-reported SoA is only mild, although our observation is likely to be confounded by the inclusion of the cases with the long-inter-stimulus-interval in the analysis. Similarly, the time-frame of the response between the two measures were different as the questionnaire was the end-summary of 135 trials, while the perceptual bias was measured immediately after each trial. Thus, interpretation of the observed perceptual delay against self-reported SoA should be exercised cautiously. Another issue in using SoA scores for evaluating and differentiating more complex controllers is the fact that we have to delay the visual feedback in our paradigm. For a control design aiming at optimizing user experience, we would need to estimate SoA ambiently so the controller can adapt online, e.g., using cognitive models (Schürmann et al., 2019). For instance, controlling the attitude of the robot toward disagreements in early phases of the interaction is considered as an important issue for HMI (Hancock et al., 2011). Therefore, perturbing the task by deferring the visual feedback is not a desirable option for many applications. Although our experimental design used an artificial setting of physical HMI, the present study successfully demonstrated the relevance of SoA in physical HMI using a controlled and a well-validated methodology. Thus, methodological improvements to monitor SoA in realistic HMI such prosthetic will be an important next step. While we need more research into how effectively we could extract the perceptual bias caused by changes in SoA, the current paradigm can be used to evaluate how external forces are (mis-)attributed as own motions in a control design. Having control over action is the important cue for sustaining SoA (Beck et al., 2017), and our study shows that the sense of control is at least partially independent of the physical effort the individual had contributed to the task. Instead, the congruency between the movement and the desired outcome is a crucial factor. Despite the variety of interesting effects and the increasing importance of autonomous agents, the interaction of those with human users is not yet fully understood. While the underlying mechanisms causing the perceptual bias observed in the intentional binding paradigm are still to be understood, we believe that this study distinctly contributes to the understanding of how a control design in physical HMI modulates SoA.

## 5. CONCLUSION

This study used an adapted intentional binding effect (IBE) paradigm to investigate whether SoA can be used to measure the quality and experience of physical HMI schemes that allow the human operator and the collaborative machine to act as a "single entity." Our study demonstrated that motion caused by an external force can be attributed to own cause when it results in a desired outcome. Furthermore, the study indicates IBE may be useful for objectively evaluating a controller, although the experimental power is considerably small and might be influenced by various other factors. Moreover, we observed IBE results to be consistent with the self-reported SoA scores from the questionnaire report. Interestingly, assistance seems to improve IBE despite being supported by another agent. Advancing the understanding of IBE will help us to isolate the true perceptual bias resulted from SoA, and extension of the paradigm to unperturbed real use cases will be essential for an adaptive usercentric HMI scheme.

## DATA AVAILABILITY STATEMENT

The data set analyses for this study can be downloaded from: https:// syncandshare.lrz.de/getlink/fiFQhwTh5EbVWJg637bzoAHt/ frontier\_endo\_2020.zip.

### ETHICS STATEMENT

The studies involving human participants were reviewed and approved by Technical University of Munich, Arcisstraße 21, 80333 München, Germany. The patients/participants provided their written informed consent to participate in this study.

### AUTHOR CONTRIBUTIONS

SE and PB developed the idea. JF executed the study. SE, JF, and SM drafted the manuscript. SH and PB equally provided critical revisions to the draft. All authors contributed equally to its conception.

### REFERENCES


### FUNDING

This research is supported by the European Union's Horizon 2020 research and innovation programme REHYB under grant agreement no 87176, the ERC Starting Grant: Control based on Human Models (conhumo) under grant agreement no 337654, and the German Research Foundation (DFG) through grant BE 5729/3&11.


**Conflict of Interest:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2020 Endo, Fröhner, Musi´c, Hirche and Beckerle. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

### APPENDIX

### Control Strategy of the Manipulandum

The manipulandum has two translational degrees of freedom in the x − y plane. The admittance controller is used to enforce compliant behavior of the planar device at the interaction point with the physically coupled human. Desired dynamics of the device, achieved with the admittance controller, is defined as a virtual mass-damper system

$$\mathcal{M}\ddot{\mathbf{p}} + \mathbf{D}\dot{\boldsymbol{p}} = \boldsymbol{f}\_h + \boldsymbol{f}\_{\tilde{\mathcal{J}}},\tag{A1}$$

where **M** = [5 5]<sup>⊤</sup> kg and **D** = [15 15]<sup>⊤</sup> Ns/m are the desired inertia and damping matrices of the device, respectively. The position of the handle is **p** = [p<sup>x</sup> py] <sup>⊤</sup>. The acceleration and velocity vectors are denoted as **p**¨ and **p**˙, respectively. The force exerted on the handle by the human is denoted as the vector **f** <sup>h</sup> . The feed-forward force **f** ff , is composed of the guiding force **f** g , exhibited at the start of the motion, and target assistive force **f** t , exhibited when the cursor collides with the target, given

$$f\_{\mathcal{J}} = f\_{\mathcal{J}} + f\_{\mathcal{E}}.$$

The guiding force, **f** g , is defined as a normal distribution curve with 10 N peak force spanned over 300 ms pushing in the direction of the middle of the target platform surface. It occurred 200 ms after the handle moved out of the starting platform and was enough to push the cart to the target without applying any additional force by the human if the handle motion was not resisted by the human, such that

$$f\_{\mathcal{S}}(t) = \begin{cases} 10e^{-\frac{(t-350)^2}{400}}, & \text{if } 200 \le t \le 500 \text{ ms and } ||\mathfrak{p}||^2 \ge 1 \text{ cm} \\ 0, & \text{otherwise} \end{cases} \tag{A2}$$

The target assistive force component, **f** t , is enforced once the target is reached, by applying f<sup>t</sup> = 20 N to the handle. The desired dynamics given with (A1) are achieved with a proportional-derivative controller. The controller is implemented in Matlab/SIMULINK. It runs on a computer with Linux PREEMPT real-time kernel (Ubuntu 14.04, 3.14.3-rt4) with a fixed-step solver at the sampling rate of 1 kHz.

**262**