EDITED BY : Michael Wininger, Panagiotis Artemiadis, Claudio Castellini and Patrick Pilarski PUBLISHED IN : Frontiers in Neurorobotics

#### Frontiers Copyright Statement

© Copyright 2007-2018 Frontiers Media SA. All rights reserved. All content included on this site, such as text, graphics, logos, button icons, images, video/audio clips, downloads, data compilations and software, is the property of or is licensed to Frontiers Media SA ("Frontiers") or its licensees and/or subcontractors. The copyright in the text of individual articles is the property of their respective authors, subject to a license granted to Frontiers.

The compilation of articles constituting this e-book, wherever published, as well as the compilation of all other content on this site, is the exclusive property of Frontiers. For the conditions for downloading and copying of e-books from Frontiers' website, please see the Terms for Website Use. If purchasing Frontiers e-books from other websites or sources, the conditions of the website concerned apply.

Images and graphics not forming part of user-contributed materials may not be downloaded or copied without permission.

Individual articles may be downloaded and reproduced in accordance with the principles of the CC-BY licence subject to any copyright or other notices. They may not be re-sold as an e-book.

As author or other contributor you grant a CC-BY licence to others to reproduce your articles, including any graphics and third-party materials supplied by you, in accordance with the Conditions for Website Use and subject to any copyright notices which you include in connection with your articles and materials.

All copyright, and all rights therein, are protected by national and international copyright laws.

The above represents a summary only. For the full conditions see the Conditions for Authors and the Conditions for Website Use. ISSN 1664-8714 ISBN 978-2-88945-490-7 DOI 10.3389/978-2-88945-490-7

#### About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

### Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

### Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

#### What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office: researchtopics@frontiersin.org

# PERIPHERAL NERVOUS SYSTEM-MACHINE INTERFACES, 2nd EDITION

Topic Editors:

Michael Wininger, University of Hartford, United States Department of Veterans Affairs, Yale University, United States Panagiotis Artemiadis, Arizona State University, United States Claudio Castellini, German Aerospace Center (DLR), Germany Patrick Pilarski, University of Alberta, Canada

For 5 years, the Peripheral Nervous System-Machine Interfaces workgroup has dedicated itself to the recruitment of researchers, clinicians, and general public in a unified effort to advance the frontier of restoration of quality of life to those with limb deficiency. Our group's mission is to bring together experts from various domains to identify promising new technologies and new opportunities for inquiry and discovery in prosthetics research. This e-Book collects 10 cutting edge research articles written by members of the workgroup, covering three domains prioritized by the workgroup: novel prosthetic technology, approaches for reducing device rejection, and prosthetic control. In our summary editorial, we four principals of the workgroup reflect on our first 5 years, and project our vision for the future, as the Society for Prosthetics.

Citation: Wininger, M., Artemiadis, P., Castellini, C., Pilarski, P., eds. (2018). Peripheral Nervous System-Machine Interfaces, 2nd Edition. Lausanne: Frontiers Media. doi: 10.3389/978-2-88945-490-7

# Table of Contents

*05 Editorial: Peripheral Nervous System-Machine Interfaces (PNS-MI)* Michael Wininger, Panagiotis Artemiadis, Claudio Castellini and Patrick Pilarski

### SECTION 1

### NOVEL TECHNOLOGY

*07 Key Insights Into Hand Biomechanics: Human Grip Stiffness Can Be Decoupled From Force by Cocontraction and Predicted From Electromyography*

Hannes Höppner, Maximilian Große-Dunker, Georg Stillfried, Justin Bayer and Patrick van der Smagt

*27 Deep Learning With Convolutional Neural Networks Applied to Electromyography Data: A Resource for the Classification of Movements for Prosthetic Hands*

Manfredo Atzori, Matteo Cognolato and Henning Müller

*37 Assessment of a Wearable Force- and Electromyography Device and Comparison of the Related Signals for Myocontrol* Mathilde Connan, Eduardo Ruiz Ramírez, Bernhard Vodermayer and Claudio Castellini

### SECTION 2

### REDUCING REJECTION


Simone Fani, Matteo Bianchi, Sonal Jain, José Simões Pimenta Neto, Scott Boege, Giorgio Grioli, Antonio Bicchi and Marco Santello


Alix Chadwell, Laurence Kenney, Sibylle Thies, Adam Galpin and John Head

*102 Corrigendum: The Reality of Myoelectric Prostheses: Understanding What Makes These Devices Difficult for Some Users to Control* Alix Chadwell, Laurence Kenney, Sibylle Thies, Adam Galpin and John Head

### SECTION 3

### CONTROL

## *104 An EMG Interface for the Control of Motion and Compliance of a Supernumerary Robotic Finger* Irfan Hussain, Giovanni Spagnoletti, Gionata Salvietti and Domenico Prattichizzo

*117 Evaluating EMG Feature and Classifier Selection for Application to Partial-Hand Prosthesis Control*

Adenike A. Adewuyi, Levi J. Hargrove and Todd A. Kuiken

*128 Combining Vibrotactile Feedback with Volitional Myoelectric Control for Robotic Transtibial Prostheses*

Baojun Chen, Yanggang Feng and Qining Wang

# Editorial: Peripheral Nervous System-Machine Interfaces (PNS-MI)

*Michael Wininger1,2,3\*, Panagiotis Artemiadis4 , Claudio Castellini5 and Patrick Pilarski6*

*1Prosthetics and Orthotics Program, University of Hartford, West Hartford, CT, United States, 2Cooperative Studies Program, United States Department of Veterans Affairs, West Haven, CT, United States, 3Department of Biostatistics, Yale University, New Haven, CT, United States, 4Department of Mechanical and Aerospace Engineering, Arizona State University, Tempe, AZ, United States, 5Robotics and Mechatronics Center, German Aerospace Center (DLR), Oberpfaffenhofen, Germany, 6Department of Medicine, University of Alberta, Edmonton, AB, Canada*

Keywords: abandonment, control, database, detection, electromyography, myoelectric, prosthetics, society

**Editorial on the Research Topic**

**Peripheral Nervous System-Machine Interfaces**

### INTRODUCTION

The Peripheral Nervous System-Machine Interfaces (PNS-MI) Workgroup is now in its fifth year of activity, and with this Editorial completes its fourth annual deliverable. Following our first Workshop at International Conference on Rehabilitation Robotics (ICORR)-2013 in Seattle, and its Proceedings document (Castellini et al.), we convened at ICORR-2015 in Singapore, and for 2016, cast a wide net for contributions to this Research Topic, seeking both to summarize the discussions in Singapore and also to gather new perspectives.

## ABOUT THE RESEARCH TOPIC (RT)

In our advertisement for this RT, we solicited contributions that either (a) advanced the frontier of prosthetic technology or (b) bridged the gap between the laboratory and the clinic. We are most delighted to be able to present ten articles written by 55 authors at 27 different institutions (representing universities, hospitals, clinics, government, and industry) from Europe, Asia, North- and South America. We Topic Editors believe that the impact of this RT may be best described as advancing the frontier in three domains: debut of novel technologies, new approaches to reducing device abandonment, and under-served or exciting new patient populations.

The new technologies introduced here extend the Workgroup's long-standing interest in EMG-based prosthetic control: utterly novel and potentially transformative detection paradigms (Höppner et al.); comprehensive testing of convolutional neural networks with simplistic architecture versus classical classification approaches to a large hand movement database (Atzori et al.); and introduction of a novel, wireless wearable biosensor for measuring force myography and electromyography simultaneously (Connan et al.).

The PNS-MI emphasis on reducing device rejection is a recent addition to our research thrusts; this RT brings together an important distinction between classifier algorithms and clinical scores/ ADL (Vujaklija et al.); finding a differential EMG-to-position mapping that ensures highest coherence with hand movements for naturalistic and intuitive control (Fani et al.); a work that coins the concept of the prosthetic homolog, and tests its putative criticality for acceptance and embodiment (Dornfeld et al.); and a new method for assessing user functionality in myoelectric control signal of upper-limb prostheses with application to in-clinic, in-lab, and also real-world (Chadwell et al.).

#### *Edited and Reviewed by:*

*Florian Röhrbein, Technische Universität München, Germany*

#### *\*Correspondence:*

*Michael Wininger wininger@hartford.edu*

*Received: 17 August 2017 Accepted: 26 September 2017 Published: 24 October 2017*

#### *Citation:*

*Wininger M, Artemiadis P, Castellini C and Pilarski P (2017) Editorial: Peripheral Nervous System-Machine Interfaces (PNS-MI). Front. Neurorobot. 11:54. doi: 10.3389/fnbot.2017.00054*

**5**

Starting with the Workshop in Singapore, the Workgroup expanded its coverage to include a wider range of patient populations and prosthetic applications. This RT expands this initiative through a hybridized EMG-mechanotechnology viz. the control of an augmentative hand prosthesis (Hussain et al.) and a classification of hand motions from different wrist positions among partial-hand amputation patients (Adewuyi et al.). And while historically our group has been interested primarily in upper-limb prosthetics, we were delighted to publish a work on vibrotactile feedback system for the control of transtibial prosthesis (Chen et al.).

### NEXT STEPS: 2017 ACTIVITY

In our fifth year, we shall cycle back to Workshops. We will next convene at the 2017 American Orthotics and Prosthetics Association (AOPA) World Congress in Las Vegas (Symposium on Multi-Scale Integration in Upper-Limb Prosthetics, Saturday, September 9). Coincident with AOPA's 100th Anniversary, we find it fitting to bring the work of our group to the AOPA World Congress, where thousands of practitioners, researchers, manufacturers, physicians, and facility owners gather from around the globe to learn, share, and collaborate.

### NEXT STEPS: WORKGROUP EVOLUTION

Our Workshop at AOPA will mark our debut as the formative Society for Prosthetics. As our group evolves its portfolio of research interests and efforts to foster collaborative exchange, we

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2017 Wininger, Artemiadis, Castellini and Pilarski. This is an open-access article distributed under the terms of the Creative Commons*  believe that it is only natural to self-identify in the tradition of academic associations that have served other sectors of medicine and engineering so well over time.

The PNS-MI Workgroup was a terrific launching pad, and accomplished its main goal perfectly: it served as the assembly point for prosthetics researchers from across the globe, to identify common objectives, opportunities for further research, and to discuss their progress with each other. But our Group wants to engage with our process partners: patients, clinical providers, payers, and students, as well as researchers from allied fields. As an informal Workgroup, we are inherently limited in our visibility and appeal to these niches; as a society, offer a much broader platform of offerings to the community.

Following our third Workshop at AOPA-2017, we will pursue a third proceedings document in 2018, wherein we shall aspire to accomplish: (1) continued dissemination of our valuable prosthetics research and (2) roadmap for the Society. We thank our many dedicated PNS-MI pioneers, and look forward to everhigher impact through the Society for Prosthetics.

### AUTHOR CONTRIBUTIONS

The four authors contributed equally to editing the Research Topic.

### ACKNOWLEDGMENTS

The authors gratefully acknowledge the participants in this Research Topic, and the members of the PNS-MI community.

*Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# **Key Insights into Hand Biomechanics: Human Grip Stiffness Can Be Decoupled from Force by Cocontraction and Predicted from Electromyography**

*Hannes Höppner <sup>1</sup> \*, Maximilian Große-Dunker <sup>1</sup> , Georg Stillfried<sup>1</sup> , Justin Bayer <sup>2</sup> and Patrick van der Smagt 2,3*

*<sup>1</sup> Bionics Lab, Institute of Robotics and Mechatronics, German Aerospace Center DLR e.V., Oberpfaffenhofen, Wessling, Germany, <sup>2</sup> Department of Informatics, Technische Universität München, Munich, Germany, <sup>3</sup> fortiss, TUM affiliated Institute, Munich, Germany*

We investigate the relation between grip force and grip stiffness for the human hand with and without voluntary cocontraction. Apart from gaining biomechanical insight, this issue is particularly relevant for variable-stiffness robotic systems, which can independently control the two parameters, but for which no clear methods exist to design or efficiently exploit them. Subjects were asked in one task to produce different levels of force, and stiffness was measured. As expected, this task reveals a linear coupling between force and stiffness. In a second task, subjects were then asked to additionally decouple stiffness from force at these force levels by using cocontraction. We measured the electromyogram from relevant groups of muscles and analyzed the possibility to predict stiffness and force. Optical tracking was used for avoiding wrist movements. We found that subjects were able to decouple grip stiffness from force when using cocontraction on average by about 20% of the maximum measured stiffness over all force levels, while this ability increased with the applied force. This result contradicts the force–stiffness behavior of most variablestiffness actuators. Moreover, we found the thumb to be on average twice as stiff as the index finger and discovered that intrinsic hand muscles predominate our prediction of stiffness, but not of force. EMG activity and grip force allowed to explain 72 *±* 12% of the measured variance in stiffness by simple linear regression, while only 33 *±* 18% variance in force. Conclusively the high signal-to-noise ratio and the high correlation to stiffness of these muscles allow for a robust and reliable regression of stiffness, which can be used to continuously teleoperate compliance of modern robotic hands.

**Keywords: grip stiffness, cocontraction, grip force, intrinsic hand muscles, interosseus muscles, electromyography, soft robotics, variable-stiffness actuators**

### **1. INTRODUCTION**

Stiffness is an important property for the interaction of any biological or mechanical system with its environment. A soft system (low stiffness) will yield to external perturbation forces, while a stiff system will withstand them. For example, when brushing one's teeth, the grip on the toothbrush needs to be soft enough for following the shape of the jaw without hurting the gum, but firm enough

#### *Edited by:*

*Michael Wininger, University of Hartford, USA*

#### *Reviewed by:*

*Agnes Roby-Brami, Institut national de la santé et de la recherche médicale (INSERM), France Xiaosu Hu, University of Michigan, USA*

> *\*Correspondence: Hannes Höppner hannes.hoeppner@dlr.de*

*Received: 13 May 2017 Accepted: 09 March 2017 Published: 22 May 2017*

#### *Citation:*

*Höppner H, Große-Dunker M, Stillfried G, Bayer J and van der Smagt P (2017) Key Insights into Hand Biomechanics: Human Grip Stiffness Can Be Decoupled from Force by Cocontraction and Predicted from Electromyography. Front. Neurorobot. 11:17. doi: 10.3389/fnbot.2017.00017* (high stiffness) for keeping the handle within a stable pose without losing it and for guiding the head of the toothbrush in the desired direction.

Stiffness is defined as a ratio of a force change to a corresponding displacement. However, additional criteria need to be fulfilled for a force–displacement relation to be considered stiffness (Latash and Zatsiorsky, 1993). These criteria are resistance, passivity, and elasticity: the direction of the force change opposes the direction of the displacement (resistance of the system against deformation); for the force change, no external energy is supplied (passivity); the force change is only dependent on the displacement and has a conservative nature (elasticity). The elasticity criterion also ensures that the reaction is instantaneous, since otherwise, the force change would not only depend on the displacement but also on the time.

A resistive response to perturbations can also be provided by the human body via reflexes, which are involuntary contractions of muscles that involve the travel of nervous signals from sensory receptors via the central nervous system to the muscles. Despite being sometimes called "reflexive stiffness," this kind of response falls outside of our definition of stiffness, because the contraction of the muscle consumes energy and the travel of the nervous signal introduces a delay. Our definition of stiffness also excludes force changes due to acceleration (inertial forces) and velocity (damping forces). Conclusively, the stiffness we measure is not a quasistiffness, reflexive stiffness, nor apparent stiffness [see also Latash and Zatsiorsky (1993)].

In biomechanics and neuroscience, our definition of stiffness is commonly referred to by using the terms *static*, *intrinsic*, or *areflexive* stiffness and is close to the stiffness of mechanical springs. It is a combination of passive stiffness stemming from the muscles, tendons, surrounding tissue, and ligaments and short-range stiffness originating from the crossbridges.

It has been shown that (a) the stiffness of a muscle increases linearly with increasing muscle force (Zajac, 1989; Shadmehr and Arbib, 1992) and the stiffness of a grip increases linearly with grip force (Höppner et al., 2011; Van Doren, 1998); (b) the slope of the linear force–stiffness curve can be modulated by changing the posture of the limb (kinematics) (Höppner et al., 2013); and (c) by simultaneously contracting flexor and extensor muscles (cocontraction), stiffness can be varied without changing posture when no force is applied to the environment (zero net force) (Osu et al., 2002). In this article, we investigate the open question whether (d) cocontraction can be used to decouple stiffness from its linear increase with force while external forces are applied and kinematics are kept constant.

Each of the stiffness modulation methods has different advantages: while changing kinematics is energy efficient, external force modulation and cocontraction allow for posture maintenance. Among these methods, we choose to investigate cocontraction as stiffness modulation mechanism, because it raises open biomechanical questions and its results can be directly applied to variable-stiffness actuators in robots. By using a perturbation device that can measure human grip stiffness related to grip force (Höppner et al., 2011, 2013), we can investigate the human mechanism of cocontraction. The device is able to measure an almost exact representation of pure stiffness—which is captured by the terms passivity, resistance, and elasticity—imposing that it is able to refrain from measuring influences from active feedback or damping and inertia. For this, the device measures forces at two static positions (see **Figure 5**), so that the acceleration and velocity are zero during the measurements, and accomplishes the transition between the two positions fast enough to exclude the possibility of reflexes. Furthermore, we use EMG—since it possibly allows measuring muscle states continuously and is thus highly relevant for teleoperation in robotics—to investigate the possibility to regress force and stiffness from the measurement of muscular activity from relevant intrinsic and extrinsic hand muscles. Note that unlike with reflexes, the metabolic energy cost for maintaining the static muscle tension does not affect the passivity criterion, because it is only used to establish the state of the system prior to the perturbation and is not affected by the displacement-related force change.

### **1.1. Stiffness in Robots**

Actively controlled compliant robotic systems (Albu-Schäffer and Hirzinger, 2002) are able to mimic an *apparent stiffness*, which makes them suitable for human–robot interaction. However, similar to the human reflex, they reach their limits at highfrequency impacts (Hogan, 1984). Thus, these systems have been extended recently by further adding an intrinsic elasticity (Vanderborght et al., 2013; Grebenstein, 2014; Wolf et al., 2016) by the use of non-linear springs—variable-stiffness actuators (VSA)—which is a concept copied from the flexibility found in biological limbs: through *cocontraction*, we can increase the stiffness and damping characteristics of our limbs, thus influencing the energy exchange characteristics with our environment. Besides (a) allowing to compensate high-frequency impacts and increasing system robustness, VSAs offer valuable properties such as (b) enriching dynamic capabilities by allowing to frequently store energy in reversal points or (c) embodying the desired behavior of a task into the mechanical structure of the robot (Visser et al., 2011). One of their main characterizing properties is their torque–stiffness diagram (Wolf et al., 2015, 2016), showing the basic coupling between torque and stiffness and how it varies with *pretensioning* of the joint—which is similar to the mechanism of cocontraction found in humans. However, biomechanics is essentially lacking similar diagrams for the human locomotor system, which might be used by robotic engineers as a template. Hence, heuristic methods have been used for setting properties of VSAs rather than clear design guidelines; e.g., most of the VSAs have a rather limited performance in decoupling stiffness from torque for the higher torques. This article is trying to close this gap in biomechanics and to find an answer to the main question: Can stiffness be significantly decoupled from its linear increase with force with cocontraction during posture maintenance?

### **1.2. Cocontraction**

Cocontraction is the simultaneous activation of at least two antagonistic muscles acting on a joint (Gribble et al., 2003). See **Figure 1** as an example of how cocontraction of antagonistic muscles affects the force and stiffness measured at an end-effector: it depicts a diagram of the force and stiffness at the fingertip of a simplified

finger actuated by two flexor and two extensor muscles. The red and black arrows denote the linear force–stiffness relations of single a-reflexive muscles with the arrow's tips pointing to the muscle's maximum force and stiffness. Although activating flexor (black arrows) and extensor (red arrows) muscles will contribute to stiffness in a positive way, the flexor muscle activation will increase the applied force and extensor muscle activation will decrease the applied force. Assuming a linear relation between force and stiffness, the reachable force–stiffness range of an antagonistic setup is defined by the vector sum of the single force–stiffness relations of the single antagonistic muscles [similar to the quadrilateral region of two antagonist muscles defined in the study by Kearney and Hunter (1990)]. If humans were able to activate all muscles independently, they would be able to reach the entire area by cocontraction. However, it is well known that due to neural and mechanical synergies, they are not able to independently activate them (De Luca and Mambrito, 1987; Milner, 2002).

In literature, it remains unclear what the notion cocontraction exactly means. Sometimes it refers to a *uniform* scaling of all muscular activations between their minimum and maximum values, resulting in an increase of force and stiffness along the direction pointing to the maximum stiffness (green arrow in **Figure 1**). Contrarily, an *isometric* cocontraction will increase stiffness only and keep the applied force constant (blue arrows)—similar to the notion *pretension* used for VSAs in robotics. Since we focus on robotics, we will ask subjects for an isometric cocontraction only and will give them a visual feedback about the applied force and stiffness. Furthermore, by referring to the notation cocontraction, we mean the simultaneous contraction of flexor and extensor muscles of thumb and index finger, which results in stable pinch grip force but increased pinch grip stiffness. A simultaneous contraction of all flexor muscles of thumb and index finger opposing each other in a pinch is not considered as cocontraction in this article.

Moreover, by referring to the notation *decoupling*, we naturally imply an increase of stiffness from its usual coupling to force. We will refrain from analyzing the possibility to decrease stiffness from its normal coupling to force—since it is expected to be impossible.

Different studies simulated, measured, and analyzed the role of cocontraction for the human locomotor system. Hogan (1984) analyzed the role of joint stiffening caused by cocontraction of an antagonistic setup for maintaining joint position (when no external torque is applied) in a simulation study in comparison to active control, asking, when do we need cocontraction and when does an actively controlled reflexive stiffness suffice? Similarly, Akazawa et al. (1983) investigated changes in stretch reflex gain and stiffness of the long thumb flexor muscles in a force-control and a constant-load position control task. Gribble et al. (2003) explored the relationship between cocontraction and the target size in a pointing task. Osu et al. (2002) investigated short- and long-term changes in cocontraction when interacting in known and unknown environments. Selen et al. (2005) analyzed in a simulation study whether cocontraction leads to more joint stability or larger fluctuations in the paradoxical situation that both stability and motor noise increase with muscle activation. Grebenstein et al. (2011) hypothesized about criteria for joint stiffening by observing examples from sports.

Cocontraction increases the stiffness of arm joints, at least in the absence of external forces (Osu et al., 2002). It is a successful strategy to stably maintain a position when internal models of the environment are imprecise, when external perturbations are expected but not predictable, or when perturbation frequencies are too high for the central nervous system to react (Akazawa et al., 1983; Hogan, 1984; Osu et al., 2002). Cocontraction can also be a successful strategy for decreasing trajectory variability and improving endpoint accuracy during multijoint arm movements (Gribble et al., 2003). The ability of cocontraction to stabilize a limb "*. . .*highly depends on levels of motor noise and sources, and on muscular architecture and skeletal properties*. . .*" (Selen et al., 2005).

Cocontraction probably also plays an important role for the absorption of impact energy (Grebenstein et al., 2011). In case of known impact energy, humans adapt joint stiffness to dissipate the impact energy over a broad range of joint motion inside the joint limits to avoid damage to the muscles. For unknown impacts, humans use a strategy of maximum cocontraction to dissipate as much energy as possible using their muscles knowing that reaching joint limits causes substantially more irreversible injuries.

However, the influence of cocontracting extrinsic and intrinsic antagonistic pairs of hand muscles on decoupling grip stiffness from its usual increase with grip force remains an open question. The investigation of the effect of cocontraction on stiffness is rather limited, and existing studies investigated the usage of cocontraction at *zero net force* only, i.e., no forces are applied to the environment. The usage of forces is highly relevant for interacting with the environment and the manipulation of objects and possibly the ability to alter stiffness at this force, too.

From VSAs in robotics, we know about their limited ability to decouple stiffness and torque for the higher torques. Is this true for human locomotor system, as well? Is the ability of decoupling force and stiffness using cocontraction limited to the lower force ranges, e.g., to zero net force, since intrinsic stiffness increases with force anyway? Or are we able to considerably decouple the two also for the higher forces? To address this question, this study will focus on human's ability to decouple stiffness from its linear increase with force using cocontraction.

Two ways of forcing subjects to cocontract are acknowledged, either by (a) the application of unstable force fields (Akazawa et al., 1983) or by (b) presenting a visual feedback about the applied muscular activity from relevant muscle groups (Osu and Gomi, 1999; Osu et al., 2002; Shin et al., 2009). Using unstable force fields seems to force subjects to increase cocontraction in a natural way but is probably limited to the production of zero net force, which means that no forces are applied by the finger or limb. On the other hand, forcing subjects to produce cocontraction based on measured *electromyography* (EMG) is an unnatural task, but allows to command different combinations of contraction and cocontraction including those leading to non-zero net force. However, so far it has been used only to investigate different levels of cocontraction at zero net force.

In this study, we will use a completely different approach (c) and present visual feedback of the applied force and stiffness of each *prior* trial to a participant, allowing him or her to learn how to modulate stiffness over the course of multiple trials.

### **2. MATERIALS AND METHODS**

We measured stiffness in subject experiments with and without voluntary cocontraction using a device that applies a fast position perturbation to a thumb–index finger grip. We used optical tracking to observe and prevent changes in kinematics and electromyography to analyze and investigate the regression of force and stiffness from muscular activity.

### **2.1. Device Description**

The grip perturbator we used in this experiment is presented in **Figure 2**. A spring (orange) is preloaded by an electromagnet (blue) fixed to a frame (black) that holds a moving part (brown).

The grip force is measured with a load cell (white). Releasing the spring causes the device to elongate by 7.5 mm within a few milliseconds (see perturbation force profile in **Figure 5**). Amendments since our previous study (Höppner et al., 2013) concern an improved guiding of the gripping force to the small load cell and allows for a smaller grip length. In addition, three markers for optical tracking and two small fans were attached to reduce the heating caused by the electromagnet. The perturbator weighs 165 g, and its length varies between 54 and 61.5 mm. The spring force is 140 N when loaded and 100 N when unloaded, i.e., considerably higher than the pinch grip force, ensuring identical experimental conditions independent of how firmly the perturbator is held. The load cell is a KM10 (ME-Messsysteme GmbH) force sensor with a nominal sensitivity of 1 mV/V and a nominal range of 100 N. The accuracy of the analog signal provided by the measurement amplifier GSV-11H (ME-Messsysteme GmbH) is 0.1 N.

### 2.1.1. Electromyography

Keeping in mind a possible application in telerobotics, we use non-invasive surface electrodes rather than invasive needle electrodes. The surface electrodes *Delsys Trigno Wireless System* have an internal amplification of 1 kV/V and provide an analog signal at 4 kHz with a constant delay of 48 ms. These electrodes complies with the requirements put forth by the Medical Device Directive 93/42/EEC, and we comply with its intended use. The EMG electrodes were attached in accordance with the recommendations of the SENIAM project (Hermens et al., 2000). Before the experiment, the subjects were asked to wash their arm with water; no soap was used. For an optimal EMG signal, the respective part of the skin was again moistened with water. As a result of earlier prestudies, we have chosen in total six muscles to be relevant for our experimental procedure: two extrinsic index flexor muscles (FDP and FDS), two extrinsic index extensor muscles (EIP and ED), and two interossei muscles in the hand (FDI and SDI; see **Table 1**). Please note that even if SDI inserts at the middle finger, we found a strong influence on our measurements and thus decided to include it.

Within the earlier prestudies, which were conducted without any tests for significance and thus not published, we analyzed in a force task the influence of index finger stiffness only. We found similar stiffness values and force–stiffness relations as measured



in a pinch grasp. Since we found the index finger predominating the measured grip stiffness, we concluded the thumb to be much stiffer than the index finger. Thus, within this study, we refrained from measuring EMG of corresponding muscles of the thumb (flexor pollicis longus, extensor pollicis brevis, and extensor pollicis longus).

Furthermore, we tested measuring the adductor pollicis muscle as well. Due to strong sweating and large movement of the underlying skin for the pinch grip, the electrodes took off very rapidly, which makes it impossible for us to measure this muscle. The electrodes were placed close to the six corresponding muscles (see **Figure 4**) by the subjects using palpation and visual feedback of the EMG signal.

### 2.1.2. Optical Tracking

The positions of arm and fingers were continuously monitored through optical tracking and corrected where necessary, so as to prevent variations from kinematics. The optical tracking system is a *Vicon Motion Capture System* consisting of 8 *MX3*+ cameras and an *MX Ultranet* controller. The cameras were arranged at distances between 0.5 and 1 m around the forearm position (for all subjects the same). The cameras have an optimal resolution of 659 (horizontal) *×* 494 (vertical) pixels at 242 frames per second, and we used them at a frequency of 400 Hz. After positioning the EMG sensors, marker sets for tracking the position and orientation of wrist and forearm and single markers to track the positions of the distal phalanx of index finger and thumb were positioned (see **Figure 4**). The optical tracking system was calibrated using the orientation of the table. The idea of the optical tracking system was to give the subject and the experimenter a feedback about variations in kinematics during the experiment to constrain it and correct when necessary, rather than using the measured optical tracking data to identify influences and their significance. We decided to use optical tracking rather than

different cuffs to constrain the kinematics since it offers more possibilities for the subjects to choose a relaxed initial posture and avoids occupying suitable EMG positions. Furthermore, there is no risk that the subjects apply wrist torque against the cuff, the influence of which on the EMG signal we would not be able to quantify.

### 2.1.3. Graphical User Interface

In addition, subjects saw a graphical representation of the measured data on a screen (see **Figure 3**). For controlling the force, two red dashed lines and one red solid line representing the required force level and the measured force were depicted. Directly after each perturbation, the measured stiffness and force were visually presented to the subject as a dot in a force–stiffness graph. This procedure allows the subject and the experimenter to check the subject's performance in the preceding trial. Furthermore, the following kinematic information was presented to the subjects: the planar positions of forearm, wrist, perturbator, thumb, and index finger; the orientation of the longitudinal perturbator axis (roll axis) in reference to the table plane; and the angular distances of wrist and forearm in reference to their initial orientations. The subjects were asked to keep the positions of the perturbator, the wrist, and the forearm within tolerance ranges, depicted as circles with a radius of 15 mm around the initial captured positions. They were furthermore asked to keep the orientations of the wrist and the forearm (displayed as angular distances in **Figure 3**) close to the initially detected ones and the roll axis of the perturbator parallel to the table plane. Note that for a successful perturbation, the force was controlled automatically to be kept within a certain force range; despite that, the positions were just visually inspected by the experimenter and not constrained to avoid fast fatigue of the subjects. As soon as the release button for valid perturbation conditions was pressed by the experimenter, the perturbation was applied after a random interval between 0.5 and 2.5 s.

The measurement setup consisted of a host computer running Linux, a real-time target computer running QNX, and a Windows computer. The real-time computer runs a MATLAB/Simulink model to control the electromagnet, to read out the force sensor at 10 kHz, and to read out the EMG sensors. The marker positions were recorded with the Windows computer and transferred to the Linux host using the DLR communication protocol *arDNet* (Bäuml and Hirzinger, 2008). A triggered recording of the Vicon data was started at 250 ms before each perturbation and lasted for 1 s. Measured force signals were calibrated before each trial since the output of the force sensor was marginally influenced by the heating of the electromagnet.

## **2.2. Experimental Procedure**

A total of 10 healthy subjects, nine male and one female (S3), seven right and three left-handed (S5, S7, S9), age 22–27 years, and all initially fully naive to the experiment, performed the two experimental protocols, with and without isometric cocontraction, as described below. For all subjects and experiments, the right hand was used, be they right or left handed, which is restricted by the design of the perturbator with its fans and optical markers. To further assist the subjects in holding their wrist and arm orientation stable during the measurements, a vacuum cushion

**FIGURE 3 | Graphical representation of measured pose and force data, which were presented to the subjects (representative)**. (Top left) Applied force (red solid line) and goal force level (red dashed lines). All previously measured perturbations were depicted as blue dots showing the applied force and stiffness, while the very last was highlighted in red. The estimation of the basic stiffness curve achieved in task 1 was depicted as a diagonal black dashed line. (Bottom left) The last perturbation was depicted for visual inspection for artifacts. Furthermore, the detected mean forces before and after perturbation as well as its beginning were shown. (Top right) The roll axis of the perturbator and its radial deflection in reference to the table plane (similar to an attitude indicator in an airplane). (Bottom right) The position of perturbator, index, thumb, wrist, and forearm depicted as dots in a plane parallel to the table. In addition, a circle with a radius of 15 mm was plotted, which indicates a tolerance around each initial measured position. If all dots were inside each circle, a text "Posture correct" was shown in green; otherwise a comment "CAUTION!! Correct posture!" was shown in red.

**FIGURE 4 | Measurement setup**. The perturbator was held by the subject between index finger and thumb, while middle finger, ring finger, and pinky had to be flexed. 6 EMG electrodes were placed to corresponding flexor and extensor muscles on the hand (FDI and SDI) and forearm (FDP, FDS, EIP, and ED). The forearm was placed in a vacuum cushion to assist subjects with holding their wrist and arm position stable. The positions of index finger, thumb, perturbator, wrist, and forearm and the orientations of perturbator, wrist, and forearm were tracked with an optical tracking system.

was used, which was adjusted to each subject. Subjects were seated in all experimental conditions.

The whole procedure lasted between 90 and 120 min per participant. No subject had a history of neurological disorder or neuromuscular injury affecting the CNS or the muscles. All subjects participated voluntarily and gave written consent to the procedures, which were conducted in partial accordance with the principles of the Helsinki agreement (non-conformity concerns the point B-16 of the 59th World Medical Association Declaration of Helsinki, Seoul, October 2008: no physician supervised the experiments). Approval was received from the works council of the German Aerospace Center, as well as its institutional board for data privacy ASDA; the collection and processing of experimental data were approved by both committees.

At first, subjects were asked to lay their arm relaxed on the table to measure the *EMG base noise* level for 5 s (see Appendix). Furthermore, the initial poses of wrist, forearm, and perturbator and the positions of index finger and thumb were measured in this relaxed pose. Second, subjects were asked to fulfill *maximum voluntary contraction* (MVC), i.e., to grip as strongly as they were able to, three times for 5 s each, while the maximum grip force and corresponding EMG levels were measured. The MVC was used to set the prescribed force levels in the following two main tasks.

### 2.2.1. Task 1—Force Task without Voluntary Cocontraction

In *task 1*, subjects were asked to stably hold six different visually presented force levels using the vertical red lines (15, 25, 35, 45, 55, and 65% of MVC) within a range of *±*5% of MVC without using any kind of voluntary cocontraction. The force levels were given to them in a randomized order four times each, leading to a total of 24 perturbations. The perturbation is a small and fast displacement of 7.5 mm of the pinch grip, and force is measured to calculate stiffness using its difference before and after perturbation. Since active response is not our scope, the measurement is finished within 40 ms. This procedure is similar to the one in our previous studies (Höppner et al., 2011, 2013), except that wrist and finger positions were measured and constrained, and EMG was measured. This force task is considered to deliver information about the subject's basic stiffness and its dependency on force.

A linear fit between force and stiffness was calculated from the measured perturbations and plotted as the basic stiffness curve in the force–stiffness graph (black dashed line in **Figure 3** top left).

### 2.2.2. Task 2—Force Task with Isometric Cocontraction

In *task 2*, subjects were asked to produce a force using the red vertical lines and to further decouple stiffness from force by using isometric cocontraction. Before *task 2*, subjects had the possibility to learn how to increase grip stiffness voluntarily by cocontraction using 10 to 20 trials that were not recorded. After this learning procedure, subjects were asked to reach 5 different force levels (15, 25, 35, 45, and 55% of MVC) given to them in a randomized order within a range of *±*5% of MVC 15 times each and use cocontraction to produce higher stiffness at a similar force than in *task 1*, leading to 75 perturbations. In other words, they had to keep the red solid line between the two red dashed lines and always produce stiffness higher than the black dashed line in **Figure 3**. After each set of 25 perturbations, the subjects paused for 5 min. During these breaks, again the EMG base noise was recorded for 5 s to detect strong deviations. After all perturbations, the subjects were asked to produce three times the MVC level for 5 s again. Note that this method does not allow commanding certain cocontraction levels. It is unfeasible to require subjects reaching a force–stiffness combination twice and can be probably only achieved after days of learning, if possible at all. This method only allows commanding the force, and the cocontraction level depends on the subject's effort.

### **2.3. Data Processing**

From the measured force data and the known position perturbation, we calculated the grip stiffness. We found out from the optical tracking data how the perturbation length is distributed to thumb and index finger. We evaluated whether and how well stiffness and force values could be predicted from EMG data and how EMG–force and EMG–stiffness relationships vary within and across subjects. We analyzed whether and how much voluntary cocontraction and the grip force before the perturbation influenced stiffness, EMG values, and kinematics.

### 2.3.1. Determination of Force and Stiffness

The methods to define the two time windows *T*bP before and *T*aP after the perturbation are similar to the one introduced in our previous study (Höppner et al., 2013) [see **Figure 5** adapted from the study by Höppner et al. (2013)], which is performed offline.

The force signals *f* were first filtered using a 21-point moving average filter. We defined the start of the perturbation *t*pert as the end of the first time interval *T*bP lasting 10 ms. *T*bP is the last time interval before *t*peak (the peak after the perturbation/maximum

**FIGURE 5 | Example for typical perturbation profile of a performed force task without cocontraction**. Force profile before, during, and after perturbation starting at *t* = 0. In addition, the time windows *T*bP and *T*aP and the mean of force for six force levels are depicted (mean force *E*TbP (*f*) subtracted). The length of *T*aP and *t*trust were found to be optimal at 18.33 and 33.3 ms, respectively [adapted from the study by Höppner et al. (2013)].

of the force signal), which has a standard deviation (SD) below <sup>5</sup> *·* <sup>10</sup>*<sup>−</sup>*<sup>4</sup> N. This number was empirically determined and led to stable results. The force before the perturbation was calculated using *T*bP. Assuming that neuromuscular feedback does not have any measurable influence within 40 ms (Höppner et al., 2013), the time *t*trust, which starts after perturbation and within which one can ignore effects of fast reflex responses, was allowed to vary between *t*pert *≤ t*trust *≤ t*pert + 40 ms and the duration *T*aP between 5 and 20 ms so as to minimize the objective function

$$\begin{split} Z = \frac{1}{n\_{\text{sub}}} \sum\_{i=1}^{n\_{\text{sub}}} \left( \frac{1}{n\_{\text{level}}} \sum\_{j=1}^{n\_{\text{level}}} \left( \tilde{\mathbf{e}}(k\_{\text{task1}\_{\text{\tilde{\eta}}}}) \right. \\ \left. + \frac{1}{n\_{\text{trial}}} \sum\_{k=1}^{n\_{\text{trial}}} \left( \tilde{\mathbf{e}}\_{T\_{\text{\tilde{\eta}}}}(f\_{\text{task1}\_{\text{\tilde{\eta}}}}) + \tilde{\mathbf{e}}\_{T\_{\text{\tilde{\eta}}}}(f\_{\text{task2}\_{\text{\tilde{\eta}}}}) \right) \right) \right) \end{split} \tag{1}$$

using all trials *n*trial, levels *n*level, and subjects *n*sub. The operator ˜*e*(*·*) *≥* 0 denotes the *coefficient of standard error* we introduced recently (Höppner et al., 2013), which combines the coefficient of variation and the standard error (SE), and which has no unit. The SE compensates the SD *σ*(*·*) for sample size *n* assessing low sample sizes with a higher SE; the coefficient of variation is a normalized measure of the SD and compensates for the sample mean *µ*(*·*). Since the objective function equation (1) mixes data sets of different size (force and stiffness) and from different dimensions (time window length and number of repetitions), we had to compensate the SD *σ*(*·*) for both. The minimum of this cost function minimizes the variation of resulting stiffness values *k* measured under *exactly* the same conditions (which is true for *task 1*, only) and the oscillations in force within time interval *T*aP of both tasks. Since subjects cannot produce the exact same cocontraction level twice (see section Experimental Procedure), and thus, the experimental conditions between perturbations in *task 2* cannot be trusted to be identical, the part of the objective function that accounts for variations in measured stiffness considers *task 1*, only. The stiffness *k* of each trial was calculated using

$$k = \frac{E\_{T\_{\rm av}}(f) - E\_{T\_{\rm br}}(f)}{\chi\_{\rm aP} - \chi\_{\rm bP}},\tag{2}$$

where *E*<sup>T</sup>bP (*·*) and *E*<sup>T</sup>aP (*·*) denote the average over time intervals *T*bP and *T*aP before and after perturbation. Note that the displacement *x*aP *− x*bP was for all experimental conditions constant (see section Device Description). The length of the second time interval *T*aP and its end *t*trust were found to be optimal under named constraints at 18.3 and 33.3 ms, respectively.

For investigating intrasubject and intersubject variability, force and stiffness were normalized subjectwise by their maximum values and divided by their SDs.

The influence of both tasks on the stiffness was analyzed statistically, as explained in the paragraph *Methods for Testing Statistical Significance* below.

#### 2.3.2. Evaluation of Optical Tracking Data

Since the optical tracking data were sometimes subject to artifacts, we detected the beginning of the perturbation within these data for each trial manually and synchronized the data sets from the real time and windows machine manually. For determining finger and thumb displacement caused by the perturbation, we applied the same time windows as for estimating stiffness from force. Furthermore, the measurements of the single markers at the index finger and thumb were not stable and sometimes flipped. Thus, we implemented a procedure that allocates these two markers according to their distance from the perturbator.

In addition, these two marker positions sometimes switched for a few milliseconds to unreasonably high values or to exact zero, which we detected automatically and discarded as missing information. For evaluating the kinematics, we used two main metrics, the SD of the distance to describe the variation in position and, if available, the SD in angular distance to describe the variation in orientation (see section Appendix). While the distance was calculated using the Euclidean norm, we calculated the angular distance between two rotation matrices *R*<sup>1</sup> and *R*<sup>2</sup> according to the study by Stillfried et al. (2014):

$$\text{angdist} := \arccos\left(\frac{\text{trace}(\mathbf{R}\_2 \cdot \mathbf{R}\_1^{-1}) - 1}{2}\right). \tag{3}$$

Since the kinematic position was controlled to be kept stable and not commanded *per se*, we refrained from analyzing the influence of kinematics on stiffness and from drawing wrong conclusions. Thus, its remaining influence is still part of the measurement noise.

### 2.3.3. Processing of the EMG Data

The oversampled EMG signal (analog card sampling inside the real-time target computer rate 10 kHz; sampling rate of the EMG signal provided by the Delsys Trigno Wireless EMG system 4 kHz) was filtered offline using a delay-free second-order Butterworth bandpass filter between 25 and 450 Hz. The produced muscular activity was evaluated using the average rectified value (ARV) over a time frame of 200 ms before the perturbation. From the relaxation task, a steady time window of about 500 ms was chosen manually (identical for all electrodes within a task), representing the EMG base noise level. The base noise of each electrode was subtracted from the EMG data subjectwise. EMG data were normalized by their maximum values and divided by their SDs for each electrode and each subject.

### 2.3.4. Regression of Force and Stiffness from EMG and Evaluation of Its Intrasubject and Intersubject Variability

We built regression models of force and stiffness from EMG using *f<sup>i</sup>* = *β*<sup>1</sup> + *β ·* **EMG** and *k<sup>i</sup>* = *β*<sup>1</sup> + *β ·* **EMG** + *β<sup>n</sup> · fi*. A clear focus is set on intersubject regression, since it allows for a subject-independent measurement of force and stiffness from muscular activity for teleoperating compliance of modern robotic hands. We divided all force and stiffness data of each subject by their SDs, since they are expected to vary considerably between subjects. The regressed models are cross-validated; for intrasubject regression, we predicted each trial subjectwise by building a model regressed from all other trials (leave-one-trialout; see section Appendix), while for intersubject regression, we predicted all trials of one subject with a model regressed from all other subjects (leave-one-subject-out). As a measure of each model fitness, the cross-validated coefficient of determination *R* <sup>2</sup> was used. For calculating the intrasubject *R* 2 cross-validated values the number of required models equals the number of perturbations per subject (leave-one-trial-out) and for the intersubject *R* 2 cross-validated values the number of required models equals the number of subjects (leave-one-subject-out) were used. Since we expected a non-linear dependency between measured EMG and force, we tested if taking the square root (Hogan, 1984) or square (Shin et al., 2009) of all EMG data improves the quality of the linear fits in force and stiffness.

### 2.3.5. Methods for Testing Statistical Significance

For significance testing, we first performed a multivariate twoway repeated-measure MANOVA to reveal whether there are significant influences of the factors *task* and *force level* and their interaction on the obtained dependent variables stiffness, kinematics, and EMG values. For the single dependent variables, we performed a univariate two-way repeated-measure ANOVA with a *post hoc* Tukey's honestly significant difference (THSD) test to reveal significant patterns of the two factors. Moreover, for testing significance of a correlation, we used a standard function in MATLAB, which provides a *p* value based on results of a *t*-test testing differences in variances. Equality of variances was tested using a two-sample F-test. Finally, Steiger's z-test was used to investigate differences between correlations (Steiger, 1980).

### **3. RESULTS**

The results of our measurements are shown as force–stiffness plots in **Figure 6**. The results are depicted as dots denoting the single perturbations. For both tasks, a linear regression between force and stiffness over all values is shown. For *task 1*, we additionally

measure of linearity.

calculated the corresponding coefficient of determination *R* 2 task1 as a measure of linearity.

The effect of force production and voluntary cocontraction on the normalized electromyogram of each of the six electrodes is depicted in **Figure 7**.

We performed a multivariate two-way repeated-measure MANOVA—including the dependent variables *stiffness*, *EMG*, *thumb*, and *index finger* displacements—to reveal whether there was a significant influence of the factors *task* and *force level*. The results showed that both factors (*p ≤* 0.001) and their interaction (*p ≤* 0.05) have a significant influence on the obtained results.

Concerning effects of learning and fatigue, we found no significant correlation between trial number to both force and stiffness for the experimental condition of *task 1*. There is a significant positive correlation for subject S6 between trial number and stiffness and a significant negative correlation for subject S5 between trial number and force for the experimental condition of *task 2*.

### **3.1. Ability to Cocontract and Decouple Stiffness from Force**

The linear regressions in **Figure 7** show the expected increase of activations across all electrodes from *task 1* to *task 2*. Results of **Figure 6** reveals clearly the expected influence of voluntary cocontraction on stiffness. Performing a univariate two-way repeated-measure ANOVA for the dependent variable *stiffness* showed that both factors *task* and *force level* (*p ≤* 0.01) are significant, but their interaction is not significant. *Post hoc* THSD tests revealed a significantly larger stiffness within *task 2* and—as might be expected—an always increasing stiffness with *force level* (*p ≤* 0.0001). Moreover, two measures for the ability to *decouple stiffness from force* are given in **Table 2** for the different force levels over the pooled trials of all subjects. The stiffness values are normalized per subject by their maximum value. The baseline stiffness at each force level is given in the first and third row as the mean of stiffness in *task 1*, *⟨k ∗ task*1*<sup>i</sup> ⟩*, and its SD *s*(*k ∗ task*1*<sup>i</sup>* ), in which

task1

subjects are asked to produce simply force without cocontraction. In the second and forth row, the mean stiffness of *task 2*, *⟨k ∗ task*2*<sup>i</sup> ⟩*, and its SD *s*(*k ∗ task*2*<sup>i</sup>* ) are given, in which the subjects try to increase stiffness by cocontraction. The difference *⟨k ∗ task*2*i−k ∗ task*1*<sup>i</sup> ⟩* and their ratio *⟨k ∗ task*2*<sup>i</sup> /k ∗ task*1*<sup>i</sup> ⟩* in the fifth and sixth row are two different measures exhibiting the average voluntary increase in stiffness through cocontraction.

### **3.2. Kinematics**

Beside minimizing the variation in kinematic orientation and position during the experiments, the kinematic data reveal insights on how the total perturbation length of 7.5 mm is distributed between thumb and index finger and give an indication of the relative stiffnesses of the two digits. **Table 3** provides the results of the finger and thumb perturbation displacements for all subjects with respect to the wrist frame, their average values, and SDs in percent; all displacements are divided by the total perturbator displacement of 7.5 mm (2.5% of the data is zero and thus deleted; see section Data Processing). Note that we related the thumb and index finger position before and after perturbation to the wrist frame instead to the world coordinate frame to get rid of forearm movements interpreted as grip displacements; anyway, both lead to similar results (world coordinate frame related data not listed).

Performing univariate two-way repeated-measure ANOVAs for the dependent variables *thumb* and *index finger* displacements showed that the factor *force level* is significant for both variables (*p ≤* 0.05), but the factor *task* is significant for the index finger displacement (*p ≤* 0.01), only. *Post hoc* THSD tests revealed no significant pattern for both factors and variables.

For details about how subjects performed in keeping the predefined position, please have a look into the Appendix.

**TABLE 2 | Mean difference and ratio between normalized stiffnesses of the two tasks for the single force levels**.


*The mean values in stiffness ⟨k ∗ task1<sup>i</sup> ⟩ and ⟨k ∗ task2<sup>i</sup> ⟩ of the two tasks and their SDs s*(*k ∗ task1<sup>i</sup>* ) *and s*(*k ∗ task2<sup>i</sup>* ) *are given. In addition, mean difference in normalized stiffness ⟨k ∗ task2<sup>i</sup> − k ∗ task1<sup>i</sup> ⟩ and their ratio ⟨k ∗ task2<sup>i</sup> /k ∗ task1<sup>i</sup> ⟩ for all force levels in percent of MVC over all subjects are listed. Note that the index i denotes the mean over subjects.*

#### **TABLE 3 | Perturbation displacement of index finger and thumb**.

### **3.3. Regressing Force and Stiffness from EMG**

We performed an intersubject regression of stiffness and force from EMG (see **Tables 4** and **5**). The results showed a large influence of the muscular activity of FDI, SDI, and force to the regression of stiffness, while all electrodes except EIP contributed equally to the regression of force. The coefficient of determination of both models highly differs between both regressions across all subjects: 72 *±* 12% and 33 *±* 18% for regressing stiffness and force, respectively. The mean correlation coefficients and their SDs between stiffness, force, and muscular activity across all subjects are listed in **Table 6**. By using these values, we conducted a paired *t*-test on the Fisher-transformed correlation coefficients on whether the correlation of EMG to force and stiffness significantly differs across subjects. The results show that only for the two intrinsic muscles in the hand, the correlation of EMG to stiffness significantly differs in comparison to its correlation to force (*p ≤* 0.001). A detailed analysis of the correlations between force, stiffness, and muscular activities for each of the two tasks can be found in the Appendix. Moreover, an overview on the contributions from the three groups of muscles—extrinsic extensors and flexors and interossei—can be found here.

Since literature inconsistently reports, we tested whether taking the square root or square of EMG data improves the quality of the linear fits of force and stiffness to EMG using Steiger's z-test (Steiger, 1980). The tests showed that the plain muscular activity provides a better correlation to both force (*p <* 0.001) and stiffness (*p <* 0.05) than taking the square of muscular activation. Moreover, no clear improvement can be found by taking the square root in comparison with plain muscular activity. Finally, taking the square root of muscular activity in comparison to the square clearly improves its correlation to force (*p <* 0.01), but not to


*Mean and SD s of index finger and thumb displacements x<sup>T</sup>b/aP between before and after perturbation in [%]. The displacements are divided by the total perturbator displacement of 7.5 mm. 2.5% of the data set was deleted.*


*–, no significance; ., p ≤ 0.05;* +*, p ≤ 0.01;* ++*, p ≤ 0.001;* +++*, p ≤ 0.0001.*

*The significance of the respective coefficients and models' coefficient of determination are listed for each subject. Note that the model is cross-validated (leave-one-subject-out).*


*–, no significance; ., p ≤ 0.05;* +*, p ≤ 0.01;* ++*, p ≤ 0.001;* +++*, p ≤ 0.0001.*

*The significance of the respective coefficients and models' coefficient of determination are listed for each subject. Note that the model is cross-validated (leave-one-subject-out).*

**TABLE 6 | Correlation between stiffness, force, and EMG**.


*Mean and SD of correlation coefficients between stiffness, force, and muscular activity across subjects and tasks. By using these values, we conducted a paired t-test on the Fisher-transformed correlation coefficients on whether the correlation of EMG to force and stiffness significantly differs across subjects. The results show that only for the two intrinsic muscles FDI and SDI, the correlation of EMG to stiffness significantly differs in comparison to its correlation to force (p ≤ 0.001).*

stiffness. Conclusively, all reported results and analyses focusing on regressing stiffness and force from EMG use the plain muscular activity.

For details about intrasubject regression, please have a look into the Appendix.

### **4. DISCUSSION AND CONCLUSION**

In this article, we analyzed the role of voluntary cocontraction for decoupling grip stiffness from its natural increase with grip force. To measure influences from cocontraction only, we minimized effects of variabilities in kinematics by providing the subject a visual feedback of the current hand and arm posture. In a first task, we asked subjects to apply a set of force levels several times without the use of cocontraction to measure the basic force–stiffness coupling. In a second task, we asked subjects to decouple stiffness from force using voluntary cocontraction while holding a specific force level. We measured EMG to investigate the possibility of regressing stiffness and force from the measurement of muscular activity.

### **4.1. Ability to Decouple Stiffness from Force by Cocontraction**

The results show that the subjects were able to increase grip stiffness between 15 and 26% of maximum stiffness by the use of cocontraction. By using the difference *⟨k ∗ task*2*<sup>i</sup> − k ∗ task*1*<sup>i</sup> ⟩*, the results show an increasing ability with force (*r* = 0.30, *p <* 0.05). Milner and Franklin reported in the study by Burdet et al. (2013) based on results of Milner (2002) a 5-fold range in modulation of wrist stiffness at zero net joint torque. Similarly, subjects in our experiment were able to modulate stiffness by cocontraction in a 5.2-fold range for the lowest force level. On average, subjects were able to vary stiffness *⟨k ∗ task*2*<sup>i</sup> / k ∗ task*1*<sup>i</sup> ⟩* with cocontraction by a 2.7 *±* 2.2-fold range (maximum at first force level of subject S5 with a 22-fold and minimum at second force level of subject S7 with a 0.8-fold modulation in stiffness).

The results provide an overview to what extent the human is able to decouple grip stiffness from force using cocontraction, while probably revealing only parts of it: First, subjects in our study had problems to stably hold the lower force levels at high cocontraction, where effects of motor noise on hand shaking are considerably higher (which confirms the supposition that cocontraction is the wrong strategy to stably hold a force level). Similarly, Kearney and Hunter (1990) reported in a study performed at the human ankle that subjects had difficulties achieving cocontractions involving high levels of muscle activations at zero net torque. Thus, subjects in our experiments probably did not use their full ability to decouple stiffness from force for the lower force levels, while they did for the higher ones. Maybe the strategy we used in our experiments of restricting subjects to exactly hold a force level is not the optimal solution for the lower levels. A better strategy might be monitoring the steadiness of force as a perturbation criterion, while the experimenter supervises the force range to help subjects reaching the higher cocontraction levels for the lower forces. Nevertheless, there is evidence suggesting that neural mechanisms of muscle inhibition and excitation exist, which limit the ability to produce all possible sets of cocontractions, probably to avoid harming the muscular system (De Luca and Mambrito, 1987). On the other hand, Milner (2002) reported that subjects were not able to voluntarily apply maximal cocontraction, but could possibly increase it by days of training similar to the study by Darainy et al. (2004). Furthermore, *task 2* in our experiments was performed up to forces of 55% MVC, only. As we found in our study (Höppner et al., 2011), this constraint avoids fast fatigue of corresponding muscles for subjects during this longlasting experimental procedure, but does not allow us to draw conclusions about forces up to 100% MVC.

To have similar cocontraction ranges at all force levels, we commanded in a former version of the experiments a combination of applied force and EMG similar to the work done by Osu and Gomi (1999), Osu et al. (2002), and Shin et al., 2009. We merged the different EMG signals into one lumped signal and asked subjects to hold different combinations of force and summarized EMG; so instead of commanding stiffness, we commanded an EMG level, which should be related to cocontraction in some way. Due to the high density of muscles in the forearm lying in different layers and thus high cross talk of multiple muscles, subjects learned to produce the EMG levels and simultaneously learned to reduce the metabolic costs for producing it. This resulted in subjects successfully solving the task without producing an increase in the measured grip stiffness. This led to the decision for a redesign of the experiments and to command grip stiffness *per se* rather than a combined EMG level.

Anyway, similar to our results reported above, Akazawa et al. (1983) found that the reflex responsiveness and stretch-evoked stiffness increase linearly with cocontraction as defined in their article. Also, the slope of this increase is steeper, the larger the tonic force is, corresponding to our result of an increasing stiffness modulation capability with higher force. However, please note that Akazawa et al. (1983) only compared the cocontraction levels of two tonic force levels achieved in the constant-load position control task and measured reflex-affected stiffness.

Finally, it needs to be mentioned that our finding of an increasing ability for decoupling force and stiffness by cocontraction is opposing the torque–stiffness plots of existing VSA mechanisms (as mentioned in section Introduction), which have a rather limited ability to decouple stiffness from torque, especially for the higher torques. The force–stiffness plots we measured within this study allow for the first time for a suitable insight and can be helpful information for robotic engineers designing VSAs.

### **4.2. Finger Displacement**

The evaluation of tracked kinematics show that for all experimental conditions the index finger got perturbed by about 2/3 and the thumb by about 1/3 of the whole displacement (see **Table 3**). This means that *the thumb is approximately twice as stiff as the index finger*. Assuming that both, the measured intrinsic stiffness and the force correspond to the number of attached crossbridges [(Burdet et al., 2013), p. 41f.], this means that *the thumb is also approximately twice as strong as the index finger*. This theory is backed by the findings of Olafsdottir et al. (2005), who showed MVC finger forces of thumb and index of 73 *±* 18 and 33 *±* 6.6 N, respectively. During their measurements, all digits were activated simultaneously and the thumb opposed the other fingers. Nevertheless, it remains unclear whether this ratio is dominated by stiffer muscles or a difference in moment arms of index finger and thumb in a pinch grip.

### **4.3. Regressing Stiffness and Force from EMG**

We built for each subject a linear model using all other subjects and used it to estimate the stiffness/force data based on muscular activity and force (leave-one-subject-out cross-validation). Even if the subject is unknown, these models provide surprisingly good results for the regression of stiffness. However, this holds for the regression of stiffness, only, and not for force. What is the reason? The significances of the coefficients for these two regressions show that the two intrinsic muscles in the hand had an unexpectedly high influence on the modeling of stiffness, while all muscles contributed almost equally to the regression of force. Looking into correlations between stiffness, force, and muscular activities shows a comparatively high correlation of the intrinsic muscles to stiffness (see **Table 6**). Moreover, the SDs of these correlations are significantly less for the two intrinsic muscles than for the extrinsic ones (*p <* 0.05) meaning that these muscles provide a stable correlation across subjects. This is possibly a result of higher signal-to-noise ratio (SNR) for the intrinsic muscles. Since the measured surface EMG signal involves the EMG pattern from other, deep, muscles—which we interpret as a lower SNR for the extrinsic muscles—the correlation of forearm muscles dropped, while the one of the intrinsic muscles in the hand did not. Similarly, Maier and Hepp-Reymond (1995) reported for almost all intrinsic hand muscles about "*. . .*high correlations to grip force with low variability, whereas the majority of the extrinsic muscles, with the exception of the long flexors, have lower correlations and higher individual variability*. . .*" in an isometric pushing task.<sup>1</sup> Conclusively, the possibility for a suitable regression of stiffness as it is influenced by voluntary cocontraction across subjects is caused by a high and stable correlation between stiffness and intrinsic muscular EMG across all subjects.

But can we conclude from these differences for regressing force and stiffness that the interossei predominate the decoupling of stiffness, perhaps by having a steeper increase of stiffness with force, while force is produced by all groups of muscles equally? Or is it just the case that the intrinsic muscles are simultaneously activated with muscles that we do not measure with EMG, but which contribute to the measured stiffness?

First, we need to acknowledge that prestudies led us to the wrong conclusion of a predominant role of the index finger on the measured stiffness, based on which we decided to exclude muscles activating the thumb from the EMG measurements. But since we find the thumb to be *just* twice as stiff, we cannot reason a dominating role of the index finger with certainty. Thus, we cannot clarify plausibly if it is causality (intrinsic muscles predominate cocontraction) or just correlation (intrinsic muscles are synergistically activated) from the conducted experiments. But the result can be interpreted from a biomechanical point of view: coactivating extrinsic flexor and extensor muscles introduces high forces on the finger joints. This may lead to instability at—in particular—the metacarpophalangeal joint: it could reduce the strain by an uncontrolled sideways, abductionlike, movement. The interossei muscles, connecting the proximal and metacarpal bones at each side of the metacarpophalangeal joint, can be used to stabilize this movement—and apparently do. A somewhat similar mechanism can be found, e.g., during pinch grip: extrinsic extensor muscles—namely extensor carpi ulnaris and extensor carpi radialis longus/brevis—are activated simultaneously with flexor muscles to prevent the wrist from moving; i.e., the intent is to contract the flexor muscles, and the extensor muscles are activated involuntarily to provide support.

<sup>1</sup> Please note that the authors of Maier and Hepp-Reymond (1995) asked subjects for the production of low isometric forces, only, and not for voluntary cocontraction. Similarly, we reported an overview of correlation coefficients between force, stiffness, and muscular activities for both tasks in the Appendix.

Moreover, a publication from Milner et al. (1995) argues in an opposite way: from an investigation of moment arms and physiological cross-sectional areas of the first dorsal interosseus and lumbricalis muscles, they revealed that these muscles must have a predominant role for controlling the force direction at the index finger, while the extrinsic muscles in the forearm act as stabilizers. Hence, they concluded that extrinsic muscles should contribute much more to finger stiffness.

It needs to be acknowledged that the SDI does neither control index finger nor thumb and controls the movement of the middle finger, only (see **Table 1**). However, due to a high influence we measured in prestudies, we decided to include this electrode. The performed experiments prove this initial finding with a large influence of the gathered SDI activity on stiffness. This is possibly caused by either a synergistic activation of this muscle or the measurement of cross talk from other muscles, e.g., first palmar interosseus.

Note that we investigated the use of non-linear regression models, as well, to improve the results: Gaussian processes (Rasmussen and Williams, 2006), linear regression with random Fourier features (Rahimi and Recht, 2007), and neural networks. None of these methods showed a significant improvement of model fitness over the linear approach, which is why we neglect them in this study. We hypothesize that the small amount of data available (approximately 100 data points for 10 subjects) does not allow to fully leverage the power of more expressive models.

On the basis of the results in the studies by Joyce and Rack (1969) and Vrendenbregt and Rau (1973), Hogan (1984) reported a linear dependency between muscle force and measured EMG activation until 30% of maximum voluntary contraction and a muscle force proportional to the square root of the pooled firing rate. Contrary, Shin et al. (2009) proposed that muscle tension follows a quadratic function of measured activation. Thus, we tested whether applying a square or square root to our processed EMG data would improve the fit. The results show that taking the square root or square of muscular activity neither improves its correlation to force nor improves its correlation to stiffness. Moreover, the results show that taking the square even makes the correlations worse. However, our measurements include levels of 55% of MVC only and do not allow us to draw conclusions for the higher force levels.

All in all, the intrinsic muscles in the hand are found to dominate our regression of stiffness and not of force, while the experiment design does not allow us to reveal whether the stiffness itself is dominated by these muscles. A good possibility to answer this question might be the use of functional electrical

### **REFERENCES**


stimulation placed on respective extrinsic and intrinsic muscles as performed for the human hand (Lauer et al., 1999) or for the intrinsic plantar foot muscles in the study by Kelly et al. (2014), which was not the focus of the experiments performed in this study. Nevertheless, the result is promising: the high SNR and high correlation to stiffness of the intrinsic hand muscles allow for a continuous measurement of grip stiffness and to explain on average 72 *±* 12% of its variance without any prior knowledge about the subject, i.e., calibration of stiffness to force and EMG in advance. This information allows to continuously teleoperate finger stiffness to actively impedance controlled robotic hands, as well as hands based on VSAs (Grebenstein et al., 2011). Moreover, it allows to continuously measure a *task-dependent stiffness* during activities of daily living: Leidner et al. (2015) started categorizing *Compliant Manipulation Tasks*into a task taxonomy, e.g., by classifying tasks of contact/no contact, in-hand manipulation/external manipulation tasks, or tasks with and without deformation of the environment. By continuously measuring stiffness of the hand, it will be possible to measure a *task dependent stiffness* during activities of daily living, such as cutting an onion, cleaning with a sponge or connecting a plug (Leidner et al., 2015), and to add a meaningful range of stiffness values to the derived taxonomy matrix.

### **AUTHOR CONTRIBUTIONS**

HH developed the idea and device, researched the literature, analyzed the data, and contributed to the acquisition of data. MG-D contributed to acquisition of data and the analysis of data. GS contributed to the acquisition and analysis of data and revised the work. JB contributed to the regression analysis and testing of machine-learning methods. PS contributed to the interpretation of data and revised the work.

### **ACKNOWLEDGMENTS**

This work has been funded by the European Commission's Eighth Framework Program as part of the project SoMa (grant number H2020-ICT-645599). We would like to thank Armin Giebel and Michael Sachs from the Department of Applied Science and Mechatronics at Munich University of Applied Sciences for their helpful assistance and advisory support. Furthermore, we would like to thank Claudio Castellini for his support in an early stage and Rachel Hornung for her assistance for questions regarding non-linear regression (both same affiliation as the first author).


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2017 Höppner, Große-Dunker, Stillfried, Bayer and van der Smagt. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## **APPENDIX**

### **A. Kinematics**

**Table A1** in Appendix lists the variation in distance of all markers over all subjects in reference to the world coordinate frame and in reference to each other (0.7% of the optical tracking data is zero and thus deleted). **Table A2** in Appendix does the same for the orientation of perturbator, wrist, and forearm (0.6% of the optical tracking data is zero and thus deleted). The marker position and orientation of the forearm of subject S8 was controlled during the experiment but not recorded for some unknown reason. The SD of the horizontal orientation of the perturbator is found to be *±*2.95 (see **Figure 3**).

The displacement of the index finger is found to be slightly decreasing (*test statistics for correlation r* = *−* 0.17; *p ≤* 0.001) and the displacement of the thumb slightly increasing (*test statistics for correlation r* = 0.20; *p ≤* 0.001) with force over all subjects, while there is no significant correlation to stiffness. Furthermore, there is a slight increase of index finger and thumb displacement (*test statistics for correlation r* = 0.16 and *r* = 0.077; *p ≤* 0.025) with the number of perturbations (duration of the experiment).

### **B. EMG Base Noise**

In the relaxing task, a mean base noise ARV of 5.8 *±* 1.7 *µ*V over all subjects and electrodes was measured, which is consistent with literature (Konrad, 2005).

### **C. Correlations between Force, Stiffness, and Muscular Activities for Both Tasks**

**Figure A1** in Appendix shows the mean correlation coefficients between force, stiffness, and the single muscle activations and their SDs across subjects for the two tasks. Maier and Hepp-Reymond (1995) reported for almost all intrinsic hand muscles about "*. . .*high correlations to grip force with low variability,

**TABLE A1 | SDs s in distance between all tracked markers**.


*SDs in distance over all subjects for the single tracked markers index finger, thumb, perturbator, wrist, and forearm inside TbP in [mm] in reference to each other and to the world coordinate system. 0.7% of the data set was deleted.*



*SDs in angular distance over all subjects for the single tracked markers perturbator, wrist, and forearm inside TbP in [◦ ] in reference to each other and to the world coordinate system. 0.6% of the data set was deleted.*

whereas the majority of the extrinsic muscles, with the exception of the long flexors, have lower correlations and higher interindividual variability*. . .*" high correlations in an isometric pushing task. We can confirm a good correlation to force in a pure pushing task. Moreover, the low intersubject variability for the intrinsic muscles in comparison to the extrinsic is obvious as well in all our experimental conditions for both force and stiffness.

Differences regarding mean values between the two tasks and between correlations to force and stiffness are clearly visible. While the correlations are similar for the condition of the isometric pushing *task 1*, they differ for the condition of *task 2* and the usage of voluntary cocontraction. Moreover, the strong correlation of the intrinsic muscles to stiffness (but not force) across both experimental conditions can be seen.

### **D. Regressing Stiffness and Force from EMG**

For regressing force and stiffness from the measured muscular activity, we normalized the force and stiffness values and divided them by their SD subjectwise (see section Data Processing). **Figure A2** in Appendix shows for each subject the mean and SD across all values for both force and stiffness.

**Tables A3** and **A4** in Appendix list the results of an intrasubject regression of stiffness and force from EMG (leave-onetrial-out cross-validation). Similar to the results of the intersubject regression, a dominant role of FDI and SDI can be seen for the regression of stiffness, while all muscles contribute equally to the regression of force. Thereby, the coefficient of determinations are 78 *±* 10% and 62 *±* 14% for regressing stiffness and force, respectively. Naturally, the intrasubject regression provides a better fit in comparison to the intersubject regression. Again, the regression of stiffness from muscular activity performs better than the regression of force.

Moreover, **Figure A3** in Appendix shows plots of measured and predicted stiffness and force data using intrasubject regression for both tasks. These plots show how much of the independence of force and stiffness can be extracted from the EMG signals. If the predicted force–stiffness points cover the same area as the measured force–stiffness points, their independence is completely retained after the prediction from EMG. If the predicted points lie on a line, their independence is completely lost and the information content of the EMG signal is reduced to one.

### **E. Contributions of Muscle Groups to the Regression of Force and Stiffness**

Besides the influence from each single muscle and electrode, it is of interest how the muscle groups—extrinsic flexors, extrinsic extensors, and intrinsic interossei, with two electrodes each—contribute to the regression of stiffness and force from EMG. Moreover, it is of interest how much the information of force adds to the regression of stiffness. The results of a linear regression on intrasubject and intersubject variability of stiffness and force from EMG (and force) are plotted in **Figure A4** in Appendix. As a measure of each model fitness, the cross-validated coefficient of determination *R* 2 is used. For calculating the intrasubject *R* 2 cross-validated values the number of required models

**FIGURE A1 | Correlation between force, stiffness, and muscle activations**. Mean correlation coefficients between force, stiffness, and the single muscle activations and their SDs across subjects for the two tasks. The SDs across 10 subjects are depicted as error bars. The diagrams on the top, middle, and below show the results of *task 1*, *task 2*, and for both tasks, respectively. Note that the results of the diagram are redundant to the information provided in **Table 6**.

stiffness for each subject across all values.

equals the number of perturbations per subject (leave-one-trialout) and for the intersubject *R* 2 cross-validated values the number of required models equals the number of subjects (leave-onesubject-out). This analysis provide 4 interesting results: (a) Using


*–, no significance; ., p ≤ 0.05;* +*, p ≤ 0.01;* ++*, p ≤ 0.001;* +++*, p ≤ 0.0001.*

*The significance of the respective coefficients and models' coefficient of determination are listed for each subject. Note that for calculating the coefficients of determination the model is cross-validated (leave-one-trial-out).*

**TABLE A4 | Intrasubject regression of force from EMG**.

muscular activity of the intrinsic muscles in the hand to regress stiffness provides a considerably better fit than using EMG of any extrinsic muscle group, which is true for both intrasubject and intersubject regression. (b) Using the intrinsic muscle, EMG works considerably better than just using force for the regression of stiffness. What's more, it seems that adding additional state information, namely, the measured grip force, does not add much to the regression of stiffness as it is decoupled from force. (c) Similar to the analysis of correlation coefficients in **Figure A1** in Appendix, the **SD** of the intrinsic muscles to regress stiffness across subjects is comparably low, which is why these muscles allow for a suitable intersubject regression, as well. (d) The regression of force from EMG works totally differently, i.e., there is no dominating role of the intrinsic muscles. If at all, the extrinsic extensors seem to dominate here. But for an adequate intrasubject regression of force from EMG, the information of all muscles is necessary. While the regression of stiffness is found to be working for intersubject regressions as well, an intersubject regression of force from EMG is not.


*–, no significance; ., p ≤ 0.05;* +*, p ≤ 0.01;* ++*, p ≤ 0.001;* +++*, p ≤ 0.0001.*

*The significance of the respective coefficients and models' coefficient of determination are listed for each subject. Note that for calculating the coefficients of determination the model is cross-validated (leave-one-trial-out).*

**FIGURE A3 | Independence of predicted data of intrasubject regression**. Results of multiple linear regression of stiffness *k*(*f*,EMG) and force *f*(EMG) and their coefficients of determination *R* 2 in comparison to the measured values for both tasks. If the predicted values are located more or less on a line, the two regression models are most likely not linear independent, and the content of information of the respective EMG signals reduces to one.

**results of performed statistical testing**. Mean leave-one-out cross-validated coefficient of determination *R* 2 for intrasubject (left; leave-one-trial-out) and intersubject (right; leave-one-subject-out) stiffness and force over all subjects for regressing stiffness from EMG or force, from EMG and force, and for regressing force from EMG using different muscle groups. The SDs over 10 subjects are depicted as error bars.

# Deep Learning with Convolutional Neural Networks Applied to Electromyography Data: A Resource for the Classification of Movements for Prosthetic Hands

#### Manfredo Atzori\*, Matteo Cognolato and Henning Müller

Information Systems Institute, HES-SO Valais-Wallis, University of Applied Sciences Western Switzerland, Sierre, Switzerland

Natural control methods based on surface electromyography (sEMG) and pattern recognition are promising for hand prosthetics. However, the control robustness offered by scientific research is still not sufficient for many real life applications, and commercial prostheses are capable of offering natural control for only a few movements. In recent years deep learning revolutionized several fields of machine learning, including computer vision and speech recognition. Our objective is to test its methods for natural control of robotic hands via sEMG using a large number of intact subjects and amputees. We tested convolutional networks for the classification of an average of 50 hand movements in 67 intact subjects and 11 transradial amputees. The simple architecture of the neural network allowed to make several tests in order to evaluate the effect of preprocessing, layer architecture, data augmentation and optimization. The classification results are compared with a set of classical classification methods applied on the same datasets. The classification accuracy obtained with convolutional neural networks using the proposed architecture is higher than the average results obtained with the classical classification methods, but lower than the results obtained with the best reference methods in our tests. The results show that convolutional neural networks with a very simple architecture can produce accurate results comparable to the average classical classification methods. They show that several factors (including pre-processing, the architecture of the net and the optimization parameters) can be fundamental for the analysis of sEMG data. Larger networks can achieve higher accuracy on computer vision and object recognition tasks. This fact suggests that it may be interesting to evaluate if larger networks can increase sEMG classification accuracy too.

Keywords: electromyography, prosthetics, rehabilitation robotics, machine learning, deep learning, convolutional neural networks

## INTRODUCTION

Transradial amputees can be highly impaired, even if equipped with the most modern prostheses. The recent advances in deep learning and convolutional neural networks may contribute to help them recover some of their capabilities by bridging the gap between the prosthetics market (that requires fast and robust control methods) and recent scientific research results in

#### Edited by:

Michael Wininger, University of Hartford, USA

#### Reviewed by:

Leslie Samuel Smith, University of Stirling, UK Ashley Kleinhans, University of Johannesburg, South Africa

> \*Correspondence: Manfredo Atzori manfredo.atzori@hevs.ch

Received: 15 July 2016 Accepted: 22 August 2016 Published: 07 September 2016

#### Citation:

Atzori M, Cognolato M and Müller H (2016) Deep Learning with Convolutional Neural Networks Applied to Electromyography Data: A Resource for the Classification of Movements for Prosthetic Hands. Front. Neurorobot. 10:9. doi: 10.3389/fnbot.2016.00009

**27**

rehabilitation robotics (that shows that dexterous and proportional control is possible).

Currently, the prosthetics market offers myoelectric prosthetic hands that are extremely advanced from a mechanical point of view and that can perform many different movements. However, the control methods are still in most cases rudimentary in order to guarantee 100% control robustness and sufficient control speed. Many myoelectric prosthetic hands are commercially available, however, few of them have the capability to reproduce many different movements. A selection of the most advanced prosthetic hands available in the market according to their movement capabilities currently include the following ones: (1) Vincent hand Evolution 2; (2) Steeper Bebionic v3; (3) Otto Bock Michelangelo; and (4) Touch Bionics i-limb Quantum (Atzori and Müller, 2015). Some of these prostheses are characterized by very high dexterity: they allow the movement of up to five different fingers independently. They allow the rotation of the thumb, to reproduce up to 36 different movements and the rotation of the wrist in near real time. In general, a commercial myoelectric prosthesis is opened or closed through the contraction of specific remnant muscles. While the mechanical characteristics of the mentioned prostheses are advanced, the control systems rely in most cases on specific movement triggers or sequential control strategies. Movement triggers link specific surface electromyography (sEMG) pulse sequences to specific movement of the prosthesis. Sequential control strategies allow to shift between a set of predefined movements through specific signals (e.g., through co-contraction, i.e., the simultaneous activation of two sEMG electrodes). Some of the considered prostheses include external sources of information in the form of active falling object prevention systems or via smartphones. Touch Bionics offers a selection of grasps according to objects located near the prosthesis (using Near-Field Communication, NFC) or according to action patterns (using accelerometer and gyroscope measurements). In the most advanced cases, pattern recognition is also used to control the prosthesis in combination with traditional methods. This solution has been proposed since 2013 by Coaptengineering and it was recently introduced by Touch Bionics to control wrist rotation. The mentioned control methods offer robust results, which are deemed to be one of the main needs in real use (Farina et al., 2014). However, the movement imagined to control the prosthesis is not natural, since it does not correspond to the movement that the amputee would have imagined to do, in order to control his real hand before the amputation. It also does not allow to control a large set of movements.

Proportional, natural and dexterous controls of robotic hand prostheses have been studied for a long time by scientific researchers. However, the current results are still not robust enough to be translated to real life use. Most of the methods rely on the use of sEMG and pattern recognition or proportional control algorithms. Pattern recognition algorithms are used to classify the movement that the subject aims to perform according to a label (Scheme and Englehart, 2011). The classification accuracy can be higher than 90–95% on less than 10 classes. However, average results are usually below 80–90% (Peerdeman et al., 2011). Simultaneous pattern recognition has been studied recently (Jiang et al., 2013; Ortiz-Catalan et al., 2013; Young et al., 2013). Proportional and simultaneous control of a large number of degrees of freedom of the prosthesis can allow achieving more natural and dexterous control using unsupervised or supervised methods (Fougner et al., 2012; Farina et al., 2014). Recently, semi-supervised methods and supervised methods were compared to evaluate the impact of precise kinematic estimations for accurately completing goal-directed tasks (Jiang et al., 2014).

Real time studies, allowing the user to adapt his response to the control software can provide a good representation of prosthesis usability (Hargrove et al., 2007; Scheme and Englehart, 2011). However, since these studies require the interaction of the user with the control system, they do not allow easy comparison with innovative analysis procedures. Another common problem in the field is that the studies are often highly specific and they are not directly comparable due to different acquisition setups, protocols and analysis pipelines. Moreover, often the datasets are not publicly available. The usefulness of benchmark databases has been demonstrated repeatedly in other fields, e.g., in the machine vision and image analysis communities (Müller et al., 2009; Everingham et al., 2010). Offline data analysis on public benchmark datasets allows the comparison of different methods and setups, accelerating the search and pushing forward progress in prosthetic control robustness. In 2014, the biggest publicly available benchmark database was released by the NinaPro project (Atzori et al., 2015). It consists of three datasets containing sEMG, accelerometer, and both hand kinematic and dynamic data recorded from 67 intact subjects and 11 amputees performing at least 50 hand movements.

Promising results have been obtained by invasive methods such as Peripheral Nerve Interfaces (Urbanchek et al., 2012), Cortical Interfaces (Chestek et al., 2011) or Targeted Muscle Reinnervation (TMR; Kuiken et al., 2009). The latter has shown very promising results, especially in transomeral or shoulder amputees (Atzori and Müller, 2015). TMR consists of the re-innervation of spare muscles of the amputee with the residual nerves of the amputated limb. However, the invasiveness of the procedure can strongly limit the application possibilities. A recent survey explored the interest of upper-limb amputees in four different techniques for prosthetic control: myoelectric, TMR, peripheral nerve interfaces, and cortical interfaces. Participants expressed the most interest in the myoelectric control, while the cortical interface elicited the lowest interest (Engdahl et al., 2015). This highlights that invasive techniques can be rejected by amputees.

Multimodal data acquisition has also been investigated. Computer vision has been combined with sEMG-based detection of movement intention to predetermine the type and size of the required grasp in relation to the object (Doˇsen et al., 2010; Markovic et al., 2014). Accelerometers showed excellent capabilities to recognize hand movements using pattern recognition and regression methods, both alone and in combination with sEMG electrodes (Atzori et al., 2014c; Gijsberts et al., 2014; Krasoulis et al., 2015).

Nevertheless, despite several improvements on the market and scientific research, the robust natural control of dexterous prosthetic hands is still missing.

Deep learning and convolutional neural networks recently revolutionized several fields of machine learning, including speech recognition and computer vision. Thus, it seems reasonable to investigate its abilities in sEMG as well.

Despite it often being considered as a new and emerging field, the birth of deep learning can be set in the 1940's. It passed through several stages and names over the years: born and known as cybernetics, it became popular as connectionism between the 1980's and 1990's, while since 2006 it started to be called with the current name (Goodfellow et al., 2016). In Goodfellow et al. (2016), the increasing dataset and model sizes are recognized as key points of the new success of this kind of approach. Thanks to the hardware and software advances it is now possible to use large networks trained with large datasets, allowing the exploitation of their capabilities.

Deep neural networks have been successful in several applications since the 1980's. However, in the field of computer vision in 2012, deep learning approaches won one of the largest object recognition challenges (the ILSVRC) decreasing the previous top-5 error rate by more than 10% (Krizhevsky and Hinton, 2010; Goodfellow et al., 2016). Since then, only techniques based on convolutional neural networks have won this competition, leading to top-5 error rates lower than 5% (He et al., 2015; Goodfellow et al., 2016). Another remarkable result in computer vision was obtained in 2012, when humanlevel results were reached using multi-column deep neural networks on computer vision benchmarks (Cires¸an et al., 2012). In the computer vision field, deep neural networks are also successfully applied in pedestrian detection (Sermanet et al., 2013) and traffic sign classification (Cires¸an et al., 2012).

Since 2010, the application of deep learning techniques to speech recognition has allowed a quick and impressive reduction of error rate (Dahl et al., 2010; Deng et al., 2010b, 2013; Hinton et al., 2012; Goodfellow et al., 2016).

Deep learning methods are also successfully applied to applications requiring to process big amounts of data, such as drug discovery (Ramsundar et al., 2015), compound activity prediction (Dahl et al., 2014) and genomic information annotation (Chicco et al., 2014). Moreover, they have also improved the performance of reinforcement learning, where a machine or software agent is able to maximize its performance by itself performing trials and errors (Mnih et al., 2015; Goodfellow et al., 2016).

As reported, there are several and continuously increasing deep neural network applications. However, convolutional neural networks have been applied to sEMG hand movement recognition mainly in a single conference article. Park and Lee (2016) used a convolutional neural network model composed of an input layer, four convolutional layers, four subsampling layers, and two fully connected layers to improve inter-user variability in six hand movements via sEMG signals. The strategy adopted was to perform a first non-adaptation experiment, applying a trained model (or classifier) and a second experiment using a retrained model (or classifier) using few labeled data. The results show a better classification accuracy for the convolutional neural network compared to Support Vector Machines (SVM) in both experiments. The highest accuracy was reached using convolutional neural networks with the retrained network.

In this article, we apply convolutional neural networks to the classification of 50 hand movements in 67 intact subjects and 11 transradial hand amputees and we compare the results with those obtained with classical machine learning methods on three Ninapro datasets (Atzori et al., 2014b). The Ninapro database is particularly useful for this analysis since it provides publicly available data and reference classification performances with classical machine learning procedures.

### MATERIALS AND METHODS

### Subjects

The data analyzed in this article are from the Ninapro database that includes electromyography data related to hand movements of 78 subjects (11 transradial amputees, 67 intact subjects) divided into three datasets. The Ninapro dataset 1 includes data acquisitions of 27 intact subjects (7 females, 20 males; 2 left handed, 25 right handed; age 28 ± 3.4 years). The second dataset includes data acquisitions of 40 intact subjects (12 females, 28 males; 6 left handed, 34 right handed; age 29.9 ± 3.9 years). The third dataset includes data acquisitions of 11 transradial amputees (11 males; 1 left handed, 10 right handed; age 42.36 ± 11.96 years). All participants signed an informed consent form. The experiment was approved by the Ethics Commission of the state of Valais (Switzerland), and it was conducted according to the principles expressed in the Declaration of Helsinki. More details about the subjects are reported in the official database description (Atzori et al., 2014b).

### Acquisition Setup and Protocol

### Acquisition Setup

Several sensors were used to record hand kinematics, dynamics and correspondent muscular activity during the experiments. Hand kinematics were measured using a motion capture data glove with 22 sensors (CyberGlove II, CyberGlove Systems LLC). A 2-axis Kübler IS40 inclinometer (Fritz Kübler GmbH) was fixed onto the wrist of the subjects to measure the wrist orientation. Hand dynamics were measured using a Finger-Force Linear Sensor (FFLS; Kõiva et al., 2012).

Two types of double differential sEMG electrodes were used to record muscule activity. Dataset one was recorded using 10 OttoBock MyoBock 13E200-50 (Otto Bock HealthCare GmbH), providing an amplified, bandpass-filtered and Root Mean Square (RMS) rectified version of the raw sEMG signal at 100 Hz. The amplification of the electrodes was set to 5. These electrodes were fixed on the forearm using an elastic armband. Dataset 2 and 3 were recorded using 12 electrodes from a Delsys Trigno Wireless System, providing the raw sEMG signal at 2 kHz. These electrodes were fixed on the forearm using their standard adhesive bands and a hypoallergenic elastic latex-free band.

The sEMG electrodes are positioned in order to combine two methods that are common in the field, i.e., a dense sampling approach (Fukuda et al., 2003; Tenore et al., 2009; Li et al., 2010) and a precise anatomical positioning strategy (De Luca, 1997; Castellini et al., 2009). Eight electrodes were positioned around the forearm at the height of the radio humeral joint at a constant distance from each other; two electrodes were placed on the main activity spots of the flexor digitorum superficialis and of the extensor digitorum superficialis (Atzori et al., 2015; identified by palpation). In dataset 2 and 3, two electrodes were also placed on the main activity spots of the biceps brachii and of the triceps brachii (also in this case, identified by palpation). More details about the acquisition setup are reported in the official database descriptor (Atzori et al., 2014b).

#### Acquisition Protocol

Data acquisitions were performed with two types of exercises. In the first one, the subjects imitated several repetitions of hand movements that were shown on the screen of a laptop in the form of movies. In the second one, the subjects repeated nine force patterns by pressing with one or more hand digits on the FFLS. Several colored bars on the screen guided the subjects to increase the force exerted by each finger up to 80% of the maximal voluntary contraction force, and then back to 0%. Intact subjects were asked to imitate the movements with the right hand, while amputees were asked to imagine imitating the movements with the missing hand, as naturally as possible.

The entire acquisition protocol included several repetitions (10 repetitions for dataset 1, 6 repetitions for dataset 2 and 3) of 40 movements and nine force patterns that were selected from the hand taxonomy and robotics literature (Kamakura et al., 1980; Cutkosky, 1989; Edwards et al., 2002; Crawford et al., 2005; Sebelius et al., 2005; Kato et al., 2006; Feix et al., 2009) also in relationship to the activities of daily living (ADL). Movement repetitions lasted 5 s and were followed by 3 s of rest.

### Data Analysis

Data analysis aims at classifying data into an average of more than 50 classes (corresponding to hand movements) with convolutional neural networks and to compare the results with classical machine learning techniques.

#### Pre-Processing

For both classical and deep learning approaches, the following steps were executed. All the data streams were synchronized by super-sampling them to the highest sampling frequency (2 kHz or 100 Hz, depending on the used myoelectric electrodes) using linear interpolation. Since the movements performed by the subjects may not be perfectly synchronized with the stimuli proposed by the acquisition software due to human reaction times and experimental conditions, relabeling was performed offline with a generalized likelihood ratio algorithm (Kuzborskij et al., 2012). Since the Trigno electrodes are not shielded against power line interferences, their electromyography measurements were filtered from 50 Hz (and harmonics) powerline interference using a Hampel filter (Kuzborskij et al., 2012).

The test set consisted of approximately 1/3 of the movement repetitions (repetition 2, 5 and 7 in database 1; repetition 2 and 5 in database 2 and database 3). The training set consisted of the remaining repetitions. This approach is different from the leaveone-out approach used by Park and Lee (2016).

For classification using convolutional neural networks, after several preliminary tests (aimed to better understand the response of convolutional neural networks on sEMG), the Delsys trigno signals were made similar to Otto Bock's by RMS rectification. Afterwards, the signal was subsampled at 200 Hz, in order to reduce computation time. Then, (both for the Delsys and the Otto Bock) the signals were low pass filtered at 1 Hz. Several normalization procedures were also tested during pre-processing in order to augment the performance of convolutional neural network classification, without leading to sensible improvement of the results.

#### Classification Using Convolutional Neural Networks

The convolutional neural network consisted of a modified version of a well known convolutional neural network (LeNet; LeCun et al., 1995), according to the implementation suggested for Cifar-10 in the package MatConvNet (Vedaldi and Lenc, 2015). The choice of a simple net, despite more complex recent ones being available, was performed in order to accelerate the training phase and to allow evaluating the effects of several pre-processing, architectural and optimization parameters according to the characteristics of the problem. While convolutional neural networks have been applied to many fields, including computer vision and speech recognition, their application to sEMG data is relatively novel (Park and Lee, 2016).

The architecture of the convolutional neural network (**Figure 1**) was structured as follows: the input data correspond to time windows of 150 ms, spanning all the electrode measurements available (10 for the Otto Bock, 12 for the Delsys). This choice corresponds well to what is usually done in the field, i.e., analyzing time windows aimed to allow control in real time (Englehart et al., 1999; Atzori et al., 2014b).

The first block of the net is composed of the following parts. First, it includes a convolutional layer composed of 32 filters. After several tests, including different shapes and sizes, the filters were defined as a row of the length of number of electrodes. Second, it includes a rectified linear unit as a nonlinear activation function.

The second block of the net is composed of the following three parts. The first one is a convolutional layer with 32 filters of size 3 × 3. The second one is a non-linear activation function (rectified linear unit). The third one is a subsampling layer that performs an average pooling with filters of size 3 × 3.

The third block of the net is composed of the following three parts. The first one is a convolutional layer with 64 filters of size 5 × 5. The second one is a non linear activation function

(rectified linear unit). The third one is a subsampling layer that performs an average pooling with filters of size 3 × 3.

The fourth block of the net is composed of the following two parts. The first is a convolutional layer with 64 filters of size 5 × 1 for the Otto Bock electrodes and size 9 × 1 for the Delsys electrodes. The second is a rectified linear unit.

The fifth block of the net is composed of the following two parts. The first one is a convolutional layer with filters of size 1 × 1. The second is a softmaxloss.

Several weight initializations were tested. Finally, the weights of the convolutional layers are initialized with random values in ranges determined in percentage according to the data range, in order to get reasonable training time and stability.

Hyper-parameters were identified via random search and manual hyper-parameter tuning (Bengio et al., 2015) on a validation set composed of two subjects randomly selected from dataset 1 and dataset 2. After several tests, the convolutional neural networks were trained using stochastic gradient descent with momentum 0.9, the learning rate was fixed at 0.001, the weight decay at 0.0005, the batch size was fixed at 256 and the number of epochs 30.

In order to increase accuracy, data augmentation was performed before training. In particular, data were doubled and white Gaussian noise was added to the new set with a signal to noise ratio equal to 25 of the measured power of the signal. Several data augmentation tests were made on the validation set, mainly changing the noise creation procedure. The selected method was chosen based on a balance between improvements and low computational time.

#### Reference Classical Classification

The procedure was based on the one described by Englehart et al. (Englehart and Hudgins, 2003; Gijsberts et al., 2014). It consisted of windowing at 200 ms, feature extraction and classification. Five signal features and three classification methods were considered, according to previous application to the Ninapro sEMG database and to sEMG in general (Englehart and Hudgins, 2003; Kuzborskij et al., 2012; Atzori et al., 2014b; Gijsberts et al., 2014). The selected signal features include: marginal Discrete Wavelet Transform (mDWT), Histogram (HIST), Waveform Length (WL), RMS and the normalized combination of all of them. The histogram (HIST) was divided into 20 bins along a 3σ threshold (Zardoshti-Kermani et al., 1995). The mDWT, was created with a db7 wavelet with three levels (Lucas et al., 2008). The used classifiers are well known, having previously been applied on sEMG in general and thoroughly described on the Ninapro data. They include: random Forests (Breiman, 2001), SVM (Cristianini and Shawe-Taylor, 2000) and k-Nearest Neighbors (Duda et al., 2001). The classification is performed on all the movements included in the database, including rest periods and the data are balanced according to the number of repetitions of movements. The reference classification procedure is described in detail in Atzori et al. (2014b).

### RESULTS

Data analysis aimed at classifying an average of more than 50 hand movement meaning with an average chance level lower than 2%. As described in detail in the ''Discussion'' Section, the results can be compared only with sEMG classification problems targeting a similar number of classes (e.g., Atzori et al., 2014b, 2015). As previously shown (Atzori et al., 2016), results higher than 90% can be easily obtained with similar approaches by reducing the number of classes, even on amputees.

As represented in **Figure 2**, the classification accuracy obtained with convolutional neural networks using the simple architecture proposed is comparable with the average results obtained from classical classification techniques, but lower than the best results obtained with classical classification techniques.

The average classification accuracy obtained using the convolutional neural network on dataset 1 is 66.59 ± 6.40%. The average classification accuracy obtained using all the classical methods on this dataset is 62.06 ± 6.07%. The best classical classification method (Random Forests with all features) obtained an average classification accuracy of 75.32 ± 5.69%.

The average classification accuracy obtained using the convolutional neural network on dataset 2 is 60.27 ± 7.7%. The average classification accuracy obtained using all the classical methods on this dataset is 60.28 ± 6.51%. The best classical classification method (Random Forests with all features) obtained an average classification accuracy of 75.27% ± 7.89%.

For amputees (dataset 3), the average classification accuracy obtained using the convolutional neural network is 38.09 ± 14.29%. The average classification accuracy obtained using all the classical methods on this dataset is 38.82 ± 11.99%. The best classical classification method (SVM with all features) obtained an average classification accuracy of 46.27% ± 7.89%.

With convolutional neural networks (as well as with classical methods) the ratio between the accuracy and the chance level is in general higher than in previous results described in the literature for hand movement recognition in sEMG, e.g., 8.5 [10 movements, accuracy 84.4%, (Li et al., 2010)], 10.56 [12 movements, accuracy 87.8%, (Tenore et al., 2009)].

The average time required to train each convolutional neural network was 1 h and 42 min. The average time required to test the network was 21.5 s using an Nvidia Titan-x GPU. This leads to a time for the classification of each time window of less than 10−<sup>3</sup> s.

Several network architectures, pre-processing parameters and hyperparameters were tested on a validation set, composed of three subjects randomly selected from dataset 1 and dataset 2. Depending on the case, the validation was made on all the movements available, or on a subset of eight movements. A summary of the results is reported in **Table 1**. The table reports the minimum Top-1 errors obtained for each parameter with the corresponding Top-5 error and epoch. Two different methods were tested: ''time window normalization'' (i.e., subtracting to each time window the mean and dividing it by the standard deviation) and ''normalization based on training data'' (i.e., subtracting to all the time windows the training data mean and dividing them by the training data standard deviation). The best results were obtained without any normalization procedure. Normalization procedures can affect the classification error up to 37%. Changing the learning rate can strongly change the minimum error for a fixed amount of epochs, while changes to the weight decay do not seem to affect the error substantially. Finally, data augmentation can reduce the classification error up to 4%, while also strongly reducing the number of epochs requested to reach it. A strong reduction of the error rate (48%) was obtained between the tests on normalization and the tests on the hyperparameters. This result was due to changes in the architecture of the net, in particular considering the first layer.

In conclusion, the classification accuracy obtained with the proposed convolutional neural network is strongly influenced by several factors (including network architectures, pre-processing parameters and optimization parameters), it provides accuracy that is more precise than the average traditional methods in

#### TABLE 1 | Tested pre-processing parameters and hyper-parameters.


The table reports the minimum Top-1 errors obtained for each parameter with the corresponding Top-5 error and epoch.

extremely little time, but it does not replicate the best classical classification methods for similar tasks.

### DISCUSSION

During the last 5 years, deep learning and convolutional neural networks revolutionized several fields of machine learning, including speech recognition and computer vision. Thus, it seems reasonable to think that they may improve the analysis of sEMG and contribute to bridge the gap between prosthetics market (that requires fast and robust control methods) and recent scientific research results in rehabilitation robotics (that show that dexterous and proportional control is possible).

In this article, we introduce a baseline for the application of convolutional neural networks to the classification of hand movements by sEMG and we compare the results with a set of classical machine learning methods on a large set of movements and subjects (including also amputees).

The electromyography data of 67 intact subjects and 11 hand amputees performing an average of more than 50 hand movements were analyzed. The data are publicly available on the Ninapro database (Atzori et al., 2014b) and they are divided into three datasets including 27, 40 and 11 subjects respectively.

The results show that convolutional neural networks with a very simple architecture are comparable to the average classical machine learning classification methods and they show that several factors (including pre-processing, the architecture of the net and the optimization parameters) are fundamental for the analysis of sEMG data. Convolutional neural network results obtained with the very simple architecture described in this article are not worse than the average of classical methods, thus we believe that they are a good avenue to explore.

The classification accuracy obtained with convolutional neural networks using the proposed architecture is 66.59 ± 6.4% on dataset 1, 60.27 ± 7.7% on dataset 2 and 38.09 ± 14.29% on amputees (dataset 3). The average results are comparable to the average results obtained with the reference classical classification, but lower than the results obtained with the best classical classification techniques. The results described in this article represent one of the first attempts to train a simple convolutional neural network on sEMG data. The literature for computer vision and object recognition showed that larger networks can achieve higher accuracy on complex tasks (Bengio et al., 2015). Thus, it may be interesting to evaluate if larger networks can improve sEMG classification too.

Regarding the overall accuracy (obtained both with convolutional neural networks and classical methods), it is fundamental to note that the results should be compared only with analyses considering a similar number of classes, i.e., approximately 50. The chance level varies with the number of classes. Therefore, considering a dataset (with a specific number of samples), a feature set and a classifier, classification accuracy is expected to decrease when the number of classes increases (Deng et al., 2010a). Thus, it is fundamental to compare accuracy only when the number of classes is comparable. It is common to see in the literature movement classification accuracy of up to 90%–95% (Castellini and van der Smagt, 2009; Tenore et al., 2009; Li et al., 2010; Peerdeman et al., 2011). However, most of these studies consider between 4 and 12 movements, with chance level between 25% and 8.33%, while the chance level of this study is inferior to 2%. Thus, a comparison of the accuracy would not be reasonable and justified by statistics. As previously shown, results over 90% of accuracy can be obtained reducing the number of classified movements to approximately 10 for amputees, even starting from lower classification accuracies (Atzori et al., 2014a, 2016). Moreover, classification accuracy can change strongly depending on several other parameters [including e.g., class balance and for amputees, several clinical parameters including forearm percentage, phantom limb sensation and years from the amputation (Atzori et al., 2016)]. Therefore, comparisons in this field must not be made lightly.

Pre-processing, net architecture and the optimization parameters seem to be fundamental for the analysis of sEMG data with convolutional neural networks, since they can strongly change the final classification accuracy in the validation set, and time to converge. The factors that influenced the most the results were the shape of the first layer of the network, the initial weights of the layers, data augmentation procedures and the learning rate.

The net architecture that was chosen is extremely simple. This choice was made on purpose, in order to make it easier to evaluate the effect of changes in the pre-processing, in the architecture of the net and in the optimization parameters. However, more complex net architectures do exist and can be trained on sEMG data, thus probably leading to higher accuracies. This fact is extremely promising for the future of sEMG data analysis and rehabilitation robotics, and may lead to increased dexterous control of robustness, thus contributing to

### REFERENCES


bridge the gap between the prosthetics market and scientific research.

In conclusion, the baseline results that have been presented in this article show that convolutional neural networks with very simple architecture can produce accurate results comparable to the average classical classification methods, and they suggest that further studies may lead to improve the overall field of sEMG controlled dexterous hand prosthetics.

### AUTHOR CONTRIBUTIONS

MA analyzed the data and wrote the manuscript. MC analyzed the state of the art and wrote the manuscript. HM supervised the analysis and wrote the manuscript.

### FUNDING

This work is partially supported by the Swiss National Science Foundation Sinergia project # 160837 Megane Pro.

### ACKNOWLEDGMENTS

The authors would like to thank all subjects for their participation in the data acquisitions and Nvidia Corporation that provided the Titan X gpu used for the deep learning data analysis.


myoelectric control? IEEE Trans. Neural Syst. Rehabil. Eng. 22, 549–558. doi: 10.1109/TNSRE.2013.2287383


IEEE Trans. Biomed. Eng. 60, 1250–1258. doi: 10.1109/TBME.2012. 2232293

Zardoshti-Kermani, M., Wheeler, B. C., Badie, K., and Hashemi, R. M. (1995). EMG feature evaluation for movement control of upper extremity prostheses. IEEE Trans. Rehabil. Eng. 3, 324–333. doi: 10.1109/86.481972

**Conflict of Interest Statement**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Atzori, Cognolato and Müller. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution and reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Assessment of a Wearable Forceand Electromyography Device and Comparison of the Related Signals for Myocontrol

Mathilde Connan, Eduardo Ruiz Ramírez, Bernhard Vodermayer\* and Claudio Castellini

Cognitive Robotics, Institute of Robotics and Mechatronics, German Aerospace Center (DLR), Wessling, Germany

In the frame of assistive robotics, multi-finger prosthetic hand/wrists have recently appeared, offering an increasing level of dexterity; however, in practice their control is limited to a few hand grips and still unreliable, with the effect that pattern recognition has not yet appeared in the clinical environment. According to the scientific community, one of the keys to improve the situation is multi-modal sensing, i.e., using diverse sensor modalities to interpret the subject's intent and improve the reliability and safety of the control system in daily life activities. In this work, we first describe and test a novel wireless, wearable force- and electromyography device; through an experiment conducted on ten intact subjects, we then compare the obtained signals both qualitatively and quantitatively, highlighting their advantages and disadvantages. Our results indicate that force-myography yields signals which are more stable across time during whenever a pattern is held, than those obtained by electromyography. We speculate that fusion of the two modalities might be advantageous to improve the reliability of myocontrol in the near future.

#### Edited by:

Ricardo Chavarriaga, École Polytechnique Fédérale de Lausanne (EPFL), Switzerland

#### Reviewed by:

Sebastian Amsuess, Otto Bock, Germany Francesco Clemente, Scuola Superiore Sant'Anna, Italy

#### \*Correspondence:

Bernhard Vodermayer bernhard.vodermayer@dlr.de

Received: 18 July 2016 Accepted: 21 October 2016 Published: 17 November 2016

#### Citation:

Connan M, Ruiz Ramírez E, Vodermayer B and Castellini C (2016) Assessment of a Wearable Force- and Electromyography Device and Comparison of the Related Signals for Myocontrol. Front. Neurorobot. 10:17. doi: 10.3389/fnbot.2016.00017 Keywords: surface electromyography, force myography, multi-modal intent detection, machine learning, human-machine interfaces, rehabilitation robotics

### 1. INTRODUCTION

The human hand is a prodigious natural tool, comprising 27 bones and 33 muscles, resulting in a total of 22 degrees of freedom (DOFs) (Biryukova and Yourovskaya, 1994); its sensorial equipment enables us to drive, browse through the pages of a book, hold and manipulate delicate objects as well as heavy tools. Due to this complexity, artificially reproducing its functions is still a challenge for the roboticians. Nevertheless, a mechatronic tool getting close to the human hand is highly desirable in the context, e.g., of dexterous hand prosthetics. Despite the fact that multi-fingered hand prostheses have appeared on the market during the last decade, their level of abandonment remains relatively high (Biddiss and Chau, 2007; Peerdeman et al., 2011). Touch Bionics' i-LIMB, Otto Bock's Michelangelo, Vincent Systems' Vincent Evolution 2 and RSL Steeper's Bebionic3 are among these examples: with as many as six DOFs, they still are a limited replacement of the hand of an amputee, not mirroring the capabilities of a real human hand. A prosthetic wrist adds at least one DOF to the device and further improves its potential dexterity, but empowering amputees to control such artifacts (myocontrol) is still an open issue.

The academic state-of-the-art of myocontrol relates to the possibility of proportionally and independently controlling each DOF of the prosthesis according to the patient's intent (Sebelius et al., 2005; Cipriani et al., 2011a); however, low stability and accuracy prevent a successful commercialization of such an approach. Myocontrol is still limited to a few DOFs (Arjunan and Kumar, 2010; Yang et al., 2014), and surface electromyography (sEMG) signals are deemed to be no longer enough (Jiang et al., 2012a). Researchers have tried to address this issue by increasing the number of sensors (Tenore et al., 2007), although it is known that four to six channels are acceptable for pattern detection (Young et al., 2012), and/or to find their optimal placement given the characteristics of the stump (Castellini and van der Smagt, 2009; Fang et al., 2015); several pattern recognition algorithms have been studied, such as artificial neural networks (Baspinar et al., 2013), linear discriminant analysis (Khushaba et al., 2009) and non-linear incremental learning (Gijsberts et al., 2014). However, one of the major drawbacks of sEMG signals is their variable nature: sweat, electrode shifts, motion artifacts, ambient noise, cross-talk among deep adjacent muscles and muscular fatigue can crucially affect them (Oskoei and Hu, 2007; Cram and Kasman, 2010; Merletti et al., 2011a; Castellini et al., 2014). In general, any change in the muscle configuration during and after the training of the machine learning algorithm (e.g., the position of the limb and the body and the weights to be lifted during grasping and carrying) must be taken into account (Scheme et al., 2010; Cipriani et al., 2011b). As a result, simultaneous and proportional (s/p) control of each DOF is slow and laborious. Therefore, the application of other types of sensors and sensor combinations is an active field of research (Fougner et al., 2011; Jiang et al., 2012a).

Among non-invasive approaches other than sEMG, electroencephalography (EEG), mechanomyography (MMG), ultrasound imaging also known as sonomyography (SMG), force myography (FMG), functional magnetic resonance imaging and more are considerable options (Lobo-Prat et al., 2014; Ravindra and Castellini, 2014; Fang et al., 2015). Such Human Machine Interfaces (HMIs) have been implemented in distinct studies. However, the research community is recently pushing the development of multi-modal sensing techniques in the field of upper-limb rehabilitation (Fang et al., 2015). For instance, experiments have shown that accelerometer sensor signals can improve the classification accuracy of EMG electrodes (Fougner et al., 2011), as well as a multimodal technique with EMG and Near Infrared Spectroscopy (NIS) (Herrmann and Buchenrieder, 2010), or a combination of EEG and electroneurography (ENG) (Rossini and Rossini, 2010). In brief, a full comparison of the advantages and disadvantages of each type of signal, as well as the possibilities offered by their fusion, is still lacking.

In this study we focus on the joint usage of sEMG and FMG sensors. Surface EMG Merletti et al. (2011b) detects Motor Unit Activation Potentials, that is, electrical fields generated by motor units during muscle contraction, whereas FMG (Phillips and Craelius, 2005; Wininger et al., 2008) detects the pressure exerted by the muscles toward the surface of the skin by volumetric changes induced during muscle activity. Due to the very different nature of the signals gathered by these two techniques, it seems reasonable that they could be proficiently fused in order to better detect a subject's intent. A simple and low-cost option to record FMG signals is represented by force-sensing resistors (FSRs), whose resistance changes according to the pressure applied to them. This is also our option of choice. These sensors are cheap and very compact. Castellini and Ravindra (2014) already proved their effectiveness, and established that finger forces can be predicted with the same accuracy than sEMG sensors (Ravindra and Castellini, 2014). Cho et al. (2016) tested their force sensing system on four amputees and demonstrated that it is possible to classify six primary grips using only FMG with an accuracy of above 70% in the residuum. We describe a novel, modular approach for joint FMG/sEMG intent detection: thanks to a newly developed, fully mobile and wireless acquisition system, we simultaneously gathered FMG and sEMG signals during an experiment in which ten intact subjects performed a repetitive sequence of wrist and hand movements. The collected data was used to assess the desirable characteristics of each modality, while self-assessment questionnaires were used to check that the device was acceptable for the subjects. In the end, we claim that a fusion of the two approaches is potentially better than using them independently for dexterous myocontrol.

### 2. MATERIALS AND METHODS

In order to assess the combined data acquisition of sEMG and FMG for myocontrol, we have built a prototype fully wearable, wireless multi-modal myocontrol system. To study its performance, its usability, and the characteristics of the obtained signals, we have involved ten intact human subjects in an experiment with the device as its core. **Figure 1** (left panel) shows the system as worn by a subject: the device is composed of three modules: a set of mixed sEMG/FMG sensors (in this case, arranged on two Velcro bracelets), a Bluetooth analog-todigital conversion board gathering and transmitting the signals, and a smartphone receiving the data via Bluetooth and able to perform myocontrol via a machine learning algorithm. The board was based upon the work of Brunelli et al. (2015), whereas the learning algorithm is Incremental Ridge Regression with Random Fourier Features (see below for more details), already been evaluated (in a non-wearable control system) by Gijsberts et al. (2014) and Strazzulla et al. (2016). Although not extensively used in this specific experiment, the machine learning algorithm can produce control signals in real time and transmit them to the sensor board, which serves as a relay routing them to a hand prosthetic device connected to it. A block diagram of the whole system is presented in **Figure 1** (right panel).

### 2.1. Experimental Setup

### 2.1.1. Sensors

Ten Ottobock MyoBock 13E200 = 50 sensors were used to gather the sEMG signals. They provide on-board amplification, rectification and filtering. Sensors of this kind are a standard in clinical applications, especially in prosthetic sockets.

FMG signals were registered by ten FSR 400 Short forcesensing resistors by Interlink Electronics. Made of a robust polymer thick film, each FSR has a 5.6 mm-diameter sensitive

area: when a force is applied to its surface, the electrical resistance of the FSR decreases correspondingly. These sensors are cheap (5€ apiece), but despite the specified remarkably large sensitivity range (0.2N–40N), they have non-negligible hysteresis at high forces, no guarantee of repeatability and a non-linear transfer function. Nonetheless, Castellini and Ravindra (2014) have shown that for small forces (0N–15N) their behavior is largely comparable and their transfer function is almost linear. In our setup, a small printed circuit board with a voltage amplifier (see **Figure 2**) provides the amplification of the FMG signals. The output of the sensor circuit is Vout = R2VCC R1+R<sup>2</sup> − R1R4VCC R1+R<sup>2</sup> × 1 RFSR , yielding a lowest admissible resistance of RFSR = R1R4 R2 = 6k, which corresponds to a theoretical maximum force observed on the FSR's surface of 3.33 N (InterlinkElectronics, 2014).

In order to provide maximum flexibility in arranging the sensors on the subject's body, specifically on the forearm or stump, uniform 3D-printed housings have been designed for both kinds of sensors. The housings are made of flexible thermoplastic polyurethane and provide adherence to the subject's skin as they are tightened to the arm by a Velcro strap. Each housing provides braces to allow sliding on the strap, so that its position can be individually adjusted and maintained, regardless of the type of sensor (see **Figure 3**). The FSR sensor housing not only serves as a retainer for the sensor and the amplifier, but furthermore comprises a structured geometric body that is divided into two parts: approximately one half is shaped like a cone and pointing toward the FSR sensor's sensitive area, the other half, shaped like a hemisphere, is pointing toward the skin. This geometric shape has been specifically designed to concentrate the force exerted by the muscles on the FSR sensor's sensitive area. The bearing of this structure is realized by a surrounding, thin and flexible membrane with a thickness of 0.5 mm and a total area of 47.5 mm<sup>2</sup> (small diameter 6.3 mm, large diameter 10 mm), linking it to the housing. Even if the elastic and damping properties of the membrane have not been subject to further investigation so far, it is assumed that the membrane may increase the signal stability, i.e., it could hypothetically add a mechanical filtering to the bio-signal. This solution offers the capability to create any combination of FMG and sEMG electrodes. In order to gather signals from the complete circumference of the forearm, the sensors are placed evenly spaced around the forearm/stump. This arrangement, also called low-density surface electrode layout or uniform electrode positioning (Fang et al., 2015), has already been proven effective for robotic hand prosthesis control in a number of previous publications (Castellini and van der Smagt, 2009, 2013).

The typical voltage output of the FSR/amplifier/housing complex has further been characterized. The measurements have been performed with a Zwick Roell ZMART.PRO compression test device providing fixture of the sensor setup as well as controlled exertion of force with an accuracy of 100 mN. The corresponding output voltage of the device has been measured with a FLUKE 289 Multimeter with an accuracy of 1 mV. The relationship between voltage output and applied force, in a range from 0 to 5.2 N, is linear with a residual average error of about 7% (see **Table 1**). It is worth noting that, during the experiment (see below), no FSR ever reached the saturation point, meaning that the FMG signals have all been correctly captured. This matches to a large extent the results obtained in Castellini and Ravindra (2014) and Ravindra and Castellini (2014) for a similar device.

#### 2.1.2. Analog-Digital Conversion and Data Transfer

A Bluetooth ADC board (**Figure 4**), consisting of a Texas Instrument MSP430F5529 microcontroller and an on-board Bluetooth chipset, provides analog-to-digital conversion (ADC) of the signals of both the sEMG and FSR sensors, and their wireless transmission. As the microcontroller natively supports only 15 AD-channels, AD-conversion of up to 32 sensors is realized via analog multiplexing, providing a maximum sampling rate of 192.5 Hz for each channel. Since the sEMG sensors already provide rectified and filtered signals with an evaluable

#### TABLE 1 | Characterization table for one FSR, including the FSR sensor, the amplifier board and the housing.


The standard deviation with respect to a linear fit is 6.9%. The provided values have been inverted by the zero-value offset voltage (4.2874V).

bandwidth limited to 10 Hz, the provided sampling frequency is an overshoot (see e.g., Castellini and van der Smagt, 2009). The same argument obviously holds for the FMG signals (Ravindra and Castellini, 2014).

The board employs two UARTs, communicating in turn with the smartphone (via a serial-over-Bluetooth connection) and the prosthesis (in our case, via a simplified RS232 protocol). Hence, the board can also relay control commands to a hand prosthesis. The final cost of the system is estimated to be below 150€.

### 2.1.3. Myocontrol Host

The wearable myocontrol system is completed by a standard commercial smartphone (Huawei Honor 6 with a commercial value of about 300e), on which the data processing is performed. This device is equipped with a quad-core Cortex-A15 processor running at 1.7 GHz, 2 GB of RAM and a 3100 mAh Li-Po battery, claimed to keep the smartphone running for 2 full days at moderate usage. Its operating system is Android 4.4 KitKat. The smartphone weighs 132 g and easily fits in a pocket (7 × 0.8 × 14 cm) with its 5-inch display. A C# application similar to the one used in Strazzulla et al. (2016) was implemented, optimized and ported to the smartphone. The application enforces the following functionalities: (a) receiving and storing the data from the ADC board's serial-over-BT port; (b) displaying a visual stimulus, both on the smartphone screen and using the prosthesis; (c) building a prediction model for the control commands; and (d) sending them off to the prosthetic hand at a 10 Hz frequency, through the ADC board's serial-over-BT port. The machine learning method of choice was—coherent with our own previous work—Incremental Ridge Regression with Random Fourier Features (Gijsberts et al., 2014). As speculated in previous work and now proven, this method provides real-time capable non-linear multivariate regression while saving a lot of computational resources to the point that the maximum usage of the smartphone's CPU showed to be at 14%. Moreover, with the program running and the cell phone display activated and fully lighted, the battery endurance is at approximately 6 h, whereas an endurance of 11 h can be achieved with the display being switched off. For receiving the sample data and sending control commands to the prosthetic hand, the cell phone's internal Bluetooth peripheral has to be activated all the time.

## 2.2. Experiment Description

### 2.2.1. Subjects

Ten intact subjects, nine of which were right-handed (subject No. 9 being the left-handed one), joined the experiment (3 females and 7 males, 28 ± 7 years old, weighing 72.4 ± 9.91 kg, 177.8 ± 12.14 cm tall). Each subject received a thorough description of the experiment, both in oral and written form. Informed written consent was obtained from all participants. Experiments with sEMG and FMG were approved by the Ethical Committee of the DLR.

### 2.2.2. Experimental Protocol

The experiment consisted of performing ten times the following sequence of wrist and hand movements: (1) wrist flexion, (2) wrist extension, (3) wrist pronation, (4) wrist supination and (5) power grasp. To enforce the opening of the hand the relaxed stance was used, in order to mimic a more natural form of myocontrol. Each movement was visually stimulated on the screen of the smartphone (the name of the required motion would appear on the screen), while the experimenter was visually checking that the movement was actually being enforced, to ensure a correct execution. Each stimulation was administered as follows: the visual stimulus would appear for 2 s to allow the subject reach the full movement, then for 6 s data were captured representing the maximal activation for that particular movement ("activation phase": only this phase was considered in the offline analysis), then the stimulus would disappear for 2 s to allow the subject return to the resting position. The sequence was administered in the same order to all subjects. The choice of this set of movements was motivated by the well-known importance of controlling at least the wrist pronation/supination (see e.g., Jiang et al., 2012b) together with grasping; for instance, pronation and supination of the wrist are operated by deep muscles (Biryukova and Yourovskaya, 1994), meaning that they are usually hard to detect using sEMG. It is worth mentioning that, to the best of our knowledge, there is so far no commercially available 2-active-DOFs prosthetic wrist, but a few prototypes are being studied [see e.g., the device embedded in the DEKA arm, https://www.youtube.com/ watch?v=KCUwoxuAdYQ, and the prototype by Ottobock which appears for instance in Amsuess et al. (2016)].

For this specific experiment, the sensors were separated in two different bracelets, the first one with ten sEMG sensors on the left forearm and the second one with ten FSR sensors on the right forearm. The bracelets were located approximately 10 cm below the subject's elbows. This further choice, rather than that of placing twenty sensors on one single forearm, was motivated by the relatively small space available on the forearm of some of the subjects, which would have potentially limited the adhesion of each sensor to the subjects' skin. (A similar problem was reported of, e.g., in Castellini and Ravindra, 2014). Of course, this diminishes the comparability of the results from the two sets of sensors, but myocontrol literature has already presented cases in which, for instance, training of a machine-learningbased method has been performed bilaterally, i.e., gathering data and training from the intact forearm and predicting using data from the impaired limb (Castellini et al., 2009; Nielsen et al., 2011). In both these works, no difference was reported in performance whether the forearm to be used was the dominant or non-dominant one.

During the whole experiment, the subjects sat in a relaxed position with their forearms over their thighs and the hands in a lateral position (with the palms looking toward each other); they were advised to perform each movement bilaterally (**Figure 5**). The recorded data kept trace of the FSR and sEMG signals as well as of a numerical identifier univocally representing the stimulated movement. This index was used as the ground truth during the supposed maximal activation of the muscles—an instance of onoff goal-directed stimuli as already used in, e.g., Sierra González and Castellini (2013).

### 2.2.3. Data Processing

The signals recorded during the experiment were stored to the smartphone's internal memory, then analyzed off-line. Lowpass filtering was applied to both signals (3rd order low-pass digital Butterworth filter with a cutoff frequency of 1 Hz) in order to remove high-frequency disturbances. This is a standard procedure in the field (see e.g., Atzori et al., 2014). For the data

processing and the subsequent statistical analysis, the following approaches have been chosen:


"leave-one-repetition-out" cross-validation was applied by training each machine on nine of the ten repetitions and testing on the remaining one. The input space was chosen to be either the FMG values, the sEMG values, or their combination, meaning that the FMG and sEMG samples were simply stacked in a 20-dimensional vector and used with the same learning method. The prediction accuracy was measured using the normalized Root Mean-Squared-Error (nRMSE) between the predicted values and the stimulus values. A one-way ANOVA test was performed to investigate whether there was a statistically significant difference in between the three inputs.

### 2.2.4. Satisfaction Surveys

Additionally, at the end of the experiment, three surveys were administered to each subject, in order to complete the envisaged system assessment with respect to its usability: the System Usability Scale (SUS) (Brooke, 1996), the NASA Task Load Index (NASA, 1986) and a reworking of the Microsoft Desirability Toolkit by Travis (2008). The SUS consists of ten questions (**Table 2**) with answers represented on a 5 point Likert scale (1 strongly disagree to 5 - strongly agree). The scoring in this survey is such that the answers to the strongly agree positive questions and to the strongly disagree negative questions generate a higher impact over the final score. The NASA Task Load Index provides an overall workload score on six subscales: Mental, Physical and Temporal Demands; Own Performance, Effort and Frustration. For each subscale, the answer could be in a range of 21 points, reaching from very low to very high (**Table 2**).

Lastly, Travis's survey consists in a series of "reaction cards" with adjectives that could be applied to the system to be tested; the user is asked to select the five cards that most closely match their personal reactions to the system. For the experiment, we used a list of 75 adjectives instead of the cards (most of them based on Travis's questionnaire), then the subject was asked to choose all adjectives he or she felt more related with the device. After that, in a more precise selection, the user had to choose only the 5 most important words and try to give a simple reason about his or her decision.

### 3. EXPERIMENTAL RESULTS

### 3.1. Stability Over Time

**Figure 6** shows typical FMG and sEMG signals obtained while a subject was performing two repetitions of the five instructed movements (plus the resting state). Due to the carefully chosen amplification/filtering stages of the sensors themselves, the amplitude of the signals obtained from both FMG and sEMG sensors are comparable, and each single produced movement appears as a distinct pattern, well separated in time from the next one as well as from the 2-s intervals allowed for resting and for preparing the next movement. Such behavior is clear, e.g., during the wrist-flexion movement enforced from 60 to 70 s in the Figure (second red bar just above the x-axis).

Visual inspection seems to indicate that FMG signals are more stable over time while the subjects are holding the position than sEMG signals: this is apparent by looking at the "plateaus" created

#### TABLE 2 | The statements found in the SUS and NASA TLX surveys.

#### SUS SURVEY

I felt comfortable with the device.


I think that I would need the support of a technical person to be able to use this device.


I needed to learn a lot of things before I could get going with this device.

#### NASA TLX SURVEY

How mentally demanding was the task?

How physically demanding was the task?

How hurried or rushed was the pace of the task?

How successful were you in accomplishing what you were asked to do?

How hard did you have to work to accomplish your level of performance?

How insecure, discouraged, irritated, stressed, and annoyed were you?

by the FSRs while each movement was enforced; as opposed to that, sEMG signals exhibit the typical oscillating down-ramp pattern due to muscular motor-unit recruitment (Merletti et al., 2011a,b). To verify that this is the case in general, we evaluated the standard deviation of the FMG and sEMG signals obtained by each subject while performing the first three repetitions of the wrist flexion movement considering the signals during the activation phases only. (Only the three sensors for each set that exhibited the highest amplitude were taken into account.) Considering **Table 3**, sEMG signals actually exhibit a significantly higher standard deviation when compared to FMG signals (mean values 0.0087 and 0.0025 in turn, Student's paired-sample t-test p < 0.01).

### 3.2. Separability of Clusters

**Figure 7** shows typical FMG and sEMG data reduced to three dimensions via Principal Component Analysis (PCA) and colored according to each movement for the cluster separability analysis carried out for both input spaces. In the figure, sEMG clusters appear more stretched than FMG clusters, a behavior very likely due to the above-mentioned oscillations while a movement is being held.

**Figure 8** shows the matrices for sEMG and FMG, while **Table 4** lists the means of Fisher's Indexes for each subject (the diagonal-zero values are not considered to evaluate the mean values). The Fisher's Index of FMG is higher (therefore better) than that of sEMG (mean values 368.82 and 94.1) with high statistical significance (Student's paired-sample t-test p < 0.001).

### 3.3. sEMG/FMG Regression for Mycontrol

**Table 5** shows the prediction accuracy obtained by each subject on one movement repetition (the nRMSE value showed is a mean of all the nRMSE obtained by the cross-validation). **Figure 9** shows the nRMSE values for all subjects. While the nRMSE values range from 0.13 to 0.21, in line with previous literature (Ravindra

TABLE 3 | Standard deviation of FMG and sEMG sensor signals obtained by each subject during the first three repetitions of the wrist flexion movement.


Considering the three sensors that exhibited the highest signal amplitude.


and Castellini, 2014), no statistically significant difference in accuracy is apparent (one-way ANOVA p > 0.05).

### 3.4. User Satisfaction

For the SUS, the total result of each subject and the mean score are presented in **Table 6**. Notice that the higher the score, the more usable the user judged the device. In this survey, the statements that had the worst scores were "I think that I would need the support of a technical person to be able to use this device" and "I felt very confident using the device."

For the NASA TLX, the total result for each subscale of each subject and overall workload are shown in **Table 7**. Here the highest the score, the more workload the user had, when using the device. A plot with average percentages of the workload by subscale is visible in **Figure 10** .

The first two surveys applied comprised suitable results with a usability score of almost 85% and an overall workload of 25%.

Finally, for the desirability survey proposed by Travis, two kinds of results were obtained, the first one using all the words chosen by the user in the first selection, and the second one considering only the 5 final selections. In order to have a different visualization, two word clouds have been created (**Figure 11**), where the bigger and darker the font is, the more often the word was selected.

Considering only the most common adjectives in the final selection, the device can be considered Simple , Intuitive , Easy to use , Familiar , Reliable and Stable. An important thing to mention about this last survey is that even though the instructions and the questions were oriented toward the device, some answers referred to the experiment performance. For instance, the adjective familiar was chosen in some cases because the subject had already performed other experiments with EMG sensors before taking part in this specific experiment.

To highlight the results of the user satisfaction surveys, a radial chart (**Figure 12**) was built, which separates the results in different categories (Low Workload Demand, Stability, Task Accomplishment, Interface, Easy to Use, Comfort and Setup) and represents the main features appreciated by the user. The figure for Low Workload Demand has been inverted on the scale to achieve better comparability with the other figures (i.e., Workload Demand is positively rated if being low). All of these categories can furthermore be separated in two kinds of classes: usability related features (Interface, Easy to Use, Comfort and Setup) and performance related features (Workload Demand, Stability and Task Accomplishment).

### 4. DISCUSSION

This work had two main aims: to assess whether a wearable combined sEMG/FMG device would be accepted by human subjects using it, and to determine whether significant qualitative and quantitative differences could be observed between sEMG and FMG signals obtained during a simple experiment aimed at myocontrol.

 0.1526 ± 0.013

 0.1367 ± 0.0048

 0.2054 ± 0.0267

 0.1736 ± 0.0139

 0.1658 ± 0.0139

 0.1636 ± 0.0066

 0.1595 ± 0.0161

 0.1385 ± 0.0083

 0.1608 ± 0.0199

Comb.

 0.1649 ± 0.0137

TABLE 4 | Mean Fisher's Index values for each subject and type of sensors.

TABLE 6 | Subject's system usability total scores.


#### TABLE 7 | NASA TLX workload percentages.


(Subject #9 actually gave a uniform scoring.)

### 4.1. Acceptance of the Device

Consider Section 3.4, in particular **Figure 12**. The results clearly show that the device, together with its user interface, posed no problems to the subjects using it, and even had some appeal. Even though the subjects involved in the experiment are not part of the potential user population (i.e., amputated subjects controlling a self-powered prosthesis), their opinions and impressions about the device are helpful for future improvements and corrections. The subjects uniformly reported that the device felt reliable and stable; that the setup was easy to use, simple and comfortable, with a low frustration rate; that the user interface was friendly, intuitive and well structured; and that the data acquisition still required a considerable amount of time (this aspect having the highest workload demand).

The keywords obtained in Travis's survey, the workloads obtained in the NASA TLX survey, and the successful performance reflected in the SUS test, already give hints about the usage risk of a possible future medical device. Of course, this cannot replace a thorough risk analysis and a widespread usability study. Furthermore, it must be remarked that the positive results obtained in the surveys could have been influenced by the subjects' behaviors themselves, as explained by Travis. All in all, it is worthwhile to stress that no online experiment was performed in this study, therefore, from the user survey results, we can provide no conclusive results about either the performance of this device in general, or the usefulness of a combined sEMG/FMG approach in myocontrol.

### 4.2. Comparison of sEMG and FMG

In previous literature (Yungher et al., 2011; Ravindra and Castellini, 2014) it has been shown that FMG shows higher overall stability over time than sEMG, meaning that, e.g., the variance of its signals is lower than that of sEMG, while human subjects are engaged in repetitive, fatiguing tasks. This is probably due to the lower influence that muscle fatigue has on FMG signals, due to its nature. Now, from the qualitative/quantitative comparison of sEMG and FMG carried out in Sections 3.1 and 3.2, two statistically significant differences between the two approaches emerge, namely (a) that FMG signals are more stable over time during single movements too, and (b) that they generate better separated patterns in the input space. Most likely, (b) is a consequence of (a); we speculate that this difference might arise from the very nature of the sEMG signals, which exhibit noise due to the recruitment of motor units while keeping an isometric hand/wrist posture. Of course, FMG would not be affected by this problem. All things considered, it seems reasonable to claim that FMG signals are more stable than sEMG ones.

About pattern separateness: if a classification approach were to be used to enforce myocontrol using such signals, better pattern separability would definitely represent a further advantage of FMG with respect to sEMG; in the case of simultaneous and proportional control, however, it is not clear whether this is an advantage or not. This kind of control requires some way to understand not only what pattern the subject desires, but also how much force/torque is involved; distant and smaller clusters for the maximal activations might contain less information about this specific feature. In our case, the regression method used in Section 3.3 is trained on maximal and minimal activation signals only, whereas it predicts the intermediate activation values by non-linear interpolation. We are in no position at this time, to claim that FMG or sEMG is better in this case (and this is reflected in the non-stastically-significant accuracy results obtained by such method, see **Table 5** again).

On a more qualitative side, we note that FMG, at least enforced using this cheap approach (that is, Force-Sensing Resistors), presents the drawback of being affected by hysteresis. Although the return to the resting state is apparent, this induces the FMG signals even to rise during the resting states, but not to come back exactly to zero or the previously measured resting states after a movement is performed. As opposed to this, sEMG signals remain almost at zero even when the user is not in the same initial position; this seems reasonable, since the resting phase involves no muscle activation. This problem can be countered by employing a smarter technique to gather FMG signals, for instance the capacitive approach, or (high-density) tactile sensing. Also, a fully-fledged, online FMG approach will need to take into account the inevitable artifacts generated by the

arm/forearm movement (i.e., accelerations inducing pressure on the sensors) and those induced by touching the socket, bumping into objects, laying the stump on a table, etc.

What we can conclude, we believe beyond any reasonable doubt, is that sEMG and FMG can be miniaturized and employed in such a framework, and that they carry different kinds of information, leading to different behaviors and signal features. We speculate that a structured sensor-fusion approach (that is, deeper than simply stacking the signals as we have done in this work) might lead to a better exploitation of each modality's characteristics.

One last remark is in order about the cost of each approach. Apparently, FMG is up to two orders of magnitude cheaper than sEMG, but this is mainly due at this stage to (a) it being enforced through Force-Sensing Resistors, which might not live up to the expectations as previously remarked; and (b) the necessity, in the very end, to produce a medically certified FMG approach and device, which might dramatically raise its costs. Again, we believe that an integrated approach is the way ahead to improve myocontrol using this still novel technique alongside sEMG. To this aim, the assessment of the wearable sEMG/FMG device we carried out is promising, as it shows that at least the required electronic machinery can be embedded in an effective, light and acceptable device.

### 4.3. Conclusions and Future Work

In this article, we have described a wearable, integrated sEMG/FMG system, targeting myocontrol and human intent detection. The experiment we conducted endorses the system's high degree of usability, indicating that it has the potential to become an integrated medical product. Still, the autonomous mobility of about 11 h is restricted by the cell phone battery,

### REFERENCES


thus recharging strategies during the patient's daily use or the deployment of a secondary device could be a solution for now. In near future, the development of a highly integrated, low energy system, where the phone serves only for teaching and displaying information, seems to be advisable. Additionally, we explored the application of FMG as a potential complement to sEMG and provided evidence that the two techniques can be integrated, but that a smart sensor fusion approach might be required to obtain the best results. Results from user satisfaction surveys presented in the paper give strong indication for the setup to be on the right track. In the near future, more experiments are planned to check the feasibility of mixed sEMG/FMG for online myocontrol, possibly down to the level of individual finger movements, and immersed in daily-life activities.

### AUTHOR CONTRIBUTIONS

System Design BV, MC, CC; Software Design MC, BV; Test setup and conduction of user Surveys ER, MC, CC; Analysis and Interpretation of data ER, CC; Drafting of manuscript MC, ER, BV; Internal Revision BV, CC; Critical Revision BV, CC, MC, ER; Final Approval MC, ER, BV, CC; Agreement on accountability by MC, ER, BC, CC.

### ACKNOWLEDGMENTS

The authors thank Mr. Miguel Neves of the DLR for designing and building the housings of the sensors. This work was partially supported by the DFG project "TACT\_HAND: Improving control of prosthetic hands using tactile sensors and realistic machine learning" (Sachbeihilfe CA1389/1-1).


transradial amputees. IEEE Trans. Neural Syst. Rehabil. Eng. 19, 260–270. doi: 10.1109/TNSRE.2011.2108667


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Connan, Ruiz Ramírez, Vodermayer and Castellini. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Translating Research on Myoelectric Control into Clinics—Are the Performance Assessment Methods Adequate?

Ivan Vujaklija1,2\*, Aidan D. Roche<sup>3</sup> , Timothy Hasenoehrl <sup>4</sup> , Agnes Sturma3,5 , Sebastian Amsuess <sup>6</sup> , Dario Farina<sup>2</sup> and Oskar C. Aszmann3,7

<sup>1</sup>Clinic for Trauma Surgery, Orthopaedic Surgery and Plastic Surgery, Research Department for Neurorehabilitation Systems, University Medical Centre Göttingen, Goettingen, Germany, <sup>2</sup>Department of Bioengineering, Imperial College London, London, UK, <sup>3</sup>Christian Doppler Laboratory for Restoration of Extremity Function, Medical University of Vienna, Vienna, Austria, <sup>4</sup>Department of Physical Medicine, Rehabilitation and Occupational Medicine, Medical University of Vienna, Vienna, Austria , <sup>5</sup>Master Degree Program "Health Assisting Engineering", University of Applied Sciences FH Campus Wien, Vienna, Austria, <sup>6</sup>Otto Bock Healthcare Products GmbH, Vienna, Austria, <sup>7</sup>Division of Plastic and Reconstructive Surgery, Department of Surgery, Medical University of Vienna, Vienna, Austria

Missing an upper limb dramatically impairs daily-life activities. Efforts in overcoming the issues arising from this disability have been made in both academia and industry, although their clinical outcome is still limited. Translation of prosthetic research into clinics has been challenging because of the difficulties in meeting the necessary requirements of the market. In this perspective article, we suggest that one relevant factor determining the relatively small clinical impact of myocontrol algorithms for upper limb prostheses is the limit of commonly used laboratory performance metrics. The laboratory conditions, in which the majority of the solutions are being evaluated, fail to sufficiently replicate real-life challenges. We qualitatively support this argument with representative data from seven transradial amputees. Their ability to control a myoelectric prosthesis was tested by measuring the accuracy of offline EMG signal classification, as a typical laboratory performance metrics, as well as by clinical scores when performing standard tests of daily living. Despite all subjects reaching relatively high classification accuracy offline, their clinical scores varied greatly and were not strongly predicted by classification accuracy. We therefore support the suggestion to test myocontrol systems using clinical tests on amputees, fully fitted with sockets and prostheses highly resembling the systems they would use in daily living, as evaluation benchmark. Agreement on this level of testing for systems developed in research laboratories would facilitate clinically relevant progresses in this field.

#### Keywords: myoelectric prosthesis, prosthetic assessment, myoelectric control, SHAP, box and blocks

### INTRODUCTION

Recent progresses in active prosthesis control for the upper limb include the introduction of novel control approaches (Scheme and Englehart, 2011; Jiang et al., 2014a; Amsuess et al., 2016), sensor types and sensor fusion algorithms (Weir et al., 2003; Dosen et al., 2010; Cipriani et al., 2014; Ortenzi et al., 2015; Nissler et al., 2016), surgical techniques (Kuiken et al., 2004; Aszmann et al., 2015), as well as advanced hardware (Cipriani et al., 2011; Grebenstein et al., 2011; Catalano et al., 2014).

### Frontiers in Neurorobotics | www.frontiersin.org

Edited by: Claudio Castellini,

Matei Ciocarlie, Columbia University, USA \*Correspondence: Ivan Vujaklija

Citation:

DLR - German Aerospace Center, Institute of Robotics and Mechatronics, Germany Reviewed by: Nicholas P. Fey,

University of Texas at Dallas, USA

ivan.vujaklija@bccn.uni-goettingen.de

Vujaklija I, Roche AD, Hasenoehrl T, Sturma A, Amsuess S, Farina D and Aszmann OC (2017) Translating Research on Myoelectric Control into Clinics—Are the Performance Assessment Methods Adequate? Front. Neurorobot. 11:7. doi: 10.3389/fnbot.2017.00007

Received: 17 July 2016 Accepted: 01 February 2017 Published: 14 February 2017

**50**

Nonetheless, the impact of these advances towards improving the experience of the everyday end user is still limited. The discrepancy between myoelectric solutions which academia develops and promotes, and the systems available on the market is indeed substantial. This issue has been previously discussed (e.g., Hill et al., 2009; Jiang et al., 2012; Farina and Aszmann, 2014) and relates to the conditions in which new methods are tested.

The necessity for testing prosthetic solutions in a greater number of amputees than currently done is a widely recognized problem. Moreover, the tests used often fail to include clinically relevant metrics. Performance metrics prevalent in laboratory research may be poorly associated to the clinical outcome, as noted previously (Simon et al., 2011; Jiang et al., 2014b; Ortiz-Catalan et al., 2015). In this perspective article, we support these arguments to further substantiate the relevance of this problem.

Transferring myoelectrical systems developed in the laboratory to clinical settings is a challenge that requires multidisciplinary efforts. Clinical tests, although not ideal, offer the most realistic prediction of the system performance in the daily use. These tests account for several of the challenges that laboratory-based assessment methodologies tend to neglect. For example, noiseless laboratory-based evaluation platforms fail to account for the end effector loads, poor socket fitting and sweating.

Here, we briefly introduce the evaluation methods regularly applied for prosthetics use, with a focus on offline approaches and some selected clinical measures. Moreover, we provide experimental data on seven conventional myoelectric users. The literature review and the experimental data are limited to the primary aim of providing our view on assessment procedures for myocontrol and suggestions for their improvement.

### PERFORMANCE EVALUATION

Laboratory-based techniques and tests for measuring the performance in controlling a myoelectric interface are numerous and, in case of offline techniques, have been mainly derived or adapted from the machine learning literature. On the other hand, initially, clinicians have mostly adapted established hand and arm impairment assessment tools to the evaluation of functional recovery with prostheses. However, in recent years, new clinical measures have been introduced to specifically target the amputee patient population.

### Laboratory Metrics

Evaluation and assessment techniques for myocontrol in strictly laboratory conditions can be broadly divided in two groups—those quantifying the system performance through offline metrics and those based on online assessments using virtual prostheses or games.

Depending on the type of the evaluated control algorithm, offline performance is most commonly assessed using either classification accuracy (Ortiz-Catalan et al., 2013) or the R 2 error with respect to a given prompt (Ameri et al., 2014). The first approach relies on the number of correct estimates that the tested classifier makes, given the new, unseen data. The second compares the estimated command with respect to a reference cue. It has been shown that offline analysis fails to reflect the performance exhibited in online scenarios (Jiang et al., 2014b; Ortiz-Catalan et al., 2015). This is classically attributed to the fact that offline analyses do not account for adaptation of the user to non-stationary signal features.

Several virtual reality (VR) based assessment benches have been proposed in recent years. These systems simulate the online use of the prosthesis, at various levels of abstraction, while still being research-based settings. They offer the advantage of not dealing with the full implementation of the system, avoiding the challenges of socket design and hardware implementations. These VR systems are sometimes abstract with respect to the intended control (Ison et al., 2016) and commonly consist in steering a computer avatar in multiple directions to assess the performance when controlling specific degrees of freedom. Alternatively, computer games can be presented to the users, e.g., controlling a cursor to hit targets on a computer screen (Ameri et al., 2014; Jiang et al., 2014a). Finally, users can also be instructed to move a virtual arm into a target posture (Simon et al., 2011), as a part of an elaborate VR test bench.

The online systems are superior to the offline evaluations since they include the user in the loop and therefore account for his/her adaptation to the system. Parameters such as completion rate, path efficiency, number of overshoots or throughput, provide a solid quantitative evaluation of online performance. Further, the Fitts' law (Fitts, 1954) has also been applied in evaluating myocontrol. It provides a single statistical measure to characterize online control (Fimbel et al., 2006; Park et al., 2008; Scheme and Englehart, 2013). Nonetheless, even if some of these test benches offer realistic testing scenarios, they have limitations. For example, weight bearing by the prosthesis and stump dynamics causing pressure changes within the socket fitting are important realistic factors of influence (Daly et al., 2014), not included in these tests. On the other hand, VR systems have found relevant applications in patient training (Roche et al., 2015; Sturma et al., 2015) and can be combined with table-top prosthetics (Stubblefield et al., 2011).

### Clinical Metrics

Clinical and rehabilitation specialists rely on a set of tests as well as questioners for assessing the user performance in myoelectric control. These tests prompt users to manipulate a variety of objects and to execute tasks mimicking those of daily living. The majority of the clinical scores validate the capability of executing certain tasks by quantifying the completion time. A battery of clinical tests requires the presence of certified examiners.

The box and blocks (B&B) test is one of the simplest and most commonly used clinical tests for evaluating the severity of upper limb deficiency. It consists of transporting, one by one, a number of square wooden blocks over a barrier using the prosthesis. The quantitative performance index for this test is the number of blocks that are successfully moved in a fixed time interval (usually 1 min). This test is simple to implement but only focuses on a limited number of DoFs and requires a minimal skill by the user.

The Clothes Pin Relocation Test (CPRT) requires the user to move a set of clothes pins of various resistances from a horizontal to a vertical bar. Since this is primarily a rehabilitation tool, the exact evaluation procedure has not been defined yet. However, most therapists use four clothespins of different resistances (1, 2, 4 and 8 lbs) and prompt the subjects to relocate them from the lowest horizontal bar to the most convenient position on the vertical bar. The time of execution is then recorded from the starting neutral position to the final neutral position. The CPRT requires activation of several degrees of freedom, although it often promotes compensatory movements which are not accounted for in the final outcome score.

The Southampton Hand Assessment Protocol (SHAP) is one of the most elaborate hand impairment evaluation tests (Light et al., 2002). It consists of 26 individual tasks that include six grips and their combinations. It can be separated into abstract object handling and execution of activities of daily living (ADL). Its final outcome is a number in the range 0–100, where 0 corresponds to absence of hand function and 100 to a healthy hand function, which mainly reflects the time needed for completing the tasks. SHAP is a very detailed hand assessment tool and therefore it tends to be lengthy and tiring for the patients, especially those with limited capabilities. Additionally, it mainly quantifies the time needed for execution and does not account for the way in which the tasks are completed.

The Action Research Arm Test (ARAT) is a global arm function assessment procedure. It is divided into four subscales—grasp, grip, pinch and gross movement—that evaluate abstract object manipulation strategies. The maximum ARAT score is 57, corresponding to normal upper limb function. This score is based on the opinion of certified examiners that rate the quality of execution of each task on a scale from 0 (cannot perform) to 3 (performs normally).

In addition to the above, several other clinical tests and questioners have been devised targeting different functions and ways of assessing upper limbs, such as the Assessment of Capacity for Myoelectric Control (ACMC; Hermansson et al., 2005) and the Jebsen-Taylor Test of Hand Function (JTHF; Davis Sears and Chung, 2010). Contrary to the other tests discussed, ACMC is a clinical evaluation test specifically tailored for myocontrol rather than generically for hand function. Nonetheless, it suffers of a relatively large subjective component which has so far limited its use.

Although being the best test bench available so far, existing clinical tests are still limited in fully representing the functional benefit of the prosthetic system for the patients. The main limitation that needs to be addressed in the field is the lack of objective clinical metrics to quantify the way movements are performed with respect to natural motor tasks. Different control algorithms may score similarly for clinical tests that quantify the time needed to perform a set of standard tasks but yet provide very different ability for the user to perform movements with natural postures (Aszmann et al., 2016).

FIGURE 1 | Correlation between clinical scores and classification accuracies. (A) Correlation between the clinical Southampton Hand Assessment Protocol (SHAP) score and offline classification accuracy. The offline scores have been obtained in realistic conditions with the patients wearing their prostheses and training and testing performed on sets of data obtained in different arm positions. Despite the realistic conditions, the associations shown here are not strong. For example, a SHAP score of approximately 40 may correspond to classification accuracy lower than 70% or greater than 85% depending on the user. The SHAP requires precise manipulation over short periods of time which is not captured by this offline metrics. (B) The correlation between the clinical Box&Blocks (B&B) test and the offline classification accuracy shows almost complete absence of association between the two. For instance, the two patients who achieved classification accuracies >95% were radically different for the number of blocks they could transfer. When computed in less realistic conditions (without prosthesis and testing on the same arm posture as training) the offline scores were greater than in the presented conditions but showed almost no correlation with clinical tests, since the majority of the patients were not able to conclude the clinical evaluation without substantial retraining.

### EXPERIMENTS

We provide data on amputees that compare the accuracy estimated offline, for one of the classic control schemes developed over the past decades, with clinical scores. These data serve the purpose of representatively supporting the need for clinical tests for myocontrol developments. Therefore, the experiment and results do not aim at providing general conclusions on all myocontrol schemes and evaluation methods but rather at exemplifying the view presented in this perspective article.

Seven male transradial myoelectric users agreed to participate. They were all fit with custom-made sockets and with the Michelangelo hand (Ottobock Healthcare GmbH, Austria) with additional wrist rotation and flexion/extension units. The study was performed in accordance with the recommendations of the local ethics board of the Medical University of Vienna (Ethics Commission number 1044/2015), with written informed consent from all subjects. Subjects were fully briefed on the study protocol and possible adverse effects in presence of a clinical staff. All given consents are in accordance with the Declaration of Helsinki. All involved participants were transradial amputees with previous experience in using commercially available prosthetic devices. Before participation in the experiment medical state of each participant has been checked by the clinical staff.

The control of the prosthesis was based on the common spatial pattern (CSP) based classifier, as described by Amsuess et al. (2016). The EMG signals were recorded with 8 bipolar surface electrodes (Otto Bock raw signal electrodes 13E200 = 50AC). The control system allowed the subjects to access seven prosthetic functions—wrist flexion/extension, wrist pronation/supination, hand open, pinch, and key grip. All the motions were recorded in three arm positions (relaxed, fully extend arm in front of the ipsilateral shoulder, and fully extended arm across the contralateral shoulder) and at three forces (30%, 60% and 90% relative to the EMG level at maximum voluntary contraction force) while wearing the full prosthetic fitting. For offline accuracy assessment, the classifier was trained by data collected in only one arm position and tested against the remaining two data sub-sets. The average of the three scores was the reference performance of the subject. The entire data set was used for training the same CSP classifier that allowed execution of the B&B and SHAP tests. These particular clinical tests were chosen since they cover a wide range of assessment goals while being entirely objective. Additionally, these two tests have been widely recognized and familiar to academic and industry-based developers as well as clinical experts.

The performance scores in both offline and clinical tests are presented in **Figure 1**. The offline classification accuracies are slightly lower than in other studies (Ahsan et al., 2010; Liu et al., 2013) because of the different arm positions used for training and testing as well as the full prosthetic fitting which is not usual in offline evaluation studies. Although with these choices we have presumably maximized the prediction capacity of offline indexes for clinical scores, still the clinical scores did not strongly correlate with the offline performance measures. For example, there were two patients who achieved a similar SHAP score just below 40 but with very different classification accuracies of <70% and >85% (**Figure 1A**). Similarly, two patients who had similar classification accuracies of 70%–75% had SHAP scores of 27 and 47 (**Figure 1A**). The B&B test requires less skill to be performed than the SHAP. However, the B&B score was even less associated to the offline classification than the SHAP (**Figure 1B**). For example, subjects with an offline accuracy >95% performed very differently in this test (**Figure 1B**). Furthermore, when considering strictly the hand movements—hand open, fine pinch and key grip—that are primarily used for this test,

FIGURE 2 | Classification output for two patients with substantially different outcome of the B&B test but very similar classification accuracies over all motions. The focus here is on the three hand motions that are most relevant for the B&B task—hand open, key grip and fine pinch. The offline accuracy for these motions is lower for the subject with the higher clinical score.

the mismatch between this test and offline performance was even more substantial. This was observed consistently in all patients but it is shown representatively for only two patients in **Figure 2**. For these patients, the average classification rate across the three hand motions was 89% and 79% whereas the transferred blocks (score of the B&B) were 5 and 12, respectively.

When the offline evaluation was performed by using data collected without wearing the prosthesis and tested on the same arm position as the training, as more commonly done in laboratory tests (e.g., Englehart et al., 1999; Hargrove et al., 2009; Li et al., 2010; Ortiz-Catalan et al., 2014b), the resulting offline classification rates were high and comparable to those reported in the literature (>90% on average). However, once fully fitted, the majority of patients were unable to successfully conclude the clinical evaluations without retraining, suggesting that the classic offline evaluation procedure performed in several research studies, even though indicative, does not necessarily vouch for superior clinical performance.

### DISCUSSION

Abandonment rates among upper limb myoelectric prosthetic users are still very high (Burrough and Brook, 1985; Glynn et al., 1986; Østlie et al., 2012). At the same time, research efforts have provided several new solutions for myocontrol that have been proven to be highly functional strictly under laboratory conditions. The limited transfer from research to real world applications likely depends on an insufficient level of evaluation procedures.

Using novel prototypes of myoelectric systems in daily life would provide the ultimate assessment, but this strategy would often require official certification by notified bodies, which often goes beyond the possibilities of academic development. The COAPT system (Coapt LLC, 2016) is one of the first systems that has reached this level of testing. Clinical evaluations at earlier stages are a compromise between laboratory conditions and real-life tests. Although not perfect, clinical tests are closer to the conditions of interest for the users than offline assessments or online tests using virtual prostheses which provide valuable, but not always sufficiently transferable scores. Here, we have presented an example of this dissociation on a small sample of amputees and focusing on offline metrics, for demonstration purposes. We have compared clinical scores with offline indexes of performance extracted in the most realistic offline conditions (patients wearing a prosthesis, training and test sets obtained on different arm postures). Despite these conditions rarely being met in the offline studies, the prediction capacity for clinical outcome was not strong. On the other hand, when the offline indexes were obtained in more common laboratory

### REFERENCES

Ahsan, M. R., Ibrahimy, M. I., and Khalifa, O. O. (2010). Advances in electromyogram signal classification to improve the quality of life for the disabled and aged people. J. Comput. Sci. 6, 706–715. doi: 10.3844/jcssp.2010. 706.715

conditions without the prosthesis and for the same arm posture for test and training, the clinical information they provided was minimal (indeed with this training, once fitted with the prosthesis patients could not even finish the clinical tests without re-training). Further extrapolating, it is obvious that an offline analysis performed in these simple conditions and, in addition, on able-bodied individuals instead of patients, is of rather poor clinical value. While we are fully aware that in the initial evaluation of a new myocontrol scheme the strict laboratory tests on healthy individuals are valuable and needed for assessing the basic algorithmic working principles, there is also the need to make efforts in continuing the evaluations of promising algorithms in clinically-relevant settings (and to further develop clinical tests that fully represents the functional benefits). We believe that the evaluation stages after the laboratory level have had so far a slower progress, and less academic interest, with respect to the proposal of new algorithms.

Considering the discrepancy presented in the literature (Jiang et al., 2014b; Ortiz-Catalan et al., 2015) and further supported here, it seems necessary that novel myoelectric systems that passed laboratory testing are then fully clinically evaluated for assessing their performance. For this purpose, researchers and clinicians should jointly devise a standardized testing framework for quantitatively and qualitatively assessing the performance of upper limb prosthetic devices and their users to boost the process of commercialization and, as a consequence, availability for the patients. This need does not only relate to the feed-forward control aspects, on which we focused here, but also to fully closed-loop systems that include sensory feedback integration (Gonzalez and Yu, 2009; Jorgovanovic et al., 2014; Ortiz-Catalan et al., 2014a).

### AUTHOR CONTRIBUTIONS

IV, DF and OCA: substantial contributions to the conception; IV, ADR, SA, DF and OCA: design of the work; IV, ADR, TH, AS and SA: the acquisition; IV, ADR, TH, AS, SA, DF and OCA: analysis; IV, ADR, TH, DF and OCA: interpretation of data for the work; IV, ADR, DF, OCA: drafting the work and revising it critically for important intellectual content . The major writing of the report was completed by IV, DF and OCA. Final approval of the version to be published was given by all authors.

### FUNDING

This work was supported by the European Union's Horizon 2020 research and innovation program under grant agreement number 687795 (project INPUT) and by the Christian Doppler Research Foundation of the Austrian Federal Ministry of Science, Research and Economy.

Amsuess, S., Vujaklija, I., Gobel, P., Roche, A. D., Graimann, B., Aszmann, O. C., et al. (2016). Context-dependent upper limb prosthesis control for natural

Ameri, A., Scheme, E. J., Kamavuako, E. N., Englehart, K. B., and Parker, P. A. (2014). Real-time, simultaneous myoelectric control using force and position-based training paradigms. IEEE Trans. Biomed. Eng. 61, 279–287. doi: 10.1109/tbme.2013.2281595

and robust use. IEEE Trans. Neural Syst. Rehabil. Eng. 24, 744–753. doi: 10.1109/TNSRE.2015.2454240


Weir, R. F., Troyk, P. R., DeMichele, G., Kuiken, T., and Ku, T. (2003). ''Implantable myoelectric sensors (IMES) for upper-extremity prosthesis control- preliminary work,'' in 25th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, (Cancun, Mexico), 1562–1565. doi: 10.1109/IEMBS.2003.1279658

**Conflict of Interest Statement**: The handling Editor declared a past collaboration with one of the authors DF and states that the process nevertheless met the standards of a fair and objective review.

The other authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Vujaklija, Roche, Hasenoehrl, Sturma, Amsuess, Farina and Aszmann. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution and reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# **Assessment of Myoelectric Controller Performance and Kinematic Behavior of a Novel Soft Synergy-Inspired Robotic Hand for Prosthetic Applications**

*Simone Fani 1,2 \*, Matteo Bianchi <sup>1</sup> , Sonal Jain2,3, José Simões Pimenta Neto2,4 , Scott Boege<sup>2</sup> , Giorgio Grioli <sup>5</sup> , Antonio Bicchi 1,5 and Marco Santello<sup>2</sup>*

*<sup>1</sup> Centro di Ricerca E. Piaggio, Università di Pisa, Pisa, Italy, <sup>2</sup> Neural Control of Movement Laboratory, School of Biological and Health Systems Engineering, Arizona State University, Tempe, AZ, USA, <sup>3</sup> PES Institute of Technology, Bangalore, India, <sup>4</sup> Pontifical Catholic University of Minas Gerais, Belo Horizonte, Brazil, <sup>5</sup> Advanced Robotics Department, Istituto Italiano di Tecnologia, Genova, Italy*

#### *Edited by:*

*Michael Wininger, University of Hartford, USA*

#### *Reviewed by:*

*Calogero Maria Oddo, Sant'Anna School of Advanced Studies, Italy Valery E. Karpov, National Research University – Higher School of Economics, Russia*

> *\*Correspondence: Simone Fani simonefani89@gmail.com*

*Received: 15 July 2016 Accepted: 29 September 2016 Published: 17 October 2016*

#### *Citation:*

*Fani S, Bianchi M, Jain S, Pimenta Neto JS, Boege S, Grioli G, Bicchi A and Santello M (2016) Assessment of Myoelectric Controller Performance and Kinematic Behavior of a Novel Soft Synergy-Inspired Robotic Hand for Prosthetic Applications. Front. Neurorobot. 10:11. doi: 10.3389/fnbot.2016.00011* Myoelectric artificial limbs can significantly advance the state of the art in prosthetics, since they can be used to control mechatronic devices through muscular activity in a way that mimics how the subjects used to activate their muscles before limb loss. However, surveys indicate that dissatisfaction with the functionality of terminal devices underlies the widespread abandonment of prostheses. We believe that one key factor to improve acceptability of prosthetic devices is to attain human likeness of prosthesis movements, a goal which is being pursued by research on social and human–robot interactions. Therefore, to reduce early abandonment of terminal devices, we propose that controllers should be designed so as to ensure effective task accomplishment in a natural fashion. In this work, we have analyzed and compared the performance of three types of myoelectric controller algorithms based on surface electromyography to control an underactuated and multi-degrees of freedom prosthetic hand, the SoftHand Pro. The goal of the present study was to identify the myoelectric algorithm that best mimics the native hand movements. As a preliminary step, we first quantified the repeatability of the SoftHand Pro finger movements and identified the electromyographic recording sites for able-bodied individuals with the highest signal-to-noise ratio from two pairs of muscles, i.e., flexor digitorum superficialis/extensor digitorum communis, and flexor carpi radialis/extensor carpi ulnaris. Able-bodied volunteers were then asked to execute reach-to-grasp movements, while electromyography signals were recorded from flexor digitorum superficialis/extensor digitorum communis as this was identified as the muscle pair characterized by high signal-to-noise ratio and intuitive control. Subsequently, we tested three myoelectric controllers that mapped electromyography signals to position of the SoftHand Pro. We found that a differential electromyography-to-position mapping ensured the highest coherence with hand movements. Our results represent a first step toward a more effective and intuitive control of myoelectric hand prostheses.

**Keywords: prosthetics, assistive robotics, rehabilitative robotics, myoelectric control, kinematics**

### **1. INTRODUCTION**

The introduction of myoelectric devices has profoundly modified prosthetics, especially in upper-limb amputation. Upper-limb myoelectric devices use electrical activity associated with muscle contraction (electromyographic signals, EMG) recorded from residual muscles in individuals with upper-limb loss to control movements of terminal devices, i.e., arm and/or hand prostheses. At the same time, electric actuators address some of the drawbacks of body-powered devices, such as heavy harness and limited functionality (Van Lunteren et al., 1983). Furthermore, myoelectric prostheses can offer better esthetics, greater pinch strength, and ease of operation (Biddiss and Chau, 2007). However, despite these advantages relative to body-powered devices, the rejection rate of myoelectric prostheses remains very high. Specifically, more than 23% of myoelectric prosthesis users abandon their devices (Biddiss and Chau, 2007). The main causes for abandonment of myoelectric devices include higher maintenance needs, greater costs, heavier weight, and low intuitiveness of control. These limitations suggest that improving intuitiveness of myoelectric control might contribute to reducing the rejection rate of myoelectric devices. In the present study, we propose that this objective could be attained by improving the correspondence between natural and prosthetic hand movements. The underlying assumption of our proposition is that individuals with upper-limb loss might prefer using a prosthetic device that can be controlled using similar muscle activation patterns underlying the control of the native hand, as opposed to being forced to learn abstract EMG patterns to fit the design of the terminal device. This approach could thus play a crucial role for increasing both acceptability and the sense of embodiment, i.e., the sense of the prosthesis becoming part of the user's body by transitioning from an extracorporeal to a corporeal structure (Fraser, 1984; Scarry, 1994; Murray, 2008).

Within the above conceptual framework, anthropomorphism may be an additional factor to be considered to improve users' acceptance of myoelectric prostheses, a concept that has been extensively studied in the field of human–robot interaction (Bartneck et al., 2009; Riek et al., 2009; Dragan and Srinivasa, 2014). Specifically, myoelectric prostheses represent a category of bidirectional human–robot interactions, as users control the terminal device through muscle activation. To improve effectiveness of prostheses performance, an additional aspect to consider is the correct identification of body sites for the acquisition of EMG through surface electrodes (Micera et al., 2010). This is an important factor to ensure optimal EMG signal quality, which in turn is necessary to improve control of the terminal device, while trying to compensate for unavoidable issues such as sweat or movement of the electrode relative to the target site.

In the literature on myoelectric prostheses, one can distinguish two approaches: (1) minimalistic mapping for standard one degree of freedom (DOF) hand prostheses, where it is customary to use two EMG electrodes located on an antagonistic muscle pair, e.g., residual wrist flexor and extensor muscles, to control the closing and opening of the prosthesis (Ajoudani et al., 2013) and (2) mapping EMG signals from multiple muscles to multi-DOFs devices [for a survey on the usage of multi-sEMG signals for robotic hand control, the reader is referred to Ison and Artemiadis (2014)]. Regarding (1), Ajoudani et al. (2012) examined the incorporation of the users' intent in the control command not only in terms of desired motion or equilibrium position but also as stiffness profile estimated on the master side through a suitable human–machine interface. This approach, called Tele-impedance, was proven to be a viable solution also to overcome stability problems in force reflecting teleoperation and enable a more human-like task execution (Ajoudani et al., 2012). Ajoudani et al. (2014) used the major finger antagonist muscle pair, i.e., the m. extensor digitorum communis (EDC) and m. flexor digitorum superficialis (FDS), to map the reference commands by leveraging upon a modified hyperbolic tangent shape. With regard to (2), interfacing EMG signals from multiple residual muscles with a multi-DOF systems requires a direct mapping from multi-EMG patterns to control commands, and this is often accomplished by using machine learning (ML) techniques. However, one of the main issues for a deployment of these methods in clinical settings is its low reliability (Biddiss and Chau, 2007). To mitigate this drawback, the concept of simultaneous and proportional (s/p) myocontrol was proposed by Jiang et al. (2009), which enables mapping of EMG signals to activation of several DOFs based on regression techniques, rather than a classifier (Ison and Artemiadis, 2015). Another valuable contribution in this area is the idea of interactive and incremental learning: since calibration of ML methods usually employs a one-shot initial phase to strengthen its robustness, the function approximation can be further refined after this phase, thus leading to a model update that can correct control instabilities (González and Castellini, 2013); for further details on these topics, the reader is referred to Santello et al. (2016). A different approach consists of invasive myo-controlled prosthetic devices, which combines surgery, bionic reconstruction, and engineering, in some cases, enhanced by sensory feedback [see, for example, Ortiz-Catalan et al. (2014), Raspopovic et al. (2014), andAszmann et al. (2015)].

In the present work, we focused on the control of the SofHand-Pro (SH-P), the prosthetic myoelectric version of an existing robotic hand, the Pisa/IIT SoftHand (SH) (Catalano et al., 2012, 2014; Ajoudani et al., 2014). The SH is a humanoid robotic hand that combines the concept of kinematic motor synergies (Santello et al., 1998) and soft robotics. By combining the concept of hand postural synergies and soft robotics, the result is an artificial system actuated with only one motor, which implements the first human hand synergy in grasping in free-hand motion. At the same time, the SH, is also adaptable and robust, and hence able to grasp different types of items. The prosthetic version, the SH-P, can be controlled using two EMG signals from a couple of agonistic–antagonistic (Ag–Aag) wrist or finger muscles.

The present study was designed to identify the myoelectric controller that could generate SH-P finger movements with the greatest reliability and degree of similarity of finger movements of the native hand. Our investigation consisted of three steps.

**Abbreviations:** Ag–Aag, agonist–antagonist; DOF, degree of freedom; ECU, extensor carpi ulnaris; EDC, extensor digitorum communis; EMG, electromyography; FCR, flexor carpi radialis; FDS, flexor digitorum superficialis; ML, machine learning; PGM, P-gain modulation; RMSD, root mean square difference; sEMG, surface electromyography; SH, Pisa/IIT SoftHand; SH-P, SoftHand Pro; SNR, signal-tonoise ratio.

First, we quantified the kinematic consistency of SH-P movements across a large number of movements to evaluate the repeatability of its performance. The objective of this evaluation was twofold: (1) to assess the reliability of SH-P response to motor commands and (2) to ensure that the evaluation of similarity between SH-P and native finger movements would not be biased by random inconsistencies in SH-P response to motor commands. Second, we evaluated the most suitable body sites on the forearm for recording EMG defined in terms of SNR. As described for the first objective, this evaluation was necessary to ensure that myoelectric controllers could be reliably compared by using the best possible EMG signals as inputs to the controllers. Furthermore, high SNR of EMG signals contain greater information about modulation of muscle activity responsible for finger movements, and could, therefore, be better exploited by the myoelectric controllers. The third evaluation, which is the core objective of the present work, compared three myoelectric control algorithms. The optimal controller was defined as the algorithm that generated SH-P finger kinematics that best resembled finger kinematics of the native hand.

Although there is an extensive literature on myoelectric prosthesis controllers [early work dates back to the 50s as (Battye et al., 1955), for a review, see, e.g., Castellini et al. (2014)], literature on the use of EMG signals for the control of synergistic movements is moving its first steps. Within this growing framework, it is important to note that the aim of this work is to provide an assessment of myoelectric controller performance and kinematic behavior of a specific device, the SH-P. For these reasons, algorithms were specifically tailored to be used with the SH-P, although other commercial prostheses are endowed with similar controllers. Results of this paper represent a stepping stone toward more in-depth investigation on the extent to which the hardware and software of the SH-P can overcome limitations of existing hand prostheses.

### **2. MATERIALS AND METHODS**

### **2.1. Participants**

We performed three experiments. Experiment 1 quantified the repeatability, reliability, and consistency of the SH-P movements. Experiment 2 was designed to identify the agonist–antagonist muscle pair characterized by the highest SNR. Experiment 3 quantified the EMG-to-position mapping algorithm that provided the best correspondence between finger movements of the native hand and SH-P.

*Experiment 1* did not involve testing of human subjects as the SH-P was controlled by artificial commands (see below). For *Experiment 2*, we tested four able-bodied volunteers (1 female, 3 males; age range: 21–26 years, mean *±* SD: 22.5 *±* 2.06). For *Experiment 3*, we tested fourteen able-bodied volunteers (5 females, 9 males; age range: 18–27 years, mean *±* SD: 20.36 *±* 2.43).

All subjects were tested on their dominant hand (right hand; self-reported hand dominance). All participants were naive to the experimental purpose of the study and had no history of neuromuscular disorders. Before data collection, subjects signed an informed consent to participate in the experiment. The experimental protocols were approved by the Institutional Review Board at Arizona State University in accordance with the Declaration of Helsinki.

### **2.2. Apparatus**

For the present investigation of myoelectric controller algorithms for hand prosthesis, we used the SoftHand-Pro (SH-P) (**Figure 1**). The SH-P is the prosthetic version of the Pisa/IIT SoftHand (SH), a robotic hand that had been designed for humanoid robots (Catalano et al., 2012, 2014).

The main design concept underlying the development of the SH (Catalano et al., 2012, 2014) is the construction of a robust, safe, low-cost, and simple robotic hand. The SH finger movements occur along the first hand postural synergy as defined by Santello et al. (1998), i.e., the first principal component extracted from static hand postures used to grasp a wide variety of imagined objects. This first postural synergy was implemented in the SH-P by combining it with a soft-robotics approach, through which a reference position of a "virtual hand" attracts the real hand, thus resulting in the concept of "soft synergy" (Bicchi et al., 2011). The soft-synergy approach enables a better control of the interaction forces between the hand and the grasped object, thus allowing the SH-P to grasp a large variety of items.

Another important feature of the SH design is that it is underactuated (Birglen et al., 2007). Specifically, the SH has 19 DOFs, 4 on each finger, and 3 on the thumb, but they are all actuated by only one motor (**Figure 1**). This feature effectively reduces weight, costs, and control complexity, thus making it an ideal candidate for prosthetics applications, since it only requires two EMG signals from an agonist–antagonist muscle pair (Zhao et al., 2015). The actuator is a DC motor that pulls a tendon running across the "phalanges" through a system of pulleys. Movements of the SH are controlled using a PID controller, which takes as inputs the desired angular position of the motor and the real position measured by a 12-bit magnetic encoder (resolution: 0.0875°) applied to the motor. All the structural components of the palm are built with rapid prototyping material (ABSplus – Stratasys), all the phalanges are fabricated using injection molding except for the "metacarpo-phalangeal" joint of the thumb, which is fabricated using a computer numeric control machine.

The original design of the SH was modified for prosthetic applications. Specifically, the overall size of the device was reduced to obtain the size of an average human male hand. The electronics were reduced in size and modified to enable interfacing with commercial EMG electrodes (Otto Bock, Germany). The electronics, previously placed at the base of the hand, was moved to the back to obtain a more compact design (**Figure 1**). The original SH motor (15-W Maxon motor RE-max-21 24 V) was substituted with a smaller and lower voltage motor (14-W Maxon motor DCX-22-S 12 V) and battery. The dimensions of SH-P used for the present work are 210 mm from the tip of the thumb to the tip of the little finger, and 170 mm from the base of the palm to the tip of the middle finger (225 mm relative to the wrist interface). The thickness of the palm is different between the thumb and the little finger sides of the hand, i.e., 40 mm on the little finger side, where the EMG connectors are placed, and 53 mm at the motor location (the thumb side), respectively. The SH-P is powered using an

markers placed on the distal phalanges were used for the experiment, whereas additional markers in a triangular configuration were placed on the palm (not visible in

11.1-V AR.Drone 2.0 HD Battery 1500 mAh (Parrot) connected

via standard cable connection. For further information about the technical characteristics of the SH-P, the reader is referred to Catalano et al. (2014).

### **2.3. Experiments**

this picture).

### 2.3.1. Experiment 1: Testing Repeatability of SH-P Finger Kinematics

Experiment 1 was designed to assess the extent to which the SH-P could be used for assistive or rehabilitation purposes by quantifying the repeatability, reliability, and consistency of the SH-P finger movements in free-hand motions. As myoelectric control through a human user could contribute to across-trial variability in SH-P finger kinematics, this experiment removed this potential confound by driving the SH-P through artificial commands. Thus, this design allowed us to isolate the causes of potential variability of SH-P kinematics to its mechanical design.

### *2.3.1.1. Apparatus*

The SH-P was placed vertically, fixed to the table through the wrist interface, and connected via a USB-A to micro USB-B cable to the laptop that was used to send commands to the SH-P and collect data from the on-board hand encoder. We used an active motion tracking system (PhaseSpace Motion Capture system, PhaseSpace Inc.) to record SH-P finger movements through 10 cameras (optical resolution: 3600 *×* 3600 – impressive sub-pixel resolution: 360,000 *×* 360,000 at 960 Hz). We placed a total of 22 light-weight infrared active LED markers (jitter *<*0.5 mm) on the SH-P, including each SH-P "phalanx" and "metacarpo-phalangeal" joints, and on the palm to create a local reference system (**Figure 1**). For the purpose of this experiment, we analyzed only the markers on the distal phalanges and palm. The orientation of the hand was defined to minimize marker occlusion and maximize capture of all markers by at least 3 cameras for at least 90% of the trial duration.

### *2.3.1.2. Experimental Protocol*

Custom software was used to control the timing of SH-P finger position (update rate: 8 Hz). We tested a total of sixty-three closing–opening movement cycles that were performed at increasing velocities. The SH-P velocities were expressed in motor steps per sample and ranged from 10 to 1000 steps/sample. We tested twenty-one SH-P velocities (3 trials per target velocity), corresponding to 10 up to 200 steps/sample with increments of 10 steps/sample, whereas the twenty-first velocity was 1000 steps/sample. Note that 10 steps/sample is the smallest step resolution that can produce an observable change in SH-P finger motion. Although the 1000 step/sample velocity does not capture a realistic SH-P finger velocity associated with activities of daily living, being it faster than the usual velocities, the SH-P moves during the normal usage; this velocity was chosen as a stress test of the SH-P hardware. We recorded three-dimensional position of the active markers on the SH-P during all trials.

### *2.3.1.3. Data Processing and Analysis*

When markers occlusion occurred, marker position data were interpolated. We found that most of the marker occlusions occurred at the end of the movement, i.e., at closed fist posture. Therefore, all makers, hence finger kinematics, could be reliably tracked from the initial position (fully open hand) and throughout the entire closing movement, with the exception of a small subset of markers when the hand was fully closed. The next step of kinematic data processing was to transform all marker positions defined with respect to an inertial system of reference, in the local reference frame, defined through the markers attached to the palm of the hand in a triangular arrangement. The new reference frame was defined in order to have the x-axis directed as follows:

from the little finger side of the hand to the thumb one, with its origin being the center of the base of the triangle. The z-axis was directed from the origin to the upper vertex of the triangle, whereas the y-axis, according to right hand rule, pointed inside the hand (**Figure 1**). The path of the fingertips was visualized in three ways: (1) all paths defined by the markers on the fingers were plotted in 3-dimensional space; (2) the value of the threedimensional position of each fingertip was defined with respect to the position of the motor of the SH-P, averaged across trials, and the mean *±* SD values for each fingertip position were plotted; and (3) the position of each fingertip was plotted against timenormalized trial duration. Using the data processed as described in (2), for each fingertip, we computed the root mean square difference (RMSD) between fingertip position from individual trials and the mean of fingertip position computed across all trials

$$\text{RMSD} = \sqrt{\frac{\sum\_{t=1}^{n} \left(\hat{\boldsymbol{\chi}}\_{t} - \boldsymbol{\chi}\_{t}\right)^{2}}{n}}$$

where ˆ*y<sup>t</sup>* is the sample of the single trial, *y<sup>t</sup>* is the mean of the sample averaged across all trials.

### 2.3.2. Experiment 2: Analysis of EMG Signal-to-Noise Ratio

The goal of this experiment was to identify the muscle pair characterized by the highest signal-to-noise (SNR) ratio as a means to reliably compare the performance of three myoelectric controller algorithms (Experiment 3). To allow a realistic comparison with using myoelectric controllers in individuals with upper-limb loss, we did not perform an exhaustive search of target muscles to record EMG from to try to minimize cross talk among adjacent wrist and extrinsic finger muscles, e.g., specific finger compartments within the target finger muscle. Rather, we focused on selecting an agonist–antagonist muscle pair whose EMG activity (1) was characterized by the greatest SNR and (2) could be reliably elicited and recorded across trials when performing reach-tograsp movements.

### *2.3.2.1. Apparatus*

We used two surface EMG electrodes that are commonly used for myoelectric prostheses (linear-proportional 13E200 MYOBOCK electrodes, Otto Bock, Germany). These electrodes are equipped with a logarithmic sensitivity adjustment and high commonmode rejection in the low frequency range (*>*100 dB at 50 Hz). The electrodes were applied in correspondence of the wrist and finger muscles. We targeted two wrist muscles (m. flexor carpi radialis, FCR, and m. extensor carpi ulnaris, ECU) responsible for wrist flexion and extension, respectively, and two extrinsic finger muscles (m. flexor digitorum superficialis, FDS, and m. EDC) responsible for finger flexion and extension, respectively. We used a LabView program (National Instruments) to record EMG data (sampling rate: 1 kHz).

### *2.3.2.2. Experimental Protocol*

We asked subjects to reach, grasp, lift, and replace a cylindrical glass bottle (weight: 400 g) whose position was aligned with the right shoulder of the subject on the sagittal plane. In the start position, the subject was instructed to open the hand with the palm parallel to the sagittal plane of the body. The distance between the subject's hand start position and the bottle was 25 cm. We asked subjects to perform two block of reach-go-grasp trials. In the first block (15 trials), subjects performed slow reach-tograsp movements to be completed within 13 s (reach onset to object grasp). In the second block (5 trials), subjects were asked to perform the same movement, but at a faster speed such that the whole movement had to be completed within 7 s. In each reach-to-grasp trial, subjects were asked to move the hand in a direction perpendicular to her/his body to reach the cylinder. The experimenter gave a "go" signal to cue subjects to start the reach onset. In our instructions to subjects, we emphasized that reachto-grasp movement should be performed in a natural fashion, i.e., similar to the way one reaches for a cup on a desk. Once the hand had reached the bottle, subjects were asked to grasp, lift (~1 cm height), hold (~1 s), and replace it on the table and place the hand back to its stat position. We used the same EMG electrode gain for both muscle pairs.

### *2.3.2.3. Data Processing and Analysis*

The data were analyzed by computing the SNR of the EMG signals during the two blocks of reach-to-grasp action and isometric maximal voluntary contractions as

$$\text{SNR}\_{\text{dB}} = 10 \log\_{10} \frac{\text{RMS (Signal)}}{\text{RMS (Noise)}}$$

where the noise level was measured at rest in between trials. Noise measurement and SNR computation were performed as described in Solnik et al. (2008). We used the Wilcoxon rank sum test to determine statistically significant differences in the SNR of the two muscle pairs. We computed a pairwise comparison between SNR from finger muscles EMG versus SNR from wrist muscles EMG by pooling all trials (slow and fast) and extensor and flexor muscles within each category.

### 2.3.3. Experiment 3: Identification of Optimal Myocontroller Algorithm

The goal of this experiment was to find the most effective EMG mapping algorithm that could generate SH-P finger kinematics during reach-to-grasp movements that best resembled native hand kinematics. To attain this objective, we compared the performance of three EMG-to-SH-P movement mapping algorithms, in terms of effectiveness of movement activation (i.e., the ability of a given algorithm to generate SH-P finger motion) and similarity in the kinematics of SH-P and native hand.

#### *2.3.3.1. Apparatus*

We used three pairs of identical cylindrical objects (cans) with different radii and weight (52 mm and 250 g; 74 mm and 450 g; 85 mm and 600 g). One of the objects in each pair was to be grasped by the subject with his/her native hand, whereas the object was grasped by the SH-P (**Figure 2**). The SH-P grasping movement was controlled by two EMG electrodes placed on the subjects forearm as he/she reached to grasp the target object.

The object to be grasped by the subject was positioned on the table surface and aligned with the sagittal plane passing through the subject's right shoulder to avoid large wrist movements. The reach distance between start hand position and target object was 25 cm. The object to be grasped by the SH-P was placed within the palm of the SH-P. View of the SH-P during the reach-to-grasp movement with the native hand was blocked by a screen. To further prevent subjects from getting distracted, they wore headphones with white noise. We used the same motion tracking system used in Experiment 1 to record SH-P and native hand kinematics. The SH-P and native hand were outfitted with markers. For the SH-P, we placed markers on each fingertip (on the back of the third phalanx), and three reference markers on the wrist as done in Experiment 1. On the subject's hand, we placed a marker on the tip of each nail and eight markers on a 3D printed wrist structure

taped on the subjects wrist to define a local reference frame [for details, see Gabiccini et al. (2013)]. Markers were also placed on each of the target objects. As done in Experiment 1, the hand and the objects were positioned to ensure that markers could be viewed by at least 3 cameras. The SH-P parameters were managed through custom software by a laptop. Custom software was also used to record SH-P motor position data. A LabView program (National Instruments), running on a second computer, was used to record EMG and motion tracking data. Data collection by the two computers was synchronized offline via software at 90 Hz.

#### *2.3.3.2. Experimental Protocol*

Before performing the reach-to-grasp experiment and test each of three myoelectric algorithms (see below), the SH-P was calibrated by asking subjects to perform a maximal voluntary isometric contraction during finger flexion and extension while recording EMG of each muscle from the target muscle pair (FDS and EDC, respectively). These muscles had been identified in Experiment 2 as those with the highest SNR. The EMG amplitude recorded during calibration is used by the controller to define the threshold EMG amplitude above which finger motion occurs as well as the upper limit of EMG amplitude. The SH-P was mounted horizontally on a support fixed to the table (with the palm parallel to a vertical plane, thumb up) (**Figure 2**). To increase the sensitivity of the SH-P to EMG signals, the EMG threshold amplitude obtained with the calibration phase obtained through maximal voluntary contractions was reduced to 8 and 14% of the maximum FDS and EDC EMG amplitude recorded. We used the same EMG electrodes and recording procedures we used in Experiment 2. During calibration, the subject sat in the same position used during the experiment. Once the hand was calibrated, the subject started the experiment. Subjects performed one block of 60 trials each for each of the three myoelectric controller algorithms with a 5-min break between blocks. We tested three EMG controller algorithms. On each trial, custom software was used to select in a pseudorandom order, a given myoelectric controller algorithm, before the start of data collection start. Each algorithm was presented the same number of times. We tested the following algorithms:

1. *EMG differential*: this EMG-to-motor position mapping uses the difference between the EMG signals recorded by the two EMG electrodes minus the value of their respective thresholds. The sign of the EMG amplitude difference dictates the SH-P finger movement direction, whereas the amplitude of the difference defines finger movement velocity (**Figure 3A**).


In **Figure 4**, it is possible to observe how the different algorithms manage the recorded EMG signals on the SH-P.

Each of the three algorithms were presented in two ways that differed depending on whether a modulation of the proportional gain, driven by the ratio between the two EMG signals, was implemented. Thus, for each subject, we considered trials with versus without the EMG-driven P-gain modulation (see above; **Table 1**). For the P-gain modulation trials, the proportional gain of the PID controller of the position control of the motor increased with increasing amplitude of the EMG signal with the smallest amplitude, normalized to the EMG signal highest amplitude. Upper and lower bounds for the proportional gain were estimated through pilot data by selecting the range of values in which the

**FIGURE 4 |** *Experiment 3***: this picture presents how the different algorithms manage the EMG signals**. The plots of the real EMG signals, the EMG signals after the elaboration, and the application of the thresholds inside the SH-P firmware and the motor positions are presented. The presented data are obtained during a real trial of Experiment 3.

**TABLE 1 |** *Experiment 3***: activation rate with the different algorithms, considering (W PGM) and discarding (W/o PGM) the proportional gain modulation**.


*Motion was considered successfully activated when SH-P motor position reached or overcame 25% of the whole range.*

SH-P moved smoothly until reaching the reference position. SH-P movement velocity and impedance were modified by changing the proportional gain value. The reach-to-grasp task instructions and experimental procedures were the same as those described for Experiment 2; however, no constraint was imposed on the time required to complete the task. We instructed the subjects to contact the object and exert grip with all the fingers. More specifically, participants were required to keep their hand with the palm parallel to the sagittal plane and with the line from the wrist to the middle finger parallel to the transverse plane. During the experiment, subjects' movements were visually inspected by experimenter in order to verify that the aforementioned instructions were correctly executed. If the experimenter noted any major

deviation from the desired behavior, the trial was repeated. As subjects reached toward the object, the SH-P moved accordingly based on the EMG-to-position mapping selected for that trial. The three object sizes were presented in a pseudo-random order and for an equal number of trials for each algorithm.

#### *2.3.3.3. Data Processing and Analysis*

For each algorithm, our analysis focused on estimating (1) how many times the subject was able to activate the SH-P movements with a normal reach-to-grasp action and (2) quantifying the similarity between the SH-P and native hand finger kinematics. To define the extent to which SH-P responded to EMG-based commands, we computed the number of times that EMG signals could trigger SH-P finger movements. To verify whether EMG signals triggered SH-P finger movement, we examined the angular position of the motor. Movement activation was defined as the angular position above the minimum input value required to reach the 25% of the total SH-P movement range. The threshold in the position of the motor was the only parameter used to discriminate trials with successful from failed activation of the SH-P. Note that the analysis of the activation rate does not take in account eventual delays or deactivations after the threshold crossing; this last event will impact the index shown above. For the identification of the optimal myoelectric controller algorithm, we used trials in which SH-P finger movements could be generated by EMG signals and quantified the similarity of the paths of the SH-P and native hand fingertips during reach-to-grasp movements. This analysis consisted of computing a numeric index between 0 and 1 corresponding to no similarity and identical kinematics between finger movements in the SH-P and native hand, respectively. The vector distances (three components) between the thumb and each

of the four fingers were computed for both the SH-P and the native hand. These values were computed for each sample of each trial and normalized by dividing each component of the vector by the amplitude of the vector itself. This procedure removes the effect of different sizes of the SH-P and native hand on the comparison of their vector distances. We then computed the dot product between vectors pairs obtained from the native and artificial hand. The obtained four values, one for each thumb-finger vector, were summed and divided by 4 to compute the average similarity index. The mean similarity index was computed for each myoelectric controller algorithm.

### **3. RESULTS**

### **3.1. Experiment 1**

Experiment 1 was designed to assess the consistency of SH-P finger movements across trials and velocities. **Figure 5** shows

respectively.

the three-dimensional coordinates of each fingertip of the SH-P recorded during multiple hand opening–closing cycles. It can be seen that the fingertip paths during hand opening and closing (dark and light data points) are fairly consistent across movement cycles. To further visualize the consistency of fingertip paths, **Figure 6** shows these data projected on the three axes of the reference frame and expressed with respect to the motor angular position.

The maximum root mean square difference (RMSD) across all paths shown in **Figure 6** were 9.4 mm for the thumb (x-axis), 5.7 mm for the index finger (x-axis), 6.5 mm for the middle finger (z-axis), 4.4 mm for the ring finger (y-axis), and 5 mm for the little finger (y-axis). A high degree of consistency in fingertip paths can also be appreciated across different movement velocities when plotting finger paths as a function of normalized time (**Figure 7**).

Some shifts with respect to time and motor position, considered as offsets, can be observed. This is more evident in the final phase of the opening of the thumb (x-axis and y-axis) and in the central phase of the middle finger (z-axis). The data plotted in **Figure 6** also show hysteresis in the finger path during hand opening versus closing movements. However, this hysteresis, which is due to the elastic elements embedded in the joints that enable the passive opening of the SH-P, does not affect finger path consistency across trials and movement velocities. The small RMSE of fingertip paths indicate that SH-P finger movements are very consistent across multiple hand opening–closing cycles and movement velocities.

## **3.2. Experiment 2**

The objective of Experiment 2 was to determine the muscle pair characterized by the highest signal-to-noise ratio (SNR) to be used for comparing myoelectric controllers in Experiment 3. We found that SNR was significantly greater for the finger than wrist muscles EMG signals (27.73 and 21.88 dB; Wilcoxon rank sum test: P = 0.0015). Therefore, we chose to use the FDS-EDC muscle pair for Experiment 3. This choice was also motivated by the objective of allowing for a more natural and intuitive myoelectric control of the SH-P. Indeed, during a reach-to-grasp action, the time course of finger muscles' EMG signals is closely related to the time course of the SH-P finger motion (**Figure 8**). Specifically, the two EDC EMG peaks correspond with SH-P opening during pre-grasp and at object release at the end of the trial. Similarly, the FDS EMG signal exhibits a reciprocal activation pattern relative to EDC, i.e., most of its EMG activity is found between the EDC peaks during SH-P closing. In contrast, this correspondence between EMG patterns and SH-P finger motion was not being captured by wrist muscles EMG.

### **3.3. Experiment 3**

Experiment 3 was designed to identify the myoelectric controller algorithm that could elicit SH-P movements with the greatest degree of consistency and whose kinematics best resembled finger kinematics of the native hand.

**Table 1** shows the rate at which each myoelectric controller algorithm could elicit SH-P finger movements. The EMG

**FIGURE 8 |** *Experiment 2***: time course of EMG activity from finger and wrist muscles (top and bottom row, respectively) as a function of normalized time on the trial duration**. "0" and "1" denote the starting moment of the movement of reach and the end of the trial within the predefined time, when the hand was back in starting position, respectively.

mapping algorithm characterized by the highest activation rate was the EMG-differential (58.57%), whereas the EMG-FCFS algorithm was characterized by the lowest activation rate (10.12%).

However, even the best performing algorithm failed to generate SH-P movement in a consistent fashion. Further examination of EMG data revealed that such failure occurred more often in subjects who exhibited a lower ability to selectively activate one of the two muscles independently from the other and/or to inhibit muscle activity when switching from hand opening to closing. An important observation was that often the SH-P responded better to hand-closing than hand-opening EMG signals. This, however, happened significantly less often when EMG signals were mapped to SH-P movements through the EMG-differential algorithm. This could have been due to the fact that, during the object release, subjects not always fully extended the fingers but rather relaxed them just enough to trigger object release, but not a full extension of the SH-P fingers. Thus, the EMG amplitude associated with this action might not have been sufficient to trigger opening of the SH-P.

Note that SH-P activation rates were not affected by the implementation of the EMG-driven proportional gain modulation (PGM, **Table 1**; chi-squared test = P *>* 0.05).

**Table 2** shows the similarity index computed on the SH-P and native hand kinematics using only the trials that successfully elicited SH-P movements as described above.

All EMG mapping algorithms resulted in SH-P kinematics that was very similar to native hand kinematics (range of similarity index: 0.80–0.88). We found no statistically significant difference in the similarity index across the three EMG myoelectric controller algorithms. Taking into account the significantly greater activation rate for the EMG-differential algorithm, we conclude that this EMG mapping algorithm is preferable since it ensures both a more reliable activation of SH-P movement in response to EMG signals while resembling native hand kinematics.

**TABLE 2 |** *Experiment 3***: kinematic similarity index between SH-P and real user movements, for the different mapping algorithms, considering (W PGM) and discarding (W/o PGM) the proportional gain modulation**.


*A score close to 1 indicates a high level of similarity. The absent data are relative to the conditions with activation rate 0%. Indeed, in this analysis, we only considered trials where the SH-P was activated.*

### **4. DISCUSSION**

The overarching goal of our three experiments was to characterize the performance of a myoelectric hand prosthesis, the SoftHand-Pro, to assess its feasibility as an assistive and rehabilitative tool. Specifically, we sought to quantify the extent to which the SH-P finger movements resembled human finger movements. We reasoned that this factor should be evaluated as it might contribute to the extent to which individuals with upper-limb loss might accept the SH-P as a prosthetic device. To achieve this objective, we also quantified the consistency of SH-P finger movements and identified the muscles that should be used to extract EMG signals to control the SH-P. Below, we discuss our results and future research directions.

### **4.1. The SH-P Mechanical Design Enables Repeatable Finger Movement Kinematics**

Successful control and performance of myoelectric prostheses rely on three factors: (1) the intrinsic properties of the terminal device (i.e., hardware), which affect how it responds to given EMG signals, (2) how EMG signals are processed (i.e., software) to generate motion of the terminal device, and (3) the extent to which the user and his/her motor commands (extracted through EMG signals) can adapt to the terminal devices hardware and software characteristics. When studying a human subject "in the loop," these three factors interact in complex ways, thus making it difficult to understand the role of each factor on the terminal device performance. To ensure that we could identify the role of each of these three factors independently, we removed the potential effect of (3) from (1) by driving multiple SH-P movement cycles through artificial, rather than EMG, control signals (Experiment 1). We should note that, although previous studies have examined the performance of the SH and SH-P through myoelectric control (Godfrey et al., 2013, 2014; Ajoudani et al., 2014; Bonilla et al., 2014; Zhao et al., 2015), the present study is the first to examine repeatability of finger kinematics without the confound of trial-to-trial and/or between-subject variability of myoelectrical signals.

We found that SH-P finger movements were highly consistent across movement cycles and velocities (**Figures 5**–**7**). Thus, the soft synergy-based design allows the control of 19 degrees of freedom through the action of one motor without compromising the reliability of multi-finger motion. This is an important result when considering the needs of individuals with upper-limb loss to perform repeatable hand movements through EMG control. In particular, relying on repeatable kinematics for given EMG inputs should facilitate the adaptation of motor commands/EMG and learning to use the SH-P effectively.

### **4.2. Finger Muscles EMG as Control Signals for the SH-P**

To control multi-fingered hand prostheses through EMG signals, the classic approach is to employ signals collected from multiple muscles and then to use these signals as inputs to ML systems. This approach results in creating a direct mapping from multi-EMG patterns and multi-DOF devices (Castellini et al., 2009; Tenore et al., 2009; Ison and Artemiadis, 2014). In contrast, the design of the SH-P enables the use of a minimal number of EMG signals (two) to control one degree of actuation. However, thanks to its adaptability, the SH-P can exploit the external environment as a multiplier of its DOF, thus enabling the performance of a wide of range of activities of daily living (Godfrey et al., 2013; Ajoudani et al., 2014; Centro di Ricerca Enrico Piaggio et al., 2016; Santello et al., 2016).

Before, we could address the question of what myoelectric controller optimizes SH-P performance, we had to identify the muscle pair whose EMG signals was characterized by the greatest signal-to-noise ratio (SNR). As noted for the testing of SH-P finger kinematics repeatability, these questions were not addressed in previous SH-P studies. We found that SNR was significantly greater in finger than wrist muscles. Although the goal of our study was not to perform a systematic evaluation of SNR across many pairs of finger or wrist muscle pairs, this result was important to rule out a potential confound in our evaluation of myoelectric controllers, i.e., negatively biasing the responsiveness of the SH-P to EMG signals or performance of a given myoelectric controller due to using EMG signals with poor SNR. Our EMG data also suggest that, because of the close correspondence between finger flexor/extensor EMG activity patterns and hand opening–closing (**Figure 8**), residual finger muscles – if available, and if the quality of EMG signal is acceptable – should be targeted for myoelectric control of hand prostheses. This recommendation is consistent with the above goal of enabling individuals with upper-limb loss control, a terminal device, in a way that closely mimics how they would have controlled their own hand, which may contribute to their acceptance of the hand prosthesis.

### **4.3. Defining the Myoelectric Controller for Optimal Reliability and SH-P Performance**

Experiment 3 compared the effect of three EMG mapping algorithms for controlling SH-P finger motion during reach-to-grasp. This evaluation was based on the extraction of EMG signals from subjects using their native hand (as done for Experiment 2), while the SH-P was fixed in the proximity to a sensorized object. We found that the EMG-differential (**Figure 3**) was the most reliable algorithm for activating SH-P finger motion (59% of trials; **Table 1**). However, analysis of similarity of SH-P and native hand kinematics revealed that the EMG-FCFS-Advanced algorithm elicited the best performance (**Table 2**). As no significant statistical difference was found across the three myoelectric controller algorithms, and taking into account the greater ability of activating SH-P finger motion, we conclude that the EMGdifferential algorithm should be used for future studies of SH-P prosthetic applications.

The superiority of the EMG-differential algorithm over the other two algorithms can be explained taking into account the criteria underlying each EMG-to-position algorithm. The EMG-FCFS and the EMG-FCFS-Advanced algorithms both rely on the existence of an EMG threshold. Specifically, the threshold is used to manage the signals by choosing the leading one and the direction of the movement. However, this threshold cannot be adjusted while operating the SH-P to guarantee the correct functioning of the algorithm. For this work, we selected the thresholds after analyzing previous recorded EMG signals path (during Experiment 2) so as to permit to the SH-P to move also in presence of weak EMG signals. The choice of a lower threshold would have resulted negative effects due to the noise recorded by the sEMG electrodes. This feature can account for the very low percentage of SH-P activations, since the signals generated during normal hand use did not result always strong enough to exceed the threshold. On the contrary, the EMG-differential control scheme does not use an EMG threshold. Therefore, any signal, even if it is small, could activate SH-P finger movements.

With regard to the EMG-differential, although this algorithm was characterized by the highest SH-P activation rate, it was not close to 100%. We believe that this is due to the fact that EMG signals were recorded during a natural reach-to-grasp movement with the native hand, rather than subjects attempting to move the SH-P through EMG signals. Furthermore, subjects were prevented from viewing the SH-P during the task and were, therefore, unaware of whether or how the SH-P moved. This was done to further remove the potential confound of EMG signals adapting to visual feedback of SH-P performance. Whereas this is a desirable phenomenon in terminal device training, for the purpose of the study, we had to remove this potential confound to focus on the ability of natural EMG activation patterns to elicit SH-P movements. Therefore, we believe that the percentage of trials characterized by SH-P activation could significantly increase by having subjects adapt their EMG activation patterns to viewing the SH-P during reach-to-grasp tasks.

Lastly, we also examined the effect of P-gain modulation on myoelectric controller performance as it can influence movement velocity, i.e., increasing the proportional gain of the PID controller of the SH-P could allow for a better trajectory tracking. We found that EMG drive P-gain modulation had no effect on the similarity between SH-P and native hand kinematics. Therefore, this technique could be used to enable other SH-P features, such as impedance control of the hand during interaction with objects (Ajoudani et al., 2014), without affecting the similarity with human finger kinematics.

## **5. CONCLUSION**

Our results support the feasibility and potential of the SH-P as an effective hand prosthesis for individuals with upper-limb loss. Specifically, we found that the mechanical design of the SoftHand-Pro, combined with using EMG from finger muscles through the appropriate EMG myocontroller algorithm (EMG differential), enables a reliable motion of all SH-P digits through EMG recorded from only two finger muscles. A particularly encouraging result, which we believe is unique in the literature of hand prostheses, is that EMG-driven motion of SH-P fingers is very similar to native hand kinematics. This is important because, given the high abandonment rate of upper-limb prostheses, the movement and design anthropomorphism of the SH-P may contribute to greater acceptance of prostheses by individuals with upper-limb loss. We should also point out that the SH-P anthropomorphism offers great potential for using the SH-P as a novel assistive and sensorimotor rehabilitation device for individuals affected by neurological disorders, e.g., using the SH-P as a supernumerary limb (Prattichizzo et al., 2014).

While this work focuses on the control of SH-P, future work will address the usage of multi-EMG inputs [see Santello et al. (2016), for a review on this topic] for the control of multiple degrees of actuation, which were already implemented in a new purely robotic version of the SH (Della Santina et al., 2015).

### **AUTHOR CONTRIBUTIONS**

SF, MB, AB, and MS designed the study. GG implemented the controllers of the SoftHand Pro. SF, MB, SJ, JN, and SB performed the experiments and data analysis. All authors contributed to writing the manuscript.

### **ACKNOWLEDGMENTS**

The authors would like to thank Jose Michael Delgado Barbosa for his support in the code development, Dr. Qiushi Fu for his input on the experimental protocol design, Keivan Mojtahedi

### **REFERENCES**


and Dr. Alycia Gailey for their help with data collection, and Manuel Giuseppe Catalano and Marco Gabiccini for their help with hardware development and data analysis, respectively.

### **FUNDING**

This work was partially supported by the Grainger Foundation, the European Research Council under the Advanced Grant Soft-Hands "A Theory of Soft Synergies for a New Generation of Artificial Hands" (No. ERC-291166), the EU H2020 projects "SoftPro: Synergy-based Open-source Foundations and Technologies for Prosthetics and Rehabilitation" (No. 688857), the EU FP7 project (No. 601165) "WEARable HAPtics for Humans and Robots (WEARHAP)," and the Eunice Kennedy Shriver National Institute of Child Health and Human Development of the National Institutes of Health (NIH) under Award Number R21HD081938. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2016 Fani, Bianchi, Jain, Pimenta Neto, Boege, Grioli, Bicchi and Santello. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Is the Prosthetic Homologue Necessary for Embodiment?

Chelsea Dornfeld1, 2, Michelle Swanston1, 3, Joseph Cassella1, 4, Casey Beasley 1, 5 , Jacob Green1, 6, Yonatan Moshayev 1, 7 and Michael Wininger 1, 8, 9 \*

<sup>1</sup> Prosthetics and Orthotics Program, University of Hartford, West Hartford, CT, USA, <sup>2</sup> ABC Prosthetics and Orthotics, Orlando, FL, USA, <sup>3</sup> Mountain Orthotic and Prosthetic Services, Lake Placid, NY, USA, <sup>4</sup> New England Orthotics and Prosthetics Systems, Branford, CT, USA, <sup>5</sup> Hanger Clinic, East Syracuse, NY, USA, <sup>6</sup> Hanger Clinic, Somersworth, NH, USA, <sup>7</sup> Orthocraft, Inc., Brooklyn, NY, USA, <sup>8</sup> Department of Biostatistics, Yale School of Public Health, Yale University, New Haven, CT, USA, <sup>9</sup> Cooperative Studies Program, Department of Veterans Affairs, West Haven, CT, USA

Embodiment is the process by which patients with limb loss come to accept their peripheral device as a natural extension of self. However, there is little guidance as to how exacting the prosthesis must be in order for embodiment to take place: is it necessary for the prosthetic hand to look just like the absent hand? Here, we describe a protocol for testing whether an individual would select a hand that looks like their own from among a selection of five hands, and whether the hand selection (regardless of homology) is consistent across multiple exposures to the same (but reordered) set of candidate hands. Pilot results using healthy volunteers reveals that hand selection is only modestly consistent, and that selection of the prosthetic homologue is atypical (61 of 192 total exposures). Our protocol can be executed in minutes, and makes use of readily available equipment and softwares. We present both a face-to-face and a virtual protocol, for maximum flexibility of implementation.

#### Edited by:

Ricardo Chavarriaga, École Polytechnique Fédérale de Lausanne, Switzerland

#### Reviewed by:

Zhong Jian, University of California, Davis, USA Elisa Canzoneri, École Polytechnique Fédérale de Lausanne, Switzerland

\*Correspondence:

Michael Wininger wininger@hartford.edu

Received: 08 July 2016 Accepted: 30 November 2016 Published: 20 December 2016

#### Citation:

Dornfeld C, Swanston M, Cassella J, Beasley C, Green J, Moshayev Y and Wininger M (2016) Is the Prosthetic Homologue Necessary for Embodiment? Front. Neurorobot. 10:21. doi: 10.3389/fnbot.2016.00021 Keywords: prosthetics, embodiment, aesthetics, human-machine interface, design

### INTRODUCTION

For those with limb loss, prosthetic technology is the target intervention for restoring quality of life (Murray, 2008). However, notwithstanding the ever-improving functionality—and despite the high cost (Zuniga et al., 2015) of an investment in a prosthetic limb—, device rejection rates remain formidable: typical reports are between 20 and 30%, and as high as 50% in some patient populations (Postema et al., 1999; Datta et al., 2004; Biddiss E. A. and Chau T. T., 2007; Biddiss E. and Chau T., 2007; Castellini et al., 2014). Cosmetic appeal, in particular, is considered a design factor of critical importance (Roeschlein and Domholdt, 1989; Gaine et al., 1997). But "cosmetics" is a vague notion, with little extant research to guide future prosthesis design. While it is well-known that color-matching to a patient's skin tone leads to greater device satisfaction (Derwentwood, 1917; Brown, 1947; Weinberg and De Hinrichs, 1952; Bryson, 1965; Newell et al., 1974; Pillet, 1983; Pohjolainen et al., 1990; Campbell et al., 1992; Pereira et al., 1996; Leow et al., 2006; Ehrsson et al., 2008; Kini et al., 2010), there is as yet only a few studies describing the importance of form factor on embodiment (Lamb, 2004; Kini et al., 2010; Jaidev et al., 2013; Raghu et al., 2013; Kamble et al., 2014).

Here, we ask a question that has hitherto not been asked: how "self-like" must a prosthetic hand be in order to be acceptable to the user? The context for the question is as follows: prosthetic hands are expensive and disused for reasons related to lack of embodiment, and cosmetic technology has progressed to where a skilled technologist can make an identical replica of absent anatomy (Altman, 2016), but typically at the

cost of intensive resource investment. Thus, it is incumbent to ask: how exacting must the prosthesis be? We propose a user-preference survey as a platform for testing the need for a prosthetic homologue.

Our protocol builds on previous studies on prosthetic aesthetics, notably lines of inquiry into the uncanny valley (Gee et al., 2005; Cabibihan et al., 2006; Poliakoff et al., 2013), and user preference in cosmetic appeal (Millstein et al., 1986; Carrozza et al., 2005; Kargov et al., 2007; Dalley et al., 2009), and in particular, we believe we put into an empirical framework one of the most-oft recognized design priorities in prosthetic manufacture (Biddiss E. A. and Chau T. T., 2007; Biddiss E. and Chau T., 2007). The protocol presented here in extends naturally on these studies, in doing so, provides an answer to a critical question: given a set of candidate hands which are all equally lifelike, would the user manifest preference for their own hand, and would their preferences be consistent? Whereas these other protocols present hands that of variously life-like or not-so-life-like character—and the degree of lifelikeness is a matter of subjectivity—, our protocol presents only images collected from real humans, which are therefore inherently life-like. This protocol best suits those investigators whose objective is to measure the (putative) importance of prosthetic homology to upper limb prosthetic patients. This protocol is designed to test the following hypotheses: (1) whether individuals show consistent preferences for hand designs, and (2) whether individuals show preferences for hand designs that are homologous to their own hand. Here, we present a full protocol and preliminary results from pilot testing in our laboratory.

### MATERIALS AND EQUIPMENT

### Face-To-Face

Implementation of this protocol in the face-to-face setting will require the following materials:

	- i. An image manipulation software: Adobe Photoshop, GNU Image Manipulation Program (GIMP), Corel PaintShop or similar. The purpose of this software is to create a binarized image of the hand based on the photo, to standardize the image size, and to count the proportion of black pixels ("pixel count").
	- ii. A spreadsheet software: Microsoft Excel, OpenOffice Calc, GoogleDocs Spreadsheet, or similar. The purpose of this software is to create a convenient reference list for sorting images by pixel count.
	- iii. A document or presentation design software: Microsoft PowerPoint, Microsoft Word, or similar. The purpose of this software is to create a series of hand line-ups for the subject to review.

We note that for experienced programmers, this suite of softwares can be obviated by use of a numerical computing environment: Matlab, Octave, R or similar.

	- i. Hand preference data: Most likely a single-page is sufficient; the purpose of this survey is to document which hands were selected by the patient.
	- ii. Demographic and psychological profile: Key variables to collect include age, sex, hand dominance, and history of hand, or arm pathologies. We recommend that some measure of anxiety or body image be used. Particularly well-regarded instruments include the Brief Fear of Negative Evaluation Scale (Leary, 1983), and the Appearance Schemas Inventory: Personal Opinions Questionnaire (Cash and Labarge, 1996). It may also be desirable to ask direct questions about the importance of aesthetics, or design priorities in the hypothetical scenario where the subject needs a prosthetic hand.

### Virtual

Implementation of this protocol in the virtual setting will require the following materials:

	- i. Investigator: Most likely this be a laptop or desktop. Softwares will be needed as described above (Face-to-Face, Item 1).
	- ii. Subject: This can be either a computer or a handheld device, e.g., smartphone or tablet. Most likely, no

additional softwares will be needed. It is important to reduce the software requirements on the subject so as to make the study accessible to a wide participant pool.

We note that both parties will need access to a mutually compatible teleconferencing software, e.g., Skype, Webex, GoToMeeting or similar.


## STEPWISE PROCEDURES

### Face-To-Face

	- i. The subject should see only a chair and the screen.
	- ii. Behind the screen should be the light box, the decoys, and the camera.

blinding; all details will be revealed at the conclusion of the session.

	- i. Ask them which hand is their dominant hand.
	- ii. Place the decoys near to their non-dominant hand and ask them to feel around until they can find the decoys.
	- iii. Investigator should spread the fingers to conform to the template of the hands in the hand bank.

∗Steps ii and iii should be performed simultaneously: Step ii is intended to distract the subject from Step iii.

	- i. Select by pixel count.
	- ii. Select two hands with greater pixel count and two with lesser pixel count.

hand images. Please identify the hand that you find most aesthetically pleasing in each lineup. In particular, we want to know: if you had to receive a prosthetic hand, which hand would you most like your prosthesis to look like?" Note all six choices.


### Virtual


## ANTICIPATED RESULTS

### Image Bank

We have recruited 20 volunteers for our development of our image bank. We exhibit representative samples of these banked images in **Figure 2**. Following our Supplement, each image required ∼90 s to convert from raw image to binarized image with noted pixel count.

### Preliminary Results

We have pilot-tested our Face-to-Face protocol on 32 subjects. Our population comprised healthy volunteers with completely intact anatomy: 13 M/19 F, 26 ± 11 years (range: 19–53 years), all right-hand dominant (with one ambidextrous subject), with an average score on the Brief Fear of Negative Evaluation Score (BFNES) of 33 ± 6 (range: 21–39; range: 0, not at all fearful–60, maximally fearful). Homology scores, i.e., the number of times the subject selected their own hand as the most appealing (max = 6) are shown in **Figure 4**; the mode value was zero selections of one's own hand (**Figure 4A**). Consistency scores, i.e., the number of times the hands selected in lineup pairs were the same (max = 3) were as follows: 1 (n = 16), 2 (n = 10), and 3 (n = 6; **Figure 4B**); all subjects had at least on consistent matching. We note that among those slide-pairs showing consistency within the pair, four subjects (12.5%) selected their own hand in all

three pairs, while 20 subjects with consistent pairings consistently picked a hand that was not their own (**Figure 4C**).

Preliminary hypothesis testing was performed to assess selection of one's own hand, consistency, and the rate of consistent selection of one's own hand, both according to sex, and score on the BFNES, dichotomized about the median. Due to the non-normality of the data, non-parametric tests were used. None of these analyses revealed significant differences between groups (**Figures 5**, **6**). Given that the frequency selection of one's own hand from the hand bank is low, and does not suggest systematic trends by the variables collected in our pilot study, we draw the inference that the prosthetic homologue may not be a particularly important target in custom prosthesis design for all users. However, there will likely be a sub-group of patients for whom a prosthetic homologue is important, though it is not evident that these patients are readily identifiable by sex or BFNES; our pilot data set is insufficiently powered to test for interaction effects.

## NOTES

We make some notes of relevance to the protocol procedures:

1. It is very important that all hand silhouettes used in this study have the same approximate posture; we recommend fingers spread comfortably: sub-maximally, but so that each finger can be seen for its true shape. In our own work, we target angular separation between thumb and little finger of 70◦–90◦ , with middle finger bisecting and parallel to the radius and ulna (**Figure 7**).


FIGURE 5 | Breakout of selection of one's own hand (A), consistency within slide pairs (B) and rates of one's own hand appearing in matched slide pairs (C), breakout by sex.

Hawken, 1996; Papathomas, 2007). Thus, we re-emphasize the importance of scaling images appropriately so that it is the shape of the hand, and not the imprint or size of the hand that drives the subject's selection.

4. Continuing on Note #3, we designated pixel count as the single parameter best poised to match the participant's hand to the hand bank, based on two justifications: physiological, and practical. As described above, the amount of black versus white content in a binarized image will have substantial impact on the viewer's attention; given this, we wanted to avoid biasing participants to picking hands with extremely high pixel counts: rather, we concluded that by selecting images with similar pixel counts, we could eliminate this source of variance, increasing the likelihood of detecting preferences associated with hand shape. Regarding pragmatics, we could have extracted features from the images, e.g., based on hand contour or finger spread, etc., but discarded these designs on the basis that their complexity would inhibit adoption of this study by a wide range of researchers; by selecting pixel count—a feature that is easily extracted from any image without sophisticated image processing steps—we hope to make this protocol more accessible among the diverse community of investigators with interest in aesthetics and prosthetic design.


the Virtual experiment truly compromises the subject naïveté remains to be tested, but must be recognized as a possible source of bias. Creative use of decoys will most likely be helpful here.

7. This hand bank is prepared prior to opening the study to enrollment so that there are an ample set of candidate hands to present to first participant. We propose soliciting volunteers to allow their hands to be photographed, and to binarize these images, saving them in a folder for accessing during the data collection phase. We suggest that this pre-collection phase of hand bank creation will give the study critical opportunity to refine their image binarization technique. Over time, our team was able to reduce the time required to binarize an image from 5 min to <30 s; this is a valuable time savings in terms of maintaining participant focus. The reduction in time burden is attributable primarily to improved acquaintance with the routine for binarization; familiarity greatly improves efficiency. We refer the reader to our Supplementalary Materials for step-by-step procedures for image binarization, and recommend MS PowerPoint or equivalent for creating the lineups.

Regarding the implications of this protocol: we firstly recognize the limitations of using a 2-dimensional test (selection of images) for an inherently 3-D problem, i.e., embodiment of a prosthetic device. However, we believe that is a useful first line of inquiry into the necessity and sufficiency of a design criterion based on homology between the prosthetic device and the anatomy that it replaces, and that this is an attractive paradigm than the alternative: 3-D printing a hand for each subject.

It is prudent to discuss two terminologies used here: embodiment, and the homologue. Embodiment is generally defined as the integration of an artificial limb into one's own body schema, i.e., the fusion of body and perception (Mulvey et al., 2009). This concept is often invoked in the context of someone asked whether they are able to "make it feel like me." While our protocol does not utilize actual prostheses, we believe that our prompt to the participants ("Which hand would you most like your prosthesis to look like") brings our study comfortably into the realm of embodiment. Further, in this work, we are coining the term prosthetic homologue, which has heretofore not been described in any scientific literature known to the authors or their collaborators. Our use of the term is meant to pair the concept of similarity (to one's own anatomy; the "homologue") to anatomical replacement (the prosthesis). We note the concept of homology is occasionally referred to in the literature related to the uncanny valley (Chaminade, 2006; Tondu, 2012; Kaerlein, 2015), but is used in a way that connotes similarity to human anthropomorphism, and not per se any one person's anatomy (or one's own anatomy). In many settings, the prosthetic homologue cannot directly be assessed: a limb-deficient individual has no anatomy from which to draw the comparison; the homologue in this case would be an abstract construct, that can only be approached conceptually, but not tested. However, in healthy persons, while there is no direct opportunity for prosthetic replacement, the anatomy is present, can be measured, and preferences in the hypothetical can be tested.

By presenting pilot data as a preliminary result, we intend to provide investigators with basis for formulating study designs, i.e., requisite sample sizes for pre-specified parameters related to statistical power. This entire protocol can be executed via freewares, i.e., at no expense beyond the base materials; given the ubiquity of digital cameras and web access, we believe that this is accessible protocol that can be readily implemented in a wide variety of settings. Lastly, we note that all data collection activities described in this manuscript were performed under the authorization of the University of Hartford Institutional Review Board, following provision of written informed consent.

### ETHICS STATEMENT

University of Hartford Human Subjects Committee (FWA00003578). All participants signed an informed consent prior to participation in this study.

### AUTHOR CONTRIBUTIONS

MW: Lead investigator, primary study architect. All authors participated in the design of the study, drafting and approval of the manuscript, and take responsibility for its contents.

### ACKNOWLEDGMENTS

The authors gratefully acknowledge the work of Amber Sayer, Steve Sousa, and Chris Welch.

### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fnbot. 2016.00021/full#supplementary-material

### REFERENCES


and L. Fortunati (Cham: Springer International Publishing), 77–88. doi: 10.1007/978-3-319-15672-9\_7


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The reviewer EC and handling Editor declared their shared affiliation, and the handling Editor states that the process nevertheless met the standards of a fair and objective review.

Copyright © 2016 Dornfeld, Swanston, Cassella, Beasley, Green, Moshayev and Wininger. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Alix Chadwell, Laurence Kenney\*, Sibylle Thies, Adam Galpin and John Head*

*Centre for Health Sciences Research, University of Salford, Salford, UK*

Users of myoelectric prostheses can often find them difficult to control. This can lead to passive-use of the device or total rejection, which can have detrimental effects on the contralateral limb due to overuse. Current clinically available prostheses are "open loop" systems, and although considerable effort has been focused on developing biofeedback to "close the loop," there is evidence from laboratory-based studies that other factors, notably improving predictability of response, may be as, if not more, important. Interestingly, despite a large volume of research aimed at improving myoelectric prostheses, it is not currently known which aspect of clinically available systems has the greatest impact on overall functionality and everyday usage. A protocol has, therefore, been designed to assess electromyographic (EMG) skill of the user and predictability of the prosthesis response as significant parts of the control chain, and to relate these to functionality and everyday usage. Here, we present the protocol and results from early pilot work. A set of experiments has been developed. First, to characterize user skill in generating the required level of EMG signal, as well as the speed with which users are able to make the decision to activate the appropriate muscles. Second, to measure unpredictability introduced at the skin–electrode interface, in order to understand the effects of the socket-mounted electrode fit under different loads on the variability of time taken for the prosthetic hand to respond. To evaluate prosthesis user functionality, four different outcome measures are assessed. Using a simple upper limb functional task prosthesis users are assessed for (1) success of task completion, (2) task duration, (3) quality of movement, and (4) gaze behavior. To evaluate everyday usage away from the clinic, the symmetricity of their real-world arm use is assessed using activity monitoring. These methods will later be used to assess a prosthesis user cohort to establish the relative contribution of each control factor to the individual measures of functionality and everyday usage (using multiple regression models). The results will support future researchers, designers, and clinicians in concentrating their efforts on the area that will have the greatest impact on improving prosthesis use.

#### *Edited by:*

*Claudio Castellini, German Aerospace Center, Germany*

#### *Reviewed by:*

*Dennis J. McFarland, Wadsworth Center, USA James Wright, Western Sydney University, Australia*

> *\*Correspondence: Laurence Kenney l.p.j.kenney@salford.ac.uk*

*Received: 13 June 2016 Accepted: 25 July 2016 Published: 22 August 2016*

#### *Citation:*

*Chadwell A, Kenney L, Thies S, Galpin A and Head J (2016) The Reality of Myoelectric Prostheses: Understanding What Makes These Devices Difficult for Some Users to Control. Front. Neurorobot. 10:7. doi: 10.3389/fnbot.2016.00007*

Keywords: prosthesis, myoelectric, activity monitoring, control, upper limb, functionality assessment

## 1. INTRODUCTION

Statistics relating to the prevalence of limb absence and provision of prostheses are poor. However, data from the United States in 2005 show that ~41,000 people were living there with major upper limb absence (Ziegler-Graham et al., 2008), which equates to 1 in 10,000 people. Furthermore, each year in England ~5–6,000 major limb amputations are carried out (NHS Choices, 2014), of which approximately a fifth are undertaken on the upper limb (Lusardi et al., 2013; NHS Scotland, 2014) and most commonly at the trans-radial (forearm) level (UNIPOD – United National Institute for Prothetics and Orthotics Development, 2010/11). In addition, congenital deformities contribute significantly to the number of people living with upper limb absence, although data on prevalence are somewhat inconsistent (Kyberd et al., 1997; UNIPOD – United National Institute for Prothetics and Orthotics Development, 2010/11; Head, 2014).

Three types of upper limb prostheses are available to people with limb absence: cosmetic prostheses, which are primarily designed to restore appearance and symmetry; and functional prostheses, which are either body-powered via the use of a harness and cables, or electrically powered via rechargeable batteries. Electrically powered prostheses, commonly known as myoelectric prostheses, are controlled by electromyographic (EMG) signals created in the residual musculature, which are picked up by electrodes housed within the prosthetic socket. However, despite the potential offered by myoelectric hands, prosthesis users report these devices to be challenging to control (Biddiss and Chau, 2007a; Peerdeman et al., 2011; Head, 2014; Engdahl et al., 2015) and to still be limited in function (Biddiss and Chau, 2007a; Peerdeman et al., 2011). These user reports are supported by the results of clinical assessment tests in which upper limb prosthesis users generally perform at less than 50% of the level of their anatomically intact counterparts (Mathiowetz et al., 1985; Grice et al., 2003; Farrell et al., 2005; Kyberd et al., 2009; Metzger et al., 2010; Bouwsema et al., 2014; Sobuh et al., 2014). Unsurprisingly, passive-use and rejection of myoelectric prostheses have been reported as problems (Biddiss and Chau, 2007b), leading to over-use injuries of the intact limb (Jones and Davidson, 1999; Gambrell, 2008; Østlie et al., 2011).

In response to user feedback, attempts have been made to improve the control of myoelectric prostheses. Since current clinically available devices are "open loop" with respect to the user, promoting reliance on visual feedback, recent advances have frequently focused on providing users with tactile feedback (Antfolk et al., 2013; Kim et al., 2014; Tee et al., 2015; Oddo et al., 2016; Xu et al., 2016). However, Saunders and Vijayakumar (2011) demonstrated that, although the introduction of feedback can improve control of myoelectric prostheses, other characteristics of the prosthesis may be equally, or even more important in determining the ability of the user to control their prosthesis. In their study, participants demonstrated that when using a "perfect" fast-responding prosthesis they were able to demonstrate good levels of control over grip force even in the absence of any feedback; however, in the presence of "*uncertainty*" as to how the hand would react (presented in the form of random delays in prosthesis response time), their control of the prosthetic hand decreased. Saunders concluded that if the central nervous system (CNS) is able to produce accurate predictions of anticipated prosthesis behavior (forward models), then reliance on feedback from the hand is reduced. Saunders and Vijayakumar (2011) also noted that a degree of "*uncertainty*" was an inherent part of myoelectric prosthesis use. This observation was further investigated by Head (2014) who identified that the standard method for housing electrodes in prosthetic sockets can result in EMG signal artifacts, or loss of electrode contact with the skin, leading to "*unpredictability*" in the response of the prosthesis to muscle contractions. Finally, despite recent findings in anatomically intact subjects (Terlaak et al., 2015) challenging the assumption, there is a widely held belief that there is a relationship between the level of skill in producing the required EMG signals and prosthesis control.

In summary, despite technological advances, control of myoelectric prostheses remains challenging, leading to device rejection and associated over-use injuries of the intact limb. Introducing feedback into the system is one possible solution to enhance prosthesis control for improved "*functionality*" and "*everyday usage*," however, research into the different aspects of the prosthesis control chain (e.g., "*unpredictability*" in the system and "*EMG skill*" of the user) may be equally important. Here, we introduce a novel protocol, including purpose-built, portable instrumentation that has been designed for the assessment of these individual factors contributing to feed-forward prosthesis control in relation to aspects of overall upper limb performance. Specifically, the protocol assesses how well a myoelectric user can control their EMG signals ("*EMG skill*") and how reliably the electrodes pick up the signals ("*unpredictability*"). These outcomes are then related to measures reflecting how close the kinematic and gaze patterns of the user are to healthy norms ("*functionality*") during performance of a structured multistage manual task, and how often the myoelectric prosthesis is used in everyday life ("*everyday usage*"). It is important to note the separation of these two performance measures. Literature has shown that an increase in upper limb "*functionality*" as assessed using clinical tests may not necessarily correspond to an equivalent improvement in everyday arm use (Bailey et al., 2015). By comparing the control factors against "*functionality*" and "*usage*," it should be possible to identify which factor(s) has/have the greatest impact on overall performance. Longer-term, researchers, designers, and clinicians can then ensure that their efforts are concentrated on the area(s) that will be of greatest benefit to prosthesis users.

In this paper, we introduce the experimental procedures to characterize key factors contributing to the feed-forward prosthesis control chain, namely skill in controlling the EMG signals ("*EMG skill*") and "*unpredictability*" in transduction of the EMG signal (between the skin and the electrode). We also describe the measures designed to capture the user's overall upper limb performance ("*functionality*" and "*usage*"). Initial results of our pilot work and their discussion are included to demonstrate the feasibility of the protocol. Furthermore, we propose a data analysis method to be used in the main study, which attributes variance in measures of performance to one or more elements of the control chain.

As the protocol is complex and involves the description of several experimental setups, we have kept the detail in the main body of the paper to a minimum and make use, where appropriate, of Supplementary Material.

### 2. METHODS AND ANALYSIS

### 2.1. EMG Skill

The muscle groups used to control the opening and closing of myoelectric hands and their associated neural pathways differ from those used in the anatomical hand (Bongers et al., 2012). It is, therefore, reasonable to assume that opening/closing the hand with this "new" set of muscles in response to a relevant prompt may be less intuitive and require an increase in mental processing time as reflected in an increased response time (Kuiken et al., 2004, 2007). It is also reasonable to assume that practice using this "new" set of muscles to open/close the hand may decrease the response time (Ando et al., 2002; Radhakrishnan et al., 2008).

To establish the mental processing time to activate the "new" set of muscles, the subtractive method developed in the 1860s by Donders (1869) can be used. Donders proposed the use of different types of reaction time tests to establish the time spent undertaking cognitive and motor processes. Such reaction time tests include the simple reaction time (SRT) test in which participants are aware of the required response before stimulus presentation, and the choice reaction time (CRT) test where the stimulus dictates the required response. Donders segmented the response time into the time taken to perceive the stimulus (signal perception), time taken to decide how to respond (decision time), and time taken to activate the neurons (motor response) (**Figure 1**). For the SRT test, there is no decision time as the required response is already known and the person is primed to react. Donders also declared that the time for signal perception and motor response does not vary between the tasks. Consequently, he suggested that subtracting the SRT from the CRT provides information as to the decision time to undertake the CRT test, or in this case, how long it takes the person to decide which muscles to activate to operate the prosthesis ("*Decision Time*"). Accordingly, this study uses reaction times measured under two different conditions to characterize the "*Decision Time*," and associated tests are termed "*Reaction Time Tests*" (see section 2.1.2).

Furthermore, the ability to control the amplitude of the EMG signal using the musculature of the residual limb can be measured through the performance of a series of continuous signal tracking tasks. There are two main types of tracking tasks: static and dynamic. For a static tracking task the subject is required to match their EMG signal to a target amplitude (Alcaide-Aguirre et al., 2013), while a dynamic tracking task involves modulating the amplitude of the EMG signal to match a moving target (Guo et al., 2009; Corbett et al., 2011; Alcaide-Aguirre et al., 2013; Lobo-Prat et al., 2014). Most clinically prescribed myoelectric prostheses are equipped with proportional control, meaning that it is not only important that a user is able to generate a signal strong enough to activate the hand but that they can also modulate the level of the signal to allow for control of the hand speed and the grip force. Dynamic tracking tasks take different forms: some contain a repetitive signal modulation, such as a sinusoidal wave of a set amplitude (Guo et al., 2009), while others vary the amplitude at random (Corbett et al., 2011; Lobo-Prat et al., 2014). For this study, we use a commercially available software package originally designed for the clinical training of myoelectric prosthesis users, which provides us with a means to test user performance in tracking both static and random amplitude modulated targets, using their EMG signal. The approach also allows us to use clinical EMG electrodes (rather than laboratory-standard EMG gel electrodes), thereby reflecting the transduction, signal processing, and amplification used in practice. We term the set of static and dynamic tracking tasks to be "*Tracking Tasks*" (Section 2.1.3).

Details of the number of repeats for each test are provided in **Table 1A**.

#### 2.1.1. Electrode Placement

The "*EMG skill*" analysis tests require an "*ideal*" electrode placement on the residual limb to ensure that the participant is able to perform to their best ability. This "*ideal*" placement requires the electrode to be placed in the optimal location, with the optimal gain and good contact with the skin.

Slight variations exist in the methods used to find the optimal location for the electrodes; for this protocol, we use the methods taught to student prosthetists at the University of Salford. Rather than use the participant's own electrodes, which would necessitate dismantling the prosthetic socket, we use one of two standard electrodes, selected to best match the participant's own type of electrode (Ottobock 13E200 = 50 or RSL Steeper SEA200). Optimal settings for the selected electrode are found using the clinical assessment tool Myoboy® (Ottobock Gmbh). Initially, the gain for each electrode is set at a mid-level of 3–4. Participants are then asked to repeatedly and consistently contract the muscle to a comfortable level. The electrode is initially placed in the center of the muscle bulk and the signal level is noted. The electrode is then moved in each of four directions (up, down, left, and right) from the starting location, by half an electrodes width. If the amplitude

#### TABLE 1A | Protocol summary – tests for the assessment of "*EMG skill*".


#### TABLE 1B | Protocol summary – tests for the assessment of "*Unpredictability*".


#### TABLE 1C | Protocol summary – tests for the assessment of "*Functionality*" and "*Everyday Usage*".


of the signal is greater in any of these new locations, the process is repeated using the new location as the starting point. This is continued until the position with maximum signal amplitude is found, and the location marked using an indelible pencil.

The gain settings are adjusted until the participant is able to comfortably achieve a post-processed signal amplitude, recorded by Myoboy®, between 30 and 60 and separation between the two signals greater than 5.1 To achieve consistent good contact of the electrodes with the skin, they are bandaged in place using elasticated bandages. The difference between the optimal location and gains, and the location and gains for the participant's own prosthesis, is noted.

#### 2.1.2. Reaction Time Tests

For these tests, the "*ideal*" electrode placement (Section 2.1.1) is used to control a MyoHand VariPlus Speed (Ottobock Gmbh).

A schematic of the experimental setup is shown in **Figure 2A**. The participant begins each trial with the prosthetic hand in a neutral position. In front of the participant is a custom-made

<sup>1</sup>The manufacturers do not disclose the details of the scale used to represent signal amplitude, hence, the units are not reported.

reaction time box with two 10-mm red LEDs serving as stimuli for hand opening (top) and closing (bottom), and one 5-mm red LED in their middle to focus the subject's attention at the start of each trial. The anatomical hand is placed on a large blue button situated on the bottom portion of the box. The trial begins when the participant indicates that they are ready by pressing the blue button. Each trial then starts with the 5-mm LED illuminating for 1 s to attract the participant's attention. At a randomly generated time between 2.5 and 3 s (Poliakoff et al., 2013) after the subject pushes the button, one of the 10-mm LEDs will then illuminate for 1 s. Once the 10-mm LED turns on, the participant should then either open (if top LED) or close (if bottom LED) their prosthetic hand in response. For the SRT Test, the subject is aware which LED will illuminate, i.e., which response is required. For the CRT Test, the subject needs to respond with either hand opening or closing, dependent on whether the top or bottom LED is illuminated. For all reaction time tests, an electronic goniometer (Biometrics Ltd) is attached across the proximal knuckle of the index finger to measure the movement of the prosthetic hand, thereby allowing for identification of the onset of hand opening or closing.

Details of the instrumentation used in SRT and CRT trials are shown in **Figure 2B**. The reaction time box and goniometer are controlled via Arduino Leonardo boards (www.arduino.cc) communicating over serial with Matlab (The Mathworks Inc.). The wait time and LED number are sent from Matlab to Arduino1 to start the test. Arduino1 waits for acknowledgment that the user is ready, based on their button press. Arduino1 then initiates recording of the goniometer data on Arduino2 and controls the LEDs on the reaction time box. Matlab then analyzes the goniometer data establishing the reaction time, which is sent back to Arduino1 and displayed to the participant. A T9545 goniometer adaptor (Thought Technology Ltd.) and TT Sensor Isolator ST9405AM (Thought Technology Ltd.) are used to interface between the goniometer and Arduino2.

#### 2.1.3. Tracking Tasks

This test uses commercially available assessment tools from Ottobock Gmbh that are routinely used in clinical care. The Myoboy® hardware is designed to measure the signal from the clinical electrodes and send it to a computer. Using the PAULA (Prosthetist's Assistant for Upper Limb Architecture) software, the signal can be viewed and the user can then undertake activities to train and improve signal control. The "*ideal*" electrode placement (Section 2.1.1) is used with the electrodes connected to the Myoboy® hardware. Two different aspects of the PAULA software are used, one for the "*static tracking task*" and one for the "*dynamic tracking task*."

The "*static tracking task*" uses the myo-testing signal visualization screen (**Figure 3A**). The boundary lines within this screen are adjustable and in this protocol are set to 39 and 51; these values were determined through pilot work as a level that is sufficiently challenging for the more skilled participant, yet somewhat achievable for the least able. The participant is given three contraction attempts to keep their signal amplitude within the boundaries for each muscle. Each contraction is 3 s long from the moment the signal first crosses the lower boundary line. Participants are scored on the percentage of time the signal remains within the boundaries.

The "*dynamic tracking task*," on the other hand, uses the training "car game" within PAULA. The task involves steering a car through gaps in approaching walls that fluctuate in height (**Figure 3B**). The game level is set in the middle of the available options at 5, and the training time is 1 min which during pilot work proved to be long enough that no one achieved a perfect score, without being too long that people who were struggling stopped trying. The height of the car is controlled using the EMG signal; muscle contraction elevates the car on the screen and muscle relaxation drops the car to the bottom of the screen. Beginning with the hand-open signal the participant must steer the car through the approaching gaps that cycle between being high and low (contraction and relaxation). Participants are given two attempts to get the best score they can achieve, defined as the percentage of gaps successfully passed through without crashing (Part 1a). The task is then repeated for the hand-close signal (Part 1b). Finally, the participant must control two cars at once using both signals (Part 2). During part 2, the cars are set up so that when one muscle is contracted the other one should be relaxed, assessing the ability of the participant to separate their signals, while cycling between hand opening and closing.

### 2.2. Effects of Electrode Interface Condition on EMG Signal Transduction

Good electrode contact with the skin is required for reliable transduction of the EMG signal. Prosthesis electrodes (known as myoelectrodes) are "dry" metal electrodes housed in a plastic case; a small gap in the prosthesis socket is designed to house the myoelectrode; two rubber projections extend from each end of the casing, which locate within pre-manufactured slots in the socket walls. Although a surprisingly neglected area, it is established that the design of prosthetic sockets and associated electrode housings can lead to problems in the transduction of the EMG signal. For example, applied load may cause the socket to move relative to the residual limb and, hence, produce signal artifacts, or electrode contact may be lost altogether (Head, 2014). Furthermore, it is possible that re-donning of the socket may lead to the electrodes moving from the optimal location (see Electrode Placement), leading to crosstalk from other muscles. These factors constitute "*unpredictability*" in the transduction of the EMG signal, leading to "*uncertainty*" as to the response of the prosthetic hand to neural commands.

Our protocol builds on previous work in this area (Head, 2014) to assess two key aspects of "*uncertainty*": (1) whether the hand responds when the user desires it and (2) whether the hand activates unexpectedly. Specifically, to assess the impact of the socket-housed electrode fit on these two "*uncertainty*" measures, participants complete a set of tasks with the forearm held at two different angles, under three electrode interface conditions (**Figure 4**): (1) "*Ideal*" – the electrodes are placed in the optimal position on the residual limb and held in place using elastic bandage as in Section "2.1.1" (**Figure 4A**). The electrodes are connected to the MyoHand VariPlus Speed (Ottobock Gmbh) as in Section "2.1.2," which is sat on the table top; using this method, there should be minimal or no movement of the electrodes in relation to the skin. (2) "*Normal*" – the prosthesis is worn as normal, and the electrodes are housed in the prosthetic socket (**Figure 4B**). From this part of the study onward, the participant uses their own prosthesis with the electrode location and gain settings which they would use in everyday life. (3) "*Additional load*" – the prosthesis is worn as normal; however, an additional 500 g load is strapped to the hand to simulate the weight of an object, such as a full jar (**Figure 4C**).

Tasks are undertaken with the arm in postures that are representative of those encountered during daily activities, such as reaching to a shelf, or down into a drawer, corresponding to ~45° above and below the horizontal. Forearm angles from the horizontal are measured using an inertial measurement unit (IMU). For this study, an Xsens MTw sensor (Xsens Technologies B.V.) is used. The IMU is placed on the back of the wrist just proximal

to the ulnar styloid. The *x*-axis is aligned along the forearm axis pointing toward the hand. For our pilot work, a proprietary algorithm was used, which outputs orientation components based on Euler angles (XYZ earth fixed); however, in the longer-term, this will be replaced with an algorithm that calculates the orientation of the *x*-axis relative to gravity (Sun et al., in press).

The set of tasks performed at each of the two arm orientations, for each of the three electrode interface conditions are described in the following two sections.

#### 2.2.1. Reaction Time Tests

The impact of the electrode interface conditions on variability in reaction times is assessed using the equipment described in 2.1.2 above. Participants begin with the "*ideal*" electrode interface; the simple reaction time (SRT) test is undertaken at each of the two arm postures. The test is then repeated for the other two interface conditions ("*Normal*," "*Additional Load*") at each of the two arm postures. The number of repeats is detailed in **Table 1B**. The spread in reaction times is compared across the electrode interface conditions, between the "*ideal*" interface and the two socket-housed electrode conditions ("*Normal*," "*Additional Load*").

#### 2.2.2. Transitioning between Arm Postures

Transitions from one posture to another may, in the case of a poor fitting socket, cause an EMG artifact and, hence, cause the

prosthetic hand to open or close when the user does not desire it (Head, 2014). Such an event could lead to the user dropping or squashing an object. Therefore, between each set of "*reaction time tests*" (see Section 2.2.1), prosthetic hand posture is recorded as the arm moves between the two arm postures. The hand begins each transition either completely open or completely closed, and prosthetic hand posture is recorded throughout the transition using the goniometer (see Section 2.1.2); any undesired activation, i.e., opening or closing of the hand is recorded.

### 2.3. Functionality Assessment

Upper limb prosthesis user functionality is typically appraised using an appropriate, validated assessment tool, such as the Southampton Hand Assessment Procedure (SHAP) (Light et al., 2002). In common with a number of other clinical tests, functionality is evaluated on the time taken to successfully complete specific tasks. Faster completion times are assumed to correspond to higher levels of functionality. Although the duration of task performance is one measure of functionality, it provides no information on how tasks are completed. A number of studies have shown that combining several outcome measures provides a more complete picture of the functional abilities of prosthesis users (Hill et al., 2009; Wright, 2009; Lindner et al., 2010; Bouwsema et al., 2012). Kinematic outcome measures and gaze behavior can be recorded during the performance of multistage tasks to provide information that complements speed of performance measures. It has previously been shown that performance characterized using these measures clearly differentiates amputees from anatomically intact controls (Bouwsema et al., 2012; Sobuh et al., 2014).

When faced with a novel task, young children are known to try a number of different movement trajectories, allowing the CNS to build a representation of the optimum trajectory (Schneiberg et al., 2002). When faced with structured multistage manual upper limb tasks, novice prosthesis users have been shown to demonstrate similar trends (Major et al., 2014). During the first few task attempts, variability in the linear acceleration patterns of the forearm is high; however, after practice with the prosthesis, variability has been shown to decrease (Sobuh, 2012). Moreover, Bouwsema et al. (2010) demonstrated that prosthesis users demonstrate a later onset of hand opening during reach-to-grasp movements than anatomically intact subjects, and a plateau in the hand aperture between opening and closing around the object.

Furthermore, previous studies undertaken by Bouwsema et al. (2012) and Sobuh et al. (2014) have shown that the gaze behavior of inexperienced prosthesis users differs from that of anatomically intact controls, however, with practice gaze patterns approach those of controls. A more functional user would be expected to demonstrate a larger number of "look-ahead-fixations" and spend less time concentrating on the prosthetic hand. In a multistage task, "look-ahead-fixations" involve gaze fixation on an area of the task critical to a future task component (such as looking at the object to be grasped, or the location it will be moved to while completing the reach, rather than concentrating on the hand). Fewer transitions between areas of interest (AOIs, e.g., hand, grasp area of the target) would also be expected. Interestingly, participants who self-report rarely using their devices in everyday life have been shown to demonstrate more gaze transitions, irrespective of their functional ability with the device (Bouwsema et al., 2012). Prosthesis users are reliant on visual feedback, as such it would be expected that patterns in gaze behavior may be related to a person's knowledge as to how their hand will respond. If a participant cannot accurately predict the response of their prosthesis, it is possible that this will be reflected in the number of gaze transitions.

We, therefore, assess participants' performance with their prosthesis using a structured multistage manual task, which involves the reaching for, grasping, then placing and releasing of a cylinder in a tube. Three levels of task difficulty are available to the participants (as described below). Performance is then characterized based on number of successfully completed trials and task difficulty, time to complete the task, aperture onset delay, plateau time during reach-to-grasp, kinematic variability in movements (using accelerometry) and gaze behavior over successive trials.

#### 2.3.1. Task Design

Previous work has suggested that certain movements are prone to cause users with poor fitting sockets particular difficulties in prosthesis control, possibly as a result of artifacts caused by electrode movement in relation to the skin or separation from the skin (Head, 2014). These include movements that would be achieved through "pronation" or "supination" in anatomically intact participants. A set of three multistage unilateral tasks ("*cylinder tasks*") have been developed (termed "*tasks A–C*"), the harder of which ("*tasks B and C*") encompass these movements and, hence, present a significant challenge to some participants. Each participant attempts 10 trials (**Table 1C**) of 2 of the 3 tasks, as follows. All participants begin with the medium difficulty level ("*task B*"). Using the prosthesis, participants reach to grasp a cylinder (dia. 52 mm, length 200 mm, weight ~350 g), lift and rotate it through 90° to the horizontal, place it into a horizontally orientated tube (inner dia. 64 mm, length 100 mm), and then release it returning their hand to the starting position. Participants who have a prosthesis with a wrist rotator are asked not to use this function during completion of any of the manual tasks. If the participant is successful in completing over 80% of the trials without dropping the cylinder, they move to "*task C*" in which the tolerance between the cylinder (dia. 52 mm, length 200 mm, weight ~350 g) and the tube (inner dia. 58 mm, length 100 mm) is reduced. If they are unsuccessful in completing 80% of the medium difficulty trials ("*task B*") they perform the easier task ("*task A*"), in which the cylinder is placed vertically into a vertically orientated target tube with the same dimensions as "*task B*" (inner dia. 64 mm, length 100 mm).

As before, participants wear sensors allowing kinematics to be assessed and an eye tracker to record gaze behavior. IMUs (Xsens MTw) are worn on the wrist of the prosthesis and on the chest,2 an electronic Goniometer (Biometrics Ltd.) is worn across the proximal knuckle of the index finger, and participants wear a Dikablis Professional Eye Tracker system (Ergoneers). All data are sampled at 50 Hz. The three systems are synchronized using an arcade style button.

#### 2.3.2. Performance Evaluation

To score performance, the task is split into five movement components and the participants gain points for each section they complete successfully. Points are gained for successful completion first time of (1) reach-to-grasp, (2) lifting and rotating the cylinder, (3) placing the cylinder all the way into the tube, (4) releasing the cylinder, and (5) returning to the start position. Half points are allocated if the movement component was completed after a second attempt (provided the cylinder was not dropped or knocked over). Additional points are added to the total to reflect the level of difficulty the subject was able to perform the task at (0, easy task "*A*"; 1, medium task "*B*"; 2, difficult task "*C*"). The participant starts and ends the movement with their hand on an arcade style button from which timestamps are generated, allowing for task duration to be calculated.

#### 2.3.3. Quality of Movement

Quality of movement encompasses both the pattern of hand aperture during reach and the movement variability throughout the task. It is possible to determine the end of the reach phase by analyzing the goniometer data. When the task begins, the hand is completely closed; the hand then opens, before closing again around the cylinder to generate a "*transport plateau*" as the object is transported. It has been shown that prosthesis users demonstrate a delay in opening the hand at the start of reach, demonstrated by an "*onset plateau*," and a delay between opening and closing the hand, termed the "*reach plateau*" (Bouwsema et al., 2010). The start of the "*transport plateau*" is taken as the end of the "reach-to-grasp" component of the task. By segmenting the "reach-to-grasp" component of the task, the delay in onset of hand movement (the length of the "*onset plateau*") is calculated as a percentage of the "reach-to-grasp," and the length of the "*reach plateau*" is calculated as a percentage of the "reach-to-grasp." Furthermore, using the wrist-mounted IMU, the kinematic variability in the linear acceleration of the forearm between tasks is assessed using the methods developed by Thies et al. (2009).

#### 2.3.4. Gaze Behavior

For the purposes of analyzing the eye tracking videos, the task area is split into six AOIs: (1) start point (button), (2) prosthetic hand, (3) "grasp critical" area (GCA) (bottom half of the cylinder for "*tasks A and B*," top half for "*task C*"), (4) other "location critical" half of the cylinder (LCA) that is required to be placed into the tube, (5) tube, and (6) LED. The percentage of time spent looking at each AOI is calculated, alongside the number of times that the gaze location transitions between each of these areas. Finally, the percentage of time spent looking at areas of the task relevant to subsequent components of the task ("look-ahead-fixations") is calculated for each point in the task (e.g., the cylinder and tube during "reach-to-grasp," or the tube during manipulation and transport).

### 2.4. Everyday Usage

Current methods of quantifying everyday prosthesis use involve self-report (Roeschlein and Domholdt, 1989; Sherman, 1999; Gallagher and MacLachlan, 2000; Raichle et al., 2008), which is known to be prone to recall and bias errors (Metcalf et al., 2007; Brown and Werner, 2008). Accelerometer-based activity monitoring (Noorkõiv et al., 2014) provides an opportunity to observe actual prosthesis use outside of the clinical environment; however, to date no studies have been published on a cohort of upper limb prosthesis users. We have adapted a protocol developed for stroke patients (Bailey et al., 2015). This research

<sup>2</sup>The trunk sensor is used for setting up the cylinder tasks; the distance of the cylinder from the resting hand position should allow the participant to reach the cylinder without leaning forwards. The trunk sensor will also record trunk compensatory movements during task performance.

involved participants wearing an activity monitor (Actigraph GT3X+) on each of their wrists while they went about their normal daily activities. The Actigraph monitors provide continuous logging of raw accelerometer data (sampled at 30 Hz). The data are downloaded using proprietary software, filtered and down sampled to 1 Hz. The processed data are expressed as activity counts (0.001664 g/count) (Actigraph Corp, 2015), which are converted into vector magnitudes (sum of the counts along each axis *x y z* 2 2 <sup>2</sup> + + ). For each second of data, Bailey et al. (2015) combined the vector magnitudes from each of the two wrist worn monitors (dominant and non-dominant arm) to inform on the magnitude of activity across both arms, expressed as the "*bilateral magnitude*" (VMDom + VMNonDom), and the contribution of each arm to the activity, expressed as the "*magnitude ratio*" [ln(VMNonDom + VMDom)].

Bailey found that in healthy, anatomically intact controls, the median "*magnitude ratio*" was around zero (symmetrical bilateral arm use); however, in the stroke cohort, the "*magnitude ratio*" was skewed toward unilateral Non-Paretic (unaffected) arm use. In general, participants in the stroke cohort who demonstrated higher levels of functionality [according to the Action Research Arm Test (McDonnell, 2008)] also demonstrated "*magnitude ratios*" closer to those of the healthy control subjects; nevertheless, a third of participants demonstrated a median "*magnitude ratio*" representing unilateral non-paretic arm use, regardless of their functionality with the paretic arm.

For our study, the activity monitors are placed on the anatomical wrist and the wrist of the myoelectric prosthesis. The monitor is not transferred to other prostheses the participant may wear (e.g., body-powered), as only the times when the myoelectric prosthesis is in use are of interest to this study. Participants are invited to wear the monitors for 1 week.

## 2.5. Pilot Study

### 2.5.1. Recruitment

The purpose of this pilot study was to assess the robustness and feasibility of the protocol before undertaking the main study with a cohort of myoelectric prosthesis users. Ethical approval was granted by the University of Salford School of Health Sciences Research Ethics committee (*REF: HSCR 15-130*) to pilot the above protocol with anatomically intact subjects using a prosthesis simulator designed to fit over their intact arm (**Figure 5**), and myoelectric prosthesis users recruited from the University of Salford Prosthetics and Orthotics Professional Patient Database. Inclusion criteria for the latter were (1) an amputation or congenital limb loss at the trans-radial level, (2) owning a myoelectric prosthesis, and (3) over 18 years of age. Exclusion criteria were (1) bilateral limb loss, (2) injury to the residual limb at the time of testing, and (3) using single site muscle control.

#### 2.5.2. Data Analysis

### *2.5.2.1. Factors Affecting Prosthesis Control*

As described above (see Sections 2.1 and 2.2), "*EMG skill*" and "*uncertainty*" both affect control of the prosthesis. Multiple variables are generated as part of this protocol that characterize these factors, and which must be combined into overall scores for skill in controlling the EMG signals ("*EMG Skill Score*"), and "*uncertainty*" introduced by the electrode interface ("*Uncertainty Score*"). For this reason, the pilot study data were reduced to ordinal data.

Specifically, the "*EMG Skill Score*" is devised of the reaction time difference between the choice and simple reaction times (termed "*Decision Time*," see Section 2.1), and the scores from the "*Tracking Tasks*." To ensure that the reaction times reported were not biased by early or late reactions, any responses faster than 100 ms or slower than 1000 ms were excluded from the analysis (Press et al., 2005). When combining the three scores contributing toward "*EMG Skill*," the "*Dynamic Tracking Task*" score was given a higher weighting, since accurately proportionally controlling a dynamic and noisy signal poses a greater challenge than the "*Reaction Time Tests*" or the "*Static Tracking Task*."

A combined score for the "*uncertainty*" introduced by the electrode interface must also be generated ("*Uncertainty Score*"). This is an ordinal score based on the reaction time spread across the conditions highlighted in Section 2.2.1 and the number of unwanted activations during the transitions (see Section 2.2.2).

### *2.5.2.2. Prosthesis Functionality and Everyday Usage*

The variables characterizing prosthesis "*functionality*" and "*usage*" were split into four key areas as described in Sections "2.3" and "2.4." Of the three possible functional tasks (easy "*A*," medium "*B*," and hard "*C*"), all participants attempted two that were analyzed independently.

Initially, a basic performance evaluation was undertaken. A "*performance score*" was generated based on the scores for the completed movement components of the task (see section 2.3.2) and the task duration (in seconds). Participants were penalized up to a total of 1 s for movement components they failed to complete successfully.

For all trials where the participant completed the "reachto-grasp" component of the task, the hand aperture profile was analyzed to establish the percentage of reach consumed by the "*reach plateau*" period and the "*delay plateau*." Using the methods developed by Thies et al. (2009), variability in the linear acceleration of the forearm was also calculated, for both the "reach-tograsp" component and the full task.

Analysis of the eye tracking data used a coding scheme to record the AOIs on which the gaze was concentrated for every frame, allowing for the time spent in each AOI and the number of transitions between AOIs to be calculated. Furthermore, the time spent looking ahead to the next component of the task was calculated.

Finally, analysis of the activity counts recorded by the Actigraph activity monitors allowed for the calculation of the "*bilateral magnitude*" and "*magnitude ratio*" using the methods described in section "2.4." By combining the raw data with the activity diary, it was also possible to establish the wear time of the prosthesis. For the purposes of this study, the results are reported based on the data recorded throughout the week, irrespective of whether the prosthesis was worn. However, to allow fair comparison of the "*magnitude ratio*" between prosthesis users and stroke patients (Bailey et al., 2015), analysis was also undertaken based only on the periods when the prosthesis was worn. For this secondary analysis, overnight removal of the prosthesis was excluded based on visual assessment of the raw accelerometer data and activity counts from the monitor worn on the prosthesis. Data were excluded from the last activity count on 1 day until the first count on the next day (activity count spikes during these non-wear periods, lasting less than 1 min with at least 10 min of non-use either side, were also excluded). If visual analysis of the raw data showed long periods (>1 h) of no prosthesis activity during the day, these periods were also excluded based on the activity counts, with the assumption that the prosthesis was removed. For more information on the visual analysis, please see the Supplementary Material. Similarly to Bailey's data, the median "*magnitude ratio*" was reported to avoid the effects of skewness.

### *2.5.2.3. Relationships between Factors Affecting Prosthesis Control and Functionality/Usage*

The early pilot work was not intended to draw conclusions on the relationship between the different factors. However, the main study will aim to establish how measures of "*functionality*" and "*everyday usage*," can be explained by the factors affecting myoelectric prosthesis control. Using multiple regression techniques, factors affecting prosthesis control (i.e., "*EMG skill*" and "*uncertainty*") will be related to measures of "*functionality*" and "*everyday usage*," specifically:

• the sum of the points gained for successful completion of the task,


To further characterize upper limb performance, measures of "*everyday usage*" will be correlated against measures of "*functionality*" collected within the clinic. These may include association of the "*magnitude ratio*" and wear time with:


Based on the findings of these analyses, it should be possible to establish the relative contribution of the factors affecting prosthesis control to each measure of "*functionality*" and "*everyday usage*."

### 3. INITIAL PILOT STUDY RESULTS AND DISCUSSION

In this section, we use early results from initial pilot work with anatomically intact subjects using a prosthesis simulator and prosthesis users to demonstrate the feasibility of this protocol. Data collected from two prosthesis users (both male, age 44–45, with congenital limb absence, and 1.5–35 years using a myoelectric prosthesis) and one anatomically intact subject using a prosthesis simulator (male, age 21, no experience) are presented.

### 3.1. Data Collection

The data collection period lasted between 4 and 5 h, including breaks, which was longer than desired, however, the protocol included tests that have since been removed. In the format presented above, the protocol would, therefore, be expected to last less than 4 h. For our study in this reduced format, the first 40 min consisted of finding the "*ideal*" electrode placement (see 2.1.1) and undertaking the "*tracking tasks*" (see 2.1.3). The "*reaction time tests*" for "*EMG skill*" analysis (see 2.1.2) took 20–30 min while the "*uncertainty tests*" (see 2.2.1 and 2.2.2) lasted a further 50–60 min. Finally, 40–50 min were spent setting up and undertaking the "*cylinder task*" (see 2.3). Breaks were provided at set points in the protocol to ensure participants' attention was maintained.

### 3.2. Initial Analysis

3.2.1. Reaction Time Tests for the Analysis of EMG Skill During the "*reaction time tests*" (see section 2.1.2), data were recorded from the goniometer both before and after the stimulus (LED) was presented. **Figure 6** shows example data recorded during the second of stimulus presentation. The red circle identifies the time point identified as the moment of hand movement onset in response to stimulus presentation. More detail on the algorithms employed to identify movement onset are presented in the Supplementary Material.

It is widely accepted that the mean reaction time for collegeaged individuals undertaking simple reaction time (SRT) tests with light-based stimuli is around 190 ms (0.19 s) (Kosinski and Cummings, 2004). During tests where the stimulus determines the reaction (CRT tests), times are often slower; exact speeds depend on the test. Moreover, the inherent delay introduced within the prosthesis would be expected to produce prosthesis reaction times that are longer than the anatomical reaction times. Initial results demonstrated measured SRT of 270–290 ms (**Figure 7**); furthermore, an increase in reaction time was seen

FIGURE 6 | Analysis of goniometer data recorded after the LED stimulus presentation in the reaction time experiment. The red marker signifies the point identified as the onset of movement.

difference between the mean CRT and the mean SRT.

between the Simple and Choice Reaction Times of 45–100 ms ("*Decision Time*"). It is worth noting that reaction times and consistency improve after first introduction to a new task (Kosinski and Cummings, 2004); this may show as a learning effect in the "*decision time*" over the small number of repeats. However, we decided not to randomize the order so that all participants underwent the same sequence of testing: the SRT first, then the CRT (see section 4). The "*decision times*" presented in **Figure 7**

suggest that Prosthesis User 2 was less skilled at deciding which muscles to activate than the other two participants.

### 3.2.2. Tracking Tasks for the Analysis of EMG Skill

The "*static tracking task*" (see section 2.1.3) assessed the participant's ability to maintain a specified signal level. This task demonstrated that different levels of "*EMG skill*" can be measured and did not show a ceiling effect; i.e., no participant achieved 100% (**Figure 8A**).

The simulator user appeared to perform better than either of the prosthesis users. It is interesting to note that during a sustained contraction, both prosthesis users demonstrated co-contraction or crosstalk for one of the two muscle groups (**Figure 8B**).

All participants were able to complete the "*dynamic tracking task*" (see section 2.1.3). Two participants performed better when only one car (muscle signal) was under assessment (Part 1), with a 20–40% higher success rate than when presented with 2 cars (Part 2). Prosthesis User 2, who demonstrated large amounts of co-contraction or crosstalk when activating the close signal (**Figure 8B**), did not fit this trend, instead a 10% improvement was seen in the success rate for the close signal for Part 2, and a 60% reduction in success with the open signal. During this second part of the dynamic task when two cars were being controlled, the participant was unable to relax the open signal while contracting the close muscle. This meant that the "open car" was guaranteed to "crash" for at least 50% of the gaps. It is possible that this participant, therefore, changed strategy to concentrate on the easier to control close signal. Alternatively, it is possible that this participant was unable to visually track the two cars and struggled with focusing equally on controlling each signal. One further suggestion is that this links with the reaction time results, which showed that this participant found deciding which muscle to activate harder than the other participants.

At this stage, it is not possible to draw any firm conclusions based on these results. However, we have demonstrated that both "*tracking tasks*" offer the possibility of differentiating between different levels of skill in controlling the EMG signal. Based on these tracking tasks, the simulator user demonstrated a higher level of skill than the two prosthesis users.

### 3.2.3. Effects of Electrode Interface Condition on EMG Transduction

Both prosthesis users experienced some difficulty in completing the tasks designed to measure the extent of "*unpredictability*" in transduction of the EMG signal leading to "*uncertainty*" ("*uncertainty tests*," see section 2.2). User 1 had a good level of control over the prosthesis, and was able to operate it as desired, however, the residual limb was very short. Consequently the participant found the addition of the 500 g mass fairly difficult to hold, reporting discomfort at the elbow. User 2 had a longer residual limb and reported feeling the additional load in his shoulder muscles. Both participants were happy to undertake the task with a 500 g load attached to the hand but would have struggled to support the prosthesis if the mass was much heavier.

The anatomically intact participant using the simulator did not exhibit any clear difficulty with completing the reaction time portion of the task (Section 2.2.1), however, when moving between the arm postures (see section 2.2.2), four unwanted activations occurred (two with the "*ideal*" interface condition and two with the "*additional load*"). The reaction time data from Prosthesis User 1 showed a large amount of variation in reaction times for all three interface conditions (**Figure 9**); however, this user did not experience any unwanted activations of the hand. Finally, Prosthesis User 2 only experienced a small amount of variability in reaction times (**Figure 9**) when undertaking the task with the "*ideal*" electrode interface condition (electrodes bandaged to the limb). However, when the socket was introduced ("*normal*" interface condition and "*additional load*"), the participant encountered a large amount of difficulty in getting the prosthesis to react as desired. For 13 of the 20 open tasks, the hand closed when the participant attempted to open it; and for those tasks where the participant did manage to open the

hand, the movement trajectory was not smooth. **Figure 10** shows a comparison of the goniometer data between Prosthesis User 2 and the other two participants. It is worth noting that each participant used a different prosthetic hand for this assessment and that the total aperture for the hand used by Prosthesis User 2 was much smaller than for the other two participants, hence the difference in range. Moreover, Prosthesis User 2 had a much looser socket fit than User 1. Consequently, as the "open muscle" contracted, the limb seemed to push against the socket moving the "close electrode" away from the skin and activating the close movement instead. This "*unpredictability*" in socket fit was also highlighted by the seven unwanted activations when transitioning between the different arm positions.

#### 3.2.4. Functionality Assessment

All participants began with the medium difficulty task ("*task B*"); completion of the task ranged from 100% (Prosthesis User 1) to less than 50% (Prosthesis User 2) of trials. Both Prosthesis User 1 and the simulator user completed over 80% of trials of "*task B*" and, therefore, moved on to the harder task ("*task C*"). Prosthesis User 2 experienced difficulty grasping the cylinder, and often dropped it as he rotated it to the horizontal. When attempting the easier task ("*task A*"), he completed 90% of the trials; however, during two of these trials, he missed the cylinder on the first attempt of "reach-to-grasp."

As introduced in Section "2.3.1," data were collected using wrist and chest-mounted IMUs, an electronic goniometer and an eye tracker. The systems were synchronized using the button press; pilot data demonstrated that synchronization was successful. The task durations, based on the button timestamps, illustrate that Prosthesis User 2 performed the medium difficulty task ("*task B*") at a slower rate than the other two participants (**Figure 11**). Prosthesis User 1 was the most consistent regarding

FIGURE 11 | (A) Mean task duration for each of the difficulty levels (Easy "A," Medium "B," and Hard "C"), (B) mean aperture "reach plateau" length as a percentage of the reach phase, (C) mean aperture onset delay as a percentage of the reach phase.

the time taken to perform the task, and as noted above, the most successful. Furthermore, Prosthesis User 1 demonstrated aperture patterns more similar to the healthy norms with a shorter "*reach plateau*" in the reach phase (**Figure 11**); onset delay was similar across the three participants (**Figure 11**).

As highlighted above, Prosthesis User 2 struggled to complete "*task B*," dropping the cylinder during rotation of the arm; the screenshots in **Figure 12** summarize the technique employed by the participant to overcome this "*unpredictability*." Unlike Prosthesis User 1 and the simulator user, Prosthesis User 2 waited until the last minute, when the cylinder was in contact with the tube, before rotating the cylinder to the horizontal. The participant's "*uncertainty*" as to how the hand would respond is highlighted in the results of the eye tracking. The eye tracking videos (**Figure 12**) were individually coded frame by frame to establish where the participant was looking. As can be seen in the images at the top of **Figure 12**, both prosthesis users looked at their hand during "reach-to-grasp," however as can be seen in **Figure 13**, there were noticeable differences in the gaze patterns of these two users. Prosthesis User 2 spent the majority of the time looking at the hand and the cylinder, tracking its movement, while Prosthesis User 1 showed a higher level of confidence in the hand, looking ahead to the cylinder and the tube. During the "reach-to-grasp" component of the task, Prosthesis User 1 looked

FIGURE 12 | Example eye tracking video – the crosshair shows the point of gaze fixation. Top: both Prosthesis Users looked at the hand at a point in the reach to check their hand aperture. Bottom: the different strategies employed to complete "task B" can be seen – left: simulator user, middle: Prosthesis User 1, and right: Prosthesis User 2 – Prosthesis User 2 struggled to complete this task and would drop the cylinder when the arm was brought to the horizontal, therefore, he delayed this movement.

ahead of the hand for 76% of the time, while Prosthesis User 2 relied on looking at the hand for over 50% of the time.

### 3.2.5. Everyday Usage

As explained in Section "2.4" participants were asked to undertake activity monitoring over the period of 1 week. For the purposes of this pilot study, data were only collected for the two prosthesis users; however, to check the methods against Bailey's data (Bailey et al., 2015) (see section 2.4), one separate anatomically intact participant underwent activity monitoring using their anatomical arms. The anatomical results echoed Bailey's findings with symmetrical use across the two arms represented by a median "*magnitude ratio*" of 0.11 (IQR = 3.28) (**Figure 14A**).

At present, no algorithm exists allowing for differentiation between non-wear and passive-use of the prosthesis using the wrist worn Actigraph monitors. Consequently, participants were asked to complete activity diaries, which subsequently showed that Prosthesis User 2 only wore his device for 3 of the 7 days,

count of the number of data points. (A) Top: bilateral arm use for anatomically intact control subject. Arm use is symmetrical across both arms, regardless of limb dominance. (B) Middle: bilateral arm use for Prosthesis User 1. (C) Bottom: bilateral arm use for Prosthesis User 2.

while User 1 wore his all week. This non-wear is reflected in the activity monitor data with purely unilateral use of the anatomical arm on these days and no activity counts for the prosthesis. From the activity diaries, we know that both participants generally wore their prosthesis for 10 h or more on the days when they were worn. It is, therefore, important that the data are collected over the week long period to ensure that representative data for each user is collected.

**Figures 14B,C** illustrate that both prosthesis users rely on their anatomically arm to a greater extent than the stroke patients participating in Bailey's study (Bailey et al., 2015). Both prosthesis users demonstrated median "*magnitude ratios*" of −7 [IQR = 5.40 (Participant 1) IQR = 0 (Participant 2)] (unilateral use of the intact arm) similar to the group of Bailey's stroke participants who rely most on their non-paretic arm. However, when only the data collected while the prosthesis was worn is included in the comparison, the median "*magnitude ratios*" reduce to −2.55 (IQR = 6.42) for Prosthesis User 1, and −2.42 (IQR = 6.76) for Prosthesis User 2. It is interesting to note that although Prosthesis User 1 wore the device for more hours during the week, both participants demonstrated similar median "*magnitude ratios*." Furthermore, it is notable that the "*bilateral magnitude*" of User 1's activity was of a level much closer to the stroke patients, while User 2 demonstrated activity to the same magnitude as Bailey's healthy controls.

### 4. LIMITATIONS AND FUTURE WORK

For the purposes of this study, the prosthesis control chain has been characterized up to the point of EMG signal transduction. In reality, however, further to the "*EMG skill*" and "*uncertainty*" addressed above, an inherent electromechanical delay will be introduced by the prosthesis itself, which will also impact on the ease of controlling the device. This delay is a culmination of the delays introduced through processing of the EMG signals and stiction/backlash in the prosthetic hand mechanisms. The measurement of these delays is complex requiring artificial activation of the electrodes. Previous work has been undertaken to calculate the optimal controller delay in a prosthesis (Farrell and Weir, 2007); however, the delays in clinically available prostheses are not available. Future work would involve integrating the measurement of electromechanical delays into the protocol detailed above.

Other limitations are the assumptions that have been made with respect to the "*Reaction Time Tests*" (Section 2.1.2). Reaction time experiments involving simple and choice reaction times would normally be randomized, and undertaken in large numbers, to overcome learning or attentional effects. This study involves the comparison of performance in these tasks between participants, therefore, it is important that all participants experience the same tasks in the same order. Furthermore, time constraints limit the number of repeats that can be undertaken. Although different participants may learn at different rates, it is assumed that as the task is novel to all participants the results will be comparable.

Furthermore, the tube used in the "*cylinder task*" is transparent, meaning that when the cylinder is within the tube it can be difficult to identify whether the participant is looking at the cylinder or the tube (likely both). Similarly when the gaze is on the GCA of the cylinder, as the hand approaches and blocks the view, it is not clear whether the AOI should be coded as the hand or the GCA. An inter-rater reliability study will be undertaken to assess the proposed approach to coding the gaze data.

As discussed in Section "2.4," the analysis methods for the assessment of everyday upper limb use were borrowed from the study by Bailey et al. (2015). A limitation of this method is that it does not inform on actual hand use. Therefore, it is not possible to confirm whether the activity counts recorded relate to the prosthetic hand being used in an active or passive manner. For future studies, it would, therefore, be worth including a system to also monitor hand movements. This approach was advocated by Sobuh et al. (2010) and a recent paper by Rowe et al. (2014) demonstrated the potential for a similar approach in the monitoring of anatomically intact upper limb movements.

Finally, reliability and validity of the experimental setups and corresponding outcome measures need yet to be explored. Reliability can be established through a test–retest study in a subset of our planned cohort. Validity of measures, where possible, may be investigated via comparison to related, established measures, for example, by comparing functional measures during "*cylinder task*" performance to SHAP and/or Box and Blocks test scores. For validation of measures characteristic of prosthesis control, we may utilize a known-groups assessment to investigate their sensitivity to distinguish between novice and experienced myoelectric prosthesis users, and we could further conduct a responsiveness study in novice myoelectric prosthesis users to identify whether an individual measure of prosthesis control responds to effects of training to perform the corresponding experimental set up of the protocol.

### 5. CONCLUSION

In this paper, we presented a protocol for the assessment of user skill in controlling EMG signals ("*EMG skill*") and "*unpredictability*" in the acquisition of these signals. These are to be assessed against overall user "*functionality*" and "*everyday usage*" of the myoelectric prosthesis. To demonstrate the protocol, results of initial pilot work were presented.

Pilot work and initial analysis of the results suggest that this protocol will be able to successfully identify differences in the "*EMG skill*" level of participants and characterize the "*unpredictability*" at the electrode interface. Data have been successfully collected for each aspect of the functional task that will allow analysis of how each control factor affects "*functionality*." Furthermore, analysis of the activity monitoring data will allow assessment of control factors against "*everyday usage*."

Although the results presented are not sufficient to draw firm conclusions, Prosthesis User 2 appeared to demonstrate a lower level of "*functionality*" than User 1, which could be attributed to either of the control factors at this stage. By collecting data across a larger cohort of prosthesis users, it should be possible to identify the relative contributions of these factors.

Finally, although the protocol is relatively long, pilot participants were provided with regular breaks and were happy with the distribution of the tasks; the length of the study was not felt to be excessive. Performing all tasks in a single test session (including breaks to avoid fatigue) has the advantage that it facilitates protocol completion in myoelectric prosthesis users, who are largely part of the working population and, hence, could prove difficult to schedule on multiple occasions within a reasonable time frame. Nevertheless, each experimental setup has been designed in such a way that it could be performed in isolation of other parts of the protocol, providing useful insights on the isolated factor the experiment is concerned with. Hence, while the complete protocol may be predominantly used by researchers due to its complexity, individual parts could be adopted by clinicians to support their decision making.

### ETHICAL CONSIDERATIONS

Ethical approval for the work reported here was granted by the University of Salford ethics committee (*REF: HSCR 15-130*). All participants provided informed written consent.

## DISSEMINATION OF RESULTS

It is intended that the results of the main study will be published in peer-reviewed journals and will be uploaded as part of the lead author's PhD thesis onto the University of Salford SEEK website. Additionally, it is intended that the results of this early work and the final study will be presented at conferences both in the UK and internationally.

### AUTHOR CONTRIBUTIONS

The study was initially conceived by LK. All authors contributed to the design of the study, which was implemented by AC. All authors were involved in the drafting and approving of the manuscript.

### ACKNOWLEDGMENTS

The authors would like to acknowledge the contributions of K Hesselink (Universiteit Twente) to the initial design of the "*cylinder task*," and JC de Vries (Universiteit Twente) who contributed to its subsequent refinement. Dr. J Gardiner (University of Salford) should also be recognized for valuable suggestions to improve the protocol design.

### FUNDING

AC is funded by the University of Salford as part of the Graduate Teaching Studentship scheme, without which this work would

### REFERENCES


not have been possible. Furthermore the Ottobock Myohand VariPlus Speed used in this study was funded through an IPEM innovation and research award.

### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online athttp://journal.frontiersin.org/article/10.3389/fnbot.2016.00007


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2016 Chadwell, Kenney, Thies, Galpin and Head. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Corrigendum: The Reality of Myoelectric Prostheses: Understanding What Makes These Devices Difficult for Some Users to Control

#### Alix Chadwell, Laurence Kenney\*, Sibylle Thies, Adam Galpin and John Head

Centre for Health Sciences Research, University of Salford, Salford, United Kingdom

Keywords: prosthesis, myoelectric, activity monitoring, control, upper limb, functionality assessment

#### **A corrigendum on**

#### **The Reality of Myoelectric Prostheses: Understanding What Makes These Devices Difficult for Some Users to Control**

by Chadwell, A., Kenney, L., Thies, S., Galpin, A., and Head, J. (2016). Front. Neurorobot. 10:7. doi: 10.3389/fnbot.2016.00007

#### Edited and reviewed by:

Claudio Castellini, Robotik und Mechatronik Zentrum, Deutsches Zentrum für Luft- und Raumfahrt (DLR), Germany

> \*Correspondence: Laurence Kenney l.p.j.kenney@salford.ac.uk

Received: 18 February 2018 Accepted: 19 March 2018 Published: 03 April 2018

#### Citation:

Chadwell A, Kenney L, Thies S, Galpin A and Head J (2018) Corrigendum: The Reality of Myoelectric Prostheses: Understanding What Makes These Devices Difficult for Some Users to Control. Front. Neurorobot. 12:15. doi: 10.3389/fnbot.2018.00015 In the original article, there was an error. The equation for the Magnitude Ratio presented in the Methods and Analysis was incorrect and should be written ln(VMNonDom/VMDom).

A correction has been made to Methods and Analysis, Everyday usage (Section 2.4), paragraph 1: Current methods of quantifying everyday prosthesis use involve self-report (Roeschlein and Domholdt, 1989; Sherman, 1999; Gallagher and MacLachlan, 2000; Raichle et al., 2008), which is known to be prone to recall and bias errors (Metcalf et al., 2007; Brown and Werner, 2008). Accelerometer-based activity monitoring (Noorkõiv et al., 2014) provides an opportunity to observe actual prosthesis use outside of the clinical environment; however, to date no studies have been published on a cohort of upper limb prosthesis users. We have adapted a protocol developed for stroke patients (Bailey et al., 2015). This research involved participants wearing an activity monitor (Actigraph GT3X+) on each of their wrists while they went about their normal daily activities. The Actigraph monitors provide continuous logging of raw accelerometer data (sampled at 30 Hz). The data are downloaded using proprietary software, filtered, and down sampled to 1 Hz. The processed data are expressed as activity counts (0.001664 g/count) (Actigraph Corp., 2015), which are converted into vector magnitudes (sum of the counts along each axis p x <sup>2</sup> + y <sup>2</sup> + z 2 ). For each second of data, Bailey et al. (2015) combined the vector magnitudes from each of the two wrist worn monitors (dominant and non-dominant arm) to inform on the magnitude of activity across both arms, expressed as the "bilateral magnitude" (VMDom + VMNonDom), and the contribution of each arm to the activity, expressed as the "magnitude ratio" [ln (VMNonDom/VMDom)].

The authors apologize for this error and state that this does not change the scientific conclusions of the article in any way.

### REFERENCES

Actigraph Corp. (2015). ActiGraph White Paper: What Is a Count? Pensacola, FL.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Chadwell, Kenney, Thies, Galpin and Head. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Irfan Hussain1 \*, Giovanni Spagnoletti1 , Gionata Salvietti1,2 and Domenico Prattichizzo1,2*

*1Department of Information Engineering and Mathematics, Università degli Studi Siena, Siena, Italy, 2Department of Advanced Robotics, Istituto Italiano di Tecnologia, Genoa, Italy*

In this paper, we propose a novel electromyographic (EMG) control interface to control motion and joints compliance of a supernumerary robotic finger. The supernumerary robotic fingers are a recently introduced class of wearable robotics that provides users additional robotic limbs in order to compensate or augment the existing abilities of natural limbs without substituting them. Since supernumerary robotic fingers are supposed to closely interact and perform actions in synergy with the human limbs, the control principles of extra finger should have similar behavior as human's ones including the ability of regulating the compliance. So that, it is important to propose a control interface and to consider the actuators and sensing capabilities of the robotic extra finger compatible to implement stiffness regulation control techniques. We propose EMG interface and a control approach to regulate the compliance of the device through servo actuators. In particular, we use a commercial EMG armband for gesture recognition to be associated with the motion control of the robotic device and surface one channel EMG electrodes interface to regulate the compliance of the robotic device. We also present an updated version of a robotic extra finger where the adduction/ abduction motion is realized through ball bearing and spur gears mechanism. We have validated the proposed interface with two sets of experiments related to compensation and augmentation. In the first set of experiments, different bimanual tasks have been performed with the help of the robotic device and simulating a paretic hand since this novel wearable system can be used to compensate the missing grasping abilities in chronic stroke patients. In the second set, the robotic extra finger is used to enlarge the workspace and manipulation capability of healthy hands. In both sets, the same EMG control interface has been used. The obtained results demonstrate that the proposed control interface is intuitive and can successfully be used, not only to control the motion of a supernumerary robotic finger but also to regulate its compliance. The proposed approach can be exploited also for the control of different wearable devices that has to actively cooperate with the human limbs.

#### *Edited by:*

*Michael Wininger, University of Hartford, USA*

#### *Reviewed by:*

*Alejandro Linares-Barranco, University of Seville, Spain Andrew Paul Tosolini, University College London, UK*

#### *\*Correspondence: Irfan Hussain*

*irfan.hussain@unisi.it*

*Received: 25 July 2016 Accepted: 24 October 2016 Published: 11 November 2016*

#### *Citation:*

*Hussain I, Spagnoletti G, Salvietti G and Prattichizzo D (2016) An EMG Interface for the Control of Motion and Compliance of a Supernumerary Robotic Finger. Front. Neurorobot. 10:18. doi: 10.3389/fnbot.2016.00018*

Keywords: wearable robotics, supernumerary robotic fingers, compliance control

### 1. INTRODUCTION

Wearable robotic devices have been mainly used in substitution of lost limbs [e.g., prosthetic limbs (Carrozza et al., 2004)] or for human limb rehabilitation [e.g., exoskeletons (Pons, 2008)]. Besides traditional wearable robotic structures, a very promising research direction aims at adding robotic extra limbs to humans, rather than substituting or enhancing the human limbs (Davenport et al., 2012; Wu and Asada, 2014). The advantage of using wearable robotic extra limbs is twofold. From one side, this addition can enable humans to augment their capabilities (Llorens-Bonilla et al., 2012). On the other side, extra limbs can compensate the missing abilities of impaired limbs, e.g., in case of chronic stroke patients (Salvietti et al., 2016).

We recently started to investigate how an extra (supernumerary) robotic finger can be used in cooperation with the human hand. We mostly focus on two possible applications: compensate the missing abilities of stroke patients with a paretic hand and augment the human healthy hand so as to enhance its capabilities. Concerning grasp compensation in stroke patient, we noted that, in last decade, many wearable devices have been proposed, especially for hand rehabilitation and functional recovery (Heo et al., 2012; Lum et al., 2012). However, only 5–20% of patients show a complete recover of upper limb 6 months after the stroke (Nakayama et al., 1994). We have, thus, proposed a wearable extra finger device that allows the patient to regain the grasping function of the hand when the deficit is stabilized (Hussain et al., 2015b; Salvietti et al., 2016). The main idea was to have the robotic finger and paretic arm acting as the two parts of a gripper to hold an object. The human user was able to control the flexion/ extension of the robotic finger through a switch placed on a ring, while being provided with vibrotactile feedback about the forces exerted by the robotic finger on the grasped object. In Salvietti et al. (2016), we introduced an EMG interface that captures the frontalis muscle activation to control the finger flexion/extension. Finally, in Hussain et al. (2016), we proposed an underactuated compliant extra finger as well as an EMG interface embedded in a cap. Concerning augmenting human healthy hand, in Prattichizzo et al. (2014a), we presented a preliminary version of a robotic extra finger showing how this wearable device is able to enhance grasping capabilities and hand dexterity in healthy subjects. In Prattichizzo et al. (2014b), we presented an objectbased mapping algorithm to control robotic extra limbs without requiring explicit commands by the user. The main idea of the mapping was to track human hand by means of dataglove and reproduce the main motions on the extra finger. Although the earlier presented works on extra-robotic fingers clearly report the impact of the research, the presented robotic devices and their control interfaces are not enough general. In fact, the proposed systems could manage only few inputs (e.g., few predefined closing trajectories), and no solutions have been proposed to modulate the compliance of the robotic finger so as to control the force on the grasped object. Since supernumerary robotic fingers are supposed to closely interact and perform actions in synergy with the human limbs, the control principles of extra finger should have similar behavior as human's ones. Humans can dynamically change their arm stiffness depending on the environment and the tasks being executed (Ajoudani et al., 2012). For instance, stiffness can be increased by muscle cocontraction when we want to make a precise positioning, or when we hold heavy loads. So that, making the actuators and sensing capabilities of the robotic extra finger compatible to implement stiffness regulation control techniques is of primary importance (Hogan, 1985). Second, we believe that the user should directly control through an interfaces of the stiffness of robotic fingers.

The main contribution of this work is the development of a novel EMG interface that can be used to control both the motion of the supernumerary robotic finger and its compliance and thus the tightness of the obtained grasp. In particular, we relate different finger motions to different gestures of the human hand. We used a commercial EMG interface (Myo Armband, ThalmicLab) for hand gesture recognition. For the compliance control, we used a dedicated surface one bipolar EMG channel to read the user biceps signal. The separation of the two EMG reading allows the user to better control independently grasp tightness and device motion. We also present an updated version of the prototype of robotic extra finger where the adduction/abduction motion is realized through ball bearing and spur gears mechanism. The proposed system can be used both by patients for grasp compensation and by healthy subjects for grasp augmentation. We performed a pilot study to demonstrate the feasibility of the approach both with healthy hand for augmenting its abilities and simulated paretic hand to compensate missing grasp abilities. We involved four healthy subjects to perform two different sets of experiments involving the augmentation of a healthy hand or the compensation of a simulated paretic hand. In both cases, the interface resulted sufficient to effectively control the extra-robotic finger so as to fulfill the proposed task. In all the experiments, the wearable device was worn in one arm, whereas the control interface was worn on the other. In fact, while healthy subjects could potentially wear the interface on the same arm where the device is worn, patients cannot properly control hand motion and muscle contraction in their paretic upper limbs. Use the healthy arm is a possible solution as well as delocalizes the EMG reading in another part of the body, see, e.g., Hussain et al. (2016). Note that the hand gestures are necessary only to select a predefined behavior of the device, so it is not necessary to keep a certain gesture for long period. This is important in bimanual tasks where both hands can be used.

The rest of the paper is organized as follows. In Section 2, we present the materials and methods. In particular, the details of the design and development of the proposed supernumerary robotic finger and the proposed EMG control interfaces are explained in details. In Section 3, the experiments using the proposed system are presented. The results are detailed in Section 4 and discussed in Section 5. Finally, in Section 6, conclusion and future work are outlined.

### 2. MATERIALS AND METHODS

### 2.1. The Supernumerary Robotic Finger

The proposed supernumerary robotic finger is composed of modules connected to partially resemble the human finger mechanical structure. Human hand fingers, excluding the thumb, consist of four phalanges connected by three joints (Jones and Lederman, 2006). The structure of the thumb is different since it has two joints at the base for the anterposition or retroposition combined with the radial or palmar abduction motions. The other fingers are capable of both adduction–abduction and flexion–extension motions. The finger's kinematic model is typically approximated by using simple revolute joints. This approximation is an effective means of modeling, as these are, in fact, the same as compared to proximal and distal joints of humans. The proximal and distal interphalangeal articulations can have only flexion/extension motion capabilities and typically are represented with a single DoF revolute joint. The metacarpal joints have both adduction/abduction and flexion/extension motion capabilities and can be modeled as a 2-DoFs joint that is composed of two revolute joints with orthogonal rotation axis (universal joint). We designed the kinematic structure of the robotic extra finger such that one motor is adopted to actuate each DoF of the robotic finger so as to replicate the flexion/extension motion of the human finger. While, at the robotic finger base, two motors realize the adduction–abduction and flexion–extension motion to replicate metacarpal joint. We used four modules in a pitch–pitch configuration for the flexion–extension motion of the finger so as to approximate the average length of the whole hand (Taylor and Schwarz, 1955). The adduction/abduction motion of base joint is obtained using spur gears that allows to transmit motion and power. One of the spur gear is mounted on the shaft of the servo motor, whereas the other is placed on the base of the finger. We used bearings to decrease the friction during rotation.

The finger design is based on the principle of modularity. Each module consists of a servomotor, a 3D printed structure (Acrylonitrile Butadiene Styrene, ABSPlus, Stratasys, USA) and a soft rubber part mounted on front to increase the friction at the contact area. The actuators used are the HS55 MicroLite servo motors. The modules are connected so that one extremity of each module is rigidly coupled with the shaft of the motor through screws, while the other has a pin joint acting as revolute joint. The exploded view and the prototype of the device are shown in **Figure 1**.

The servo motors are pulse width modulation (PWM) controlled. The PWM signals are generated by a microcontroller At-mega 328 installed on an arduino nano board. The portability and wearability of the device is improved by enclosing all the electronics circuitry in a 3D printed housing which is attached to the finger base support. An external battery pack (5 V) is used to provide power to the actuators. Technical details on the device are summarized in **Table 1**.

### 2.2. The EMG Control Interface for the Supernumerary Robotic Finger

As explained in the introduction, we combined the EMG signals associated with the activation of more muscles for the proposed interface. In particular, we used two EMG interfaces on the arm, one to record the continuous EMG amplitude aiming to regulate the compliance of the device and the second

FIGURE 1 | On left, the exploded cad view, whereas on right, the prototype of the robotic extra finger. Four modules are used for the flexion/extension motion, while the revolute joint based on bearings and spur gears mechanism at the finger base is used for the adduction/abduction motion. The device can be worn on the forearm through an elastic band.

TABLE 1 | The technical details of supernumerary robotic finger.


to recognize different hand gestures to be associated with the motion of the robotic finger. Both EMG interfaces are placed on one of the arms, one at the biceps and other at the forearm, while having the robotic finger on the other arm as shown in **Figure 2**. We developed the circuit acquisition and signal conditioning board for one channel EMG electrodes to measure continuously the biceps muscle EMG signal variations. We used the Myo Armband to recognize the gestures at forearm position.

**Figure 3** shows the block diagram of the proposed system. Both, EMG one channel interface and Myo Armband are connected to a computer through Bluetooth communication. The PC runs MATLAB that is used to process the EMG signal for the compliance control. In order to stream data from Myo Armband to the robotic finger, we used MyoMex. The PC communicates with the robotic device controller (Arduino) through serial communication which in turn controls the motion and compliance of the supernumerary robotic finger.

Section 2.2.1 describes the development of the acquisition and signal conditioning board for one channel EMG interface followed by the compliance regulation of the robotic device through the amplitude variation in the acquired biceps EMG signal. In Section 2.2.2, we describe the gesture recognition through the Myo Armband and their association with the motion control of the supernumerary robotic finger through a finite state machine (FSM).

### 2.2.1. One Channel EMG Electrodes Interface and Robotic Device Compliance Regulation

We used non-gelled reusable silver/silver-chloride electrodes for the EMG one channel interface. These are recommended for biopotentials recording since they present the lowest noise interface (Merletti et al., 2009). The design and development of the EMG signal acquisition board is carried out, while considering

FIGURE 3 | Block diagram of complete system. On top left, the block diagram of EMG one channel interface is shown, where (a) surface electrodes, (b) snap leads, (c) acquisition board, and (d) control board. On top, right, the myoarm band with its major components (e) logo LED, (f) status LED, (g) expandable flex, (h) micro USB charging port, and (i) electrical sensor.

the requirements associated with bandwidth, dynamic range, and physiological principles. The typical EMG waveform is characterized with a spectral content between 10 and 250 Hz with amplitude up to 5 mV, depending on the particular muscle (Merlo and Campanini, 2010). The first stage of the signal conditioning board is developed by using an instrumentation amplifier (INA333) which offers an high common-mode rejection ratio (110 dB @ *G* ≥ 10), while the second stage contains a low-noise high speed operational amplifier (AD869x) to perform band-pass filtering and amplification of the acquired EMG signal. **Figure 4** shows the block diagram of the implemented EMG circuit board. Three electrodes are interfaced to the board; two of them (*VIN+* and *VIN*−) are connected to the inputs of an instrumentation amplifier (In-Amp) and third one called "reference electrode" is connected to a mid-supply reference voltage (*Vss* = 1.65 V). This configuration improves the quality of EMG signal acquisition as it increases the common-mode rejection ratio (CMRR). The first stage of the EMG board is an In-Amp with an additional stage of AC coupling. This configuration allows a precise control of DC levels rejecting undesired DC offset voltage introduced by electrode–skin interface. The DC component is subtracted by feeding the output signal back to the reference input of the In-Amp, by an integrator feedback network, which results in the first-order high-pass response. The second stage of the EMG board is a 4th order low-pass Butterworth filter. An active topology (a Sallen–Key circuit implementation – 4th order low-pass filter cascading two stages of 2nd order) was chosen to get a better performance and less complexity than a passive one. The acquired EMG signal is sampled at 1 kHz (double EMG band) to avoid aliasing.

The reference value of received EMG was normalized using maximum voluntary contraction (MVC) technique (Farina and Merletti, 2000). This solution avoids the problems related to the high influence of detection condition on EMG signal amplitude. In fact, amplitude can greatly vary between electrode sites, subjects, and even day-to-day measures of the same muscle site. We implemented an autotuning procedure based on the MVC in order to better match the user-dependent nature of the EMG signal. The implemented MVC routine consists of a 3-s time window in which the user slowly starts increasing the contraction of the biceps muscle to reach their maximum effort.

placed on the biceps muscle.

**Figure 5A** shows the relation between EMG (percentage of MVC) signal at biceps and time (milliseconds). The relationship between EMG signal and muscle tension is non-linear. The MVC value itself is not calculated as a single peak data point because that would allow too much variability. In order to obtain a more stable reference value, we have implemented an algorithm using a sliding window technique of 500 ms duration to compute the mean amplitude of the highest signal portion acquired during the 3-s time window.

The technical details of the EMG acquisition board are listed in **Table 2**.

The EMG signal acquired through the developed one channel interface is used to control the stiffness of each module of the robotic device through the implemented control scheme based on servo motor.

In the following, we will explain how stiffness regulation has been obtained using servomotors. Generally, in active compliance control framework, the equation relating the motor torque to its position is given by

$$\pi = k \Delta\_q = k(q\_{\text{des}} - q\_m)$$

where *qdes* is the desired (reference) joint position, *qm* is the measured (current) joint value, and *k* is the stiffness constant (Siciliano et al., 2010). Note that the compliant (or stiff) behavior of the joint is achieved by virtue of the control, differently from what happen in mechanical systems with a prevalent dynamics of elastic type. This controller is typically used with actuator that can be torque controlled. Servo motors are position controlled actuators where it is not possible to directly command the exerted torque. A small reference position variation in the clockwise direction is counterbalanced by a large amount of torque in the counterclockwise direction to compensate for this. This behavior is regulated by the controller embedded in the servo motor and cannot be modified. This torque–position relationship defines the standard stiffness of the servo motor (*kc*) that cannot be changed by the user. The only servomotor parameter that can be commanded is its desired position *qdes*. We considered that, at time instant *t*, the desired position for the *i*-th servomotor is obtained as

*q t q t q t des*, , *i m i i* ( ) = − () () 1 1 + ∆ − ,

where

$$
\Delta q\_i(t) = k\_d k\_\varepsilon (q\_{des} - q\_m). \tag{1}
$$

The scaling factor *kd* is introduced to modulate the position error. In order to vary the parameter *kd*, we used the EMG signal acquired at the user biceps. In particular, the range of EMG signals was linearly mapped in the range 0.4–3 of parameter *kd*. In **Figure 5B**, a plot of the relation between biceps contraction and commanded displacement is reported for one module. In presence of a rigid grasped object, the measured positions of the extra finger joints do not change due to the object constraints. So that, changing the desired position of the servomotors through the scaling factor, we can control the force exerted by the device onto the object. In other words, changing the value of *kd*, it is possible to command a position of the module that results in a higher force applied onto the object.

In order to regulate the stiffness between modules, we set priorities. We considered two distinct cases. If only the fingertip module is in contact with the object, all the other modules change their stiffness accordingly. This solution allows to control the stiffness of modules that are not in contact

#### TABLE 2 | Technical details of EMG signal acquisition and conditioning board.


FIGURE 5 | On left, (A) the maximum voluntary contraction (MVC) proportional to the biceps muscle contraction is shown. While on right, (B) the graph between Δ*q* and percentage of MVC for different values of *kd* is plotted.

with the object in precision grasps. In power grasps, in order to obtain suitable contact points, we set different priorities according to the position of the module in the finger. If the fingertip module comes in contact first, the remaining modules change their stiffness accordingly. If another module comes in contact first, modules below to it regulate their stiffness, while the module above no. The same methodology is followed for other intermediate modules. The contact of a module is detected comparing the desired angle commanded to the servo motor (*qdes*) with the actual position read by the encoders (*qm*). When Δ*q* = ||*qdes* − *qm*|| overtake a predefined threshold a contact is recognized. After the contact is achieved, the compliance can be regulated.

### 2.2.2. EMG Armband Gesture Recognition and Robotic Device Motion Control

We used a Myo Armband at forearm to recognize the hand gestures that control the device motions. This device has electrically safe setup with low voltage battery and Bluetooth LE protocol, eight surface EMG sensors working at frequency of 2200 Hz and 9-DoF IMU working at 50 Hz. The provided software development kit (SDK) is suitable for working with the recorded data and for developing standalone applications. EMG signals are filtered through notch filters at frequencies of 50 and 60 Hz in order to take out any power-line interference. For the sake of simplicity, we considered the five gestures available with the SDK. These gestures mainly involve flexion/extension of fingers and flexion/ extension of hand.

We implemented specific types of grasps for both kinds of users in order to make better suitable to use the robotic finger with healthy hand or paretic hand. In particular, in case of healthy hand, we defined *anatomically impossible* grasps and ulnar grasps (see **Figures 6A,B**). In case of *anatomically impossible* grasp, the supernumerary robotic finger coordinates with human hand to grasp big size objects that cannot be grasped using only one hand. In *ulnar* grasp configuration, the robotic device coordinates with ring and pinkie fingers to grasp and hold an object, while the upper part of the hand (thumb, index, and medium fingers) is left free to do another task allowing, for instance, to hold multiple object in one hand or to unscrew a bottle cap with a single hand. In case of paretic hand users, we defined *power* and *precision* grasp as shown in **Figures 6C,D**. In the former, each module flexes with a fixed step size in order to wrap the finger around the object. In the latter, the target is to hold small size objects between the paretic limb and the device fingertip pad. To this aim, the fingertip is kept parallel to the paretic limb during flexion motion. The contact is expected to occur between the object and the fingertip module. Finally, the supernumerary robotic finger can actively be wrapped around the wrist as a bracelet when not used (see **Figure 6E**). We implemented a trigger-based FSM to control the motion of the robotic device (see **Figure 7B**). All the gestures were associated with a unique trigger signal. In **Figure 7A**, the gestures recognized through the Myo Armband are shown. In particular, *fist*-(event *e*1) switches the device from bracelet position to working position and vice versa. *Double tap*-(event *e*2) changes the grasp modalities. Patients with paretic hand can switch between precision and

*power* and *precision grasp*.

power grasp. When augmentation purpose is concerned, the user can switch between ulnar and anatomically impossible grasp. *Wave out*-(event *e*3) corresponds to flexion, and *Wave in*-(event *e*4) is associated with extension. Finally, *Finger spread*-(event *e*5) can stop the motion of the robotic finger.

## 3. EXPERIMENTS

In Section 2, we introduced a novel EMG interface to control motion and compliance of a supernumerary robotic finger. In the following, we demonstrate how this interface and the wearable device can be effectively used both to compensate paretic hand functions and to augment healthy human hand capabilities. We performed a proof-of-concept study involving four healthy subjects (three male and one female, aged 29–40 years). Written informed consent was obtained from the participants. The procedures were in accordance with the Declaration of Helsinki. The aim of this study was to verify the potential of the approach and to understand how rapidly the subjects can successfully interact with the wearable device by using the proposed EMG control interface. The experiments were divided into two categories. The first set of experiments was related to compensation of grasping function, whereas the second was related to augmentation of hand capabilities. In particular, the compensation experiments, shown in Section 3.1, have been carried out asking to the subjects to simulate a paretic hand. We focused mainly on bimanual tasks of activity of daily living (ADL). The augmentation experiments shown in Section 3.2 were performed with the healthy hand to show the effectiveness of the device in increasing the hand grasping abilities and workspace, e.g., allowing to grasp big size objects which can not be grasped using a single hand or holding multiple objects using the augmented hand, i.e., human hand and the supernumerary robotic finger. In both the experimental sets, the subjects used the EMG interface on one arm (three subject used the right arm, one the left), whereas the supernumerary robotic finger was worn on the other arm. The Myo Armband was positioned at the forearm, while the one channel electrodes interface on the biceps (see **Figure 2**).

### 3.1. Compensation of Paretic Hand Functions

Among the different ADL, we focused on those involving "hold and manipulate" tasks. Such activities are generally bimanual tasks where one hand is used to restrain the motion of one object, while the other operates on it, e.g., unscrew the cap of a bottle, open a beans can, etc. The proposed supernumerary finger can be an effective aid in such tasks (Hussain et al., 2016). To demonstrate how the EMG interface can be used by patients, we asked to the subjects to execute different ADL involving a hold and manipulate task (see **Figure 8**). In particular, the subjects were asked to grasp an object using the gestures of the hand and to regulate the grasp tightness acting on the stiffness of the device. We used a subset of objects from the YCB grasping toolkit (Çalli et al., 2015). This toolkit is intended to be used to facilitate benchmarking in prosthetic design, rehabilitation research, and robotic manipulation. The objects in the set are designed to cover

a wide range of aspects of the manipulation problem. It includes objects of daily life with different shapes, sizes, textures, weight, and rigidity. We considered six objects with different shapes to show how the robotic finger can adapt to the shape of the objects to realize a stable grasp. We mainly targeted the objects used in kitchen and in other ADL. During all the tests, subjects simulated the paretic hand and device was positioned on the arm as supposed to be used with the patients.

The subject was asked to perform different bimanual ADL without using the hand grasping ability where the device was worn. The controlateral arm was always used to control the device motion and joints stiffness. **Figure 8** shows the ADL tasks performed by simulating a paretic hand. All the targeted tasks normally require two healthy hands but have been successfully executed with the aid of the robotic extra finger even if one hand was non-functional. The robotic finger and paretic hand was used to constrain the object, while healthy hand was used for manipulation. **Figures 8A,B,D–F** show the example of opening the cans, box, and bottle with various shapes and different caps. **Figure 8C** reports the task of pouring water from a bottle while holding the glass with the help of robotic device and paretic arm. All the task were fulfilled controlling the device through the proposes' interface. The subjects used hand gestures to shape the finger around the object. Later, they controlled the grasp stiffness by contracting the controlateral arm biceps. Note that all the "opening" tasks required stiffness control to be executed. In fact, while compliant joints are preferable to adapt the shape of the finger to the object to grasp, a stiff device is necessary to achieve the stable grasps necessary while unscrewing the caps.

### 3.2. Augmenting Healthy Hand Function through the Proposed System

In this experiment, the subjects were asked to grasp a set of objects with the augmented hand to prove the effectiveness of the extra-robotic finger in enlarging the human hand workspace

that are impossible to do with human hand only. (A) Unscrewing a cap of bottle, (B) grasping bigger box, (C) grasping two balls, (D) opening door,

(E) soldering a board, and (F) grasping plate and glass.

and dexterity. We targeted tasks involving either *anatomically impossible* grasp or *ulnar* grasp, as defined in Section 2. In the former case, the subjects were asked to grasp relatively big size objects which cannot be grasped using only one hand. In the latter case, the users tried to grasp objects only using the ring and the pinkie fingers opposite to the sixth finger and to perform another operation with the remaining fingers (thumb, index, and middle). In **Figure 9A**, the user is unscrewing a cap from a bottle using only one hand. Ulnar grasp is used to keep firm the bottle, while the other fingers can unscrew the cap. **Figure 9B** shows the example of grasping big size box with the augmented hand that is impossible to grasp with the human hand only. Holding multiple objects with the augmented hand is shown in **Figures 9C,F**. The example illustrated in **Figure 9D** involves the task of opening the door using the handle, while carrying a heavy bag with the hand. The user was able to turn the handle to open the door using the robotic device, while keep holding the bag with the hand. The 9-e is another example where the user can solder a circuit board, while holding the board by robotic finger, ring, and pinkie. The thumb, index, and middle finger are used to hold soldering gun. Note that, all the tasks are either impossible or at least very difficult to be carrying out with a single hand. All these tasks were successfully fulfilled by all the subjects with the help of the EMG interface and the supernumerary extra finger. Also in this subset of examples, the possibility to control both motion and joint stiffness of the device was exploited by the users.

### 4. RESULTS

In Section 3, we described the tasks performed by the subjects to prove the usability of the proposed EMG interface and the novel supernumerary finger prototype. In the following, we will give the details of the position of the device and the forces exerted on the grasped object for two particular type of grasps, i.e., power and precision grasps. **Figures 10**–**13** reported the behavior of the device during power and precision grasping, respectively. In particular, **Figures 10** and **11** refer to the power grasp reported in **Figure 8A**, whereas **Figures 12** and **13** refer to the precision grasp reported in **Figure 9E**. We report only these examples for the sake of brevity. **Figures 10**–**13** represent the average of five repetitions of the same subject. To measure the forces exerted on the objects, we equipped each module of the extra finger with a Force Sensing Resistor (FSR) (408, Interlink Electronics Inc., USA). The user was asked to command the supernumerary finger till the grasp is obtained. Once the device was in contact with the object, the user increased the stiffness of the device by cocontracting his/her biceps (see **Figure 14**). The contraction of the biceps was read by the EMG interface, and the value of *kd* in equation (1) was increased (see **Figure 15**). This variation produced a variation in the desired angle *qdes* of the modules, while the read actual position of the modules remained the same due to the constrain imposed by the object (see **Figures 11** and **13**). The variation in the desired angles produces, however, an increase of the force exerted by the device onto the object, as shown in **Figures 10** and **12**. So that, by cocontracting the biceps the user can regulate the grasp tightness. As expected, in power grasps, all the modules move of a similar angle so as to wrap the object. All the modules also contribute to the grasp tightness applying force on the object. Differently, in precision grasp, the fingertip module is the only module exerting force. The module motion is opposed to the direction of the other three modules so as to leave the fingertip parallel to the hand.

After the experiments, we investigated the users' subjective satisfaction and possible concerns related to the proposed system. We proposed a questionnaire to the subjects to evaluate their satisfaction and usefulness of the proposed system. Questionnaires and interviews are recommended methods for user feedback and what features they particularly like or dislike in the system (Nielsen, 1994). The subjects were asked to fill the Usefulness Satisfaction and Ease of use questionnaire (USE) (Lund, 2001) that focuses on the experience of the system usage. This questionnaire uses a seven-point Likert rating scale. Mean and SD of the questionnaire factors are presented in **Table 3**.

The proposed EMG interface and the novel robotic extra finger prototype successfully enabled the users to complete all the targeted tasks both related to augmentation and compensation. The experiments proved that the presented system can be an effective aid both in augmenting the healthy human hand and in compensating its missing abilities in case of a disease. The proposed EMG control interface resulted to be intuitive and simple. The users were able to generate multiple control inputs without using sensorized gloves on human hand and were able to modulate the compliance of the robotic device in proportional to the EMG signal amplitude variations in biceps. Moreover, the upgraded version of the device with additional adduction/abduction degree of freedom increased the dexterity of the robotic device allowing more complex operation, especially when hand augmentation was considered.

### 5. DISCUSSION

Supernumerary robotic limbs are a new generation of wearable robots which aims at assisting natural limbs by closely interacting with them. In order to realize safe and natural interaction of human limbs with the extra-robotic limbs, the control principles, actuation, and sensing capabilities of extra limbs should have similar behavior as humans ones, e.g., their ability to regulate compliance. In this regard and to overcome the limitations of the control interfaces presented in state of the art for supernumerary robotic fingers, we propose a novel EMG interface. In particular, to obtain multiple user control inputs to control the motion of extra-robotic finger, as well as to regulate its compliance, we have

presented an EMG-based control interface that can be used to control different trajectories for finger flexion/extension and can regulate the finger compliance and thus the tightness of the grasp. The exploitation of the supernumerary robotic fingers in compensating and augmenting the human hand grasping abilities is at an early stage. One of the major challenges in augmenting/compensating human capabilities through robotic extra limbs concerns the development of a suitable control interfaces for the integration of the device motion with that of the human. We better demonstrate this fact by recalling the approaches presented in literature and their limitations. Wu and Asada (2014) presented a control algorithm enabling a human

#### TABLE 3 | Questionnaire factors and relative marks.


*The mark ranges from "1* = *strongly disagree" to "7* = *strongly agree." Mean and SD are reported.*

hand augmented with two robotic fingers to share the task load together and adapt to diverse task conditions. Postural synergies were found for the seven-fingered hand comprised of two robotic fingers and five human fingers through the analysis of measured data from grasping experiments. In Prattichizzo et al. (2014b), a mapping algorithm able to transfer to an arbitrary number of robotic extra fingers the motion of the human hand has been presented. The mapping algorithm was based on the definition of a virtual object obtained as a function of a set of reference points placed on the augmented hand (human hand and robotic fingers). The mapping algorithm allowed to move the extra fingers according to the human hand motions without requiring explicit command by the user. Both the approaches used an instrumented glove to track the human hand presenting some limitations which affected their practical application. Patients with a paretic hand cannot properly control finger motions, thus a dataglove interface cannot be used. The estimation of the human hand posture and fingers motion implies a reliable and computationally expensive hand tracking. Moreover, datagloves can be only used for position control of the robotic device without having any control on force or stiffness regulation. As a preliminary solution to the above mentioned issues, we implemented a trigger-based control approach (Hussain et al., 2015a,b). The trigger signal was activated by a wearable switch placed on a ring. A single switch activation regulated the stop/motion of the finger along a predefined flexion trajectory, while a double activation switched from flexion to extension and vice versa. Although the ring-based control approach resulted simple and intuitive, this control interface involved human hand thumb, thus, limiting the use of thumb in completion of tasks. Moreover, it offers few user control inputs to control the motion of the robotic finger and force control is not straightforward. The control approach and the device presented in this paper are a possible solution of the above mentioned issues of the techniques presented in literature. In Section 3, we reported several tasks where a supernumerary finger can be used both for grasping compensation of paretic limb and to augment human hand capabilities. In Section 4, we showed how the EMG interface can be effectively used to control the position of the finger and the force exerted on the object.

All the experiments were performed involving healthy subjects. We are currently starting to test the system with stroke patients showing a residual mobility of the arm. We delineate the patients' condition for being included in the pilot experiments. Patients have to score ≤2 when their motor function is tested with the National Institute of Health Stroke Scale (NIHSS) (Brott et al., 1989), item 5 "paretic arm." Moreover, the patients has to show the following characteristics: (1) normal consciousness (NIHSS, item 1a, 1b, 1c = 0), absence of conjugate eyes deviation (NIHSS, item 2 = 0), absence of complete hemianopia (NIHSS, item 3 ≤ 1), absence of ataxia (NIHSS, item 7 = 0), absence of completely sensory loss (NIHSS, item 8 ≤ 1), absence of aphasia (NIHSS, item 9 = 0), absence of profound extinction and inattention (NIHSS, item 11 ≤ 1).

### 6. CONCLUSION

In this paper, we present an EMG control interface for a supernumerary robotic finger that can be used to control motion and joint stiffness. The aims are grasping compensation in chronic stroke patients and augmentation of human healthy hand to enhance its grasping capabilities and workspace. The motion of the robotic finger is controlled through gesture recognition and its compliance is regulated by EMG signal amplitude variations. In particular, we proposed Myo Armband to recognize the user gesture to control the motion of the robotic device. We developed EMG one channel electrode interface to modulate the compliance of the robotic device through a control scheme based on servo motor. We developed a five DoFs device that can be worn on the user wrist by an elastic band. We validated the use of device in augmenting and compensating the human hand grasping abilities. In particular, we showed how the supernumerary robotic finger can play the role of an extra thumb enlarging the human hand workspace and the hand dexterity and how it can compensate the missing abilities of the non-functional hand in case of stroke patients. We demonstrate through experiments that the same interface can be used by patient and healthy subjects to control different flexion trajectories and to regulate the grasp tightness.

As future work, we are improving the portability of the system, in particular, we are realizing a Bluetooth communication of EMG interfaces with the robotic device controller. We are also testing the EMG interface with stroke patients so as to collect interesting insights for the extra finger development.

### AUTHOR CONTRIBUTIONS

IH contributed in literature review, design, and development of robotic finger; implementation of control, calibration, testing, experiments, and paper writing. GSp contributed the development of EMG interfaces design and development. GSa contributed in control development and in paper writing. DP supervised the project overall and contributed in writing and proof reading.

### ACKNOWLEDGMENTS

The authors are grateful to Dr. David Cioncoloni and Prof. Simone Rossi of the Dipartimento di Scienze Mediche, Chirurgiche e Neuroscienze, UOC Neurologia e Neurofisiologia Clinica, Brain Investigation and Neuromodulation Laboratory (Si-BIN), Siena, Italy, for their suggestions and the help in recruiting patients. The research leading to these results has received funding from the European Union's Horizon 2020 Research and Innovation Programme under Grant Agreement No. 688857 of the project SoftPro: Synergy-based Open-source Foundations and Technologies for Prosthetics and Rehabilitation.

### REFERENCES


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2016 Hussain, Spagnoletti, Salvietti and Prattichizzo. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

*Adenike A. Adewuyi1,2,3\*, Levi J. Hargrove1,2,4 and Todd A. Kuiken1,2,3,4*

*1Department of Biomedical Engineering, Northwestern University, Chicago, IL, USA, 2Center for Bionic Medicine, Rehabilitation Institute of Chicago, Chicago, IL, USA, 3 Feinberg School of Medicine, Northwestern University, Chicago, IL, USA, 4Department of Physical Medicine and Rehabilitation, Northwestern University, Chicago, IL, USA*

Pattern recognition-based myoelectric control of upper-limb prostheses has the potential to restore control of multiple degrees of freedom. Though this control method has been extensively studied in individuals with higher-level amputations, few studies have investigated its effectiveness for individuals with partial-hand amputations. Most partial-hand amputees retain a functional wrist and the ability of pattern recognition-based methods to correctly classify hand motions from different wrist positions is not well studied. In this study, focusing on partial-hand amputees, we evaluate (1) the performance of non-linear and linear pattern recognition algorithms and (2) the performance of optimal EMG feature subsets for classification of four hand motion classes in different wrist positions for 16 non-amputees and 4 amputees. Our results show that linear discriminant analysis and linear and non-linear artificial neural networks perform significantly better than the quadratic discriminant analysis for both non-amputees and partial-hand amputees. For amputees, including information from multiple wrist positions significantly decreased error (*p* < 0.001) but no further significant decrease in error occurred when more than 4, 2, or 3 positions were included for the extrinsic (*p* = 0.07), intrinsic (*p* = 0.06), or combined extrinsic and intrinsic muscle EMG (*p* = 0.08), respectively. Finally, we found that a feature set determined by selecting optimal features from each channel outperformed the commonly used time domain (*p* < 0.001) and time domain/ autoregressive feature sets (*p* < 0.01). This method can be used as a screening filter to select the features from each channel that provide the best classification of hand postures across different wrist positions.

Keywords: pattern recognition, electromyography, partial-hand amputee, myoelectric control, intrinsic hand muscles, feature selection

#### *Edited by:*

*Michael Wininger, University of Hartford, USA*

#### *Reviewed by:*

*Noman Naseer, Pusan National University, South Korea Youngjin Choi, Hanyang University, South Korea*

#### *\*Correspondence:*

*Adenike A. Adewuyi adenike-adewuyi@fsm.northwestern. edu*

> *Received: 30 July 2016 Accepted: 04 October 2016 Published: 19 October 2016*

#### *Citation:*

*Adewuyi AA, Hargrove LJ and Kuiken TA (2016) Evaluating EMG Feature and Classifier Selection for Application to Partial-Hand Prosthesis Control. Front. Neurorobot. 10:15. doi: 10.3389/fnbot.2016.00015*

**Abbreviations:** ANOVA, analysis of variance; AR, autoregressive; EMG, electromyogram; LDA, linear discriminant analysis; LNN, linear neural network; LogDet, log-detector; MAV, mean absolute value; MdF, median frequency; MnF, mean frequency; MP, mean power; PF, peak frequency; PSD, power spectrum descriptors; QDA, quadratic discriminant analysis; RMS, rootmean-square; SFS, sequential forward searching; SI, separability index; SSC, slope-sign changes; TD, time domain; TDAR, time domain and autoregressive; VAR, variance; V-ord, V-order; WAMP, Willison amplitude; WL, waveform length; ZC, zero crossings.

## INTRODUCTION

Pattern recognition-based myoelectric control of externally powered prostheses has demonstrated remarkable potential to restore function to individuals with upper-limb amputations. This control method has shown promise in laboratory settings (Kuiken et al., 2009; Scheme and Englehart, 2011), and a pattern recognition myoelectric controller is now clinically available for individuals with high-level upper-limb amputations (Uellendahl et al., 2016). However, this population comprises less than 10% of all upperlimb amputations in the United States (Dillingham et al., 2002; Ziegler-Graham et al., 2008). The majority of amputations are distal to the wrist (i.e., partial-hand amputations) (Dillingham et al., 2002). Since this level of amputation can involve a variety of clinical presentations, it is difficult to treat successfully with a prosthesis (Lake, 2004). Though, partial-hand amputations are often termed "minor" amputations (Ziegler-Graham et al., 2008), successful treatment is of significant importance because the effects of partial-hand amputation on employment and selfimage are comparable to those of more proximal amputations (Burger et al., 2007; Hebert and Burger, 2016). Partial-hand amputees perceive themselves to be at a higher disability level than do individuals with unilateral transradial or transhumeral amputations (Davidson, 2004; McFarland et al., 2010), they are more likely to reject their prosthesis (Biddiss and Chau, 2007), and more than half are unable to return to their previous occupation (Burger et al., 2007).

Though externally powered myoelectric prostheses for more proximal upper-limb amputees have been commercially available for decades (Parker and Scott, 1986), they have only recently become available to partial-hand amputees, in part because of the technological complexities of replacing the motor function of a finger within the size limits of a prosthetic digit (Uellendahl and Uellendahl, 2012). Externally powered partial-hand prostheses, such as the i-limb quantum (Touch Bionics Inc.) and Vincentpartial (Vincent Systems GmbH) have independently functioning digits and, thus, offer a wide range of articulated grasps not previously available to partial-hand amputees. Commercial prostheses use conventional control algorithms that use an estimate of the EMG amplitude for proportional control of the speed of an actuated joint (Phillips et al., 2012; Uellendahl and Uellendahl, 2012). Though pattern recognition control has the potential to intuitively restore control of more degrees of freedom than conventional methods (Englehart and Hudgins, 2003; Hargrove et al., 2007; Kuiken et al., 2009), it has not yet been shown to be sufficiently robust for partial-hand prosthesis control.

Partial-hand amputees often retain the ability to move their wrists, and preservation of residual wrist motion is critical for functional everyday activities. Montagnani et al. (2015) showed that when non-amputees are limited to two degrees of freedom at the wrist (pronation/supination and flexion/extension) and one degree of freedom at the hand (open/close), they perform similarly to when they are limited to a one degree-of-freedom wrist (rotation) coupled with their intact, twenty-two degree-offreedom hand. Thus, a clinically successful partial-hand pattern recognition control system must maintain high performance while allowing the individual to use their wrist. Our previous studies demonstrate that varying wrist position adversely affects pattern recognition performance in offline and real-time virtual studies, though the severity of this wrist position effect is diminished by training the classifier with data from multiple wrist positions and combining EMG data from the extrinsic and intrinsic muscles of the hand (Adewuyi et al., 2016; Earley et al., 2016).

The selection of effective features and robust classifiers are critical in the design of pattern recognition-based control systems. Previous studies that investigated classifiers, such as artificial neural networks (Hudgins et al., 1993), hidden Markov models (Chan and Englehart, 2005), linear discriminant analysis (LDA) (Englehart and Hudgins, 2003), support vector machines (Al-Timemy et al., 2013), Gaussian mixture models (Huang et al., 2005), and quadratic discriminant analysis (Scheme and Englehart, 2011) found little difference in classification error between different classifiers within non-amputee and amputee groups (Scheme and Englehart, 2011). An LDA classifier is used most commonly because it provides a good balance between classification performance and computational efficiency. However, because most studies have focused on individuals with more proximal amputations, it remains unclear whether these findings are true for partial-hand amputees whose forearm muscle activity is significantly modulated by wrist movement during a task (Mogk and Keir, 2003; Johnston et al., 2010).

Pattern recognition of EMG signals is dependent on the user's ability to generate repeatable and differentiable muscle contractions. Effective EMG features are those that both provide unique information about limb motion and are minimally sensitive to factors that degrade performance by altering the EMG signals – such as electrode shift (Young et al., 2012), muscle fatigue, muscle contraction effort (Tkach et al., 2010), force variation (Al-Timemy et al., 2015), and limb position (Al-Angari et al., 2016). The robustness of numerous features to such factors has been evaluated; however, typically, the performance of features and feature combinations are evaluated across all channels. Few studies have investigated the importance of selecting individual features from different channels, and no studies, to our knowledge, have specifically evaluated which feature subsets are most robust to changes in wrist position. To search for important subsets in the feature/ channel space, Oskoei et al. (2013) used separability indices and classification rate as objective functions and a genetic algorithm as a search strategy, whereas Khushaba and Al-Jumaily (2007) used classification rate as an objective function and particle swarm optimization as an evolutionary computation search technique. Both of these studies aimed to increase the efficiency of pattern recognition by finding optimal feature subsets, but the selection of best features and channels was not done simultaneously. More recently, Al-Angari et al. (2016) used feature/channel subset selection (using correlation-based and distance-based methods) to determine whether selecting optimal features from each channel would improve the limb position effect.

This work evaluates several strategies in non-amputees and partial-hand amputees for improving classification of hand grasps performed with varying wrist positions. In this study, we (1) compare the performance of linear and non-linear classification techniques and (2) evaluate the performance of optimal EMG feature subsets that are most robust to wrist position variation.

### MATERIALS AND METHODS

### Data Collection

Data from non-amputee subjects, previously collected by Adewuyi et al. (*n* = 7) (Adewuyi et al., 2013, 2016) and Earley et al. (*n* = 9) (Earley et al., 2014) using similar protocols, were combined and used for this study. According to Adewuyi et al., nine selfadhesive bipolar surface Ag/AgCl EMG electrodes (Bio-Medical Instruments) were evenly spaced around the dominant forearm with an inter-electrode distance of 2.5 cm: five electrodes on the proximal forearm, 2–3 cm distal to the elbow and four electrodes on the distal forearm, 7–8 cm proximal to the wrist. However, for the data from Earley et al., eight self-adhesive bipolar surface Ag/ AgCl EMG electrodes (Bio-Medical Instruments) were evenly spaced around the forearm: six electrodes on the proximal forearm and two electrodes on the distal forearm (one on the anterior side and one on the posterior side). EMG data from intrinsic hand muscles were recorded with four electrode pairs on the hand. Two electrode pairs were placed on the palmar side (over the thenar and hypothenar eminence) and two electrode pairs were placed on and dorsal sides (over the first and third dorsal interossei). Data from partial-hand amputee subjects (*n* = 4), previously obtained by Adewuyi et al.*,* were also evaluated (Adewuyi et al., 2013, 2016). All subjects gave written consent, and experiments were performed at the Rehabilitation Institute of Chicago under an approved Northwestern University Institutional Review Board (IRB) protocol.

### EMG Signal Processing

EMG signals were acquired using a custom-built EMG amplifier with a software gain of 2000× for each channel. All EMG data were digitally sampled at 1000 Hz using a custom-built A/D converter based on a TI AD1298 24-bit bioamplifier chip and band pass filtered (30–350 Hz) with a Type 1, eighth-order Chebyshev filter.

### Procedure

Custom-designed computer software was used to visually prompt subjects to perform two functional hand grasps (key grip and chuck grip), one open hand posture, or a rest posture. All four hand postures were performed with a neutral wrist position and repeated while the subjects held their wrist in the following comfortable positions: flexion, extension, pronation, supination, abduction, and adduction, for a total of seven wrist positions. Each hand posture was held for 3 s. Subjects from Earley et al. performed four repetitions of each hand posture in each wrist position (Earley et al., 2014), and subjects from Adewuyi et al. performed 10 repetitions of each hand posture in each wrist position (Adewuyi et al., 2013, 2016).

### Data Analysis

Offline analyses were performed using MATLAB 2015a software (The Mathworks, Natick, MA, USA). For all conditions, data were segmented into 200-ms windows with a 20-ms frame increment (Smith et al., 2011).

### Effect of Classifier Type on Classification Error

A combination of four EMG time domain (TD) features [mean absolute value (MAV), number of zero crossings, waveform length (WL), and number of slope-sign changes] and six coefficients of a sixth-order autoregressive (AR) model features (hereafter called TDAR features) was extracted from each EMG data window. Four classifiers were examined: (1) an LDA classifier, (2) a quadratic discriminant analysis classifier (QDA), (3) a multilayer perceptron neural network with linear activation functions in its one hidden layer (LNN), and (4) a multilayer perceptron artificial neural network with non-linear hyperbolic tangent sigmoid activation functions in its one hidden layer (MLPANN). The LDA was selected because it is the most commonly used for the classification of limb movements using EMG. It was compared to a QDA because they make very similar assumptions about the data except that it allows non-linear boundaries between data. These were compared to a LNN and MLPANN because they are on the opposite side of the spectrum in that they make no assumptions about the underlying distribution of the data.

All classifiers were trained using data from (1) only extrinsic muscle EMG data, (2) only intrinsic muscle EMG data, or (3) a combination of all extrinsic and intrinsic muscle EMG data. Data were divided into training data sets (50% of all data), testing data sets (30% of all data) and validation data sets (20% of all data). The validation data sets were used to minimize overfitting of the neural networks; training of the neural networks stopped once the classification error of the validation sets began to increase. First, the training and testing data sets were used to train and test the classifiers, respectively. The other 50% of the data (previously used for testing and validation in the first group) was used for training and 30% of the data (previously part of the training set in the first group) was used for testing. The results of these two groups were then averaged. Seven hidden layer neurons were empirically chosen for the MLPANN, and the LNN had four neurons in its hidden layer. Since the LNN has linear activation functions, it simply maps the weighted inputs to the output of each neuron and is, thus, mathematically equivalent to a reduced two-layer input–output model (Haykin, 1999). The neural networks were trained using scaled conjugate gradient descent (Møller, 1993).

An exhaustive search was performed to determine the optimal number of wrist positions needed for classifier training. An LDA classifier was trained using data from one to seven wrist positions and tested on data from all seven wrist positions. All possible combinations of data from *n* wrist positions were evaluated, and the combination with the lowest error was chosen for each subject and plotted as a function of number of wrist positions.

### Effect of EMG Feature Subset on Classification Error

Twenty five time and frequency domain features were extracted from each EMG channel. Nineteen of these features were: MAV, zero crossings (ZC), slope-sign changes (SSC), WL, Willison amplitude (WAMP), root-mean-square (RMS), variance (VAR), v-order (order of 3), log-detector (LogDet), AR coefficients (order of 6), mean frequency (MnF), median frequency (MdF), peak frequency (PF), and mean power (MP). The frequency domain features MnF, MdF, PF, and MP were derived from the short-time Fourier transform using Hamming windows. Previous studies have shown that feature sets based on the shorttime Fourier transform perform better than TD features and are comparable to feature sets based upon the wavelet transform and the wavelet packet transform (Englehart et al., 1999). The remaining six features were a set of power spectrum descriptors (PSD) proposed by Al-Timemy et al. (2015). These features were derived as the orientation between features extracted from a non-linearly mapped EMG record and the original EMG record and as such the resultant features were shown to be less affected by different contraction efforts.

Two main approaches can be used to select an optimal feature subset: the filter or the wrapper. The filter approach typically evaluates features based on their discriminative power using their content (e.g., within- and between-cluster separability, distance measures). The wrapper approach applies a classifier to evaluate feature subsets by minimizing classification error. Here, we used the Bhattacharyya distance as a filter function and an LDA as a wrapper function.

The Bhattacharyya distance is used as an important measure of the separability between distributions (Bhattacharyya, 1946; Park and Lee, 1998). Because it evaluates features based on their discriminative power using their content, it is independent of the classifier type and can be generalized to other classifiers. We evaluated and defined the separability index for each feature/ channel combination (SI) as:

$$\text{SI} = \min\_{\mathcal{X}=1:\mathcal{N}\_{\boldsymbol{\varepsilon}}=1,\boldsymbol{Y}=\boldsymbol{X}+1:\boldsymbol{N}\_{\boldsymbol{\varepsilon}}} D\_{\mathbf{B}} \left\{ \boldsymbol{c}\_{\boldsymbol{X}}, \boldsymbol{c}\_{\boldsymbol{Y}} \right\}$$

where *N*c is the total number of classes available, which for this study was 4. DB{*c*1,*c*2} is the Bhattacharyya distance between the distributions of classes *cX* and *cY*. *SI* is, therefore, the minimum separability between all classes, for a given feature/channel combination. This was calculated using data from all the wrist positions. The larger the separability index, the greater the feature's ability to distinguish one class from another, thereby leading to an increased likelihood of correct class selection by a pattern recognition classifier. The separability indices were sorted in descending order. The final number of feature/channel combinations selected from this ordered list was equivalent to the number of features in the TDAR feature sets.

The wrapper method used an LDA classifier in combination with a feature selection algorithm based on the sequential forward searching (SFS) method (John et al., 1994). In SFS method, there are two sets: set A that is initially empty and set B that includes all the features. This algorithm employs an iterative search method where it selects the feature from set B that produces the minimum classification error as the first selected feature in set A. It then pairs each of the remaining features in set B with all the features in set A. The feature in set B paired with all the features in set A that generates the minimum classification error is identified and moved to set A. In each iteration, one feature in set B is selected and added to set A as the most informative feature. This method, thus, does not just select individual features that have the lowest classification error but selects features that result in the lowest classification error when paired with other features. This was performed using EMG data from the (1) extrinsic, (2) intrinsic and, (3) combination of the extrinsic and intrinsic muscles. In total, five feature sets were compared. They were as follows: TDAR features, TD features (MAV, ZC, SSC, and WL), SI features (features selected from each channel based on separability index), SFS features (features selected from each channel using the SFS method), and all features. The final number of features in the SI and SFS feature subsets was equivalent to the number of features in the TDAR feature sets. The five feature subsets were compared using an LDA classifier alone.

To test the reliability of these feature sets, sensitivity and specificity were calculated where sensitivity was defined as the number of recognized true hand motion classes divided by the total number of true hand motion classes. Specificity was defined as the number of rejected false hand motion classes divided by the total number of false hand motion classes.

To determine which features were most important, the features were added one-by-one as inputs into an LDA classifier in the order of their separability index or in the order of selection by the SFS method. Principal component analysis (PCA) was also used to transform the data into a new coordinate system such that the greatest variance in the data was explained by the first coordinate and the least variance in the data was explained by the last coordinate. The newly transformed coordinates were added one-by-one, as feature inputs into an LDA classifier in descending order of the amount of variance explained by each principal component. The minimum classification error was determined for all methods and the feature set was reduced to the set of X features that decreased error by 99%. This was done separately for extrinsic, intrinsic, and combination extrinsic and intrinsic muscle EMG data for non-amputees and amputees. The frequency of selection of each feature in this set of X features was determined and averaged across subjects.

#### Effect of EMG Feature Subset on Classification Error

To determine the effect of classifier type on classification error, a two-way repeated measures analysis of variance (ANOVA) test was performed with subject as a random effect, and muscle set and classifier type as fixed effects. This analysis was performed separately for amputees and non-amputees. To determine the effect of feature set on classification error, a two-way repeated measures ANOVA test was performed with subject as a random effect, and muscle set and feature set as fixed effects. *Post hoc* comparisons were made using a Bonferroni correction factor to determine significance. All analyses were performed separately for amputees and non-amputees using Minitab 17.3.1 (Minitab Inc. PA, USA), with a significance level set at α = 0.05.

### RESULTS

### Effect of Classifier Type, Muscle Set, and Wrist Position on Classification Accuracy

For non-amputees, performance was comparable across classifiers, except that the QDA performed significantly worse than all other classifiers. The combination of extrinsic and intrinsic muscle EMG performed significantly better than either intrinsic or extrinsic muscle EMG alone (*p* < 0.001). Using EMG from intrinsic muscles alone was significantly better than EMG from extrinsic muscles alone (**Figure 1A**) (*p* < 0.001). There was no significant interaction between the two factors (*p* = 0.06). For amputee subjects, the QDA also performed worse than all other classifiers, though this was not statistically significant (*p* = 0.2). Performance using combined EMG data from extrinsic and intrinsic muscles was significantly better than using intrinsic or extrinsic muscle EMG alone. Unlike the non-amputee data, there was no difference in performance when using EMG from extrinsic or intrinsic muscles (*p* = 0.86) (**Figure 1B**).

**Figure 2** shows the relationship between the number of wrist positions and classification error. For amputees, classification error decreased as the number or wrist positions increased, but no significant decrease in error occurred when more than four, two, or three positions are included for extrinsic (*p* = 0.07), intrinsic (*p* = 0.06), and the combination of extrinsic and intrinsic muscle EMG (*p* = 0.08), respectively. For non-amputees, error continued to significantly decrease with each additional wrist position for the extrinsic muscle and combined extrinsic and intrinsic muscle EMG. For the intrinsic muscles, no significant decrease in error occurred when more than four wrist positions were included (*p* = 0.09).

analysis; LNN, neural network with linear activation functions; MLPANN, neural network with non-linear activation functions. Error bars represent SE

### Effect of Feature Selection on Classification Error

**Figure 3** shows the average classification errors across five EMG feature sets. For both amputees and non-amputees, there was a main effect of muscle set and feature set and no significant interaction between these factors (*p* = 0.98, *p* = 0.1, respectively). The SFS feature set performed better than all other features, including feature sets that used all features, and performed significantly better than the TDAR feature set, TD feature set, and SI feature set (**Table 1**). For amputees, the SFS feature set also performed the best but was only significantly better than the TD feature set (*p* = 0.03). The analysis of the sensitivity and specificity of the feature sets revealed the same trends observed with classification accuracy and are presented in Table S1 in Supplementary Material.

**Figure 4** shows the relationship between classification error and number of features using SFS, separability indices (SI), and PCA as feature selection methods. For both non-amputees and amputees, and across all muscle groups, feature selection using SFS reached a minimum error rate at a much faster rate and with fewer features than the PCA or SI methods. For example, with the SFS method, a minimum error of 6.18% was achieved with 139 features, but with only 36 features, classification error had decreased by 99%, to 6.625%.

The probability of selection of each of the 25 features in the subset of features that account for 99% of the maximum classification

partial-hand subjects. Error bars represent SE.

(\*significantly lower than LDA, LNN, and MLPANN).

TABLE 1 | *P*-value table for pair-wise comparisons between different EMG feature sets for non-amputees.


*Bold values in the table are statistically significant (i.e. p* < *0.05).*

accuracy, averaged across subjects, is presented using the SI method (Figure S1 in Supplementary Material) and using the SFS method (**Figure 5**). Using the SI method, the features that were most and least often selected were generally consistent between amputees and non-amputees. The autoregressive features were much less likely to be selected for both non-amputees and amputees using the SI method than using the SFS method. Moreover, though the importance of the features was relatively consistent across muscles using the SI method, the importance of features differed drastically across muscle sets for the SFS method for amputees. For example, the MAV, WL, and SSC features were, respectively, the 18th, 19th, and 16th most often chosen feature from extrinsic muscle EMG data for amputees, but were the 5th, 3rd, and 8th most often chosen features, respectively, from intrinsic muscle EMG data in amputees. Some features, however, were consistently selected across EMG datasets, such as the first power spectrum descriptor (PDS1), which was the most commonly selected feature across all muscle sets, non-amputees and amputees.

### DISCUSSION

The application of pattern recognition techniques for control of externally powered myoelectric partial-hand prostheses promises to restore more function to partial-hand amputees than previously available. This work evaluated two approaches for improving the robustness of pattern recognition control against the effect of wrist position: (1) comparison of linear and non-linear classification schemes and (2) the selection of the best features taken from each channel.

Overall, the performance of all classifier types was comparable for amputees and non-amputees though the QDA performed worse than all other classifiers. This may be because unlike the LDA, the QDA is a more complex model that allows for the heterogeneity of covariance matrices for each class of data. Consequently, it requires more data to estimate more parameters and achieve high accuracies. It is also possible that the QDA performed worse because of overfitting of the training data. Although the average performance of non-amputees and amputees was different, the relative performance of different classifiers was consistent within the two groups. These findings are consistent with those of Scheme and Englehart (2011), who evaluated offline classifier performance for individuals with transradial amputations.

Among numerous possible combinations of features, TD and TDAR features (MAV, SSC, ZC, WL, and autoregressive coefficients) are commonly used. Our results show that the optimal feature set determined by sequentially adding one feature from each channel using the SFS method outperformed all other feature sets. Few studies have investigated the importance of selecting the best features from different channels. Al-Angari et al. (2016) used the Mahalanobis distance and a correlationbased method to determine the best features in each channel that were most resistant to changes in limb position. They also found a significant variation in the probability of selection of the AR features using the two feature selection methods. This is most likely because the Bhattacharyya distance, such as the Mahalanobis distance, looks at the separability of different classes for each feature, whereas SFS indirectly considers the mutual information between each feature and class and selects the feature that best improves error in conjunction with other features already in the chosen set.

Not only are some features more important only in the context of other features, but also the muscle group from which EMG is extracted greatly affects feature selection. The commonly used time-domain features MAV, WL, ZC, and SSC, which have been found to be effective in classifying hand postures, were among the least important features selected from the extrinsic muscle EMG, and the most important features selected from the intrinsic muscle data. This is most likely because these features are significantly affected by changes in extrinsic muscle EMG in

sequential forward selection; SI, feature selection using the separability indices of each feature; PCA, principal component analysis. LDA classifiers were trained and tested with data from seven wrist positions to recognize four motion classes. Shaded error bars represent SE. (A) Extrinsics: non-amputees, (B) intrinsics: non-amputees, (C) extrinsics and intrinsics: non-amputees, (D) extrinsics: amputees, (E) intrinsics: amputees, and (F) extrinsics and intrinsics: amputees.

different wrist positions, but the intrinsic muscles, which do not cross the wrist joint, are less affected by changes in wrist position. Because the majority of partial-hand amputations are caused by trauma (Ziegler-Graham et al., 2008), the intrinsic muscles can be severely damaged, or absent and, thus, not viable for EMG-based control. In such cases, it becomes more important to optimize control using extrinsic muscle EMG by selecting the appropriate features.

An optimal feature is one that both allows for discrimination between hand postures across multiple wrist positions as well as providing information that is distinct from other features. Methods, such as the SFS method, that select the best performing features that provide distinct discriminatory information about hand grasps patterns could be useful for proper pre-selection of features for classification of different hand postures in different wrist positions. We found that the time-dependent PSDs proposed by Al-Timemy et al. (2015) were reasonably well selected for both SI and SFS methods across all muscle groups, suggesting that they are less affected by changes in wrist position and provide good classification of hand grasps. The set of PSDs are extracted directly from the TD using Fourier transform relations and Parseval's theorem and, thus, keep computational costs low. Given their consistently good performance across muscle sets and subject groups, these features should be taken into consideration for future clinical implementation of pattern recognition-based systems for partial-hand prostheses.

We collected data from seven wrist positions, which can be burdensome for the user especially as the user trains the pattern recognition system with more hand grasps. We found that for amputee subjects, training in more than two to four positions provided no significant additional improvement. This study has a potential limitation in that the analyses for

non-amputees were performed offline. Some previous research has demonstrated a minimal correlation between offline performance and usability with a virtual task (Lock et al., 2005; Jiang et al., 2014); however, other studies have shown significant correlation between offline classification error and real-time control (Smith et al., 2011; Young et al., 2011). The real-time implementation would involve the pre-selection of appropriate features from each channel using SFS. Once complete, realtime classification would proceed only using those preselected features. As this method only selects relevant features, it would involve the selection of a fewer number of features than TDAR features from all channels. Given the improvement in offline performance using the SFS method particularly for the extrinsic muscles, we would expect that preselecting features that are least sensitive to wrist position would result in better performance than the TDAR features though the relationship between offline and real-time performance is unclear. Thus, further analysis of data from amputees completing tasks with the wrist in different positions in a virtual environment or with a physical prosthesis is warranted.

## CONCLUSION

In order for pattern recognition techniques to be used for control of partial-hand prostheses, the control system must be robust enough to main good control when the user moves their wrist. This research study compared the performance of linear and non-linear classification schemes and evaluated the performance of different EMG feature sets for improving pattern recognition control of hand grasps in multiple wrist positions. We found that the commonly used LDA classifier performed just as well as linear and non-linear artificial neural networks for amputees and non-amputees. We also found that selecting the best features from each channel using an SFS algorithm resulted in significant improvements over the commonly used TD feature sets and optimal feature sets. Finally, our results suggest that some of the widely used TD features are better suited for use with intrinsic muscle EMG data than extrinsic muscle data for good control across multiple wrist positions.

## AUTHOR CONTRIBUTIONS

AA helped in conceiving the study concept, collecting, analyzing, and interpreting the data; drafting the manuscript; and obtaining funding. LH helped in conceiving the study concept, interpreting the data, critically revising the manuscript for important intellectual content, obtaining funding, and supervising the study. TK helped in conceiving the study concept, interpreting the data, obtaining funding, and supervising the study. All authors read and approved the final manuscript.

### ACKNOWLEDGMENTS

The authors would like to thank A. Barlow, Ph.D. for reviewing the manuscript, and K. Turner OTR/L for help with subject recruitment and the CBM electronics team for support.

### FUNDING

The contents of this paper were developed under a grant from the Department of Health and Human Services, Administration for Community Living (ACL), NIDILRR grant number 90RE5014- 02-00, the NIH grant number T32 HD7418, the NICHD grant 1F31HD078092-01 and the UNCF-Merck Graduate Dissertation Fellowship. The contents do not necessarily represent the policies of the Department of Health and Human Services or the official views of the National Institutes of Health, and you should not assume endorsement by the Federal Government.

### REFERENCES


### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at http://journal.frontiersin.org/article/10.3389/fnbot.2016. 00015

TABLE S1 | Average specificity and sensitivity of the five feature sets for all hand motion classes, averaged across subjects.

FIGURE S1 | Probability of selection of the 25 features using the separability indices (SI). Features are ordered from most to least often selected for amputee subjects. Mean absolute value (MAV), zero crossings (ZC), slope-sign changes (SSC), waveform length (WL), Willison amplitude (WAMP), root-mean-square (RMS), variance (VAR), v-order (V-ord, order of 3), log-detector (LogDet), autoregressive (AR1–AR6) coefficients, mean frequency (MnF), median frequency (MdF), peak frequency (PF), mean power (MP), and power spectrum descriptors (PSD1–PSD6). Error bars represent SE. (A) Extrinsic muscles, (B) intrinsic muscles, and (C) extrinsic and intrinsic muscles.


*of Orthotists and Prosthetists 42nd Academy Annual Meeting and Scientific Symposium*, Orlando.


**Conflict of Interest Statement:** LH and TK have ownership interest in Coapt LLC., a start-up company that sells myoelectric pattern recognition control systems. No Coapt products were used as part of this research. The remaining author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The handling editor declared a past co-authorship with the author TK and states that the process nevertheless met the standards of a fair and objective review.

*Copyright © 2016 Adewuyi, Hargrove and Kuiken. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

*Baojun Chen, Yanggang Feng and Qining Wang\**

*The Robotics Research Group, College of Engineering, Peking University, Beijing, China*

In recent years, the development of myoelectric control for robotic lower-limb prostheses makes it possible for amputee users to volitionally control prosthetic joints. However, the human-centered control loop is not closed due to the lack of sufficient feedback of prosthetic joint movement, and it may result in poor control performance. In this research, we propose a vibrotactile stimulation system to provide the feedback of ankle joint position, and validate the necessity of combining it with volitional myoelectric control to achieve improved control performance. The stimulation system is wearable and consists of six vibrators. Three of the vibrators are placed on the anterior side of the thigh and the other three on the posterior side of the thigh. To explore the potential of applying the proposed vibrotactile feedback system for prosthetic ankle control, eight able-bodied subjects and two transtibial amputee subjects (TT1 and TT2) were recruited in this research, and several experiments were designed to investigate subjects' sensitivities to discrete and continuous vibration stimulations applied on the thigh. Then, we proposed a stimulation controller to produce different stimulation patterns according to current ankle angle. Amputee subjects were asked to control a virtual ankle displayed on the computer screen to reach different target ankle angles with a myoelectric controller, and control performances under different feedback conditions were compared. Experimental results indicated that subjects were more sensitive to stimulation position changes (identification accuracies were 96.39 ± 0.86, 91.11, and 93.89% for able-bodied subjects, TT1, and TT2, respectively) than stimulation amplitude changes (identification accuracies were 89.89 ± 2.40, 87.04, and 85.19% for able-bodied subjects, TT1, and TT2, respectively). Response times of able-bodied subjects, TT1, and TT2 to stimulation pattern changes were 0.47 ± 0.02 s, 0.53 s, and 0.48 s, respectively. Furthermore, for both TT1 and TT2, the absolute error of virtual ankle control reduced by about 50% with the addition of vibrotactile feedback. These results suggest that it is promising to apply the vibrotactile feedback system for the control of robotic transtibial prostheses.

Keywords: vibrotactile feedback, volitional myoelectric control, human-centered closed-loop control, vibrotactile stimulation, position control, robotic transtibial prostheses

### 1. INTRODUCTION

The control loop of human movement is closed by combining efferent motor output and afferent sensory feedback. For amputees, the loss of limbs not only reduces the ability of motor control but also causes the absence of some useful sensory feedback. Therefore, it is necessary to rebuild the human-centered control loop for prosthetic limbs. However, most existing studies on robotic

#### *Edited by:*

*Claudio Castellini, German Aerospace Center, Germany*

#### *Reviewed by:*

*Manfredo Atzori, University of Applied Sciences Western Switzerland Valais, Switzerland Sasha Blue Godfrey, Istituto Italiano di Tecnologia, Italy*

> *\*Correspondence: Qining Wang qiningwang@pku.edu.cn*

*Received: 24 May 2016 Accepted: 25 July 2016 Published: 22 August 2016*

#### *Citation:*

*Chen B, Feng Y and Wang Q (2016) Combining Vibrotactile Feedback with Volitional Myoelectric Control for Robotic Transtibial Prostheses. Front. Neurorobot. 10:8. doi: 10.3389/fnbot.2016.00008*

lower-limb prostheses are focused on motor control (Sup et al., 2008; Au et al., 2009; Hitt et al., 2009; Bergelin et al., 2010; Cherelle et al., 2014; Lawson et al., 2014; Wang et al., 2015), while works on sensory feedback are limited. Though amputees could still receive some haptic feedback through the interaction between residual limbs and prosthetic sockets, the information might be insufficient for the control of a robotic prosthesis with complex functionality. In addition, the lack of sensory feedback makes it difficult for amputee users to accept prostheses as their "own limbs." As a consequence, affording amputee users the ability to "feel" prosthetic limbs is a challenge for the development of robotic prostheses.

The goal of robotic prosthesis control is allowing amputee users to control prosthetic limbs in a natural and intuitive way, which is similar with that of controlling intact limbs. However, most existing lower-limb prostheses are controlled by their intrinsic controllers, and do not afford amputee users the freedom to directly control prosthetic joints. The control strategy is quite different from that of intact limbs, resulting in the absence of ownership feeling of prostheses. In recent years, several studies have been carried out to explore the potential of realizing volitional control of robotic lower-limb prostheses by amputee users with myoelectric controllers (Au et al., 2005; Ha et al., 2011; Dawley et al., 2013; Hoover et al., 2013; Wang et al., 2013; Huang et al., 2014; Chen et al., 2015). Ha et al. (2011) presented a volitional myoelectric controller for the control of a prosthetic knee during non-weight-bearing activity. Position control of the knee joint could be realized by estimating angular velocity of the knee joint using surface electromyographic (EMG) signals measured from the hamstring and quadriceps muscles. Hoover et al. (2013) developed a finite-state myoelectric controller for stair ascent with a powered transfemoral prosthesis. The controller combined proportional myoelectric torque control with a state-determined knee impedance to estimate knee torque using surface EMG measurements of muscles in the residual thigh. In our previous study, we designed a myoelectric controller for a robotic transtibial prosthesis (Chen et al., 2015). With the proposed controller, amputee users were able to volitionally adjust control parameters by actively contracting residual muscles in the shank, and could adaptively walk on the ground with varied slopes. These studies validated the promise of rebuilding the pathway of efferent motor output in the human-centered control loop for robotic lower-limb prostheses. However, the control loop is not closed due to the lack of sufficient feedback from prostheses, and control performance could, therefore, be limited.

Sensory substitution is an effective approach to provide feedback for prosthesis control (Antfolk et al., 2013). It transfers the feedback information through a different sensory channel or in a different modality (Kaczmarek et al., 1991). After training for a period of time, amputee users are able to understand the feedback information transferred by the sensory substitution system. There are several different sensory substitution methods, such as visual sensory substitution (Zambarbieri et al., 1998), auditory sensory substitution (Bamberg et al., 2010; Gonzalez et al., 2012; Yang et al., 2012), and tactile sensory substitution (Sabolich and Ortega, 1994; Wall and Kentala, 2005; Buma et al., 2007; Bark et al., 2008; Cipriani et al., 2008, 2012; Fan et al., 2008; Alahakone et al., 2010; Wheeler et al., 2010; Gopalai et al., 2011; Rusaw et al., 2012; Stepp et al., 2012; Witteveen et al., 2012; Erwin and Sup, 2014, 2015; Crea et al., 2015). Most visual sensory substitution systems are not portable, which makes them inconvenient to use in daily life. Auditory sensory substitution has a high requirement for quietness, and its performance may be impacted when talking with others. Therefore, auditory sensory substitution is not a satisfactory approach for daily application. Compared with sensory substitution systems based on visual or auditory feedback, tactile sensory substitution systems might be more practical for daily use, because they are usually wearable and will not disturb daily activities. Tactile feedback is usually provided through electrotactile stimulation (Sabolich and Ortega, 1994; Buma et al., 2007) or vibrotactile stimulation (Bark et al., 2008; Cipriani et al., 2008, 2012; Fan et al., 2008; Wheeler et al., 2010; Wentink et al., 2011; Rusaw et al., 2012; Stepp et al., 2012; Erwin and Sup, 2014, 2015; Crea et al., 2015). Compared with electrotactile stimulation, vibrotactile stimulation is more comfortable, which makes it easier to be accepted by amputee users (Kaczmarek et al., 1991). Vibrotactile stimulation systems usually produce different senses by changing stimulation parameters, such as stimulation position, frequency, amplitude, and duration. Different pieces of feedback information are given to users by activating corresponding stimulation patterns with specific combinations of stimulation parameters. Several studies have been carried out to explore the potential of applying vibrotactile feedback for the control of upper-limb prostheses (Bark et al., 2008; Cipriani et al., 2008, 2012; Wheeler et al., 2010; Stepp et al., 2012; Witteveen et al., 2012; Erwin and Sup, 2014, 2015). Among these studies, grasping force (Cipriani et al., 2008, 2012; Stepp et al., 2012) and joint position (Bark et al., 2008; Wheeler et al., 2010; Witteveen et al., 2012; Erwin and Sup, 2014, 2015) were two mostly used feedback information. Cipriani et al. (2012) proposed a vibrotactile feedback system to provide force feedback for an EMG controlled prosthetic hand. Grasping force of the prosthetic hand was measured by five cable tension sensors when grasping tasks were performed. To transfer the information of grasping force, vibration stimulation with a frequency proportional to the measured force was given to the user. Erwin and Sup (2015) presented a haptic feedback system for a virtual wrist prosthesis. The virtual wrist was controlled by a surface EMG-based controller, and a three-node tactor array was used to transfer the information of wrist joint position to subjects. Compared with the efforts made to develop vibrotactile feedback systems for upper-limb prostheses, limited studies were carried out for lower-limb prostheses (Fan et al., 2008; Rusaw et al., 2012; Crea et al., 2015). In these studies, useful movement information was provided to subjects through vibrotactile feedback, which helped the subjects to adjust their own body to improve walking stability. Fan et al. developed a haptic feedback system, which had four pneumatically controlled balloon actuators mounted on a cuff worn on the middle thigh. Four piezoresistive force sensors were integrated into a shoe insole to measure contact forces of four critical points of the foot. Sensory input from the foot was relayed to the leg by driving corresponding balloon actuators, and users were expected to "feel" the contact force of the foot by perceiving sequential stimuli (Fan et al., 2008).

Rusaw et al. (2012) proposed a similar vibratory feedback system, which produced vibration stimulations using four tactors. The system provided transtibial prosthesis users with vibratory feedback proportional to the signal received from force transducers located under the prosthetic foot. Experimental results suggested that the use of vibratory feedback improved postural stability in transtibial prosthesis users. Unlike the two previous studies, Crea et al. (2015) presented a tactile feedback system that transferred the information of gait-phase transitions rather than the contact force of the foot to walkers. Gait-phase transitions were detected with pressure-sensitive insoles, and stimulations were produced by three vibrators placed on the thigh. When different gait-phase transitions occurred, corresponding vibrators would be activated. The feedback was expected to be helpful for gait control in lowerlimb amputees. Though the above vibrotactile feedback systems could provide helpful information for lower-limb amputees to adjust their own bodies and improve walking stability, the feedback information given to amputees were used for the control of intact limbs, rather than prosthetic joints. Most of these feedback systems were tested on amputee users wearing passive prostheses, which had low requirement for prosthetic joint control. However, the feedback information might be insufficient for some robotic prostheses, whose prosthetic joints are controlled by volitional myoelectric controllers. Therefore, to close the human-centered control loop for prosthetic joints, it is necessary to rebuild the pathway of afferent sensory feedback by providing movement information of prosthetic joints to amputee users. Chew (2006) designed a vibrotactile feedback system, which embedded nine vibrator motors with the prosthetic socket liner. When a virtual ankle displayed on the computer screen was in different positions, vibrations of corresponding vibrotactile mapping patterns would be displayed. With the vibrotactile feedback, subjects were directed to control the virtual ankle to desired positions using a handled knob. Though this study explore the potential of mapping vibration patterns with ankle angle, the feedback system was not tested when working together with a myoelectric controller, which is thought to be a promising approach for amputee users to volitionally control robotic prostheses.

In this research, we aim to design a vibrotactile feedback system that feeds back the information of ankle joint position to amputee users. The system consists of six vibrators, which are placed on the anterior and posterior side of the thigh. By combining it with a volitional myoelectric controller, human-centered closed-loop control of robotic transtibial prostheses could be realized. To evaluate the promise of applying the proposed feedback system for prosthetic ankle control, eight able-bodied subjects and two transtibial amputee subjects participated in this study and four experiments were performed. The first two experiments were performed to evaluate subjects' performance of discriminating vibrations applied on different positions or with different vibration amplitudes. The third experiment was performed to verify whether subjects were able to make fast responses to stimulation position changes when continuous vibrations were applied on the thigh. In the fourth experiment, the two amputee subjects were asked to control a virtual ankle displayed on the computer screen to reach different target ankle angles using a volitional myoelectric controller. To validate the necessity of combining vibrotactile feedback with myoelectric control, control performances of the virtual ankle under different feedback conditions (no feedback, vibrotactile feedback, and two types of visual feedback) were compared. Experimental results showed that subjects had a better performance of perceiving vibration positions than discriminating vibration amplitudes. In addition, subjects were able to perceive stimulation position changes with small time delay. Control performance of the virtual ankle with vibrotactile feedback was much better than that without any feedback, and comparable with that under visual feedback conditions. These results suggest that it is promising to apply the proposed vibrotactile feedback system for robotic transtibial prosthesis control and achieve improved control performance by combining it with volitional myoelectric controllers.

The remainder of the paper is organized as follows. We introduced the hardware for vibrotactile stimulation and EMG measurement in Section 2.1, followed by the illustration of experiment protocol in Section 2.2. In Section 3, results of four experiments were reported. The discussion was presented in Section 4, and we concluded in Section 5.

### 2. MATERIALS AND METHODS

### 2.1. Hardware

### 2.1.1. Vibrotactile Stimulation System

The vibrotactile stimulation system has six miniaturized vibrators (pager motors), which are 12 mm in diameter, 3.4 mm in height, and 1.7 g in mass (**Figure 1A**). For this kind of vibrators, the vibration amplitude and frequency are coupled together. Therefore, only vibration amplitude is controlled in this study. Each vibrator is driven by a pulse width modulation (PWM) signal, and vibration amplitude is determined by the duty cycle of the PWM signal. In this research, the vibrators are divided into two groups: three of them (V1, V2, and V3) are placed in a line on the anterior side of the thigh, while the other three (V4, V5, and V6) on the posterior side of the thigh (**Figure 1B**). The distance between adjacent vibrators is about 7 cm. To improve the comfortability of wearing the stimulation system, vibrators

are pasted on a thin and stretchy sleeve worn by each subject. The vibrotactile stimulation system is controlled by a self-designed driver circuit, which receives control commands sent from a host computer through a RS232 serial interface. Control commands include IDs of activated vibrators, corresponding vibration durations, and vibration amplitudes.

### 2.1.2. EMG Measurement

Wet-gel Ag/AgCl surface electrodes (Ambu, NF-50-K) are used for EMG measurement. Two channels of EMG signals are collected from the dorsiflexor and plantar flexor muscles of amputee subject's residual shank (**Figures 2A,B**), respectively. Positions for electrode placement are determined by palpation. One electrode is placed on the bony area of the knee as the reference electrode. EMG signals are differentially amplified with a gain of 1000, fullwave rectified and lower-pass filtered with a Butterworth filter, whose cutoff frequency is 2.0 Hz. Then, the signals are amplified with a gain of 10. The above signal processing is accomplished by a self-designed circuit. The processed signals are transmitted to a host computer through a data acquisition (DAQ) card (National Instruments, NI-USB-6009). The sampling rate for signal collection is 1000 Hz.

### 2.2. Subjects and Experiment Protocol

Eight able-bodied subjects and two transtibial amputee subjects (TT1 and TT2) participated in the research and provided written and informed consent. The experiment was approved by the Local Ethics Committee of Peking University (Beijing, China). Able-bodied subjects had an average age (mean ± SD) of 26.6 ± 2.7 years, height of 176.3 ± 5.4 cm, and weight of 67.6 ± 8.0 kg. TT1's age was 34 years, height was 172 cm, and weight was 66 kg. He has been amputated (left side) for 17 years. The length of his residual shank was 12 cm (from patella to the amputated site), while the length of his sound shank was 42 cm (from patella to malleolus lateralis). TT2's age was 27 years, height was 172 cm, and weight was 75 kg. He has been amputated (right side) for 5 years. The length of his residual shank was 22 cm, while the length of his sound shank was 43 cm.

Four experiments were performed in this study. The first three experiments were designed to evaluate the sensitivity of the thigh to different types of stimulations produced by the vibrotactile feedback system. The fourth experiment was designed to evaluate the performance of controlling a virtual ankle to reach different target positions using a myoelectric controller under different feedback conditions. Both amputee and able-bodied subjects participated in the first three experiments, while only two amputee subjects participated in the last experiment. When the four experiments were performed, subjects were asked to sit on a chair, and vibrators on the posterior side of the thigh should not contact with the seat.

### 2.2.1. Perception of Stimulation Position Changes

The first experiment was designed to evaluate subjects' ability to perceive the change of stimulation positions. In each trial of the experiment, subjects received two discrete vibrations sequentially. The duration of each vibration was 200 ms, the interval between them was 400 ms. These two vibrations were produced by vibrators placed at either the same position or different positions. Vibrators were activated at the maximum amplitude. Note that these two vibrations were applied on the same side (the anterior side or the posterior side) of the thigh in a single experiment trial. After these two vibrations were produced, subjects were required to judge whether these two vibrations were applied at the same position or different positions (and more specially, moving up or down), and then clicked corresponding button displayed on the computer screen. For example, if the first vibration was produced by V1 and the second vibration was produced by V2 or V3, subjects should click the button denoting "moving up"; if the first vibration was produced by V5 and the second vibration was produced by V4, subjects should click the button denoting "moving down"; if these two vibrations were produced by the same vibrator, subjects should click the button denoting "unchanged." Button click should be completed within 3 s, otherwise it would be considered as a false identification. With the proposed vibrotactile stimulation system, 18 (3 × 3 + 3 × 3) different combinations of vibrations could be produced. In this experiment, each combination was repeated for ten times. Therefore, a total of 180 trials were tested. Test orders of experiment trials with different vibration combinations were randomly determined. Before the test trials began, subjects were asked to take several training trials. The training period terminated when subjects were familiar with the experiment task, and it usually took about 10–20 min.

### 2.2.2. Perception of Stimulation Amplitude Changes

The second experiment was designed to evaluate subjects' ability to perceive the change of stimulation amplitudes. Similar to experiment 1, in each trial of the experiment, subjects received two discrete vibrations sequentially. The duration of each vibration was 200 ms, and the interval between them was 400 ms. These two vibrations were produced by the same vibrator. The amplitude of each vibration could be 0, 50, or 100% maximum amplitude. Each experiment trial had three periods: the first stimulation period, the interval, and the second stimulation period. To avoid mistaking vibrations with 0% maximum amplitude (i.e., no vibration was produced) happened in the first

muscles, respectively.

stimulation period as the second stimulation, the current experiment period (the first stimulation, the interval, or the second stimulation) was displayed on the computer screen. After the two stimulations were produced, subjects should judge whether stimulation amplitude increased, unchanged or decreased (the second stimulation was compared with the first one), and then clicked corresponding button displayed on the computer screen. The button should be clicked within 3 s, otherwise this identification would be considered as a false one. For each vibrator, there were 9 (3 × 3) different combinations of vibrations. In this study, all the stimulation combinations were tested for all the six vibrators and repeated for three times. Therefore, a total of 162 test trials were taken. The test order of experiment trials with different vibration combinations was randomly determined. Before test trials began, subjects took several training trials to get familiar with the experiment task.

#### 2.2.3. Response to Stimulation Position Changes

The third experiment was designed to evaluate subjects' ability to make fast responses when the stimulation position changed. In each trial of the experiment, a continuous stimulation sequence with vibration position changes was applied on the subjects. For each stimulation sequence, a row of three vibrators on the anterior side or posterior side of the thigh were activated sequentially from up to down or from down to up. Vibration amplitude of each vibrator was set to be the maximum amplitude. In this research, four kinds of stimulation sequences were tested: V1 → V2 → V3, V3 → V2 → V1, V4 → V5 → V6, and V6 → V5 → V4. To avoid subjects predicting the moment of stimulation position changes, vibration duration of each vibrator was randomly ranged from 1 to 2 s. When the activation of one vibrator was terminated, the next vibrator would be activated immediately. Subjects should click a button displayed on the computer screen as soon as possible after they perceived the change of vibration position or the beginning of the stimulation sequence. As a consequence, the button should be clicked for three times in each experiment trial. In this study, each stimulation sequence was tested for 15 times and, therefore, a total of 60 test trials were taken. The test order of experiment trials with different stimulation sequences was randomly determined. Between two adjacent test trials, 5-s rest was allowed. To quantitatively evaluate the performance of each subject, response time (*TR*) of perceiving stimulation position changes is calculated by

$$T\_R = \mathbf{t}\_C - \mathbf{t}\_s,\tag{1}$$

where *tC* denotes the moment of button click and *tS* denotes the moment of stimulation position changes.

#### 2.2.4. Virtual Ankle Control

The fourth experiment was designed to validate the necessity of combining vibrotactile feedback with volitional myoelectric control. In this experiment, amputee subjects were asked to control a virtual ankle displayed on the computer screen to reach target ankle angles under four different feedback conditions (**Figure 3**).

The experiment task is similar to that performed in our previous study (Chen et al., 2014). With a previously designed myoelectric controller [please refer to Chen et al. (2014) for

more details], amputee subjects were able to volitionally control the virtual ankle by actively contracting residual plantar flexor muscles and dorsiflexor muscles. To train the myoelectric control model, subjects were asked to consciously perform dorsiflexion and plantar flexion of the "phantom" ankle with different muscle contraction intensities. EMG signals were collected and processed using 10-ms adjacent sliding windows. Average amplitudes were calculated for the two channels of EMG signals measured by the circuit, and then they were normalized by dividing the maximum amplitudes of the two channels, respectively. The normalized data were mapped to the joint angle of the virtual ankle with the myoelectric control model. Position of the ankle joint was updated every 10 ms. Myoelectric controller was trained before the experiment, and it took about 10 min.

Movement range of the virtual ankle is from −17.5° to 17.5°, where positive value denotes dorsiflexion and negative value denotes plantar flexion. In this experiment, 7 target ankle angles were tested: 0°, ±5°, ±10°, and ±15°. Each target position was tested for 6 times and, therefore, a total of 42 test trials were taken. The test order of experiment trials with different target ankle angles was randomly determined. Before each test trial began, there was a 5-s preparation period and the value of target angle was displayed on the computer screen for subjects to be prepared. When the test trials began, only 1 s was left for subjects to control the virtual ankle to reach target ankle angles, and average angle of the last 200 ms was calculated for the evaluation of control performance. The test trial was designed to mimic the scenario of the swing period during walking. In this period of a gait cycle, position control of the ankle joint plays an important role in preventing the foot from dragging along the ground and absorbing shocks when the foot strikes on the ground.

The above test trials were performed under four feedback conditions: no feedback, visual feedback 1, visual feedback 2, and vibrotactile feedback. Absolute errors of virtual ankle control under these four feedback conditions were compared. In "no feedback" condition, no feedback was given to amputee subjects when they controlled the virtual ankle. In "visual feedback 1" condition, the movement of the virtual ankle was displayed on the computer screen, but the target position was not marked. In "visual feedback 2" condition, the movement of the virtual ankle was displayed on the computer screen, and the target position was also marked. In "vibrotactile feedback" condition, subjects received vibration stimulations produced by different vibrators in real time. The ID of activated vibrator was determined by current joint angle of the virtual ankle (**Table 1**). Experiment trials of virtual ankle control with vibrotactile feedback were performed as follows. At the beginning of each trial, amputee subjects relaxed their residual muscles and no vibrator was activated (ankle angle should be about 0°). When subjects volitionally control the virtual ankle to different positions, corresponding vibrators would be activated. By perceiving the change of vibration position, subjects could be aware of whether to increase or decrease the intensity of residual muscle contraction.

### 3. RESULTS

### 3.1. Perception of Stimulation Position Changes

The average identification accuracy of perceiving stimulation position changes over eight able-bodied subjects was 96.39 ± 0.86% (i.e., mean ± SEM). The performance of discriminating vibrations applied on the anterior side of the thigh (97.36 ± 0.73%) was a little higher than that of the posterior side (95.42 ± 1.20%). To determine whether the difference was statistically significant, a paired-samples *t*-test was performed. There were no outliers in the data, as assessed by inspection of a boxplot. The assumption of normality was not violated, as assessed by Shapiro–Wilk's test (*p* = 0.170). Result of pairedsamples *t*-test revealed that accuracies of discriminating vibrations applied on the anterior side and posterior side of the thigh showed no statistically significant difference (*p* = 0.087). To make a further understanding of how identification errors



distributed, we calculated the identification accuracy for each vibration combination (**Table 2**). For vibrations applied on the anterior side of the thigh, most of the errors happened when V1 → V2 and V2 → V1 were performed. For vibrations applied on the posterior side of the thigh, most of the errors happened when V4 → V5, V5 → V4, V5 → V6, and V6 → V5 were performed.

Identification accuracies of TT1 and TT2 were 91.11 and 93.89%, respectively. Similar to the result of able-bodied subjects, TT1 had better identification performance when vibrations were applied on the anterior side (92.22%) than applied on the posterior side (90.00%) of the thigh. Most of the errors happened when V2 → V3 and V3 → V2 were performed on the anterior side, and V5 → V4, V5 → V6, and V6 → V5 performed on the posterior side (**Table 3**). For TT2, the performance of discriminating vibrations applied on the posterior side (96.67%) was better than that applied on the anterior side (91.11%), and most of the errors were caused by the misidentification of V2 → V3 (**Table 4**).

### 3.2. Perception of Stimulation Amplitude Changes

The overall identification accuracy of perceiving stimulation amplitude changes over eight able-bodied subjects was



*The top table shows the result of discriminating vibrations applied on the anterior side of the thigh, while the bottom table shows the result of discriminating vibrations applied on the posterior side of the thigh.*

TABLE 3 | Identify accuracies (%) of TT1 for different combinations of vibrator activation.


*The top table shows the result of discriminating vibrations applied on the anterior side of the thigh, while the bottom table shows the result of discriminating vibrations applied on the posterior side of the thigh.*

89.89 ± 2.40%. For each individual vibrator V1, V2, V3, V4, V5, and V6, average accuracies over eight able-bodied subjects were 90.74 ± 3.21, 89.81 ± 3.34, 89.81 ± 2.29, 89.81 ± 2.50, 89.81 ± 2.78, and 89.35 ± 3.09%, respectively (**Figure 4**). To make a further understanding of how amplitude combinations influenced the identification performance, we calculated average identification accuracies over the six vibrators for 0 vs. 0%, 0 vs. 50% (including 0 → 50% and 50 → 0%), 0 vs. 100% (including 0 → 100% and 100 → 0%), 50 vs. 50%, 50 vs. 100% (including 50 → 100% and 100 → 50%), and 100 vs. 100% maximum amplitude, respectively (**Figure 5A**). Corresponding accuracies of these stimulation combinations were 98.61 ± 1.39, 96.18 ± 1.48, 99.65 ± 0.35, 86.11 ± 5.14, 76.74 ± 5.96, and 79.17 ± 7.70%, respectively. A two-way repeated measures ANOVA was performed to

TABLE 4 | Identify accuracies (%) of TT2 for different combinations of vibrator activation.


*The top table shows the result of discriminating vibrations applied on the anterior side of the thigh, while the bottom table shows the result of discriminating vibrations applied on the posterior side of the thigh.*

determine the effects of different vibration positions and vibration amplitude combinations on identification performance. There were no outliers, as assessed by examination of studentized residuals for values greater than ±3. There was no statistically significant interaction between vibration position and vibration amplitude (*p* = 0.440). The main effect of vibration position showed no statistically significant difference in identification accuracy (*p* = 0.975). But the main effect of vibration amplitude combination showed that there was a statistically significant difference in identification accuracy (*p* < 0.01). *Post hoc* pair-wise comparisons showed that identification performance of the first three stimulation combinations were significantly better than those of the last three combinations (*p* < 0.05 for all pair-wise comparisons, i.e., any combination from the first three was compared with any combination from the last three). In addition, all pair-wise comparisons among the first three combinations and those among the last three combinations were not statistically significant.

Experimental results of TT1 and TT2 were similar with those of able-bodied subjects. For TT1, identification accuracies of vibrations applied on V1, V2, V3, V4, V5, and V6 were 92.59, 85.19, 85.19, 96.30, 81.48, and 81.48%, respectively (**Figure 4**). Average accuracies over the six vibrators were 100.00, 88.89, 91.67, 88.89, 77.78, and 77.78% for 0 vs. 0%, 0 vs. 50%, 0 vs. 100%, 50 vs. 50%, 50 vs. 100%, and 100 vs. 100% maximum amplitude, respectively (**Figure 5B**). For TT2, identification accuracies of vibrations applied on V1, V2, V3, V4, V5, and V6 were 88.89, 88.89, 77.78, 85.19, 81.48, and 88.89%, respectively (**Figure 4**). Average accuracies over the six vibrators were 100.00, 100.00, 100.00, 72.22, 69.44, and 55.56% for 0 vs. 0%, 0 vs. 50%, 0 vs. 100%, 50 vs. 50%, 50 vs. 100%, and 100 vs. 100% maximum amplitude, respectively

(**Figure 5C**). For both able-bodied and amputee subjects, most of the errors were caused by the misidentification of 50 vs. 50%, 50 vs. 100%, and 100 vs. 100% maximum amplitude. The results indicate that it is more difficult for subjects to discriminate the amplitude of a vibration (50 and 100% maximum amplitude) than judge whether a vibration happens.

We also performed a paired-samples *t*-test to compare identification performance of vibration position changes and vibration amplitude changes. No outliers were found in the data, as assessed by inspection of a boxplot. The assumption of normality was not violated, as assessed by Shapiro–Wilk's test (*p* = 0.508). Compared with the performance of discriminating stimulation amplitude changes, identification accuracy of discriminating stimulation position changes had an increment of 6.50 ± 5.73%, and the difference was statistically significant (*p* = 0.015).

### 3.3. Response to Stimulation Position Changes

The average response time to stimulation position changes was 0.47 ± 0.02 s over eight able-bodied subjects. For different vibration sequences, average response times to V1 → V2 → V3, V3 → V2 → V1, V4 → V5 → V6, and V6 → V5 → V4 were 0.46 ± 0.02, 0.47 ± 0.03, 0.46 ± 0.02, and 0.48 ± 0.02 s, respectively (**Figure 6**). To determine whether the difference was statistically significant, a one-way repeated measures ANOVA was performed. There were no outliers in the data, as assessed by inspection of a boxplot. Response time was normally distributed for each vibration sequence, as assessed by Shapiro–Wilk's test (*p* > 0.05). Mauchly's test of sphericity showed that the assumption of sphericity had not been violated (*p* = 0.603). Response times to different vibration sequences showed no statistically significant difference (*p* = 0.212). The overall response time of TT1 and TT2 were 0.53 and 0.48 s, respectively. Response times of TT1 were 0.56, 0.53, 0.53, and 0.50 s, respectively, for V1 → V2 → V3, V3 → V2 → V1, V4 → V5 → V6, and V6 → V5 → V4. As for TT2, corresponding response times were 0.50, 0.46, 0.49, and 0.48 s, respectively. The results indicated that the average response time of amputee subjects was a little longer than that of able-bodied subjects. In addition, the four types of stimulation sequences did not make a significant difference to response performance.

### 3.4. Virtual Ankle Control

For TT1, the overall absolute errors of virtual ankle control under different feedback conditions (i.e., no feedback, vibrotactile feedback, visual feedback 1, and visual feedback 2) were 4.38°, 2.18°, 2.49°, and 1.88°, respectively (**Figure 7A**). For different target ankle angles, control performances under different feedback conditions were not exactly the same. When the target ankle angle was −15°, −10°, −5°, 0° or +15°, TT1 produced much larger errors for virtual ankle control under no feedback condition than the other feedback conditions. When the target ankle angle was +5° or +10°, control performances under no feedback, vibrotactile feedback, and visual feedback 1 were close to each other, but a little worse than that under visual feedback 2 condition. For target ankle angles of −15°, −10°, −5°, 0°, +5°, +10°, and +15°, average control errors over the four feedback conditions

were 2.38°, 2.27°, 2.37°, 0.68°, 2.18°, 2.00°, and 3.01°, respectively. Though the control error was small for 0° target angle, it was not 0. It is probably caused by the variation of EMG signals and external signal noise. Control performance of TT2 was similar with that of TT1. The overall absolute errors of virtual ankle control under the four feedback conditions were 4.79°, 2.45°, 2.43°, and 1.91°, respectively (**Figure 7B**). Average control errors over the four feedback conditions were 2.49°, 3.95°, 2.80°, 0.47°, 2.61°, 3.63°, and 2.38°, respectively, for target ankle angles of −15°, −10°, −5°, 0°, +5°, +10°, and +15°. For most of the target angles, the largest control error was produced under no feedback condition, which was in consistence with the overall performance. The experimental results indicate that the overall absolute error of virtual ankle control greatly reduced (by about 50%) when any types of feedback (vibrotactile or visual) was given. The overall control performance under vibrotactile feedback condition was similar to that under visual feedback 1 condition, and only 0.30° and 0.54° larger than that under visual feedback 2 condition for TT1 and TT2, respectively.

### 4. DISCUSSION

The long-term goal of robotic prosthesis control is allowing amputee users to control prosthetic limbs as their "own limbs." Compared with traditional prosthetic controllers, humancentered control is more similar with the control of intact limbs. It allows amputee users to play a more important role in the control loop of robotic prostheses, and makes it more effective to coordinate with the movement of intact limbs and prosthetic limbs. However, most existing human-centered controllers for robotic lower-limb prostheses are open-loop. Though amputee users are able to directly control prosthetic joints with volitional myoelectric controllers, they receive insufficient feedback from robotic prostheses, which could limit the control performance. To close the loop of human-centered control, it is necessary to add artificial feedback of prosthetic joint movement to existing control systems. Therefore, we designed a vibrotactile feedback system in this research, and performed several experiments to evaluate the promise of applying it for the control of robotic lower-limb prostheses. Though some vibrotactile feedback systems have been developed for lower-limb prostheses in existing studies, most of them focused on providing feedback information (e.g., contact force of the foot or moments of specific gait events) for the adjustment of intact limbs rather to improve walking stability. Compared with these studies, the aim of this study is to improve the performance of prosthetic joint control with a volitional myoelectric controller. To achieve this goal, a vibrotactile feedback system is combined with volitional myoelectric control to close the human-centered control loop, which could improve the intuitiveness of human–machine interaction.

To make the vibrotactile feedback system more practical for prosthesis control, its design should follow three rules. First, vibrations produced by the stimulation system should be easily perceived, and different stimulation patterns should be correctly discriminated by amputee users. In this study, subjects were found to be more sensitive to stimulation position changes than stimulation amplitude changes. To improve the performance of discriminating different stimulation patterns, only vibration positions were changed for different stimulation patterns, while the vibration amplitude was set to be the maximum amplitude. Second, the provided feedback information should be helpful for the current prosthetic control system. In this study, we aim to propose a human-centered closed-loop controller for prosthetic ankle. Therefore, feedback of ankle joint position might be more appropriate than other types of feedback (e.g., contact force of the foot). In addition, accurate control of ankle joint position is important for robotic transtibial prostheses, as it is helpful to avoid the foot dragging along the ground during swing phase and improve the adaptability of walking on uneven terrains. Third, the mapping relationships between vibration patterns and feedback information should be easy to learn, allowing amputee users to understand the transferred feedback with low cognitive burden. As dorsiflexion and plantar flexion are movements in the sagittal plane, according to our experience, it is easier to map the feedback information with vibrations applied on the anterior/posterior side than on the medial/lateral side of the thigh. Therefore, three of the vibrators were placed on the anterior side of the thigh, and the other three on the posterior side, although the medial side of the thigh was found to be more sensitive to vibrotactile stimulations than the anterior side in a previous study (Wentink et al., 2011).

In this study, seven different vibration patterns were defined: one pattern corresponds to no vibration and the other six patterns correspond to vibrations produced by six individual vibrators. If more vibrators were used in the vibrotactile feedback system, more vibration patterns could be defined, and the resolution of the feedback information would be improved. However, it is easier to cause confusion of vibrations produced by adjacent vibrators with closer distance, as most of the identification errors found in the first experiment were caused by the confusion of vibrations produced by adjacent vibrators. Due to space limitation of the thigh for vibrator placement, only three vibrators were placed on each side according to our experience. Though the range of virtual ankle movement is divided into seven segments, the resolution of the feedback information could be improved using some skills. For example, if the stimulated position is kept at the lower part of the thigh in the anterior side, current angle of virtual ankle is around +5°; if the stimulated position is varied between the lower part and middle part of the thigh in the anterior side, current angle of virtual ankle is around +7.5°.

Combining the proposed vibrotactile feedback system with volitional myoelectric control is promising for improving the performance of prosthetic ankle control. For both TT1 and TT2, compared with the performance of virtual ankle control without any feedback, the performance greatly improved by about 50% when vibrotactile feedback was provided. Furthermore, control performance with vibrotactile feedback was comparable to that with visual feedback 1, and only a little worse (the average absolute error over two amputee subjects increased by 0.42°) than that with visual feedback 2. However, unlike robotic hands, visual feedback is unpractical for robotic leg control, as amputee users cannot always looking down at their feet during walking. By contrast, vibrotactile feedback is more appropriate for the control of robotic lower-limb prostheses, as it will not cause any obvious inconvenience to amputee users during walking.

Though experimental results in this research are promising, there are still some works to do to further validate the viability of applying the vibrotactile feedback system for robotic transtibial prosthesis control. In current study, the proposed

vibrotactile feedback system was only tested when subjects were seated. Whether amputee subjects could still achieve satisfactory performance of perceiving vibrations and discriminating different stimulation patterns when walking with prostheses is unknown. Compared with seated experiment trials, amputee users will receive more tactile interference when walking with prostheses, which might cause the reduction of sensitivity to vibrotactile stimulation. In addition, the requirement for response time to stimulation position changes could also increase, especially when walking at a fast speed. As a consequence, to satisfy the above requirement, more training and better stimulation techniques might be necessary. In our future work, we aim to integrate the vibrotactile stimulation system and myoelectric controller with a robotic transtibial prosthesis to close the human-centered control loop (**Figure 8A**). When amputee users walk with robotic transtibial prostheses, they are able to volitionally control the joint angle of prosthetic ankle with the myoelectric controller during swing phase. Meanwhile, they will receive vibrotactile stimulations corresponding with current ankle angle. In this case, closed-loop control of prosthetic ankle can be realized, and control performance of ankle angle during swing phase could, therefore, be improved. The wearing of vibrotactile feedback system, volitional myoelectric control system and a robotic prosthesis [adapted from our previous prosthesis PKU-RoboTPro (Wang et al., 2015)] by a transtibial subject is shown in **Figure 8B**. We will test whether the human-centered closed-loop controller could improve the performance of adaptively walking on the ground with varied slopes, and it is a follow-up of our previous study (Chen et al., 2015).

### 5. CONCLUSION

In this study, we propose a vibrotactile stimulation system to provide feedback of ankle joint position, and explore the potential of combining it with volitional myoelectric control to close the human-centered control loop for robotic transtibial prostheses. By activating vibrators placed on different positions of the thigh, the presented vibrotactile feedback system makes it easy for users to perceive different vibration patterns and understand the ankle angle transferred by the stimulation. Experimental results of virtual ankle control on two transtibial amputees suggest that it could be helpful to add vibrotactile feedback to the control loop, and it is promising to achieve improved control performance of robotic transtibial prostheses.

### AUTHOR CONTRIBUTIONS

QW and BC designed research; BC, YF, and QW performed research; BC analyzed data; and BC and QW wrote the paper.

### ACKNOWLEDGMENTS

The authors would like to thank K. Yuan for his help in the design of vibrator driver circuit.

### REFERENCES


### FUNDING

This work was supported by the Beijing Disabled Persons Federation, the National Program for Support of Top-notch Young Professionals, the National Natural Science Foundation of China (No. 61005082, 61533001), the Beijing Nova Program (No. Z141101001814001), and the Beijing Municipal Science and Technology Project (No. Z151100003715001, Z151100000915073).

in *2013 IEEE International Conference on Rehabilitation Robotics (ICORR)* (Seattle, WA: IEEE), 1–6.


gait asymmetry in transtibial amputation gait. *Gait Posture* 36, 631–634. doi:10.1016/j.gaitpost.2012.04.004

Zambarbieri, D., Schmid, M., Magnaghi, M., Vermi, G., Macellari, V., and Fadda, A. (1998). "Biofeedback techniques for rehabilitation of the lower limb prosthetic subjects," in *Proceedings of the VII Mediterranean Conference on Medical and Biological Engineering and Computing (MEDICON)*, Lemesos, 1–5.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2016 Chen, Feng and Wang. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*