# NEUROMECHANICS AND CONTROL OF PHYSICAL BEHAVIOR: FROM EXPERIMENTAL AND COMPUTATIONAL FORMULATIONS TO BIO-INSPIRED TECHNOLOGIES

EDITED BY : Massimo Sartori, Francisco J. Valero-Cuevas, Alfred C. Schouten, Matthew Tresch, Yoshihiko Nakamura and Manish Sreenivasa PUBLISHED IN : Frontiers in Computational Neuroscience and Frontiers in Neurorobotics

#### Frontiers Copyright Statement

© Copyright 2007-2019 Frontiers Media SA. All rights reserved. All content included on this site, such as text, graphics, logos, button icons, images, video/audio clips, downloads, data compilations and software, is the property of or is licensed to Frontiers Media SA ("Frontiers") or its licensees and/or subcontractors. The copyright in the text of individual articles is the property of their respective authors, subject to a license granted to Frontiers.

The compilation of articles constituting this e-book, wherever published, as well as the compilation of all other content on this site, is the exclusive property of Frontiers. For the conditions for downloading and copying of e-books from Frontiers' website, please see the Terms for Website Use. If purchasing Frontiers e-books from other websites or sources, the conditions of the website concerned apply.

Images and graphics not forming part of user-contributed materials may not be downloaded or copied without permission.

Individual articles may be downloaded and reproduced in accordance with the principles of the CC-BY licence subject to any copyright or other notices. They may not be re-sold as an e-book.

As author or other contributor you grant a CC-BY licence to others to reproduce your articles, including any graphics and third-party materials supplied by you, in accordance with the Conditions for Website Use and subject to any copyright notices which you include in connection with your articles and materials.

All copyright, and all rights therein, are protected by national and international copyright laws.

The above represents a summary only. For the full conditions see the Conditions for Authors and the Conditions for Website Use. ISSN 1664-8714 ISBN 978-2-88945-946-9 DOI 10.3389/978-2-88945-946-9

#### About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

#### Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

#### Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

#### What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office: researchtopics@frontiersin.org

# NEUROMECHANICS AND CONTROL OF PHYSICAL BEHAVIOR: FROM EXPERIMENTAL AND COMPUTATIONAL FORMULATIONS TO BIO-INSPIRED TECHNOLOGIES

Topic Editors:

Massimo Sartori, University of Twente, Netherlands Francisco J. Valero-Cuevas, University of Southern California, United States Alfred C. Schouten, Delft University of Technology, Netherlands Matthew Tresch, Northwestern University, United States Yoshihiko Nakamura, The University of Tokyo, Japan Manish Sreenivasa, University of Wollongong, Australia

Cover image: Modified from Jane\_Mori/Shutterstock.com

The term "neuromechanics" defines an integrative approach that combines the neuromuscular control and the biomechanical aspects of physical behavior in humans and animals. Crucial to this approach is a detailed description and modeling of the interaction between the nervous system and the controlled biomechanical plant. Only then do we have the broader context within which to understand evolution, movement mechanics, neural control, energetics, disability and rehabilitation.

In addition to enabling new basic science directions, understanding the interrelations between movement neural and mechanical function should also be leveraged for the development of personalized wearable technologies to augment or restore the motor capabilities of healthy or impaired individuals. Similarly, this understanding will empower us to revisit current approaches to the design and control of robotic and humanoid systems to produce truly versatile human-like physical behavior and adaptation in real-world environments. This Research Topic is therefore poised at an opportune moment to promote understanding of apparently disparate topics into a coherent focus.

Citation: Sartori, M., Valero-Cuevas, F. J., Schouten, A. C., Tresch, M., Nakamura, Y., Sreenivasa, M., eds. (2019). Neuromechanics and Control of Physical Behavior: From Experimental and Computational Formulations to Bio-inspired Technologies. Lausanne: Frontiers Media. doi: 10.3389/978-2-88945-946-9

# Table of Contents

## CHAPTER 1

#### INTRODUCTION

*06 Editorial: Neuromechanics and Control of Physical Behavior: From Experimental and Computational Formulations to Bio-inspired Technologies* Manish Sreenivasa, Francisco J. Valero-Cuevas, Matthew Tresch, Yoshihiko Nakamura, Alfred C. Schouten and Massimo Sartori

## CHAPTER 2

#### EXPERIMENTAL AND COMPUTATIONAL TECHNIQUES

*10 The Dynamics of Voluntary Force Production in Afferented Muscle Influence Involuntary Tremor*

Christopher M. Laine, Akira Nagamori and Francisco J. Valero-Cuevas


Keivan Mojtahedi, Bryan Whitsell, Panagiotis Artemiadis and Marco Santello

*46 Forearm Flexor Muscles in Children With Cerebral Palsy are Weak, Thin and Stiff*

Ferdinand von Walden, Kian Jalaleddini, Björn Evertsson, Johanna Friberg, Francisco J. Valero-Cuevas and Eva Pontén


## CHAPTER 3

#### MODELLING FORMULATIONS

*97 Predictive Simulation of Reaching Moving Targets Using Nonlinear Model Predictive Control*

Naser Mehrabi, Reza Sharif Razavian, Borna Ghannadi and John McPhee

*109 Evaluation of a Neuromechanical Walking Control Model Using Disturbance Experiments*

Seungmoon Song and Hartmut Geyer


## CHAPTER 4

#### BIO-INSPIRED TECHNOLOGIES


Alexei V. Alexandrov, Vittorio Lippi, Thomas Mergner, Alexander A. Frolov, Georg Hettich and Dusan Husek


Andrés Úbeda, José M. Azorín, Dario Farina and Massimo Sartori

# Editorial: Neuromechanics and Control of Physical Behavior: From Experimental and Computational Formulations to Bio-inspired Technologies

Manish Sreenivasa<sup>1</sup> \*, Francisco J. Valero-Cuevas <sup>2</sup> , Matthew Tresch3,4 , Yoshihiko Nakamura<sup>5</sup> , Alfred C. Schouten6,7 and Massimo Sartori <sup>7</sup> \*

*<sup>1</sup> Department of Mechanical, Material, Mechatronics and Biomedical Engineering, University of Wollongong, Wollongong, NSW, Australia, <sup>2</sup> Division of Biokinesiology and Physical Therapy, Department of Biomedical Engineering, University of Southern California, Los Angeles, CA, United States, <sup>3</sup> Shirley Ryan AbilityLab, Chicago, IL, United States, <sup>4</sup> Department of Biomedical Engineering, Northwestern University, Evanston, IL, United States, <sup>5</sup> Department of Mechano-Informatics, School of Information Science and Technology, University of Tokyo, Tokyo, Japan, <sup>6</sup> Department of Biomechanical Engineering, Delft University of Technology, Delft, Netherlands, <sup>7</sup> Department of Biomechanical Engineering, University of Twente, Enschede, Netherlands*

Keywords: sensory-motor system, biorobotics, neuromechanical modeling, motor control, neuro-technologies, neural control of movement

**Editorial on the Research Topic**

**Neuromechanics and Control of Physical Behavior: From Experimental and Computational Formulations to Bio-inspired Technologies**

## INTRODUCTION

#### The motivation behind this research topic was to cut across conventional boundaries that separate movement neuroscience, biomechanics, and robotics. The aim was to underscore that brain and body collaborate to produce behavior in biological organisms. While this is a simple idea, compartmentalization in education and science has often artificially separated brain from body. We also bring forward research paradigms to investigate physical behavior at the interface between humans and interacting robots. Understanding human-robot physical interaction requires the understanding of the complex interplay between brain, body, and the external environment. This could be achieved by employing a neuro-mechanical approach to the study of human and robot movement.

Within the context of a neuromechanical approach, we aimed to collate a research corpus that included work on experimental and neurophysiological analysis, computational modeling, and applications in rehabilitation and bio-inspired robotics. The 22 contributions to this research topic provide a wide range of perspectives and methodologies. The high-caliber contributions to this research topic also highlight the existence of a significant community of researchers interested in an interdisciplinary view toward the study of brain-body and human-robot interactions.

## THE CONTRIBUTIONS

Several of the studies enclosed in this research topic combined experimental work with state-of-the-art technologies and novel methodologies to measure and analyze neural control in humans (Banks et al.; Mojtahedi et al.; Logan et al.; Krüger et al.; Reyes et al.; Úbeda et al.).

Edited and reviewed by: *Si Wu,*

#### *Peking University, China* \*Correspondence:

*Manish Sreenivasa manishs@uow.edu.au Massimo Sartori m.sartori@utwente.nl*

Received: *23 January 2019* Accepted: *15 February 2019* Published: *19 March 2019*

#### Citation:

*Sreenivasa M, Valero-Cuevas FJ, Tresch M, Nakamura Y, Schouten AC and Sartori M (2019) Editorial: Neuromechanics and Control of Physical Behavior: From Experimental and Computational Formulations to Bio-inspired Technologies. Front. Comput. Neurosci. 13:13. doi: 10.3389/fncom.2019.00013* A common theme among many studies was a cohesive and complimentary use of numerical modeling, signal processing and experimental approaches, with one supporting and enhancing the findings of the other to understand the basic mechanisms of movement (Guarín and Kearney; Golkar et al.; Song and Geyer; Petric et al. ˇ ; Laine et al.; Mehrabi et al.; Sreenivasa et al.; Alexandrov et al.; Lippi and Mergner; Pasma et al.; Von Walden et al.). Yet another inspiring aspect of this research topic was the crossover toward efforts in biomarkers of pathology and rehabilitation (Laine et al.; Reyes et al.; Sreenivasa et al.; Von Walden et al.; Shuman et al.; Banks et al.; Úbeda et al.) and robot control (Pasma et al.; Lippi and Mergner; Alexandrov et al.; Szczecinski et al.). Taken together, the contributions to this research topic illustrate the wide scope of research being conducted in neuromechanics and the vital role this will play in the future of rehabilitation and robotics.

The remainder of this editorial provides an overview of the scientific contributions. These are presented within three macro categories including:


## NEUROMECHANICS

Although there is detailed understanding of the mechanisms taking place in the central nervous system or in the musculoskeletal system during movement, there is far less knowledge of the interplay between these two systems during complex motor tasks (Tresch and Jindrich, 2014; Sartori et al., 2016, 2017; Sreenivasa et al., 2016; Valero-Cuevas and Santello, 2017; Cohn et al., 2018; Schouten and Mugge, 2019). This motivates a neuromechanical approach to the study of human behavior when producing movements and forces to interact with the environment.

One way to better understand the interplay between the neural and biomechanical apparatus is by identifying the system dynamics using a combination of models and experimental observations. Guarín and Kearney developed a methodology to dissociate the time-varying intrinsic and reflex components during movements at the human ankle joint. Their results indicate that the joint stiffness is modulated dynamically during non-stationary movements. The relevance of their work lies in proposing methodologies for extracting joint stiffness profiles during dynamic conditions, a step forward with respect to current methods that predominantly investigate quasi-stationary scenarios. Golkar et al. also focused on dynamic stiffness about the ankle joint, using human experiments to study the modulation of intrinsic and reflex stiffness with muscle activation. Logan et al. applied harmonic transfer function analysis to investigate how humans control upright posture and speed during locomotion. Their results support the existence of a temporal hierarchy of subtasks during locomotion, with the control upper-body posture taking precedence over others. Schumacher and Seyfarth used a feedback-based neuromuscular modeling approach to study reflex gains during hopping. Interestingly, their results indicate that different combinations of gain values enable optimizing specific hopping characteristics (e.g., performance and efficiency), and that this was invariant with respect to changes in the model's mechanical properties. With another feedback-based approach, Song and Geyer investigated the response of a walking model to simulated disturbances such as electrical stimulation and perturbations applied to the legs and to the whole-body. Their results show that the model behavior compares favorably to experimental recordings of similar disturbances during human locomotion. In a conclusion that resonates with this research topic, the authors comment on how such model approaches can complement the experimental study of human motor control.

Another branch of contributions investigated whole-body human movements (either free or when interacting with the environment), which require the coordination of multiple degrees of freedom, and are often directed toward completing specific tasks accurately and efficiently. The study of such movements provides the means to answer fundamental questions about motor control and brain-body interaction. Krüger et al. applied canonical correlation analysis to study movement variability in discrete goal-directed reaching tasks. They tested the effects of ischemia, (temporary) artificially reduced blood flow to the arms, and found that while this reduced the complexity of movement control, the endpoint variability did not increase. Petric et al. ˇ comment on the lack of movement studies on real-world tasks such as hammering a nail into wood. In their work, they investigated the kinematics and dynamics of hammering both from an experimental and modeling perspective. They found that for periodic impact tasks, a model that relates the distance moved as a logarithmic function of time provided better predictions than Fitts' Law. Le Mouel and Brette proposed the hypothesis that postural adjustments follow spatial and temporal patterns that provide impetus for future movement. In a thought-provoking article, they relate this ability to learned proficiency in movements and the impairment of these skills with aging. Mojtahedi et al. studied how two humans collaborate to complete a physical task and the strategies used to infer movement direction. Interestingly, their findings suggest that the modulation of arm stiffness during cooperative tasks could be an effective means to communicate intended direction of movement. The authors relate this to future applications in human-robot interaction.

Even the simplest of movements often require the coordinated and smooth control of several muscles. Investigations at the neuromuscular level can provide an interesting way to reverseengineer how the brain controls movements. Reyes et al. studied corticomuscular and intermuscular coherence between the muscles of the finger and the thumb during pinch grips. They found significantly reduced coherence when individuated control of the thumb and index finger was required, and interpret their findings as supporting the notion that the cortex bonds task-related motor neurons into task-dependent functional units. Laine et al. used complimentary experimental and modeling approaches to study how the neuromechanics of voluntary force production in the index finger were sufficient to produce involuntary tremor. The authors suggest that their findings represent a new category of tremor, one related to the viscoelastic response of a closed-loop musculotendon system to dynamic inputs. Moreover, they underscore the importance of coherence analysis, a well-established and neurophysiologically informative phenomenon (Farmer, 1998), to the study of muscle coordination in neuromechanics. Mehrabi et al. modeled a musculoskeletal planar arm and compared predictions using non-linear model predictive control (NMPC), to those from dynamic optimization and to experimental recordings. The authors motivate the NMPC approach by highlighting its effectiveness in simulating motion where the kinematics may or may not be prescribed, and where the target position may move during execution. Their results show that NMPC predicted the hand trajectory quite well, but not the hand velocities or muscle activations. Sreenivasa et al. developed optimal control-based predictive simulations of a child's gait with and without an anklefoot orthosis. The motivation behind their work was to provide modeling-based tools to identify the patient-specific optimal stiffness of the orthosis, while reducing the effort required for the clinical procedure of fitting an orthosis to a patient. These studies inform the debate on the extent to which optimality (as understood by engineers) occurs in the neural control of our bodies (Loeb, 2012).

## BIOTECHNOLOGY AND REHABILITATION

The knowledge gained from the study of human motor control, and the methods developed to analyze and synthesize human biomechanics are invaluable for the development of biomarkers of pathology, biotechnologies and rehabilitation processes that directly interact with the human body during complex movements. In a review article, Pizzolato et al. set an optimistic yet cautious note on the application of computational musculoskeletal models for the design and control of wearable devices. An interesting focus of the review was on bioinspired technologies that not only support or augment movement, but also foster tissue strengthening and repair through optimal tissue loading. Von Walden et al. studied the forearm flexor muscles in children with cerebral palsy (CP). They experimentally record the muscle strength and cross-sectional area, and use a modeling approach to estimate passive muscle stiffness independent of reflex activity. With a study title that conveys a concise takehome message, the authors found that the forearm flexor muscles of children with CP are weak, thin, and stiff, something that may inform personalized treatments. This underscores the plastic relationship between brain and body: damage to the neural controller can lead to physical changes in the anatomical actuators and plant.

The conclusions from Shuman et al. and Banks et al. are of significant importance to the research on muscle synergies during movements. Shuman et al. comment on the impact that filtering and scaling of recorded electromyography (EMG) data can have on the computed synergies for CP and typicallydeveloping children, and the implications this can have when comparing results across studies and laboratories. Banks et al. investigated 30 variations in the methodology used for muscle synergy analysis, and evaluated the impact of these variations in identifying responders in post-stroke population. Their results highlight the sensitivity of MSA to methodological choices, and a need to standardize and/or provide exacting detail about the methodology used in future work. Úbeda et al. proposed a method to decode spinal primitives of multi-muscle control from electroenchephalography (EEG) recordings. The authors apply their method to healthy individuals as well as to patients with incomplete spinal cord injury, and comment on the future usage toward a new class of brain-exoskeleton interfaces. This work, together with that of Reyes et al. and Laine et al. helps clarify the ongoing debate on whether and how one can detect prescriptive synergies of neural origin—as opposed to descriptive synergies of numerical origin (Brock and Valero-Cuevas, 2016).

## BIO-INSPIRED ROBOTICS

Robot sensors and actuators remain vastly different from those on humans and animals, yet bio-inspired control methods find rich application in robotics. Szczecinski et al. developed methodology to design subnetworks that perform specific mathematical operations, and can be assembled into larger networks to mimic to some extent an animal's nervous system. Humanoid robots, with their anthropomorphic form and capabilities for bipedalism, are an excellent example for the translation of insights from human movement research to robotics. One interesting focus is on maintaining balance and posture, with robots possibly learning from humans, but also providing a real-world platform to test models and hypotheses for human balance. Alexandrov et al. proposed the use of the eigen-movement control concept derived from human hip-ankle coordination, as a simpler and more stable way to control humanoid robot balance. Lippi and Mergner implemented a modular human-derived control architecture for maintaining balance and posture in a humanoid robot. Interestingly, their results show that the mechanical coupling from the robot's body was sufficient to stabilize movements in the frontal and sagittal planes, without a need for an explicit link between the respective control modules. Pasma et al. quantified human balance control using the independent channel (IC) model and implemented the IC model in a humanoid robot. Their results show that the IC model, a descriptive model in the frequency domain, can imitate human balance behavior in real world situations with a humanoid robot. This provides further evidence that the IC model is a valid description of human balance control.

## CONCLUSIONS

This research topic posed direct focus on the neuromechanics of movements and the forces generated in interaction with the environment. This is described as an integrative approach that combines the neuromuscular control and the biomechanical aspects of body and the physics of the tasks in humans. We assert that this approach is overdue and necessary to obtain the theoretical and experimental frameworks for understanding the evolution, mechanics, neural control, energetics, disability, and rehabilitation of physical behavior. Moreover, understanding the neuromechanics of physical behavior should also leverage the development of personalized wearable robotic technologies that can interact with biological tissues within the composite neuromuscular system. This is central to ultimately mimic, restore or augment motor capabilities in healthy or impaired individuals.

## AUTHOR CONTRIBUTIONS

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

## REFERENCES


## FUNDING

Research reported in this publication was supported by the National Institute of Arthritis and Musculoskeletal and Skin Diseases of the National Institutes of Health under Awards Number R01 AR- 050520 and R01 AR-052345 to FV-C. This work was also supported by the Department of Defense CDMRP Grant MR150091 and Award W911NF1820264 from the DARPA-L2M program to FV-C. MCT acknowledges funding from the NIH under the grant number NS086973. This work was partly supported by the European Research Council (ERC) Starting Grant INTERACT (No. 803035) awarded to M. Sartori, under the European Union's Horizon 2020 research and innovation programme.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Sreenivasa, Valero-Cuevas, Tresch, Nakamura, Schouten and Sartori. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# The Dynamics of Voluntary Force Production in Afferented Muscle Influence Involuntary Tremor

#### Christopher M. Laine1, 2, Akira Nagamori <sup>2</sup> and Francisco J. Valero-Cuevas 1, 2 \*

<sup>1</sup> Department of Biomedical Engineering, University of Southern California, Los Angeles, CA, USA, <sup>2</sup> Division of Biokinesiology and Physical Therapy, University of Southern California, Los Angeles, CA, USA

Voluntary control of force is always marked by some degree of error and unsteadiness. Both neural and mechanical factors contribute to these fluctuations, but how they interact to produce them is poorly understood. In this study, we identify and characterize a previously undescribed neuromechanical interaction where the dynamics of voluntary force production suffice to generate involuntary tremor. Specifically, participants were asked to produce isometric force with the index finger and use visual feedback to track a sinusoidal target spanning 5–9% of each individual's maximal voluntary force level. Force fluctuations and EMG activity over the flexor digitorum superficialis (FDS) muscle were recorded and their frequency content was analyzed as a function of target phase. Force variability in either the 1–5 or 6–15 Hz frequency ranges tended to be largest at the peaks and valleys of the target sinusoid. In those same periods, FDS EMG activity was synchronized with force fluctuations. We then constructed a physiologically-realistic computer simulation in which a muscle-tendon complex was set inside of a feedback-driven control loop. Surprisingly, the model sufficed to produce phase-dependent modulation of tremor similar to that observed in humans. Further, the gain of afferent feedback from muscle spindles was critical for appropriately amplifying and shaping this tremor. We suggest that the experimentally-induced tremor may represent the response of a viscoelastic muscle-tendon system to dynamic drive, and therefore does not fall into known categories of tremor generation, such as tremorogenic descending drive, stretch-reflex loop oscillations, motor unit behavior, or mechanical resonance. Our findings motivate future efforts to understand tremor from a perspective that considers neuromechanical coupling within the context of closed-loop control. The strategy of combining experimental recordings with physiologically-sound simulations will enable thorough exploration of neural and mechanical contributions to force control in health and disease.

Keywords: tremor, force control, muscle models, closed-loop system, dynamics

## INTRODUCTION

It is well known that humans cannot produce a perfectly stable force. Within the context of precise, goal-directed actions, involuntary force fluctuations can reveal clinically relevant information about neuromuscular control in disorders such as dystonia (Xia and Bush, 2007; Chu and Sanger, 2009), Parkinson's disease (Vaillancourt et al., 2001; Ko et al., 2015), bruxism (Laine et al., 2015b),

#### Edited by:

Sliman J. Bensmaia, University of Chicago, USA

#### Reviewed by:

Andrew Joseph Fuglevand, University of Arizona, USA Juan Álvaro Gallego, Northwestern University, USA Helen Huang, North Carolina State University and University of North Carolina at Chapel Hill, USA

\*Correspondence:

Francisco J. Valero-Cuevas valero@usc.edu

Received: 29 April 2016 Accepted: 02 August 2016 Published: 19 August 2016

#### Citation:

Laine CM, Nagamori A and Valero-Cuevas FJ (2016) The Dynamics of Voluntary Force Production in Afferented Muscle Influence Involuntary Tremor. Front. Comput. Neurosci. 10:86. doi: 10.3389/fncom.2016.00086 and essential tremor (Héroux et al., 2010), among others. In such tasks, the nature of force variability may be influenced by both central and peripheral components of sensorimotor integration.

Unfortunately, the utility of measuring involuntary force fluctuations (i.e., tremor) within scientific or clinical settings has been limited due to the large and often ambiguous set of factors which can influence such measures. In some cases, tremor may reflect a mechanical resonance whose frequency depends on the physical characteristics of the muscle/limb in question (Lakie et al., 2012; Vernooij et al., 2013). At the same time, tremor may stem from cycles of excitation around the stretch-reflex loop (Lippold, 1970; Young and Hagbarth, 1980; Christakos et al., 2006; Erimaki and Christakos, 2008). The two mechanisms likely interact, since reflex activity is itself influenced by muscle/tendon compliance (Rack et al., 1983), limb loading (Joyce and Rack, 1974), contraction history (Gregory et al., 1998), and the temporal dynamics of force production (Xia et al., 2005).

It is clear that the specific type and extent of neuromechanical coupling influencing performance of a given task are of key importance for understanding the generation of force variability. Understanding the factors which influence dynamic force control is especially important given that this is the basis of manual dexterity during activities of daily living. However, the neural and/or mechanical origins of unintended force variability are not always clear, particularly within the context of dynamic force control.

In this study, we investigated the relationship between voluntary force production and involuntary force variability in a group of healthy adults engaged in a dynamic, isometric force tracking task. Given the various links between reflex activity, contraction dynamics, and tremor, our hypothesis was that involuntary force variability would depend upon voluntary contraction dynamics. In order to better understand the potential sources of force variability within our experimental task, we used a physiologically-realistic computer simulation to determine the sufficiency of muscle-tendon mechanics and reflex pathways to reproduce our experimental results. The simulation also allowed us to characterize the sensitivity of force variability to parameters such as reflex gain.

The significance of our study is two-fold. First, we describe a novel source of tremor along with a method for its experimental induction, and strong evidence for its origin in musculotendon dynamics. Second, the sensitivity of this tremor to both neural and mechanical factors within our simulation implies that simple force tracking tasks, such as described here, may represent a novel approach to investigating peripheral components of sensorimotor integration in health and disease.

#### METHODS

All procedures were approved by the institutional review board at the University of Southern California and all participants gave informed written consent prior to participation. Ten healthy participants were recruited (4 female, 6 male, aged 23–31 years) to carry out force tracking experiments.

## Physiological Data

#### Task

Participants were seated ∼1 m from a 17-inch computer monitor which displayed a sinusoidal target with a vertical range representing forces from 5 to 9% of the maximum force that each individual could exert with the index finger of their self-reported dominant hand (see **Figure 1A**). Visual feedback of exerted force was provided in the form of a cursor which moved left to right across the computer screen for 40 s before looping back to the left. Prior to recordings, participants practiced tracking several target cycles to become familiar with the task. Each participant then tracked the 0.25 Hz sinusoidal target for two 80 s trials separated by several minutes of rest. A slow sinusoidal target is a rich behavior that is ideal for probing dynamic dependencies. For example, if tremor depended on force velocity, then tremor amplitudes would appear to follow the derivative of the target sinusoid (i.e., a cosine). If one direction of force (increasing vs. decreasing) were tied to tremor amplitudes, then tremor amplitudes would be largest along either the rising or falling phase of the target sinusoid. If the magnitude of force were most relevant, tremor amplitudes would essentially follow the target trajectory, being largest at the peaks and smallest at the valleys.

As depicted in **Figure 1A**, a miniature single-axis force transducer was fixed to the top of a plastic cylinder and located under the tip of the finger. Participants were asked to produce a downward force perpendicular to the force sensor, an action requiring contraction of the index finger slip of the flexor digitorum superficialis (FDS) muscle. This particular muscle and joint action were chosen because flexion at the proximal interphalangeal (PIP) joint is necessary for manipulation activities of daily living, and because this straightforward mechanical action is well suited for simulation.

#### Force and EMG Measurements

Surface EMG recordings were made over the distal portion of the index finger slip of the FDS muscle using an active bipolar electrode (Biometrics Ltd, Newport, UK) grounded at the wrist. Confirmation of correct electrode positioning was accomplished via palpation of the distal muscle belly (∼7 cm proximal to the crease of the wrist, on the ulnar side) during index finger flexion, as well as observation of ongoing EMG signals during PIP joint flexion/extension and during our isometric task. The EMG signals were acquired at 1000 samples per second using a Biometrics DataLog system and associated software. The measurement of force, and the display of visual feedback to participants, was accomplished using custom MATLAB (The MathWorks, Natick, MA, USA) scripts to acquire force signals from a miniature load cell (ELB4-10, Measurement Specialties, Hampton, VA, USA) using a USB-DAQ (National Instruments, Austin TX, USA). The data acquisition unit sent a synchronization pulse to the biometrics system at the start of each recording. The data were analyzed offline using custom MATLAB scripts.

## Data Analysis

#### Conversion of Sinusoidal Force to Instantaneous Phase

To uncover the slow, voluntary force associated with the intended target trajectory, the force produced by each participant was low-pass filtered at 0.5 Hz. Using the Hilbert transform, the instantaneous phase of this tracking force was calculated and expressed in degrees (0–360◦ ) over the course of each target cycle. This conversion was useful since instantaneous phase holds information about the actual dynamics of force production at a given time, regardless of tracking error. Although tracking error was not a focus of this investigation, it was still important to eliminate poorly tracked target cycles. Target cycles in which the absolute tracking error exceeded 4% of a participant's MVC level at any time point were excluded from all further analysis.

#### Calculation of Instantaneous Tremor Amplitude

To quantify the presence of involuntary force fluctuations, the sinusoidal force trajectories produced by each participant were filtered into two different frequency bands.

First, we investigated the presence and magnitude of force fluctuations at high frequencies (>6 Hz), which cover the frequency range of physiological tremor (Lippold, 1970; Elble and Randall, 1976; Burne et al., 1984; Christakos et al., 2006). These force fluctuations were extracted by band-pass-filtering the force produced by each individual between 6 and 15 Hz (zero-phase, 4th order Butterworth filter). Beyond about 15 Hz, the amplitude of force fluctuations is essentially negligible due to the low-pass filtering effects of tissue (finger pad, tendon) and muscle.

Second, we quantified slow (1–5 Hz) force fluctuations. Generally, slow fluctuations in force stem from changes in the overall drive to motor neurons (Allum et al., 1978; De Luca et al., 1982; Miall et al., 1993; Slifkin et al., 2000; Squeri et al., 2010). These slow fluctuations do include voluntary correction of tracking errors, but our analysis focused on fluctuations that were consistently present at particular phases of the target cycle, and therefore reflect an involuntary process. To extract these fluctuations, we used a 1–5 Hz band-pass-filter (zero phase, 4th order Butterworth filter).

An example of band-pass filtered force traces in relation to the target sinusoid is depicted in **Figure 1A** (right).

#### Calculation of Tremor Modulation as a Function of Tracking Phase

As described above, the tracking force produced over time by each participant was converted to a trace of instantaneous phase angles where each complete target cycle was represented as a progression from 0 to 360◦ . Each cycle was then divided into 36 phase-bins (each representing 10◦ ). Again, it should be noted that all analyses are based on the temporal dynamics of the force produced by the participants and not on the displayed target. This renders any positive or negative tracking lags irrelevant [although they would be minimal given the highly feed-forward nature of this type of task (Erimaki et al., 2013)]. To examine the relationship between tracking phase and force variability, we first converted each band-pass filtered force signal into an instantaneous amplitude signal by rectification

FIGURE 1 | Experimental task and simulation. Participants were asked to produce isometric force against a small load cell with the tip of the index finger (A). Using feedback of their applied force, participants tracked a 0.25 Hz sinusoid during 80 s trials. The range of the target sinusoid was 5–9% of each individual's maximum voluntary contraction (MVC) force. An example of the recorded force from one participant is depicted to the right (top trace). Below, the same force trace has been filtered into 2 different bands, a 1–5 Hz band which captures slow tremor and tracking error (middle), and a 6–15 Hz band which reveals physiological tremor and fast twitches (bottom trace). In this example, both high and low frequency force fluctuations appear to depend upon the phase of the tracked sinusoid. (B) Depicts the implementation of a control loop used to simulate the tracking experiment above. At the "spinal cord," three sources of input are summated. The first two sources are proprioceptive signals from the muscle spindle (via group Ia and II afferents), and Golgi tendon organ (GTO) (via Ib afferents) models, which provide positive and negative feedback to the "spinal cord," respectively. The third input is a tracking signal sent from a supraspinal controller. This "tracking controller" sends an output (C) which is continually updated according to the difference between the target and the force (F) produced by the muscle (see text for details). From the "spinal cord," a neural drive (ND) signal is sent to a Hill-type muscle model via an "activation filter" which further shapes the neural drive to account for calcium dynamics within physiological muscle. The filtered muscle activation (MA) signal sent from the "activation filter" generates contraction (Fm), accounting for physical properties of the muscle-tendon complex, such as a series and parallel elastic element (SE and PE), mass (M), viscosity (B), and pennation angle (alpha). The delays and sign associated with each feedback loop are also depicted.

and smoothing with a 200 ms Gaussian window. The magnitude of the resulting smoothed/rectified signal also serves as a simple estimation of instantaneous variance within the specified frequency band, given the equivalence between total signal power (in frequency domain) and total signal variance (in time domain) (i.e., Parseval's theorem). We then found the sum of the filtered/rectified force values associated with each phase interval, and divided each by the integral of the filtered/rectified force trace. This procedure gives the relative proportion of total force variability (within the specified frequency band) associated with each 10◦ phase interval of the target cycle.

Under the null hypothesis, the proportion of force variability in each phase bin does not depend on the phase progression of the target cycle. Thus each 10◦ phase bin would be expected to show about 2.8% (100%/36 bins) of the total force variance. To test the null hypothesis, we compared our recorded proportions to a phase-randomized distribution generated directly from the recorded data (i.e., shuffled versions of our own data). We constructed these shuffled distributions of proportion values by randomly selecting a different 10◦ phase bin in each tracked target cycle, and then calculating the proportion of total force variability, as previously described. The process was repeated 5000 times, creating a distribution of shuffled proportion values which allowed us to determine a 95% confidence interval. Proportions falling outside of this interval would then represent statistically significant deviations from chance level. Our use of a Monte–Carlo method provides a direct, conservative, and assumption-free statistical analysis. Similar methods are often used in neuroscience, where analysis of real vs. shuffled/randomized neural activity is common (Perkel et al., 1967; Tam et al., 1988; Türker et al., 1996; Rivlin-Etzion et al., 2006; Laine et al., 2012). In our case, alternative methods such as testing for differences between individual phase bins, would be ill-suited for identifying the timing of tremor modulation with respect to the target phase, and would also not account for the fact that the proportion within each phase bin is not strictly an independent measurement.

The above methodology was applied to individual participants. To evaluate the population as a whole, the proportions for each phase bin were averaged across individuals. As a statistical evaluation, we calculated, for each phase bin, the number of individuals whose tremor proportion fell above or below the 95% confidence interval. For any given phase bin, a 5% error rate might be expected. Since our analysis included 10 individuals, it could be expected that at least one may have exceeded the confidence level purely by chance. However, the binomial probability that 2 of 10 individuals should show a (false positive) significance at the 95% confidence level is 0.015. For this reason, our population significance level was set to 0.015, or 2 out of 10 individuals, for our consistency analysis.

In addition to analyzing the proportion of force variance in each phase interval, we also calculated the cross-cycle average tremor amplitude in each phase interval. This analysis yielded an amplitude profile for each individual (and frequency band), similar to the proportion profiles described above. We then recorded the maximum and minimum values observed in the amplitude profile of each individual, regardless of the particular phase at which these values were found. This allowed us to evaluate the actual extent of tremor amplitude modulation, uncoupled from any particular pattern of tremor modulation across target phases.

#### Force to EMG Coherence across Target Phases

Coherence is a frequency-domain measure of synchronization (primarily phase-locking) between signals, and is bounded between 0 (no correlation between signals) and 1 (perfect linear correlation). Coherence between rectified EMG activity and force is useful for identifying the frequency content of force-relevant neural drive to muscles, since action potential shapes/sizes and other recording artifacts only influence the EMG spectrum, but would not be synchronized with force. In addition, coherence between FDS activity and force tremor provides validation that our simulation of a dynamically activated FDS muscle is appropriate for exploring the potential origins of recorded force fluctuations.

To calculate EMG to force coherence, the force and EMG signals were concatenated across all trials from all subjects to form two long signals. These signals were then converted to timefrequency-representations (TFRs) via wavelet analysis. We chose a wavelet approach so that we could precisely determine which frequencies of force were synchronized with EMG, and at what times. The technique is common where temporal variation of spectral power or synchronization is of interest (e.g., Siemionow et al., 2010; Tscharner et al., 2011). This was accomplished through convolution of each original signal x(t) with a Gaussianwindowed complex sinusoid (a Morlet wavelet), the duration of which was set to span 3 cycles of each frequency (f) from 1 to 20 Hz. The process can be expressed by the following formula:

$$TFR\left(t,f\right) = \int \propto \left(t\right) \frac{1}{\sigma\sqrt{2\pi}}e^{-\left(t-\tau\right)^2}e^{-j2\pi f\left(t-\tau\right)}dt$$

where the standard deviation (σ) of the Gaussian window is set to 3/(2πf). The force trace (band-pass filtered between 1 and 20 Hz) as well as the EMG activity (rectified, normalized per subject to have unit variance) were thus converted to complex-valued TFRs (herein defined as TFR\_Force and TFR\_EMG, respectively). The spectral power of each signal can be calculated as follows:

Power\_Force (t, f) = TFR\_Force (t, f) • conj (TFR\_Force (t, f)) Power\_EMG (t, f) = TFR\_EMG (t, f) • conj (TFR\_EMG (t, f))

Where conj refers to the complex conjugate.

Likewise, the time-frequency cross-spectrum can then be defined as:

$$\text{TFR\\_cspec}\left(\mathbf{t}, \mathbf{f}\right) = \text{TFR\\_Force}\left(\mathbf{t}, \mathbf{f}\right) \bullet \text{ conj}\left(\text{TFR\\_EMG}\left(\mathbf{t}, \mathbf{f}\right)\right)$$

The time course of coherence can then be calculated per frequency as:

$$\frac{\text{TFR\\_Coherence (f)} = }{\text{ } \begin{aligned} \text{TFR\\_cspec (f)} \* W & \text{/} 2 \\ \text{(Power\\_Force (f)} \* W) & \text{\(Power\\_EMG (f)} \* W\) \end{aligned}}$$

Where the term ∗W represents convolution of the indicated time series with a rectangular window (W), the duration of which was set per frequency to be 7/f. The multiplication and division in the above equation are simply element-by-element operations on the time series data.

Prior to further analysis, coherence values were normalized using Fisher's r-to-z transform Fz <sup>=</sup> atanh(<sup>√</sup> C) where C is the coherence at a given time-frequency point (Benignus, 1969).

Because of the short time scales involved in the calculation of wavelet coherence, it is best to recast coherence values as a statistical deviation from chance level. Here, the chance level was derived empirically by recalculating the time-frequency coherence after reversing the concatenated EMG signal in time. This causes EMG signals from one participant to be tested for coherence with force traces produced by a different participant, and completely misaligns the signals with respect to the phase progression of the sinusoidal target. The actual coherence values for each frequency were then converted to standard Z-scores with respect to the distribution of coherence values obtained from the "fake" time series. This method helps to emphasize any synchronization which varies significantly across target phases. Values > 1.65 (the one-sided 95% confidence level for a Z-test) indicate that the time-localized coherence between EMG and force was greater than expected by chance at a given phase and frequency.

#### Simulations

#### Closed-Loop Control Overview

We used a computational model of an afferented muscle to study the dependence of tremor on the dynamics of force production. A schematic diagram of the feedback-driven control loop is shown in **Figure 1B**. Briefly, a Hill-type muscle-tendon model was driven by a neural activation signal to produce force under isometric conditions. The simulation was intended to approximate the action of the FDS muscle in our experimental data. The muscle-tendon model describes changes in force as well as the magnitude and rate of associated changes in the length of the muscle fascicle and tendon, accounting for their viscoelastic properties. Our simulation includes two spinal proprioceptive systems; the muscle spindle and the GTO. Upon muscle fiber lengthening, the muscle spindle sends excitatory feedback through primary (Ia) and secondary (II) afferent fibers proportional to eccentric changes in muscle fiber length and velocity; while GTOs send inhibitory feedback (Ib) proportional to the force in the tendon. A tracking controller, whose operation includes conduction and synaptic delays appropriate for a transcortical loop (Lourenço et al., 2006; Pruszynski et al., 2011; Sohn et al., 2015), sends a command signal (C) to the "spinal cord" which is corrected at each time step according to the difference between the target force level and the actual force output from the muscle. This tracking control signal simply ensured that the afferented muscle-tendon model could follow the target force trajectory, and is not intended to model a specific neural pathway, or to recreate human visuomotor or voluntary tracking behaviors. Signals from the tracking controller, muscle spindle, and GTO, are integrated at the "spinal cord" to generate the αmotoneuron drive to the lumped-parameter muscle model. This neural drive (ND) at each ms (t) can be expressed in the following form:

$$\text{ND(t)} = \text{la(t-15)} + \text{II(t-25)} - \text{lb(t-17)} + \text{C(t)}$$

The output of the tracking controller (C) is calculated as:

$$\text{C(t) = C(t-1) + k \bullet (Target(t) - F(t-35))}$$

where F is the force on the tendon and k is a constant. Note that the above represents a simple "iterative learning control" (ILC) scheme (Wang et al., 2009).

To translate the neural drive into force, the signal was delayed by an additional 22 ms before reaching the muscle fibers to account for conduction time along efferent fibers. At the muscle, the signal was passed through an "activation filter" which shapes the signal to account for calcium dynamics in physiological muscle. Finally, the muscle-tendon model converts this muscle activation signal to the force output of the tendon. In this simulation, the delays for each pathway have been matched to physiological recordings from humans and reflex latencies from the FDS muscle in particular (Lourenço et al., 2006).

The muscle model, muscle spindle, and GTO elements of this control loop have been published previously by various groups and will be described and referenced individually below.

## Control Loop Elements

#### Muscle Model

Our Hill-type muscle-tendon model and its mathematical derivation were adopted from previous literature (He et al., 1991; Brown et al., 1996). The schematic diagram of this muscle-tendon model is presented in **Figure 1B**. The muscle fascicle consists of a mass (M), two passive elastic elements (PE in **Figure 1B**), a viscous element (B), and a contractile element (Fm), which is connected with a pennation angle (α) to a series elastic element (SE) representing tendon and aponeurousis.

The contractile element generates muscle force as a fraction of the maximal force that the muscle is capable of producing. This is defined as the product of its physiological cross-sectional area and a constant factor (45 N/cm<sup>2</sup> ) (Holzbaur et al., 2005). Two parallel elastic elements characterize passive behaviors of muscle fascicles. The first (non-linear) spring acts against stretch of muscle fascicle, while the second (linear) spring resists compression (Brown et al., 1996). The series elastic element (SE) shown in **Figure 1B** is a lumped non-linear spring model of tendon and aponeurosis. The force produced by this element in relation to the length of the tendon has been implemented as in Brown et al. (1996). The contraction dynamics within the muscletendon unit are modeled as a second-order differential equation (He et al., 1991).

Taking the above factors into account, the output force function (F) can be summarized as follows

$$\begin{aligned} \mathbf{F(t)} &= \mathbf{MA(t)} \bullet \mathbf{FL(t)} \bullet \mathbf{FV(t)} + \mathbf{F\_PE1(t)} + \mathbf{F\_PE2(t)} \\ &+ (\mathbf{B} \bullet \mathbf{v(t)}) + (\mathbf{a(t)} \bullet \mathbf{M}) \end{aligned}$$

Where MA is the muscle activation (the output of the activation filter), FL is the force-length function, FV is the force-velocity function, F\_PE1 and F\_PE2 are the forces produced by the two elastic passive elements in the model, v(t) and a(t) are the velocity and acceleration of muscle fiber contraction, and the muscle mass (M) and viscosity (B) are constants.

Because the force produced by human participants ranged from 5 to 9% of maximal effort, we applied the same forces to the simulated FDS muscle. Given our focus on understanding the general nature of tremor modulation by dynamic force production, it was not necessary to calculate the precise, isolated contribution of the FDS muscle to the generation of index finger force in our experimental task. To simulate the FDS muscle, architectural parameters were set (**Table 1**) according to published anatomical data (Lieber et al., 1991; Holzbaur et al., 2005). For our purposes, the muscle fibers of the FDS muscle associated with the tendon acting on PIP joint of the index finger were combined into a single belly for simplicity.

#### Muscle Spindle Model

The muscle spindle model employed in this study is adapted from Mileusnic and Loeb (2006). This computational model was chosen because it is both physiologically realistic and, at the same time, is immediately compatible with the inputs and outputs of the other elements within our control loop. The model comprises three types of intrafusal fibers, namely, the bag1, bag2, and chain fibers, all of which are modeled as a second-order mechanical system (a mass, a viscous element, and parallel and series elastic elements), similar to a Hill-type muscle-tendon model. Each of the intrafusal fibers receives input describing the muscle fascicle length, velocity, and acceleration, as well as a fiber-type-specific fusimotor activation signal (dynamic or static). In this study, fusimotor activation was set to be constant during each simulation run. The fusimotor gains tested were 75, 150, and 350. Functionally, these are arbitrary units, but can be expressed conceptually as pulses per second. We chose to define our baseline value as 75, since this is near the previously published value of 70 (Mileusnic and Loeb, 2006), and we varied that parameter because fusimotor drive is known to depend upon task and individual psychology (Ribot et al., 1986; Ribot-Ciscar et al., 2000, 2009; Hospod et al., 2007). Because fusimotor drive is modified by the nature of the task independently of (and even without) α-motoneuron firing (neural drive, in our model), we chose not to assume obligatory α-γ coactivation. It is true that mechanisms other than fusimotor drive may change the effective gain of afferent activity (e.g., presynaptic inhibition). Here, variation in fusimotor drive is not only a likely physiological occurrence, but also serves to more generally

TABLE 1 | Architectural parameters of the slip to the index finger of the flexor digitorum superficialis (FDS) muscle.


represent the overall gain of spindle feedback to motor neurons. For integration with the feedback control loop, the final outputs of the spindle model were normalized to fall between 0 and 1.

#### Golgi Tendon Organ Model

The GTO model was adopted from Elias et al. (2014). This GTO model presents the overall behavior of a population of Ib fibers. It was placed in series with tendon, so that it receives tendon force as an input. The force was then converted into Ib fiber output. The transfer function described in Elias et al. (2014) was implemented using the c2d function in MATLAB. The Ib fiber output was scaled between 0 and 1, as was carried out for the spindle outputs.

#### Activation Filter

The activation filter adjusts the neural drive signal to account for the effects of calcium dynamics (release and reuptake) on crossbridge formation, as described in Song et al. (2008). The resulting muscle activation signal (MA in **Figure 1B**) is the "effective" drive delivered to the muscle model.

#### Simulation Analysis

Force tracking was simulated for 128 s with new values for each output parameter derived every ms. To be certain that only consistent, steady-state behavior was analyzed, only the last 30 cycles were used for analysis. The muscle forces produced by the simulation were analyzed in the same way as the experimentally recorded force tremor, providing a direct comparison.

## RESULTS

#### Tremor during Force Tracking

The phase of voluntary force modulation influenced both low (1–5 Hz) and high (6–15 Hz) frequency bands of involuntary tremor. For reference, the top panels of **Figures 2A,B** show the target isometric force sinusoid, which spanned from 5 to 9% of each individual's maximum voluntary force level. The panels immediately below the target in **Figures 2A,B** show the proportion of total tremor variance associated with each phase of the target sinusoid. The proportion of total force variance accumulated in each 10◦ phase bin is shown for low (**Figure 2A**, upper trace) and high (**Figure 2B**, upper trace) frequency bands. Each plot represents a grand average over all 10 participants, who together tracked a total of 357 target cycles. The largest tremor in either frequency band was observed at the beginning of the rising phase of the target. A second time period of increased tremor amplitude appeared at or slightly after the peak of the target cycle, mainly for high frequency tremor. The dashed line indicates the proportion of variance to be expected in each phase bin if force variability were evenly distributed over all phase bins.

Below these average tremor profiles are the cross-participant standard deviations associated with the mean proportions in the traces above. Variance across participants was highest at the base of the target sinusoid, where bursts of tremor (e.g., **Figure 1**) were often observed. The bottom panels of **Figures 2A,B** depict the number of participants (out of 10 total) whose tremor profiles showed statistically larger (above 0 line) or smaller (below 0

FIGURE 2 | Phase-dependent modulation of tremor amplitude. The amplitude of force variability within the 1–5 and 6–15 Hz frequency bands depended upon the phase of the target cycle (shown at the top of each column, for reference). The first data panel of (A) (left column) shows the mean proportion of total force variability (within the 1–5 Hz band) observed in each 10◦ phase bin, calculated over all participants. The same panel of the right column (B) shows this analysis for 6–15 Hz force fluctuations. The horizontal dashed line in each figure represents the proportion expected per bin if tremor amplitude were constant across target phases. The 1–5 Hz frequency band shows a clear peak at the initiation of the rising phase of the target sinusoid. A similar profile was observed for 6–15 Hz force fluctuations, but with a more pronounced tremor amplification at the peak of the target sinusoid. Below these grand averages are depicted the cross-participant standard deviation for each phase bin. The cross-participant variability is generally highest at the valley of the target sinusoid, when bursts of high amplitude tremor were more likely to occur (as can be seen in Figure 1A). The bottom panels are histograms which depict the number of individuals (out of 10), whose tremor profiles deviated significantly from chance level at each phase-bin, as determined by a Monte-Carlo test (see text for details). In these histograms, counts above the 0 line record the number of individuals who displayed greater than chance-level proportions. Counts below 0 indicate the number of individuals showing lower tremor than expected by chance at each phase. Note that these "less than chance" histogram counts are an effect of other phases showing high proportions of tremor, and do not indicate suppression of ongoing tremor, which was never observed. Histogram counts exceeding the shaded region indicate that significant effects at a given phase were more consistent across individuals than could have occurred by chance (according to a binomial test). The histograms show that the modulation of tremor by target phase was fairly consistent across individuals. The arrows on each histogram (A,B bottom), emphasize that tremor was most often larger than expected at the peaks and valleys of the target sinusoid, for both frequency bands.

line) proportions in each phase bin than expected by chance (as determined by a Monte–Carlo test, as previously described). The shaded region marks the number of participants that may have been expected to show significant effects by chance. That is, histogram counts exceeding the upper limits of the shaded range represent a consistent amplification of tremor occurred across the population of participants. Histogram counts below the 0 line are caused by the high proportions observed in other phase bins and do not represent suppression of tremor, which was never observed. Again, column A shows results for 1–5 Hz force fluctuations and column B shows results for 6– 15 Hz force fluctuations. For both frequency bands, a significant population effect was observed at the beginning of the rising phase of the target sinusoid, and at the beginning of the falling phase.

## EMG to Force Coherence

To confirm that the cross-cycle modulation of force variability was also reflected in the activation of the FDS muscle, we calculated EMG-to-force coherence. Using wavelet coherence, we were able to examine the coupling between signals at each frequency, and at each phase of the target cycle. The statistical magnitude of coherence (z-score with respect chance-level) shown in **Figure 3** for each time-frequency pixel was calculated from the full data set (all 357 tracked cycles). Pixels with values greater than 1.65 can be considered as significant at the 95% confidence level.

Importantly, the coupling between EMG and force signals closely resembles the phase progression of force tremor amplitudes, and reflects the same frequency profile. The phase-related modulation of coherence demonstrates temporal variation in synchronization between signals, which would be expected if the frequency content of neural drive depended on the phase progression of the tracking action.

## Actual Tremor Amplitudes during Force Tracking

Although our study is primarily focused on the modulation pattern of normalized tremor amplitudes as a function of voluntary force dynamics, it is also important to address actual tremor amplitudes, and the extent of amplitude modulation across target phases. Since the pattern of tremor modulation could vary across individuals (described below), we chose to record the maximum, minimum, and 1 amplitude (max-min). The latter was important for better comparability with our simulation results, since our simulation does not contain noise or ongoing physiological tremor, both of which are typically present in human participants. In general, we found tremor amplitudes fluctuated by a factor of about 2 over the course of a target cycle. **Figure 4** shows the mean and cross-participant SD for each measure.

## Tremor Modulation across Different Individuals

Although the phase-dependent modulation of force variability shown in **Figure 2** was representative of the population overall, tremor profiles did vary across individuals. **Figures 5A–D** depicts the tremor proportion profiles for high and low frequency bands in 4 individuals whose profiles differed from each other. Overall, most participants showed some degree of increased tremor

(in either frequency band) at the peaks and/or valleys of the target sinusoid. The modulation of tremor amplitudes in these individuals, shown in the bar graphs at the bottom of each column, indicate that 1–5 Hz tremor amplitude was consistently higher than 6–15 Hz tremor amplitude, but the ratio could vary across individuals.

#### Simulation Results

By embedding a modeled FDS muscle within a feedback-driven control loop, we were able to simulate the force tracking experiment carried out by human participants. Surprisingly, this simple model of an afferented FDS muscle sufficed to produce much the same pattern of tremor modulation in relation to the 0.25 Hz sinusoid as seen in **Figures 2**, **5**. **Figures 6A–C** shows the modulation of tremor obtained when the simulation was run using low, medium, and high fusimotor drive. Adjusting the fusimotor drive, and thus, the gain of afferent feedback from the muscle spindle, could produce variation in simulation results similar to the type of variation observed across different subjects (e.g., compare **Figure 5A** with **Figure 6A**, or **Figure 5C** with **Figure 6C**). Force fluctuations near the valley of the sinusoidal

deviations for these features. For both frequencies, there was a nearly two-fold

modulation of tremor amplitude across a target cycle, on average.

target were present in all cases, although the fluctuations occurring at the peak of the sinusoid was reduced as the afferent gain was increased. As with the experimental data, the rising and falling phases of the target sinusoid did not appear to be associated with consistent changes in tremor activity. At the bottom of each column in **Figure 6** are bar graphs showing the average extent of amplitude modulation. Since the minimum amplitude was nearly 0 in all cases, these bars also represent the average maximum amplitude across phases as well. Of particular importance is the fact that increasing fusimotor gain resulted in a doubling (B) and tripling (C) of high frequency tremor amplitudes, as compared with the low fusimotor drive condition (A). Low frequency tremor did not appear to be consistently influenced, but if anything, was actually reduced in amplitude as fusimotor drive was increased. It should be noted that the amplitudes measured from our simulation should not be expected to precisely match those recorded experimentally. Of greater importance is the relative relationship between high and low frequency tremor amplitudes, and how they vary across individuals or simulation parameters. That said, our simulated amplitudes appear to be smaller than those recorded experimentally by a factor of about 10, which is reasonable considering the highly reduced/simplified nature of the model and the absence of any noise.

We also ran the simulation after eliminating various elements of the control loop. **Figure 7** shows the resulting

tremor modulation pattern when the simulation was run completely feedforward (**Figure 7A**), using only feedback from the controller (**Figure 7B**), using only the controller and GTO feedback (**Figure 7C**), and using only the controller and spindle feedback (**Figure 7D**). Where spindle feedback is present, the fusimotor drive was set to 75 (as in **Figure 6A**). An increase in 6–15 Hz fluctuations occurred roughly at the peak and valley of the target sinusoid regardless of the feedback utilized in the control loop. However, the precise shape, timing, and magnitude of these fluctuations were altered by the type of feedback utilized. Inclusion of spindle feedback (**Figure 7D**) was necessary to produce realistic tremor variance patterns (compared with **Figures 2**, **5**) at the initiation of the rising phase of the target sinusoid. Also it is worth noting that tremor amplitudes (bar graphs at bottom of **Figure 7**) were drastically reduced in the absence of spindle feedback roughly by a factor of 10 for 1–5 Hz tremor and by a factor of about 50 for 6–15 Hz tremor. These observations are well aligned with previous findings where reduction of afferent feedback was associated with reduced/eliminated physiological tremor (Halliday and Redfearn, 1958; Sanes, 1985; Erimaki and Christakos, 2008). Tremor modulation was in general particularly sensitive to spindle feedback, since increasing its effective gain through fusimotor drive (**Figures 6B,C**) or removing spindle feedback

Also, the relationship between high and low frequency amplitude modulation is similar to that observed in the experimental data (Figures 4, 5, bottom).

entirely (**Figures 7A–C**) both influence the overall shape, timing, and amplitude of tremor fluctuations. This occurred even though the largest change in muscle fascicle length was only about 1.1 mm. Interestingly, removal of GTO feedback had minimal effect (**Figure 7D** compared with **Figure 6A**).

## DISCUSSION

In this study, we show that involuntary tremor can arise simply from the dynamic viscoelastic response of afferented muscles during voluntary production of isometric force. We characterized this tremor in two frequency bands as healthy adults performed a sinusoidal force tracking task. Furthermore, we simulated the spontaneous emergence of tremor from purely peripheral mechanisms using a computational closed-loop model comprised of well accepted musculotendon, spindle, and GTO computational modules. Our results extend the current understanding of how force variability arises independently of central mechanisms during production of isometric force. Importantly, our results suggest that simple force tracking tasks may provide a clinically and scientifically relevant

FIGURE 7 | Simulation results after removing sources of feedback control. Each panels (A–D) depicts the results of the simulation after removing elements of the control loop. (A) Depicts the tremor modulation profile that emerged when the simulation was run completely feedforward, with no feedback. (B) Depicts the results when using the controller output, but no neural feedback. Panel (C) shows the tremor modulation occurring when Gogli tendon organ feedback was added to the controller, while panel (D) shows the tremor profile that emerged when only controller and spindle feedback were used. The baseline condition (before removal of feedback) was the same as in Figure 6A. Removal of the GTO feedback had minimal influence on the overall shape and timing of force fluctuations (compare Figure 6A with panel D). In contrast, elimination of spindle feedback not only reduced the magnitude of force tremor, but also influenced the general shape and timing of tremor modulation (panels A–C), particularly at the initiation of the rising phase of the target sinusoid. Shown at the bottom of each column are bar graphs depicting the average tremor amplitude modulation for each frequency band (as in Figures 5, 6). The addition of spindle feedback (D) to the model was the only condition which greatly modified tremor amplitudes. Note the change of scale for the bar graph in (D) as compared with (A–C). While low frequency tremor modulation was increased by roughly a factor of 10 with the addition of spindle feedback, the high frequency tremor increased by about a factor of 50, with respect to any other condition.

window into the neural and mechanical factors which generate involuntary tremor.

Although tremor is not often attributed to the specific dynamics of voluntary force production, several investigations have suggested the existence, and potential importance, of such an interaction. For example, muscle stretch has been suggested to play a role in tremor modulation during dynamic force production. Specifically, declining isometric force is associated with at least some small degree of muscle fiber lengthening, toward resting state (Ito et al., 1998). Compared with shortening contractions, lengthening contractions are associated with increased force variability (Christou and Carlton, 2002) and increased motor unit coherence within the physiological tremor range (Semmler et al., 2002). However, it does not appear that muscle fiber lengthening, or associated spindle activity, could explain our results, since we would have expected a systematic and consistent increase in tremor during the descending phases of our sinusoidal target trajectory.

Similarly, in the production of bite force, 7–10 Hz jaw tremor is reduced in slowly increasing force ramps compared with constant or slowly decreasing force (Sowman et al., 2008). In the present study, a simple relation between force direction (increasing or decreasing) and finger tremor was not observed, likely due to differences in the physiology of bite vs. grip force control. Jaw tremor in the 7–10 Hz range depends upon the activity of periodontal mechanoreceptors (Sowman et al., 2006), and different bite-force dynamics may have led to different levels of dental intrusion (Schoo et al., 1983), and presumably adaptation of the periodontal mechanoreceptors (Sowman et al., 2008). Accordingly, both the dynamics of bite-force production and afferent feedback are important considerations when comparing healthy adults to those who suffer from bruxism (Laine et al., 2015b). While jaw tremor may depend on specific mechanical properties of the gums and their interaction with periodontal mechanoreceptors, these studies do demonstrate that tremor modulation can stem from afferent responses to dynamic force.

Stretch-reflex amplitudes have, in fact, been reported to change during the production of sinusoidal forces. For example, rhythmic (sinusoidal) pen-squeezing has been shown to produce stretch-reflex modulation in the FDS muscle (Xia et al., 2005). This is important in the context of the present study because oscillations of excitation around the stretch reflex loop are considered to be one of the major contributors to physiological tremor (Lippold, 1970; Young and Hagbarth, 1980; Christakos et al., 2006; Erimaki and Christakos, 2008). In the study of Xia et al. (2005), it was observed that stretch-reflex amplitudes were roughly modulated in a sinusoidal fashion such that increased reflex amplitudes were associated with higher background FDS EMG levels. A similar conclusion was reached by Stanislaus and Burne (2009), who reported a consistent relationship between stretch-reflex gain and overall contraction level regardless of force dynamics. If a similar sinusoidal modulation of reflex gain were responsible for tremor modulation in our study, we should have observed a sinusoidal modulation of tremor amplitude, which was not the case.

Few tasks involve only one muscle, as thus, it is possible that some tremors stem from an interaction among co-activated muscles. Due to the simplicity of our task and the posture of the hand, it is likely that any co-activated muscles were also co-modulated synergistically during tracking. It has been shown that synergistic muscles may share neural drive over a wide range of frequencies (Laine et al., 2015a). Moreover, we have shown that changes in the magnitude of an isometric fingertip force are likely produced by a simple scaling of a same muscle coordination pattern (Valero-Cuevas, 2000), while others have shown that isometric force magnitude does not influence the frequency content of shared neural drive among muscles of the hand (Poston et al., 2010). In addition to the potential for shared descending drive, neighboring co-activated muscles would likely show temporally-coordinated afferent activity and reflex responses as well. At the very least, it seems that the effects of slow sinusoidal contractions on musculotendon dynamics would be similar across all co-activated muscles, leading them to tremor at the same time relative to the slow voluntary action.

During voluntary force production, tremor may also stem from the recruitment, de-recruitment, and firing rates motor units. For example, the force level at which motor units are recruited can be lower than the force level at which they are de-recruited (De Luca et al., 1982), and due to the activation of persistent inward currents, the magnitude of neural drive needed to recruit a motor unit is often higher than the drive at which the same unit is de-recruited (Gorassini et al., 2002). Therefore, the population of motor units which generate a given force is partly determined by the recent contraction history of the muscle. Motor unit activity, especially the twitches of motor units near threshold, may contribute to isometric force tremor (McAuley and Marsden, 2000). The relevance of such mechanisms to the present study is not clear, but we can speculate that the contribution would be minimal, given the sufficiency of our simulation (which does not include firing motor units, their intrinsic properties, or any source of signal-dependent noise) to replicate the experimentally-observed tremor modulation. We would, however, assume that simulated tremor amplitudes would more closely match those observed experimentally if signaldependent noise and/or intrinsic motor unit properties were included in our closed-loop system. This is a topic which certainly merits future investigation.

It is of course possible, even likely, that many of the above mentioned sources of force variability were still present to some degree in our study, but were not very consistent across participants or across target cycles. In that sense, they may explain some of the variation in tremor profiles observed across participants. Similarly, the degree to which any differences in muscle/tendon strength, size, and compliance across individuals would have influenced our results remains a topic for future investigation. We can speculate, however, that tendon strain magnitudes would likely be important, given that the dynamics of muscle stretch influence spindle output, which had the largest influence on tremor in the present study. The fact that our model could at least partly replicate cross-subject variation in tremor modulation through manipulation of fusimotor gains, which would be expected to vary across individuals (Ribot et al., 1986; Ribot-Ciscar et al., 2000, 2009; Hospod et al., 2007), adds validity to our simulation results and helps to mitigate concerns about its simplifications/assumptions. While it is beyond the scope of this study to precisely match the tremor profiles of every individual, or to exhaustively test the influence of all possible parameters, our results should serve as an important proof of principle upon which to base future investigation.

Despite the many possible sources of tremor within our task, it is clear that the dominant phase-dependent source of tremor was peripheral neuromechanical coupling, rooted in the viscoelastic properties of muscle and tendon. While smooth tracking is initially disrupted as a mechanical consequence of musculotendon dynamics, the spindle reflex system plays an important role in determining the overall magnitude and timing of the resulting tremor.

Our results, therefore, motivate and justify the development of similar experimental paradigms for scientific and clinical applications. For example, we propose that the tremor induced by slow voluntary force modulation may provide a simple measure of reflex integrity, providing an alternative to direct, yet time-consuming and often uncomfortable, perturbations of nerves or tendons. Further, it may be that characteristic patterns of tremor modulation would emerge within the context of spasticity, dystonia, or within conditions such as Parkinson's disease or essential tremor. As a means of probing peripheral neuromechanical coupling, the type of tremor described in this study may hold potential as a tool for understanding and assessing dysfunctional sensorimotor control in those with congenital or developmental disorders, or in those with acquired dysfunction due to trauma or disease. Finally, neuromechanical coupling may contribute mechanistically to the maintenance or amplification of pathological tremor. While we did not test the effects of inserting a descending tremor-frequency input into our simulation, such investigation may be informative, and perhaps even suggest novel avenues for clinical intervention.

#### REFERENCES


## AUTHOR CONTRIBUTIONS

CL, AN, and FV contributed to the design, execution and drafting of this work, and approved the final manuscript. Experimental data was collected by CL and simulations were implemented by AN.

#### FUNDING

Research reported in this publication was supported by the National Institute of Arthritis and Musculoskeletal and Skin Diseases of the National Institutes of Health under Awards Number R01 AR-050520 and R01 AR-052345. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

#### ACKNOWLEDGMENTS

The authors thank Matteo Bertucco for assistance with data collection and participant recruitment.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Laine, Nagamori and Valero-Cuevas. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Beta Band Corticomuscular Drive Reflects Muscle Coordination Strategies

Alexander Reyes <sup>1</sup> , Christopher M. Laine<sup>1</sup> , Jason J. Kutch2 † and Francisco J. Valero-Cuevas <sup>1</sup> \* †

<sup>1</sup> Brain-Body Dynamics Lab, Department of Biomedical Engineering, University of Southern California, Los Angeles, CA, USA, <sup>2</sup> Applied Mathematical Physiology Lab, Division of Biokinesiology and Physical Therapy, University of Southern California, Los Angeles, CA, USA

During force production, hand muscle activity is known to be coherent with activity in primary motor cortex, specifically in the beta-band (15–30 Hz) frequency range. It is not clear, however, if this coherence reflects the control strategy selected by the nervous system for a given task, or if it instead reflects an intrinsic property of cortico-spinal communication. Here, we measured corticomuscular and intermuscular coherence between muscles of index finger and thumb while a two-finger pinch grip of identical net force was applied to objects which were either stable (allowing synergistic activation of finger muscles) or unstable (requiring individuated finger control). We found that beta-band corticomuscular coherence with the first dorsal interosseous (FDI) and abductor pollicis brevis (APB) muscles, as well as their beta-band coherence with each other, was significantly reduced when individuated control of the thumb and index finger was required. We interpret these findings to show that beta-band coherence is reflective of a synergistic control strategy in which the cortex binds task-related motor neurons into functional units.

#### Edited by:

Sliman J. Bensmaia, University of Chicago, USA

#### Reviewed by:

Gordon M. G. Shepherd, Northwestern University, USA Aaron J. Suminski, University of Wisconsin-Madison, USA

#### \*Correspondence:

Francisco J. Valero-Cuevas valero@usc.edu

† These authors have contributed equally to this work.

Received: 29 December 2016 Accepted: 10 March 2017 Published: 04 April 2017

#### Citation:

Reyes A, Laine CM, Kutch JJ and Valero-Cuevas FJ (2017) Beta Band Corticomuscular Drive Reflects Muscle Coordination Strategies. Front. Comput. Neurosci. 11:17. doi: 10.3389/fncom.2017.00017 Keywords: coherence, dexterity, beta-band, neuromuscular, synergy, EMG, EEG

## INTRODUCTION

Both corticomuscular and intermuscular synchronization, as quantified by coherence analysis (Rosenberg et al., 1989; Farmer et al., 1993; Conway et al., 1995), provide an important means of understanding the cortical drive to muscles (Conway et al., 1995; Baker et al., 1997; Brown, 2000; Boonstra et al., 2009a). Corticomotor drive contains a 15–30 Hz (beta-band) oscillatory component (Murthy and Fetz, 1992, 1996a,b; Sanes and Donoghue, 1993; Stancák and Pfurtscheller, 1996; Donoghue et al., 1998; Mima and Hallett, 1999; Lebedev and Wise, 2000; Witham et al., 2010), which entrains targeted motor neurons (Farmer et al., 1993, 1997; Mima and Hallett, 1999) and leads to synchronization between cortical and muscular activities in that frequency range (Murthy and Fetz, 1992; Conway et al., 1995; Baker et al., 1997; Salenius et al., 1997). Functionally-related muscles also share a common intermuscular beta-band input (Kilner et al., 1999; Boonstra and Breakspear, 2012; Boonstra, 2013), which is widely accepted as cortical in origin (Brown et al., 1999), as the motor cortex is the only well-established source for such beta-band drive.

Although beta-band cortical drive has received a great deal of attention, its functional significance for motor control remains unclear. Currently it is suggested that oscillations in this range functions to support a constant motor state (Kilner et al., 2000; Pogosyan et al., 2009; Engel and Fries, 2010). This is consistent with the observation that, during the production of a constant force, beta-band coherence is strengthened with continued sensory feedback and minimal voluntary movement (Gilbertson et al., 2005; Androulidakis et al., 2006, 2007; Lalo et al., 2007; Engel and Fries, 2010; Aumann and Prut, 2015). While the magnitude of beta-band corticomuscular coherence does correlate with force(Conway et al., 1995; Baker et al., 1997; Kilner et al., 1999, 2000, 2004; Baker, 2007; Kristeva et al., 2007; Witte et al., 2007), it disappears during movement (Baker et al., 1997; Kilner et al., 1999, 2000, 2004; Brown, 2000; Feige et al., 2000) and imagined movements (De Lange et al., 2008), and there is even evidence that the signal is not entirely feed-forward (Fisher et al., 2002; Baker et al., 2006; Witham et al., 2011). Such findings raise important questions as to the functional role of beta-band drive to muscles.

Numerous studies have investigated low force (< 5 N) precision pinch paradigms to characterize cortico-spinal interactions through the use of corticomuscular coherence (CMC) (Muir and Lemon, 1983; Lemon and Mantel, 1989; Lemon et al., 1995, 1998; Baker et al., 2003). Findings have revealed that beta-band CMC is modulated by digit displacement (Riddle and Baker, 2006), object compliance (Kilner et al., 2000), and similar studies suggest a dependence upon the history and time course of muscle contraction (Chakarov et al., 2009; Omlor et al., 2011; Nazarpour et al., 2012). In nearly all cases, findings of decreased beta-band CMC can be interpreted as reflecting either (1) a departure from steady-state control of a particular muscle, or (2) the cortical "unbinding" of muscles when individuated, rather than synergistic, activation is called for. Given that betaband cortical activity has been suggested as a "binding" signal for many years (Gray, 1994; Santello, 2014), and that such binding would naturally favor synergistic rather than individuated control of the fingers (Boonstra et al., 2009b; Danna-Dos Santos et al., 2010; Kattla and Lowery, 2010; Aumann and Prut, 2015), our overall hypothesis was that beta-band corticomotor drive should be reduced or eliminated when the degree of individuated muscle control is increased.

To address these issues, we studied beta-band corticomuscular and inter-muscular coherence (CMC and IMC, respectively) while participants applied low magnitude precision pinch forces to one of two different objects. The first object was a solid wooden dowel. Production of a constant pinch force against a solid object represents a relatively simple task for the nervous system. The second object was a custom-designed spring which buckles when compressed unless prevented from doing so through precise dynamic adjustment of thumb and index fingertip forces. The spring task described herein has been modified from Valero-Cuevas et al. (2003) and described in Dayanidhi et al. (2013b). This task requires the dynamic regulation of thumb and index fingertip force vectors in 3-D to stabilize the spring, which can be modeled as undergoing an instability similar to a subcritical pitchfork bifurcation (Venkadesan et al., 2007). The physical movement of the fingers remains negligible because large movements tend to increase the likelihood of buckling.

If beta-band coherence depends on the generation of a relatively stable pinch force, then coherence should change relatively little across these two tasks. If beta-band coherence is inherently an intermuscular binding signal, then we would expect to see little coherence during compression of the unstable spring, either in terms of CMC or IMC. Our findings support this hypothesis and suggest that the dynamic, mechanical relationships among muscles are likely critical factors shaping the frequency content of corticomotor drive.

## METHODS

## Participants

We recruited 15 healthy participants (30.3 ± 4.6 years, 6 females) who were self-reported as right-handed. There were no known prior or current neurological conditions in any of the participants, nor did they report any previous hand injuries or surgeries. This study was carried out in accordance with the recommendations of Institutional Review Board (IRB) at the University of Southern California (USC) with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the Office for the Protection of Research Subjects at USC.

## Task 1: Establishing Maximal Unstable Spring Compression Force

The Strength-Dexterity test provides a quantitative measure of hand dexterity by requiring dynamic regulation of endpoint force direction and magnitude to stabilize a slender and compliant spring prone to bucking (Valero-Cuevas et al., 2003). The instability in the spring increases with compression force, and thus the maximal compression force reached is indicative of the greatest instability the neuromuscular system can control. The force required to bring the spring to solid length was approximately 3.7–3.8 N, however the maximal compression force healthy adults can reach is less than 3.0 N (Dayanidhi, 2012). We specifically chose a spring with a low strength requirement (<15% maximal precision pinch force) to focus primarily on cortical drive involved with dexterity demand, rather than strength. The resting length of the spring measured 4.2 cm in length, weighed approximately two grams and had a spring constant of 0.86 N/cm. Custom designed 3-D printed acrylonitrile butadiene styrene (ABS) plastic end caps were glued to both ends of the spring to create flat surfaces on which to attach a force transducer. Additional ABS end caps were attached on top of the sensor for two purposes: (1) to provide a place for subjects to grasp the object and (2) to serve as a thermal barrier to prevent body heat from adding a bias to the temperature-sensitive force transducer. With the addition of the end caps and sensor, the effective resting length of the spring was 5.7 cm (**Figure 1A**).

In this task, subjects rested their right arm on a table and used their index finger and thumb to compress the spring with their hand resting on the table. During the task, we asked participants to ensure that their 3rd–5th fingers did not assist in the task by tucking them into their palm. They were given four attempts (90 s

tolerance limits as with the spring. Gray areas represent data within tolerance and dowel-low and dowel-high refer to dowel-low and dowel-high, respectively.

each) to try to compress the spring as much as possible (Valero-Cuevas et al., 2003; Dayanidhi et al., 2013a,b; Lawrence et al., 2014). The average maximal compression force reached prior to spring buckling was taken as a normalized measure of their dexterous performance. We rounded this maximal value to the nearest tenth of a Newton and defined this as the subject-specific Fmax. We then calculated 40 and 80% of the subject-specific Fmax for use in the second phase of the study.

#### Task 2: Visuomotor Force Tracking

Subjects were seated comfortably in front of a computer providing visual feedback of their precision pinch compression force. They compressed either the same slender spring or a wooden dowel (length = 5.2 cm, diameter = 0.12 cm, **Figure 1C**) to visually track a series of randomly presented step targets set to 40 and 80% of their Fmax. For three trials for a given object, target force levels were presented at 30 s intervals with 5 s of rest in between (**Figures 1B,D**). The trial lasted until five repetitions of each force level were presented. During the resting periods, the subjects would hold the object with just enough force to prevent dropping it. Breaks were given in between trials when necessary to prevent fatigue effects. Subjects then repeated the procedure with the other object for the same number of trials and force levels. This two-by-two factorial design yielded four force conditions: spring-low, spring-high, dowel-low, and dowel-high.

## Recordings

#### Force

Normal compression forces were measured by affixing a uni-axial load cell (ELFF-B4-10L, Measurement Specialties, Hampton, VA) with double-sided tape to the index finger side of either the compliant spring or the wooden dowel. The circular load cell measured 0.41 cm in height, 1.27 cm in diameter and aligned perfectly with the diameter of the objects. Signals from the sensor were differentially amplified with a custom designed circuit operating in the 0–5 N range. Data were captured using a USB Data Acquisition (DAQ) system (National Instruments, Austin, TX) sampling at a rate of 2,048 Hz. Prior to data collection, the sensor voltage was converted to Newtons by removing the DC offset and calibrating the load-cell using a four-point linear regression with fixed weights. The offset and gain of the load cell were corrected periodically to ensure accurate force recordings.

#### Electromyography (EMG)

Bipolar surface EMG were collected using a Delsys Bagnoli system (Delsys, Natick, MA) from the first dorsal interosseous (FDI) and abductor pollicis brevis (APB). Data were filtered between 20 and 450 Hz, amplified by 1000, and then sampled at a 2048 Hz. The reference electrode for the recordings was placed on the olecranon of the right arm. Recording locations were identified by palpating the muscle during force production in the direction of mechanical action for each muscle.

#### Electroencephalography (EEG)

Sixty four channels of EEG were recorded at a sampling rate of 2048 Hz (ANT Neuro, Enschede, The Netherlands). The fixed recording sites were arranged according to the international 10– 20 system for scalp electrode placement. We ensured repeatable recordings of cortical areas across subject by taking skull measurements and placing electrode Cz at the cross section of the midway point between the nasion and inion and the midway point between the left and right tragus of the ear. Electrode impedances were kept below 10 k with respect to the reference electrode CPz.

Following digitization, EEG signals were bandpass filtered between 10 and 500 Hz and both EEG and EMG signals were notch filtered at 60 Hz and its harmonics up to 500 Hz using a 4th order Butterworth filter implemented in MATLAB (Mathworks, Natick, MA) and FieldTrip, a software package for EEG and EMG analysis (Delorme and Makeig, 2004; Oostenveld et al., 2011). Subsequently, EMG were rectified to extract group activity of motor units (Halliday et al., 1995; Mima and Hallett, 1999). Data formats collected from two separate systems were synchronized by configuring the NI-DAQ to send a trigger pulse to the EEG and EMG systems via a split BNC cable. Custom scripts were created to read in trigger events and synchronize all data.

#### Trial Selection

The start of a trial was defined as the time when the on-screen target transitioned from a resting value to either 40 or 80% of Fmax, and its end was defined as the time when the target value returned back to rest. Trial windows were 30 s in duration with a 5 s inter-stimulus interval. Sample traces of the matching paradigm are shown in in **Figure 1B** for the spring (blue trace) and in **Figure 1D** for the dowel (orange trace); each showing three randomized presentations of the low and high force levels (note that the force levels are the same across objects). Force data during these steady hold phases were visually examined to determine if the task was performed correctly. Our requirement was that the hold phase must be within a ±15% tolerance of the target force value and be held within this range for at least 5 s. The force profiles meeting these criteria are shown as the gray shaded areas in **Figures 1B,D**. Force profiles not meeting these criteria were excluded from analysis. The accuracy of the force matching task was analyzed by taking the root mean square error of the steady hold force compared to the target level for each of the four conditions.

Epochs of synchronized EEG and EMG from each condition that satisfied the steady-state criteria were normalized and pooled across conditions and subjects (Amjad et al., 1997). Normalizing each epoch of data gives equal weight to each section and effectively eliminates the possibility of coherence bias which favors sections with high EMG amplitude (Amjad et al., 1997; James et al., 2008; Schoffelen et al., 2011).

## Coherence Analysis

Synchronous oscillations between cortical activity and EMG indicate functional connectivity which can be assessed through coherence analysis (Nunez et al., 1997; Mima and Hallett, 1999). Briefly, coherence measures the temporal correlation between two signals through the strength of the consistency of their phase lag as a function of frequency. The result is a coherence spectrum bounded between 0 and 1 for each frequency of interest. A value of 1 indicates a perfect temporal correlation, while 0 indicates no correlation.

For a given time series, x(t), let the auto spectrum be represented as

$$P\_{\text{xx}}(f) = \frac{1}{L} \sum\_{i=1}^{L} X\_i(f) \cdot X\_i^\*(f)$$

where Xi(f) represents the Fourier transform of the signal at segment i of L, and ∗ indicates the complex conjugate. A similar spectrum exists for the signal y(t), represented as Pyy f . The cross spectrum between the signals x (t) and y(t) is represented as

$$P\_{\mathbf{x}\mathbf{y}^\*}\left(f\right) = \frac{1}{L} \sum\_{i=1}^L X\_i\left(f\right) \cdot Y\_i^\*\left(f\right).$$

Coherence is calculated by normalizing the square of the crossspectral density between signals x (t) and y(t) by the product of their individual auto spectral densities (Baker et al., 1997; Nunez et al., 1997) as represented by

$$C\_{\rm xy} \left( f \right) = \frac{\left| P\_{\rm xy} \left( f \right) \right|^2}{P\_{\rm xx} \left( f \right) \cdot P\_{\rm yy} \left( f \right)}$$

Corticomuscular coherence (CMC) refers to the specific case where cortical and muscular signals represent x (t) and y(t). CMC was computed for each EEG-EMG electrode pair as well as FDI-APB coherence using FieldTrip, an open-source toolbox in MATLAB for the analysis of EEG and MEG data (Oostenveld et al., 2011). We used discrete prolate spheroidal sequences (DPSS) or Slepian tapers (Slepian, 1978) for the calculation of the auto and cross spectra. The multi-taper method provides several measures of the spectral estimation by multiplying the data series by a series of orthogonal tapers prior to calculating the Fourier transform (Pesaran, 2008). Three tapers were used in our analysis, providing a spectral bandwidth of ±5 Hz (Maris et al., 2007; Schoffelen et al., 2011).

#### Selection of EEG Electrodes

Although CMC can be calculated between each muscle and every scalp electrode, we limited our selection to EEG electrodes that showed high FDI-EEG coherence during the dowel-low task. Similar low-force isometric precision task have been previously used in literature for CMC analysis (Baker et al., 1997; Kilner et al., 2000; Fisher et al., 2002; Riddle and Baker, 2006; Chen et al., 2013). Raw coherence values were first normalized by conversion to standard Z-scores using the following formula

$$Z = \frac{\operatorname{arctanh}\left(\sqrt{\text{C}}\right)}{\sqrt{1/2N}}$$

Where N is the total number of tapers used in the calculation of coherence (C) (Baker et al., 2003; Laine et al., 2013, 2014). The electrode locations selected for further analysis were those at which the average Z-score exceeded a Bonferroni-corrected 99% confidence level (Z = 3.6). The correction accounts for the total number of EEG channels.

## Linear Mixed-Effects Model

A linear mixed-effect model provides a method of describing a relationship for a measurable quantity as a function of the sum of weighted independent variables (Winter, 2013). We investigated the effects of task condition on beta coherence using a linear mixed-effect model with the following format:

$$\text{CMC}\_{\beta} = \beta\_0 + \beta\_1 \cdot \text{Condition} + \beta\_2 \cdot \left(1|\text{participant}\right) + \epsilon\_1$$

where CMCβ is the average beta-range coherence of an epoch, Condition (high and low force for both the spring and dowel) is the fixed-effect term, Participant is the random-effect term, β<sup>n</sup> terms are the coefficients for the independent variables, and ǫ is the error. The random-effects term was inserted to account for subject variability since several measurements were taken for each condition. Models were generated to estimate CMCβ for the FDI and APB well as beta intermuscular FDI-APB coherence.

## RESULTS

## Baseline Corticomuscular Coherence

To confirm the spatial sensitivity and validity of our analysis procedures, we calculated FDI-EEG coherence for the dowellow condition. **Figure 2A** shows a head map of the average Ztransformed corticomuscular coherence between the FDI and all EEG channels. The locations of the four electrodes that exceeded our significance threshold (described previously) are shown in white over the left primary motor cortex and labeled C3, C1, CP3, and CP1. The spatial localization and magnitude shown over the left sensorimotor cortex coincides with previous literature findings during low force production (Witte et al., 2007; Chakarov et al., 2009; Piitulainen et al., 2013).

Cortical drive to the FDI during the dowel-high and springhigh conditions was directly compared. **Figure 2B** shows the average Z-transformed coherence spectra (5–100 Hz) of the four significant electrodes for the dowel-high (orange trace) and spring-high (blue trace) conditions. The peak coherence for both conditions appeared in the beta frequency range (gray shaded area in **Figure 2B**). The peak Z-transformed coherence during the dowel-high condition was 6.84 at 20.1 Hz. Despite matched force levels, however, the peak coherence during the spring-high condition decreased to 4.59 at 21.1 Hz.

#### Linear Mixed-Effect Model

**Figure 3** shows the results of the linear mixed-effects model which tests the effects of condition on beta FDI-EEG (**Figure 3A**) and APB-EEG (**Figure 3B**) beta-range coherence. Using an Ftest, we compared the difference in effect of spring-low and dowel-low model coefficients on beta CMC. We found no significant difference in effect of low force coefficients on either FDI-EEG or APB-EEG beta coherence. It is important to note that, in these low force conditions, both objects remain in the stable domain as the spring has not been compressed enough to exhibit instability.

A similar comparison of the difference in the linear mixedeffect coefficients for the spring-high and dowel-high conditions revealed a significance difference in effect on both FDI-EEG (p = 6.8459e-08) and APB-EEG (p = 1.6889e-05) beta-range coherence. Overall, CMC was reduced in the unstable task (spring-high condition) compared to the stable task (dowel-high condition) with matched force levels. During the high force compression conditions, the dowel remains in the stable domain, but the spring has been compressed to the point of instability.

## EMG-EMG Coherence

We tested for changes in FDI-APB beta coherence across the higher force conditions. **Figure 4A** shows the FDI-APB intermuscular coherence for the spring and dowel objects. The dowel-high condition showed a peak beta range Z-transformed coherence of 6.59 at 24.6 Hz. This peak value was significantly higher than for that of the spring-high condition, which had a peak value of 2.33 at 24.4 Hz. **Figure 4B** depicts the results of the linear mixed-effect model showing that the effects of the dowel was significantly higher than for the spring at the high force level (p = 4.9415e-10).

## DISCUSSION

In this study, we have examined the functional meaning of betaband corticomotor drive. Specifically, we tested the hypothesis that beta-band cortical drive reflects the "binding" of motor

FIGURE 3 | Results of the linear mixed-effects model for the four conditions: dowel-low (DL), spring-low (SL), dowel-high (DH), and spring-high (SH). The model was constructed to predict mean beta range coherence using condition as the fixed-effect with corrections for the random effects of participant. In each bar graph, the mean beta range CMC is shown on the vertical axis and condition is on the horizontal axis. Bars representing one standard error are shown for each condition and significance above the matched force level column pairs represent the statistical difference in the linear mixed-effect coefficients as determined using an F-test. (A) Linear mixed-effect found no differences in FDI-EEG beta coherence for the low forces, but found a significant difference between the higher force compression conditions. (B) Linear mixed-effect model results for APB-EEG coherence show the same statistical results for low and high conditions as in FDI-EEG.

neurons into functional units for synergistic cortical control, and thus should depend upon the type/degree of muscle coordination required by a task. Our findings provide evidence that beta-band corticomotor drive is inherently a reflection of intermuscular "binding" rather than steady isometric force production.

It is known that synergistically-activated muscles share beta-band cortical drive, as measured by intermuscular EMG coherence (Kilner et al., 1999; Boonstra and Breakspear, 2012; Boonstra, 2013). It has been suggested that this phenomenon reflects the "binding" of these muscles into a functional unit (or synergy) at the cortical level (Gray, 1994; Santello, 2014). As such, we predicted that beta-band CMC should emerge more prominently when the task of the motor cortex can be reduced to scaling the activation of a functional group of motor neurons (Valero-Cuevas, 2000). The idea that cortical output should reflect the dimensionality of a task is in line with the work of Rathelot and Strick (2006, 2009) who proposed that the motor cortex in humans and some higher primates may have evolved specific pathways (an "old" and "new" M1) to allow optional and flexible utilization of muscle synergies or "motor primitives."

In this study we have used compression of a slender spring to obligate time-varying fingertip force adjustment while minimizing physical movement and changes in net force (Valero-Cuevas et al., 2003; Dayanidhi et al., 2013a,b; Lawrence et al., 2014). This paradigm has been shown in an fMRI study to increase the engagement diverse brain networks (Mosier et al., 2011) beyond what is required for less-demanding actions. Succeeding at the unstable task (spring-high) requires a level of independent dynamic control of the muscles of the thumb and index finger to dynamically regulate fingertip force vectors to stabilize the spring (Johanson et al., 2001; Venkadesan et al., 2007), which is not the case for isometric tasks like when a wooden dowel is pinched (Valero-Cuevas, 2000). Accordingly, our data showed a general reduction of intermuscular coherence between the EMG signals recorded from the APB and FDI muscles when the spring was compressed at a level that induced instability. The reduction of beta-band (15–30 Hz) coherence between muscles can be interpreted as stemming from an "unbinding" of these two muscles at the level of the cortex. Changes in force alone would not explain our results, since little change in IMC was observed in the dowel-low vs. dowel high condition. Further, increased spring compression force is associated with increased instability, but not changes in spring compliance. Our results almost certainly reflect a neural reaction to the demands of controlling instabilities.

Similar to our findings for IMC, CMC in the beta-band was markedly reduced for the FDI and ABP muscles when the spring became unstable in the spring-high condition. If this phenomenon were simply due to increased noise in the EEG signal (e.g., due to increased cortico-cortical communication during the more difficult task) we would expect that IMC would have been preserved (since the muscles would still share betaband cortical drive), but this was not the case. Further, a change from static to dynamic force production is known to increase high-frequency (30–50 Hz) CMC as beta-band CMC shifts to higher frequencies (Omlor et al., 2007). We did not observe this type of shift. The simplest interpretation would be that the betaband cortical drive was fundamentally an intermuscular signal in this task, and that a shift in cortical control strategy occurred when the thumb and index finger muscles required independent control. This interpretation is also in line with evidence from a pilot study in which within-muscle motor unit coherence was reduced in the APB muscle during the same high-force spring compression task (Laine et al., 2015). A relationship between beta-band cortical drive and control strategy also agrees with its dependence on psychosensory aspects of a task (Laine et al., 2014), although typically it is the higher frequencies of cortical drive (30–50 Hz) which show the greatest sensitivity to such features (Brown et al., 1998; Omlor et al., 2007, 2011; Patino et al., 2008; Mehrkanoon et al., 2014).

Recently it has been hypothesized that beta oscillations arise from closed loops from M1 to muscle synergies back to M1 (Aumann and Prut, 2015). As such, during sustained contraction of a muscle, groups of sensorimotor neurons oscillate in synchrony to maintain the current state (Engel and Fries, 2010). Conversely during movement, de-synchrony of the local group disrupts beta rhythms. The results obtained in this study support this closed-loop hypothesis given in the fact that during the stable conditions (i.e., spring-low, dowel-low and dowel-high), synergistic muscle activations were necessary for producing the target forces. However, in the spring-high condition, it was necessary to disrupt these synergies despite the fact that a relatively constant net force was maintained.

## Confounds and Alternative Explanations

Physical differences in the objects compressed are of key importance to this study. Our interpretation is that the primary factor was the decreased stability of the spring when compressed at higher forces, however, small differences finger position, movement, object compliance, etc. could influence coherence measures and must be considered.

In a previous study, Kilner et al. found positive correlations between the magnitude of beta-band CMC and object compliance during a ramp and hold precision pinch task (Kilner et al., 2000). In our study, we did not observe a difference in beta-band CMC between the dowel-low and spring-low conditions, and at high forces, we observed a reduction rather than an increase in beta-band CMC. While the study of Kilner et al. shares many similarities with the work presented here, the compliant object used in our study was free at both ends and had the propensity to buckle and slip out of the hand when compressed at applied forces >2.2 N. In contrast, two fixed levers with programmable compliance were compressed in the study of Kilner et al. Such differences make direct comparison between studies inexact, but it is clear that the higher compliance of the spring relative to the dowel should, if anything, favor beta-band coherence (Kilner et al., 2000), as should higher force relative to lower force (Witte et al., 2007). Accordingly, our observation of decreased beta-band coherence in the spring-high condition appears to be related to a change in neuromuscular control strategy in response to the reduced stability of the spring. If very small movements can influence CMC (Kilner et al., 1999), then it is possible that the underlying reason for this may relate to the fact that movement may not allow muscles to be "bound" in the same way as is possible during static isometric force production. Although the physical positioning of the fingers can also have an influence on coherence (Riddle and Baker, 2006) the final position of the fingers during spring vs. dowel compression did not differ by more than about 1 cm, and we have no reason to believe that this would have been a major factor in the present study. Moreover, prior fMRI work in this same task has shown that the presence of instability has an effect distinct from that of compliance (Mosier et al., 2011).

It is also important to consider potential drawbacks of using EEG and surface EMG signals. For example, surface EMG will not be as sensitive as single motor unit recordings (Keenan et al., 2012) and may be influenced by signal processing techniques such as filtering or rectification (Boonstra and Breakspear, 2012; Farina et al., 2013). Even so, a systematic difference in the sensitivity of EMG signals to beta-band drive across tasks seems an unlikely explanation for our results, especially given the low levels of force required and our use of the most appropriate signal processing methods (i.e., EMG rectification) under these conditions (Farina et al., 2013).

It is relevant that compression of the spring to the point that it becomes unstable is an inherently difficult and demanding task. While there is increasing evidence that M1 activity may be involved with the perception of task goals (Shen and Alexander, 1997; Cisek et al., 2003; Scott, 2003), our suggestion is that the perception of difficulty in motor tasks is secondary to more tangible factors, such as the need for time-critical sensorimotor corrections, and individuated vs. synergistic control of muscles. In one recent study, for example, beta-band CMC was reduced when the degree of bimanual muscle coordination was increased through a visual feedback manipulation (de Vries et al., 2016). The interpretation was that beta-band CMC relates to the control of individual muscles, rather than coordination. The opposite argument could also be made, however, because in that study, errors of force may have been perceived and corrected individually for each hand in order to achieve better overall bimanual coordination. This would be in line with our coordination-based interpretation of beta-band drive to muscles. The same study also found that low-frequency IMC (∼10 Hz) across hands was highest when bimanual coordination was highest. Because ∼10 Hz CMC is not present in that study nor our own, low-frequency drive to individual muscles cannot be measured or compared concurrently with inter-muscular drive (i.e., IMC). This makes it difficult to attribute changes in IMC to inherently coordination-related aspects of our tasks, rather than other factors which might influence the production of lowfrequency neural drive in general. That said, our **Figure 4A** does visibly show low-frequency IMC reduced along with the beta-band in the spring-high condition, which certainly justifies further investigation of this issue.

Overall, our study tests the notion that beta-band cortical drive essentially reflects the dimensionality of cortical commands, and our results strongly suggest that beta-band CMC should be interpreted carefully, with special attention to

#### REFERENCES


muscle coordination and time-critical sensorimotor demands. Our current speculation, supported by the findings of this investigation, is that beta-band CMC reflects the use of a lowdimensional mode of cortical control over groups of muscles, rather than the maintenance of a steady-state force output. While fully understanding the neurophysiology and task-dependence of cortico-motor oscillations requires further study, our results justify such future work and provide a springboard for more focused investigations into the relationship between the physical requirements of a task and neural control strategies necessary to satisfy them. Finally, such insights into the origin and modulation of cortico-motor oscillations would not only clarify fundamental mechanism for sensorimotor control, but perhaps provide well-founded tasks and analyses directly translatable to clinical measures and diagnostic tests (Norton and Gorassini, 2006; Hammond et al., 2007; Pogosyan et al., 2009; Fisher et al., 2012; Ko et al., 2016).

## AUTHOR CONTRIBUTIONS

AR, JK, and FV designed research; AR performed research; AR, CL, and JK analyzed data; AR, CL, JK, and FV wrote the paper.

#### ACKNOWLEDGMENTS

This material is in part based upon work supported by NSF Grant EFRI-COPN 0836042, NIH Grant R01-052345, NIH Grant R01-050520 to FV, and Supplement R01-050520-W1 to AR.


coupling between neuronal spike trains. Prog. Biophys. Mol. Biol. 53, 1–31. doi: 10.1016/0079-6107(89)90004-7


**Conflict of Interest Statement:** FV holds US Patent No. 6,537,075 on some of the technology used in this study that is commercialized by Neuromuscular Dynamics, LLC.

The other authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Reyes, Laine, Kutch and Valero-Cuevas. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Keivan Mojtahedi <sup>1</sup> , Bryan Whitsell <sup>2</sup> , Panagiotis Artemiadis <sup>2</sup> and Marco Santello1 \**

*1Neural Control of Movement Laboratory, School of Biological and Health Systems Engineering, Arizona State University, Tempe, AZ, USA, 2Human Oriented Robotics and Control Laboratory, Mechanical and Aerospace Engineering, School for Engineering of Matter, Transport and Energy, Arizona State University, Tempe, AZ, USA*

Of particular interest to the neuroscience and robotics communities is the understanding of how two humans could physically collaborate to perform motor tasks such as holding a tool or moving it across locations. When two humans physically interact with each other, sensory consequences and motor outcomes are not entirely predictable as they also depend on the other agent's actions. The sensory mechanisms involved in physical interactions are not well understood. The present study was designed (1) to quantify human–human physical interactions where one agent ("follower") has to infer the intended or imagined—but not executed—direction of motion of another agent ("leader") and (2) to reveal the underlying strategies used by the dyad. This study also aimed at verifying the extent to which visual feedback (VF) is necessary for communicating intended movement direction. We found that the control of leader on the relationship between force and motion was a critical factor in conveying his/her intended movement direction to the follower regardless of VF of the grasped handle or the arms. Interestingly, the dyad's ability to communicate and infer movement direction with significant accuracy improved (>83%) after a relatively short amount of practice. These results indicate that the relationship between force and motion (interpreting as arm impedance modulation) may represent an important means for communicating intended movement direction between biological agents, as indicated by the modulation of this relationship to intended direction. Ongoing work is investigating the application of the present findings to optimize communication of high-level movement goals during physical interactions between biological and non-biological agents.

Keywords: human–human interaction, impedance, leader and follower, physical interaction, stiffness

## INTRODUCTION

Collaboration, defined as the act of cooperation among multiple agents toward the attainment of a common goal, is one of the most sophisticated behaviors exhibited by biological organisms. Although cooperation is ubiquitous among a wide range of species ranging from ants to primates, the level of sophistication reached by humans in their ability to cooperate is unparalleled in the animal kingdom. Of particular interest to the neuroscience and robotics communities is the understanding of how humans collaborate to perform motor tasks.

Physical collaboration between two homologous biological agents, such as two humans holding a tool or moving it across locations, entails complex sensorimotor processes. Specifically, the

#### *Edited by:*

*Matthew Tresch, Northwestern University, USA*

#### *Reviewed by:*

*Robert Gregg, The University of Texas at Dallas, USA Marc Daniel Killpack, Brigham Young University, USA*

#### *\*Correspondence:*

*Marco Santello marco.santello@asu.edu*

*Received: 30 September 2016 Accepted: 29 March 2017 Published: 13 April 2017*

#### *Citation:*

*Mojtahedi K, Whitsell B, Artemiadis P and Santello M (2017) Communication and Inference of Intended Movement Direction during Human–Human Physical Interaction. Front. Neurorobot. 11:21. doi: 10.3389/fnbot.2017.00021*

problem of physically collaborating with another agent to perform a given motor task introduces control problems that go well beyond those encountered when controlling one's own limb. For example, planning and execution of reaching or grasping movement are thought to occur through an internal model of the agent's limb that allows prediction of the sensory consequences of the motor action (Johansson and Flanagan, 2009; Wolpert et al., 2011). Examples of such phenomena are the temporal coupling of grip and load forces associated with moving an object denoting anticipation of movement-related inertial forces (Flanagan and Wing, 1997), or the anticipatory control of torque prior to lifting an object with an asymmetrical center of mass (Salimi et al., 2003; Bursztyn and Flanagan, 2008; Fu et al., 2010, 2011; Fu and Santello, 2012; Mojtahedi et al., 2015). However, when two humans physically interact with each other, sensory consequences and motor outcomes are not entirely predictable as they also depend on the other agent's actions. Therefore, the question arises as to how the central nervous system of each agent factors in the other agent's actions when physically interacting with each other to perform a collaborative task. A better understanding of this problem can help developing biologically inspired controllers supporting human–robot physical interactions, e.g., exoskeletons used for neurorehabilitation or physical augmentation, and optimizing the way these interactions can be performed.

Physical interaction between humans and robots has been mainly investigated using the notion of mechanical impedance. Hogan (1985) first proposed robot impedance controllers as a way to guarantee stable and robust behavior of a robot that interacts with a human. Since then, a plethora of robot applications involving physical human–robot interaction use control of impedance, and in most cases, this is done to purposefully impose a specific dynamic behavior to the human agent. For example, the MIT-MANUS—used extensively for upper limb rehabilitation—uses the concept of impedance control in a back-drivable system to restrict the motion of the patient's arm along a specific path, while the patient tries to reach a target *via* a manipulandum attached to his/her paretic arm (Krebs et al., 1998). For this scenario, impedance control is used to assist the human subject to reach a pre-defined target and imposes high resistive forces to motion that is not congruent with the desired trajectory.

The main objective of our study was to quantify the extent to which the human body (mainly upper limb) impedance can be used to infer intended movement direction of a cooperating agent in absence of other sensory cues (e.g., vision, hearing). Specifically, the present study sought to characterize the role of haptic information, which includes the relationship between force and displacement in a power exchange between two agents. We pursued this objective by quantifying human–human physical interactions where one agent ("follower") was asked to infer the intended direction of motion of another agent ("leader"). In this design, the follower is trying to estimate the direction that the leader would allow them to move. Our interpretation of this interaction is that (1) the leader's intended movement direction modulates this relationship in a direction-specific manner and (2) the follower can interpret this direction-specific modulation of this relationship to infer the leader's intended movement direction. Note that the impedance in formal sense is quite complicated to measure due to the involvement of inputs/responses from both leader and follower who are physically coupled. So, even if the leader hypothetically modulated impedance to "instruct" the follower, the measure of leader's impedance would not reflect the follower's behavior as they both probe *and* react to the forces and motions. Briefly, we hypothesize that the emergent dyadic behavior (follower's inference of leader's intended direction) could be captured by the relationship between resultant force and displacement. Certainly, dyad's arm impedances could affect this relationship, but certain aspect of dyad's behavior interaction such as follower's probing strategy could not be considered as impedance. Thus, the current study could only directly show and support how the relationship between force and displacement would change while we interpret the changes in the relationship as arm impedance modulation.

We also investigated the role of visual feedback (VF) in communicating intended directions *via* arm impedance modulation. We hypothesized that cooperating agents would be able to use arm impedance modulation to effectively communicate intended movement direction among cooperating agents. Previous studies have shown that humans can adapt to force fields during reaching tasks by modulating their arm impedance over time (Franklin et al., 2007; Wong et al., 2009). Therefore, we hypothesized that repeated exposure to the leader's impedance would lead to a trial-by-trial modulation of arm impedance and improvement in follower's ability to infer the leader's intended direction. Finally, we hypothesized that haptic feedback would be sufficient to enable cooperating agents to accurately communicate intended movement direction through modulation of arm impedance.

## MATERIALS AND METHODS

#### Subjects

We tested 20 right-handed subjects (12 males, 8 females; age: 18–28 years). Hand dominance was self-reported. Subjects had no history or record of neurological disorders and were naïve to the purpose of the studies. Subjects gave informed written consent to participate in the experiments. The experimental protocols were approved by the Institutional Review Board at Arizona State University and were in accordance with the Declaration of Helsinki. Five pairs of subjects (dyads) were assigned to the experiment with VF, whereas the other five dyads were assigned to the experiment with no visual feedback (NVF).

## Experimental Apparatus

Each dyad was shown 12 lines oriented 30° apart from each other denoting movement direction and a circle (5-cm radius) on a computer screen (**Figure 1**). A number (1–12) was displayed at the outer end of each line. In the VF experiment, the dyad saw a dot on the screen. The dot position was colocated with the position of the handle the two subjects were holding and was located underneath the screen. The dot displayed on the monitor moved the same amount as the handle (ratio 1:1). The handle movement was constrained by the robot arm in the horizontal plane. The

screen prevented the dyad from seeing the arm configuration of the other agent and the grip handle. In the NVF experiment, the dyad could not see the dot position but could still see the direction lines and circle.

For both experiments, we used an anthropomorphic 7-degreeof-freedom robot arm (LWR4+, KUKA) with the associated KRC robot controller and the KUKA's Fast Research Interface. We used two load cells (Model: 3140-500 kg, precision: 0.02% FS, one-axis force sensor) embedded in the grip device to measure the resultant forces of the dyad in *x*- and *z*-axis (**Figure 1**).

#### Experimental Tasks

One subject was designated as the "leader," whereas the other was designated as the "follower." At the start of the trial, the handle and corresponding dot displayed on the screen were positioned in the center of the circle. For each trial, the experimenter showed a specific number on a sheet to only the leader. This number was one of the 12 possible movement directions, which we will refer to as the "intended direction of movement" for that trial. The leader was instructed that his/her goal was to plan the movement in the direction that was shown to them by the experimenter while keeping the object as close as possible to the center of the circle (**Figure 1**). Therefore, leader thought about performing a movement rather than executing it in the direction assigned by the experimenter. The follower was instructed that his/her goal was to infer the leader's intended direction of movement. The follower was also instructed that he/she could move the grip handle as he/she desired, but that he/she had to stay within the circle. The leader was instructed to react to the forces and motion of the follower while preserving the intention to move in a given intended direction. Thus, the leader tried to hold the handle in the middle of the work space and resisted all motion. The follower explored the space to infer the intended direction of leader. Whenever the position of the grip handle and the corresponding dot moved out of the circle, the color of circle and direction lines changed from blue to red to signal that the trial had to be stopped and repeated. Therefore, both groups received VF of the error, i.e., they were shown when the grip handle crossed the boundaries of the circular workspace. The subject pairs in the VF group never moved out of range. For the NVF group, the handle moved out of range only on four trials performed by three subject pairs (0.95% of all trials across five subject pairs in NVF). The grip handle range of motion was not physically constrained. After performing each trial, the follower was asked to write the number of the inferred direction on an answer sheet. During the whole experiment, neither the follower nor the leader received any feedback about his/her performances from experimenter, nor was the leader informed about the follower's performance by the follower or the experimenter. Verbal communication between the subjects, as well as between the subjects and the experimenter, was not allowed before, during, or after the trial.

The role of each subject in the dyad can therefore be described as follows.

The "leader" was asked to


• React to the follower's forces by maintaining the handle as still as possible at the center of the circle while preserving the intention to move in the instructed direction.

The "follower" was asked to


Subjects were asked to keep their right hand close to the grip handles and wait for a "go" signal. As soon as they heard the "go" signal, they were asked to grasp the handle and started to interact with each other. Subjects initially performed 24 trials (2 repetitions per directions; Trial: 1–24) to reach a plateau in the performance, e.g., correct inference of the leader's intended movement direction. Pilot data had revealed that this number of trials had been found to be sufficient for familiarization purposes. Then, subjects continued to perform 60 more trials (5 repetitions per direction; Trial: 25–84). The order of directions was randomized in both Trial: 1–24 and Trial: 25–84. We used different randomized order across dyads. Each trial lasted 30 s. The same instructions were given to the groups with and without VF.

At the beginning of each trial, the arm posture was inspected by the experimenter to ensure the same posture would be used across trials. Handle position was always located on the sagittal plane, and the trunk was as close as possible to the frame to prevent both subjects from viewing their arm configuration. The experimenter also verified that subjects kept their gaze on the monitor on each trial.

## Data Recording, Processing, and Experimental Variables

The robot was used to restrict motion of the fixture to the horizontal plane, prevent rotation of the fixture, and record the position of the grip handle during the experiment. We synchronized collection of position and force data. Position and force data were recorded at a sampling rate of 100 Hz and run through a fifthorder Butterworth low-pass filter (cutoff frequency: 30 Hz). The first-time derivative of force or position data was also low-pass filtered with cutoff frequency of 15 Hz.

#### Percentage of Inferences

The dyad's goal was to minimize the error between the leader's intended direction and the follower's inference. Therefore, all the metrics were defined based on this task requirement. To quantify the extent to which the follower could correctly infer the leader's intended movement direction, we computed the percentage of accurate inferences (PAI) by each follower relative to the total number of trials based on his/her responses in the answer sheet. The follower's error direction with respect to the leader's intended direction was defined as the difference between leader's intended direction and follower's response. This error direction could be any number ranging between −6 and +5. An accurate inference of the follower would correspond to a difference of 0, whereas non-zero differences would denote inaccurate inferences. Positive sign of error direction (with respect to the leader's intended direction) indicated counterclockwise difference between the leader's intended direction and the follower's response (**Figure 1**). One error direction with respect to intended direction is equivalent to 30° (**Figures 1** and **2**).

In **Figure 1**, we define the directions 1, 4, 7, and 10 as *cardinal* directions and all other directions as *non-cardinal*. This distinction was motivated by the fact that accurate inferences of the leader's intended movement direction differed across cardinal versus non-cardinal directions (see Statistical Analysis).

#### Force–Displacement Relationship

In the present work, the term impedance denotes to the effect of voluntary muscle activations of mainly the upper limb to the limb dynamics. Those dynamics largely affected by the muscle activations include both the stiffness and the damping characteristics of the arm, which are only apparent when there exists force interaction with the environment, in the current case the follower. In the dyadic interaction, the resultant force denotes to the net force which both leader and follower together generated on the handles. In the current experiment, the robot is entirely passive because the friction and damping effects of the robot are feed-forwarded to the joints by a torque controller, and therefore, their magnitudes are negligible (close to 0; the robot inertia and damping are by default compensated through the built-in impedance controller of the KUKA arm). The inertia effects are also negligible due to the low acceleration in the handle motion. The average of absolute acceleration of the handle was 16.02 (±0.35) cm/s2 (1.63 ± 0.04% of gravity

acceleration). Thus, the damping and inertia effects are minimum relative to the stiffness effects due to the low velocity and low acceleration, respectively, of the handle motion. Therefore, consistent with the previous work (Mussa-Ivaldi et al., 1985; Formica et al., 2012), we assume that the total change in force [Δ*F*(*n*)] is primarily caused by the stiffness of the subjects interacting with each other.

In the present work, the estimation of the stiffness depends on the follower's exploration of the workspace, which is very different from the systematic exploration of an equal number of movement

leader's intended movement direction computed (across all subjects). (A) Trial: 1–24. (B) Trial 25–84. Data for each direction are means averaged across all subjects. The cardinal directions are 1, 4, 7, and 10. The dash lines represent the SE. The four concentric gray circles represent the ring axes of percentage. 100% is the biggest ring, while 0% is a dot in the center which is not shown.

directions as tested in previous work (Perreault et al., 2001; Krebs et al., 2003). So, we had to use a different approach to quantify the relationship between force and motion due to limitations of applying a conventional approach to estimate stiffness. We used Eq. 2 to calculate the force–displacement relationship (|*k*|):

$$F(n) = \sqrt{F\_x^2(n) + F\_x^2(n)};\ U(n) = \sqrt{U\_x^2(n) + U\_x^2(n)}\tag{1}$$

$$k(n) = \frac{\Delta F(n)}{\Delta U(n)} = \frac{F(n) - F(n-1)}{U(n) - U(n-1)}\tag{2}$$

Equation 1 describes how to calculate the force [*F*(*n*)] and displacement [*U*(*n*)] magnitudes for each time point (*n*). As we see later in **Figure 5**, the follower did not explore all the points in the circle with 5-cm radius; and it illustrates that the magnitudes of force and displacement are changing reversely regardless of their directions or vector properties, e.g., small forces and large displacements are along the intended direction, while we have large forces and small displacements for other directions (**Figures 5A,B**). Therefore, it seems that considering only magnitudes is enough to discriminate the relationship between force and displacement across directions. So, the *k* in Eq. 2 should be calculated on the points visited by the follower. To avoid the canceling effect of *k* value due to positive and negative *k* values (according to movement direction relative to start location), we computed absolute value of *k* [|*k*(*n*)|]. For example, if the follower pulled the handle in direction 1 for 3 cm distance, this creates positive values of *k* in Eq. 2. By contrast, when the leader pulls back the handle to the center to maintain the handle at the center, this creates negative values of *k* in Eq. 2.

As |*k*| value was associated with a specific position point within the circle (**Figure 1**), we averaged |*k*(*n*)| over time to obtain the best value of the force–displacement relationship for each visited point. We gave |*k*| the value of 0 to the points that were not visited. If the position of the grip handle did not

(C). The selected trial is representative of correct response.

change relative to its starting position (center of the circle), we assigned the maximum value of |*k*| of that trial (infinite) to the |*k*| at that position. The denominator is equal to 0 while the handle position does not change. So, we consider the |*k*| as infinite (the maximum |*k*| which is recorded when they did move the handle in that trial).

The position resolution in the horizontal workspace plane was 1 mm2 . We calculated the average of non-zero values of |*k*| for all the position points within each 1 mm2 and assigned that |*k*| value to that square. Therefore, each square in the horizontal plane was assigned a specific |*k*| value. By doing so, the average |*k*| associated with each direction could be obtained by calculating the average of non-zero values of |*k*| of the squares located in that direction (within ±15° of each direction). This procedure led to the extraction of 12 |*k*| values, 1 for each of the 12 movement directions with respect to the leader's intended direction (|*k*|*i*,Average; *i*: −6 to +5; *i* is movement direction with respect to intended direction). First, we calculated the average |*k*| across all movement directions (|*k*|Average; Eq. 3). Second, we normalized the |*k*| values (|*k*|*i*,Normalized; *i*: −6 to +5; Eq. 5) based on the maximum of the average |*k*| values in all directions for each trial (|*k*|Max; Eq. 4) to remove differences in |*k*| across dyads.

$$\left|k\right|\_{\text{Average}} = \frac{1}{12} \sum\_{i=-6}^{5} \left|k\right|\_{i, \text{Average}}\tag{3}$$

$$\left|k\right|\_{\text{Max}} = \max\left( \left|k\right|\_{-6, \text{Average}}, \left|k\right|\_{-5, \text{Average}}, \dots, \left|k\right|\_{\*5, \text{Average}}\right) \tag{4}$$

$$\left|k\right|\_{i, \text{Normalized}} = \frac{\left|k\right|\_{i, \text{Average}}}{\left|k\right|\_{\text{Max}}} \tag{5}$$

Briefly, we had 100 × 100 points (|*k*| estimates). We assigned a 0 value to |*k*| at the points that were not visited by the handle. For the visited points, we obtained the average value of estimated |*k*| across time samples to capture the behavior of both leader and follower at that point. We then calculated the spatial average of |*k*| values (non-zero values) within ±15° of each direction to obtain average |*k*| of that direction. Although the measure of |*k*| is not formal stiffness or impedance, it is suitable for capturing the relationship between force and motion of a dyad in our paradigm, which may imply aspects of dyad's modulation of stiffness/ impedance.

#### Statistical Analysis

#### Inference of Intended Movement Direction

We chose to break down the trials by 24 for the first block and 60 trials for the second block. Analysis of pilot data revealed that the accuracy of predicting the intended movement direction reached a maximum and converged to a steady state after the first 24 trials. The results of the current study also confirmed these pilot observations (**Figures 2A,B**).

To assess whether PAI was sensitive to the leader's intended direction along the cardinal directions and non-cardinal directions, we performed analysis of variance (ANOVA) with repeated measures on PAI with two within-subject factors (Trial; trials 1–24 and trials 25–84; two levels, Direction; cardinal and noncardinal directions; two levels) and one between-subject factor (Group; VF and NVF groups, two levels; see **Figure 3**).

#### Force–Displacement Relationship Is Modulated across Trials for the VF Group

We performed ANOVA with repeated measures on the average |*k*| of all directions, |*k*|Average, with one within-subject factor (Trial; two levels). We also performed ANOVA with repeated measures on the normalized |*k*|, |*k*|*i*,Normalized, with one within-subject factor (Direction; direction −6 to +5; 12 levels; **Figures 6C,D**). Note that |*k*| normalization was performed to only remove the strength differences between subjects.

Comparisons of interest for statistically significant differences (*p* = 0.05) were further analyzed using *post hoc* tests with Bonferroni's corrections. We conducted the normality and sphericity tests and statistical models were valid. Statistical analysis was performed using IBM Sciences Statistical Package for the Social Statistics.

## RESULTS

#### PAI of Follower

Percentage of accurate inference analysis was divided into three sections. First, we report the evolution of PAI over trials. Second, we investigate the effects of Group, Trial, and Direction on PAI by keeping the original direction in order to assess the effect of cardinal directions on PAI. Third, we assess PAI without regarding the effect of direction to assess how both groups performed when we had a common reference (i.e., 0 error direction).

#### PAI Analysis across Trials

Visual inspection of the trial-to-trial fluctuations of the PAI revealed that performance was more variable in the early trials (1–24). To minimize the effect of large random trial-to-trial PAI fluctuations in these early trials, for statistical purposes, we averaged PAI across a variable number of trials. We found that averaging PAI across 3, 4, 6, 8, and 12 trials gave approximately the same result, i.e., PAI stopped improving after the first 24 trials.

Percentage of accurate inferences improved in both VF and NVF groups (**Figures 2A,B**, respectively). In the beginning, both groups could not perform consistently above 60% of PAI. However, after approximately 24 trials, both groups reached a steady-state performance.

We analyzed the time it took followers to report inferred leader's intended direction. When VF was available, followers reported the follower's intended direction within 29.6 ± 0.2 s, whereas the response time was slightly shorter (27.4 ± 1.1 s) when VF was not available.

#### Cardinal versus Non-Cardinal Directions

**Figure 3** shows PAI for all directions. We compared the PAI associated with the leader's intended movement in the cardinal directions (1, 4, 7, and 10; **Figure 1**) versus non-cardinal directions. Although VF did not affect PAI [no main effect of Group; *F*(1,8) = 0.697, *p* = 0.428], PAI was significantly different as a function of Trial [*F*(1,8) = 28.891, *p* = 0.001, η<sup>2</sup> = 0.78] and movement direction [*F*(1,8) = 7.254, *p* = 0.027, η<sup>2</sup> = 0.47]. No significant interactions were found (all *p* values >0.320). As found earlier across all movement directions, PAI of Trial: 25–84 was significantly larger than Trial: 1–24 (*p* = 0.001). For the experimental trials, PAI associated with the leader's intended movement along the cardinal directions was significantly larger than along non-cardinal directions (93.5 and 80.5%, respectively; *p* = 0.027).

#### VF versus NVF

**Figure 4** showed the PAI of follower for the VF and NVF groups computed from Trial: 1–24 and Trial: 25–84 (**Figures 4A,B**, respectively). PAI was well above chance level (equivalent to 1 out of 12 possible directions, i.e., 8.33%). After 24 trials, PAI values were 87.33 and 83.33% when performed with and without VF, respectively. If we assume that ±1 error direction with respect to intended direction is a negligible performance error (±30°), the combined PAI were 94.67 and 96.33% for trials performed with and without VF, respectively.

The availability of VF did not significantly affect PAI [no main effect of Group; *F*(1,8) = 0.535, *p* = 0.485]. However, we found statistically significant differences in PAI as a function of Trial [*F*(1,8) = 30.444, *p* = 0.001, η<sup>2</sup> = 0.79], but no significant interaction between Group and Trial [*F*(1,8) = 0.309, *p* = 0.594]. We found that PAI from Trial: 25–84 was significantly larger than from Trial: 1–24 (**Figure 4**; *p* = 0.001). Note that we reported the effect size (partial-eta squared) as a measure of magnitude of our effect. The effect size of learning was quite large (η<sup>2</sup> = 0.79). This indicates that the significance of the result was unlikely to be marginal, for example, large variation within one subject could have driven the result. Therefore, we were confident that our sample size (five subjects per group) was adequate. The results (effect sizes) were highly consistent among the 5 subjects within each group, and—most importantly—highly consistent across the 10 subjects across both groups.

In summary, the follower's ability to infer the leader's intended movement direction was insensitive to whether the follower could view the position of the dot on the screen or not. Furthermore, PAI improved with practice, implying that the follower and leader gradually adapted to each other's actions to communicate and collaborate with each other. Specifically, the follower learned to infer the leader's intended movement direction, of the leader, while the leader learned how to react to the follower's forces. Finally, the follower was more accurate in inferring the leader's intended movement direction for cardinal than non-cardinal directions.

#### Force–Displacement Relationship Analysis

We first present one representative trial from a dyad performing our task with VF, followed by analysis of |*k*| adaptation for the VF group. We present the force–displacement relationship analysis for only VF group because they had the reference point of the center of circle. This allowed us to perform force–displacement relationship analysis relative to this reference point.

#### Representative VF Trial

**Figure 5** shows the displacement–position profile (**Figure 5A**), force–position profile (**Figure 5B**), and normalized |*k*|–position profile (**Figure 5C**) of a sample trial of a VF dyad. The displacement– position profile for the VF group reveals that the dyad performed the task as instructed, i.e., within the boundaries and close to the center of the circular workspace. Note that the dyad exhibited larger handle displacement along the leader's intended direction (**Figure 5A**). With regard to the force– position profile of the VF group, the leader could generate a reasonable force field (impedance field) for each direction as if the resultant force tended to be directed toward the leader's intended direction at each position (**Figure 5B**). Visual examination of the normalized |*k*|–position profile of the VF group reveals that the dyad exhibited lower |*k*| in the intended leader's movement direction (**Figure 5C**).

#### Force–Displacement Relationship Analysis: Dyads with VF

To elucidate the force–displacement relationship analysis in the VF group, we compared the |*k*| measured during the dyad interaction during trials 1–24 and trials 25–84. We captured the evolution of normalized |*k*| (|*k*|Normalized) across two blocks of trials (1–24 and 25–84; **Figures 6C,D**). We then performed pairwise comparisons of |*k*|Normalized within each block to investigate how dyads selectively generated |*k*|Normalized across different directions with respect to the leader's intended movement direction.

We found a main effect of Direction in |*k*|Normalized for both trials [in Trial: 1–24, *F*(11,44) = 15.182, *p* = 0.001, η<sup>2</sup> = 0.81; in Trial: 25–84, *F*(11,44) = 37.058, *p* = 0.001; η<sup>2</sup> = 0.90]. For the Trial: 1–24, we found no significant difference in |*k*|Normalized on pairwise comparisons between the intended movement direction (0, yellow bar) and directions of ±1 (orange bars), −6, ±5, and ±2 (dark brown bars, **Figure 6C**; all *p* > 0.05). However, there was significant difference in |*k*|Normalized between intended direction and adjacent directions of ±3 and ±4. Similarly for Trial: 25–84, no significant differences were found when comparing |*k*|Normalized at the leader's intended movement direction and adjacent directions (±1; orange bars). However, we found significant difference in |*k*|Normalized on pairwise comparisons between the intended movement direction (0, yellow bar) and all other directions except ±1 (dark brown bars, **Figure 6D**; all *p* > 0.05). **Figure 7** illustrates how |*k*|Normalized of movement direction with respect to intended direction changed across trials. **Figure 7** shows that the variations in |*k*|Normalized of movement direction with respect to intended direction of 0 (yellow line in **Figure 7**) were gradually settled in the gray box over the trials and also became more discriminable from |*k*|Normalized of movement direction with respect to intended direction of 1 (orange line in **Figure 7**). **Figure 7** shows how the force–displacement relationship in dyadic interaction evolves across trials, and this might imply that dyad learns to modulate their stiffness to perform the task.

To further quantify the effect of trial (practice) on force– displacement relationship modulation, we compared the average |*k*| of all directions (|*k*|Average) across two blocks of trials (1–24 and 25–84). Statistical results showed that |*k*|Average did not change significantly across Trial (*p* = 0.447).

These results indicate that, following early exposure to our task, the dyad learned to modulate force–displacement relationship across movement directions. Importantly, the dyads' |*k*|

became minimum at the leader's intended movement direction, even though the average |*k*| did not change significantly.

## DISCUSSION

The primary goal of our study was to quantify the extent to which human body impedance can be used to infer intended movement direction of a cooperating agent. We found that the success in conveying inferring intended direction of motion between two agents was correlated with the control of leader's impedance as a function of the follower's direction of motion. Hence, we were able to show that the leader was conveying the information of intended direction to the follower by controlling his/her impedance at the object they were interacting with. Therefore, our results may imply that two cooperating agents could use arm stiffness modulation during physical interaction as a viable means of communication of intended movement direction. We discuss these results in relation to previous work, potential sensorimotor mechanisms, and applications of the proposed framework to human–robot interactions.

## Effect of Practice on Accuracy of Movement Direction Inferences and Force–Displacement Relationship

A moderate amount of practice (1–24 trials) led the followers to a significantly greater accuracy of inferences of the leader's intended movement direction (**Figures 2** and **4**). This result indicates that subjects might have needed some practice to gauge and interpret each other's physical response. Nevertheless, the small number of trials leading to a very high level of accurate inferences (>83%) also suggests that humans (a) can maintain the high level of accurate responses after 24 trials (**Figures 2** and **4**), which might imply that within 24 trials the dyad was already specialized for modulating arm stiffness, (b) are very sensitive to the directional tuning of arm impedance (see force– displacement relationship analysis below; **Figure 6**), and (c) can therefore learn fairly quickly to correctly interpret such directional tuning (**Figure 7**).

The adaptation of force–displacement relationship (|*k*|) as function of intended movement direction evolved gradually (**Figures 6C,D**). Gradual discrimination of normalized |*k*| across different direction of motion (**Figure 7**) implies that dyads could convey the intended direction of motion by modulating and perceiving normalized |*k*| associated with the physical interaction. We interpret these data as follows: after performing 24 trials, the leader selectively generated less |*k*| in his/her intended direction in response to follower's force perturbations. Therefore, we interpret this finding as evidence that control of the leader's stiffness might be a critical factor in conveying the intended direction to the follower regardless of VF of the handle or arms. Our results also indicate that subjects learn this strategy by experiencing our task for the 12 directions. It is conceivable that exposure to fewer movement directions might result in faster learning across trials, or shorter exploration duration within each trial.

## Sensorimotor Integration Mechanisms for Movement Direction Inference

Visual feedback of the movement of the shared handle did not affect the extent to which followers correctly inferred the leader's intended movement direction. This result suggests that haptic feedback elicited by physical interaction is sufficient to extract intended movement direction from the perceived force– displacement relationship (**Figures 2** and **4**). VF of movement trajectory was not necessary also in tasks performed by individual subjects requiring adaptation to stable or unstable dynamics (Franklin et al., 2007). Furthermore, final adaptation was similar with and without VF even when the learning signals (proprioception and vision versus only proprioception) were different (Franklin et al., 2007). Another study found that visual information of the movement trajectory alone might not be sufficient to modulate limb stiffness in response to an unstable elastic force field applied to the limb (Wong et al., 2009). Specifically, such adaptation might rely on somatosensory feedback to a greater extent than vision because of a direct relationship with perturbing forces. In reaching tasks, visual perturbations (manipulation of the cursor position) did not result in stiffness modulation, whereas force perturbation in elastic force field caused a significant increase in stiffness (Wong et al., 2009).

Movement kinematics appears to be sensitive to whether VF is available or not during adaptation of movement trajectories. Specifically, the movement profiles were significantly more linear when VF was available in "no force" and "velocity force" fields. However, the linearity did not change for visual and no visual conditions in a position force field (Franklin et al., 2007). The current study found different movement profiles of net displacement during the physical interaction between VF and NVF groups. However, even with this difference, the subjects learned the task and performed equally well in later trials. We should note that these results do not rule out a role of VF in our physical interaction task, but rather point to the fact that haptic feedback alone was accurate enough to enable correct inference of intended movement direction.

Subjects' ability to use non-visual feedback to estimate human body (mainly upper limb) stiffness and infer intended movement direction likely arises from their ability to integrate sensory feedback about movement and force. Specifically, based on the definition of stiffness, movement direction associated with low stiffness would result in a larger displacement due to smaller force and smaller displacement due to larger force for high stiffness. Our focus on stiffness incorporates this relation between force and displacement, and our interpretation about stiffness as a means of communication includes the use of position sensing for this purpose. Specifically, we propose that position and force sensing combined was involved in the estimation of intended movement direction (see Supplementary Material; Figures S1 and S3 in Supplementary Material). As impedance cannot be sensed by a specific type of sensory receptor, impedance estimation would have to rely on integrating estimation of muscle length and force, each of which is mediated by distinct mechanoreceptors (muscle spindles and Golgi tendon organs, respectively). In Figure S3 in Supplementary Material, the logistic regression analysis is in favor of the proposed notion that force–displacement relationship (|*k*|) is a better predictor of PAI than either average resultant force or maximum displacement alone at intended direction. This proposition is also consistent with experimental evidence showing that subjects estimate object stiffness by differentially weighing feedback information provided by muscle length and force receptors (Mugge et al., 2009). Since the leader was never required to execute a voluntary movement but just to "plan" (but not execute) a movement in a prescribed direction, it is conceivable that motor cortical areas involved with upper limb control were activated, as shown by literature on motor imagery (e.g., Vogt et al., 2013; Eaves et al., 2016; Hanakawa, 2016).

## Impedance-Based Communication in Human–Human and Human–Robot Interactions

Human arm impedance control has received increased attention during the last decades due to its importance in physical interaction with robotic devices, for assistive, rehabilitation, and performance augmentation purposes. Humans can vary the dynamics of their interaction with a robot by changing the configuration of their limbs and/or modifying limb stiffness through co-contraction of opposing muscles (Perreault et al., 2001; Krebs et al., 2003; O'Neill et al., 2013; Patel et al., 2013). From a robotics point of view, Hogan (1985) showed that these dynamics can be dealt with by effectively utilizing impedance as a way of controlling the robot and its interactions with humans and external objects.

Human–robot interaction applications motivated the design of our study. Nevertheless, our results should be considered a preliminary step in the context of these potential applications due to the fact that our setup is a simplified version of tasks with more complex mechanics. Although the human–human interaction scenario we investigated is not representative of all contexts involving physical collaborations between two human agents, or human–robot agents, we believe that our work provides important insights about the feasibility of using impedance as a viable means of human–robot communication. Specifically, in a collaboration task similar to the one used for the present study, the robot arm controller could be trained to probe or sense—as the "leader" or "follower," respectively—the impedance or exploratory movements of the human agent and assist his/her movement accordingly. Further work, however, is needed to quantify the extent to which such impedance-based controller can mimic the human–human co-adaptation described in the present study.

It is worth noting that several studies have investigated physical human–robot interaction (Duchaine and Gosselin, 2009; Lecours et al., 2012; Cherubini et al., 2013) and the use of impedance in human–robot interactions (e.g., Lin et al., 2013; Nisky et al., 2013; Quek et al., 2013). The key difference between prior work and the current study is that this work is the first investigation of humans' ability to use stiffness as a means of communicating intended direction of motion. It should be emphasized that the intended movement direction was effectively communicated without generating significant motion. Thus, this result underscores humans' ability to convey and to understand intended movement direction through the modulation of stiffness *in the absence of* or *before* an actual movement. Our approach points to applications where a human or robot follower can intuitively learn to recognize when or whether the movement direction of the leader may be incorrect or hazardous. Additionally, this approach can also be utilized as a two-way method of communication for ambiguous situations during cooperative tasks. As such, our work contributes to the insights provided by research in the area of human–human and human–machine physical interaction (Reed and Peshkin, 2008; Jarrasse et al., 2012; Ganesh et al., 2014; Sawers and Ting, 2014).

With respect to the time it took followers to infer the leader's intended direction (~30 s), these latencies are too long in human– robot interaction scenarios where speed and safety are important criteria. Nevertheless, it is conceivable that the response latencies could be potentially reduced were participants to be exposed to a lower number of intended directions (e.g., four cardinal directions). Further work is needed to leverage our findings for human–robot interaction applications.

### Impedance-Based Communication of High-level Movement Goals

At least two theoretical frameworks—that differ in terms of whether a physical interaction between two agents is necessary or not—could account for our results. One of these frameworks would predict that humans modulate their arm stiffness as a function of planned movement in a given direction, regardless of whether another agent is probing their intended movement direction. If so, our findings would indicate that the follower learns how to capture the force–displacement relationship, which might imply the stiffness modulation to correctly infer the leader's intended movement direction. However, an alternative framework would predict that the leader—consciously or sub-consciously—gradually learned that modulating arm stiffness was an effective or the best way to communicate his/her intended movement direction to the follower.

Our present data do not allow distinguishing between these two alternative frameworks. Therefore, future work is needed to determine the neural mechanisms responsible for non-verbal communication of movement direction through stiffness modulation and co-adaptation of two cooperating agents. Nevertheless, the fact that our dyads improved with practice in communicating and inferring movement direction would favor the second framework as the most plausible scenario. Future work will address the underlying neural mechanisms.

In the aspect of admittance/impedance relation to describe the coupled interaction (Hogan, 1985), the way the roles of leader and follower were defined may suggest that the leader must operate as an admittance (reading an input force and responding with a motion) and the follower as an impedance (apply a force and read

#### REFERENCES


a motion). The question is raised to what extent the two actors can strictly interpret the task in this sense, in which case the leader would in fact modulate admittance not impedance. However, another scenario could be that the follower applies probing motions (not forces), senses the leader's resistive force, and observes the error caused by the leader's resistance. As the follower's task has a positional constraint (remaining within the 5-cm circle), it is more likely that the follower tries to perform a motion and senses a resistance, i.e., the follower is interpreting the leader as an impedance.

## CONCLUSION

We found that agents performing a collaborative manipulation task were able to non-verbally communicate/infer intended movement direction even when VF of arm configuration or handle was not available. With practice, the ability to correctly infer intended movement direction improved in parallel with a directionally tuned modulation of force–displacement relationship which might imply aspects of peoples' modulation of arm stiffness/impedance. We conclude that human body (mainly upper limb) stiffness, extracted through haptic feedback alone, can be successfully used to infer/ communicate intended movement direction. These results provide proof of concept for potential applications to human–robot interactions, where artificial controllers could be designed to capitalize on estimating and reacting to human limb stiffness.

## AUTHOR CONTRIBUTIONS

Data collection: KM and BW. Design, analyzing data, and preparing manuscript and figures: all the authors.

## FUNDING

This research was supported by a Collaborative Research Grant BCS-1153034 from the National Science Foundation (NSF). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of NSF.

## SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at http://journal.frontiersin.org/article/10.3389/fnbot.2017.00021/ full#supplementary-material.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2017 Mojtahedi, Whitsell, Artemiadis and Santello. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Forearm Flexor Muscles in Children with Cerebral Palsy Are Weak, Thin and Stiff

Ferdinand von Walden<sup>1</sup> , Kian Jalaleddini <sup>2</sup> , Björn Evertsson3, 4, Johanna Friberg1, 4 , Francisco J. Valero-Cuevas 2, 5 and Eva Pontén1, 6 \*

<sup>1</sup> Department of Women's and Children's Health, Karolinska Institute, Stockholm, Sweden, <sup>2</sup> Division of Biokinesiology and Physical Therapy, University of Southern California, Los Angeles, CA, USA, <sup>3</sup> Department of Neurology, Karolinska Hospital Huddinge, Stockholm, Sweden, <sup>4</sup> Karolinska Institute, Stockholm, Sweden, <sup>5</sup> Department of Biomedical Engineering, University of Southern California, Los Angeles, CA, USA, <sup>6</sup> Department of Pediatric orthopedic Surgery, Astrid Lindgren's Children Hospital, Stockholm, Sweden

Children with cerebral palsy (CP) often develop reduced passive range of motion with age. The determining factor underlying this process is believed to be progressive development of contracture in skeletal muscle that likely changes the biomechanics of the joints. Consequently, to identify the underlying mechanisms, we modeled the mechanical characteristics of the forearm flexors acting across the wrist joint. We investigated skeletal muscle strength (Grippit®) and passive stiffness and viscosity of the forearm flexors in 15 typically developing (TD) children (10 boys/5 girls, mean age 12 years, range 8–18 yrs) and nine children with CP Nine children (6 boys/3 girls, mean age 11 ± 3 years (yrs), range 7–15 yrs) using the NeuroFlexor® apparatus. The muscle stiffness we estimate and report is the instantaneous mechanical response of the tissue that is independent of reflex activity. Furthermore, we assessed cross-sectional area of the flexor carpi radialis (FCR) muscle using ultrasound. Age and body weight did not differ significantly between the two groups. Children with CP had a significantly weaker (−65%, p < 0.01) grip and had smaller cross-sectional area (−43%, p < 0.01) of the FCR muscle. Passive stiffness of the forearm muscles in children with CP was increased 2-fold (p < 0.05) whereas viscosity did not differ significantly between CP and TD children. FCR cross-sectional area correlated to age (R <sup>2</sup> <sup>=</sup> 0.58, <sup>p</sup> <sup>&</sup>lt; 0.01), body weight (<sup>R</sup> <sup>2</sup> <sup>=</sup> 0.92, <sup>p</sup> <sup>&</sup>lt; 0.0001) and grip strength (R <sup>2</sup> <sup>=</sup> 0.82, <sup>p</sup> <sup>&</sup>lt; 0.0001) in TD children but only to grip strength (<sup>R</sup> <sup>2</sup> <sup>=</sup> 0.60, p < 0.05) in children with CP. We conclude that children with CP have weaker, thinner, and stiffer forearm flexors as compared to typically developing children.

#### Edited by:

Xiaoli Li, Beijing Normal University, China

#### Reviewed by:

Ming Li, Peking University First Hospital, China Geoffrey Handsfield, University of Auckland, New Zealand

> \*Correspondence: Eva Pontén eva.ponten@ki.se

Received: 30 December 2016 Accepted: 07 April 2017 Published: 25 April 2017

#### Citation:

von Walden F, Jalaleddini K, Evertsson B, Friberg J, Valero-Cuevas FJ and Pontén E (2017) Forearm Flexor Muscles in Children with Cerebral Palsy Are Weak, Thin and Stiff. Front. Comput. Neurosci. 11:30. doi: 10.3389/fncom.2017.00030 Keywords: cerebral palsy, skeletal muscle, muscle stiffness, muscle size, upper limb

## INTRODUCTION

An insult to the immature, developing brain before the age of two results in a condition clinically referred to as cerebral palsy (CP), characterized by motor impairment (Rosenbaum et al., 2007). Despite that the brain injury is non-progressive, motor function commonly deteriorates over time and a progressive contracture formation, i.e., a decrease in passive range of motion (pROM), is common (Hagglund and Wagner, 2011). For many children, surgical treatment is needed to restore and preserve musculoskeletal function. Clinical experience suggests that contracture formation is due to shortening/stiffening of the musculotendinous complex, as the range of motion of the joint often is practically normal when tendons of contracted muscles are cut during surgery. What causes the increased stiffness and shortness of the muscle is not known, but clinically we know that it affects joint biomechanics.

Children with CP generally have less skeletal muscle mass as compared to typically developing (TD) children (Barrett and Lichtwark, 2010). Recent reports have highlighted stunted growth in the lower limb as a contributing factor to contracture development (Gough and Shortland, 2012) and slowed growth rate has been detected as early as 15-months of age (Herskind et al., 2015). Several investigators have described increased passive stiffness of the calf muscle in children with CP and different tests have shown that the calf muscle is 22–120% stiffer in CP children compared to TD (Ross et al., 2011; de Gooijervan de Groep et al., 2013; Geertsen et al., 2015). The process starts early and increased whole muscle passive stiffness in the calf has been described as early as 3 years of age (Willerslev-Olsen et al., 2013). Biochemical studies of muscle in CP have shown an increased content of intramuscular collagen (Booth et al., 2001), with an increased amount of connective tissue around fiber bundles i.e., a thickening of the perimysial extracellular matrix (de Bruin et al., 2014). The perimysium is considered to be in a physical continuum with the tendon (Passerieux et al., 2007). Thus, marked collagen deposition in the perimysium could potentially offer an increased resistance to passive stretch.

Less information is available on how skeletal muscle size correlates to joint biomechanics and muscle function for the upper limb. This is unfortunate as the clinical importance of accurate information on skeletal muscle function and biomechanical properties prior to for example tendon transfer surgery of the wrist is of great importance and a guide in clinical decision making. Therefore, we aimed to model the mechanical characteristics of the forearm flexors acting across the wrist joint in children with CP during passive stretch. Second, we aimed to correlate skeletal muscle size and biomechanics to age, body weight and strength.

#### MATERIALS AND METHODS

#### Participants

Nine children [6 boys/3 girls, mean age 11 ± 3 years (yrs), range 7–15 yrs] with CP (3 bilateral/6 unilateral) scheduled for upper limb surgery at Karolinska University Hospital, Stockholm, Sweden were consecutively included in the study. Inclusion criteria were confirmed CP diagnosis, age between 7 and 18 yrs, and cognitive ability to follow instructions. The children with CP were classified according to the Gross Motor Function Classification Scale (GMFCS) and the Manual ability classification system (MACS). GMFCS I means that the child can walk indoors and outdoors without limitations, and children in GMFCS V has no means of independent mobility. The GMFCS classification has been shown to be the best predictor of treatment results in CP (Shore et al., 2012). Eight of the children were classified as GMFCS I and one child as GMFCS III. The GMFCS III child needed a walking aid indoors and a wheel chair outdoors. MACS I means that the child handles objects easily and successfully with minimal limitation in speed and accuracy, and children in MACS V do not handle objects and has limited ability to perform even simple actions (requires total assistance). The children in the current study were classified as follows; MACS I three children, MACS II four children and MACS III two children.

As controls, a convenience sample (n = 15) of TD children was recruited (10 boys/5 girls, mean age 12 yrs, range 8–18 yrs). This study was carried out in accordance with the recommendations of the Regional Ethical Review Board in Stockholm with written informed consent obtained from all subjects and minimum one parent per subject in accordance with the Declaration of Helsinki. The protocol was approved by the Regional Ethical Review Board in Stockholm.

## Clinical Assessment and Strength Measurements

Passive range of motion (pROM) of the wrist was assessed by a physiotherapist or trained medical student using a goniometer. All children were thereafter tested for maximal grip strength using a dedicated device, Grippit <sup>R</sup> (AB Detektor, Göteborg, Sweden). The arm of the subject was placed in a neutral position (thumb upwards) with the ulnar side resting in a padded semicircular plastic tube. All subjects received verbal encouragement and were instructed to grip the handle as hard as they could and maintain the same intensity for a 10 s period. The handle size was adjustable and each child was allowed to independently choose the handle size.

## Neuroflexor Measurements and Data Processing

For wrist stiffness measurements, we used the **NeuroFlexor** <sup>R</sup> (NeuroFlexor <sup>R</sup> Scientific, Release 0.0.6, Aggero MedTech AB, Solna, Sweden). With the elbow in 90◦ , the hand was placed on the NeuroFlexor <sup>R</sup> platform, so that the axes of rotation of the wrist and the platform were aligned. Subjects were instructed to remain relaxed throughout the experiments and the device applied ramp-and-hold perturbations with a velocity of 5◦ /s, by extending the wrist from 20◦ palmar flexion to 30◦ extension. It recorded the angle of the wrist and the forces exerted by the limb and device. For each subject, we recorded 5 trials. We also recorded a trial following removal of the subject's hand from the device (no-hand trial) to account for forces generated by the mechanical properties of the apparatus only.

We subtracted the no-hand force from the with-hand forces and used a linear least-squares technique to identify the biomechanical model (Schouten et al., 2008; Meskers et al., 2015):

$$\begin{aligned} F\_{WH} \text{(t) } -\ \, \_FK\_{WH} \text{(t) } &= \, m l^2 \theta \, '' \text{(t) } +B\theta \, ' \text{(t) } +K\theta \, \text{(t) } +K\_1 \theta \, ^2 \text{(t) }\\ &+ K\_2 \theta \, ^3 \text{(t) } +mg \cos(\theta \, \text{(t))} \end{aligned}$$

where FWH(t) and FNH (t) are the recorded forces in the withhand and no-hand trials, m is the mass of the hand (point mass), l is the length of the hand from the wrist joint, θ(t) is the recorded wrist angle and θ ′ (t), θ ′′(t) are its first and second derivatives (angular velocity and acceleration) computed numerically by differentiating the recorded joint angle, g is the gravitational acceleration and finally K and B are the joint stiffness and viscous parameters, respectively. The second and third-order power of joint angles (θ 2 (t), θ 3 (t)) were also included in the model to account for nonlinear changes of joint biomechanics as a function of joint angle (Sobhani Tehrani et al., 2013). The main assumption of the model is that the neural component of the force is negligible, which is a fair assumption as the velocity of perturbation was low enough to avoid evoking reflex responses (Jalaleddini et al., 2016). Thus, the muscle stiffness we estimate and report is the instantaneous mechanical response of the tissue that is independent of reflex activity. From this model, we quantified and reported the mean and standard deviation of the stiffness and viscous parameters across the trials. **Figure 1** shows typical measured and predicted forces using the biomechanical model. The biomechanical model accounted for 95.5 ± 3.6% of the variance in measured force, on average.

#### Skeletal Muscle Ultra Sound Assessment and Image Processing

The cross sectional area of the flexor carpi radialis muscle (FCR) was measured with B-mode ultrasonographic equipment with a linear probe (BK medical Flex focus 1202, 8670, scanning frequency 12 MHz Herlev, Denmark). The subjects were instructed to place their elbow on a height adjustable table with the dorsum of the hand toward the examiner and the fingers extended. If this position was difficult for the child to maintain, the caretaker was instructed to assist and hold the arm in the above-described position. Then, the distance from the olecranon to the styloid processes of the ulna was determined using a measuring tape. Skin markings were made at the transition between the proximal and middle third of the forearm, using a black permanent marker, to ensure correct placement of the ultrasound probe. During the ultrasound examination the subject was sitting in a standardized position with the dorsum of the forearm resting flat on a height adjustable table, the elbow at approximately 60◦ of flexion, arm supinated and fingers extended. Following generous application of water-soluble gel (Gurò Medigel, Gurò s.a.s.—Catenanuova (En)—Italy) the head of the ultrasound probe was placed proximal and parallel to the line marking, and at a right angle to the radius and ulna. The interfaces between subcutaneous adipose tissue and muscle tissue, between muscle tissue and fascia and between muscle tissue and bone were identified on the ultrasonic image. The border of the muscle was outlined by free hand, and the supplied software in the ultrasonograph was used to calculate the crosssectional area.

#### Statistical Analysis

Values are reported as means ± SD. Differences in age, body weight, isometric grip strength, skeletal muscle cross sectional area, muscle passive stiffness and viscosity were investigated by the Mann-Whitney-Wilcoxon test. Linear regression was used to investigate correlations between FCR area and age, body weight, grip strength and passive stiffness and viscosity, respectively. Significance level was set at p < 0.05 for all statistical comparisons.

## RESULTS

No difference was seen between the groups with respect to age (CP 11.3 yrs ± 3.1 vs. TD 12.3 yrs ± 3.7, ns) and body weight (CP 46.8 ± 26.1 kg vs. TD 44.7 ± 14.9 kg, ns). FCR cross sectional area was 43% smaller in children with CP as compared to TD children (CP 0.84 cm<sup>2</sup> <sup>±</sup> 0.38 vs. TD 1.47 cm<sup>2</sup> <sup>±</sup> 0.75, <sup>p</sup> <sup>&</sup>lt; 0.01). As expected, grip strength (10 s isometric contraction) was also significantly lower, −65% in CP as compared to TD children (CP 58.3N ± 32.1 vs. TD 167.5N ± 93.5, p < 0.01, **Figure 2**).

Skeletal muscle passive stiffness was increased 2-fold in CP as compared to TD (CP 4.65 ± 3.42 vs. TD 1.96 ± 0.62, p < 0.05) whereas skeletal muscle viscosity didn't differ significantly between the groups (CP 1.71 ± 1.22 vs. 1.27 ± 0.86, **Figure 3**).

Skeletal muscle size correlated to age (R <sup>2</sup> <sup>=</sup> 0.58, <sup>p</sup> <sup>&</sup>lt; 0.01), body weight (R <sup>2</sup> <sup>=</sup> 0.92, <sup>p</sup> <sup>&</sup>lt; 0.0001) and strength (<sup>R</sup> <sup>2</sup> <sup>=</sup> 0.58, <sup>p</sup> <sup>&</sup>lt; 0.01) in TD children. Interestingly, the same relationship between skeletal muscle size did not correlate to age and body weight in children with CP, however still showing a significant correlation to strength (R <sup>2</sup> <sup>=</sup> 0.60, <sup>p</sup> <sup>&</sup>lt; 0.05). Skeletal muscle size did not correlate to passive stiffness or viscosity in CP or in TD children (**Figure 4**).

## DISCUSSION

This study shows that forearm flexor muscles in children with CP are stiffer as compared to TD children. Similar findings have been described for the lower limb, specifically the ankle flexors. Previous reports have estimated the passive tension to be 22– 120% higher in CP for the ankle flexor muscles (Ross et al., 2011; de Gooijer-van de Groep et al., 2013; Geertsen et al., 2015). Thus, our finding of a 2-fold increase in skeletal muscle stiffness of the forearm flexors is comparable to data on weight bearing muscles of the lower limb. Information on the current status of the muscle is of great clinical importance when planning treatments, such as stretching, use of splints and surgical interventions e.g., tendon transfer surgery. Tendon lengthening, rerouting and transfer in the upper limb aim to balance the muscles of the wrist, so that the wrist can be maintained in a functional neutral position. All these surgical techniques have a component of dosing—how tight should the tendon transfer be, how long the tendon lengthening? This dosing is highly dependent on how stiff the transferred or lengthened muscle is. Long-term results are difficult to predict, as there often is a progression of contracture formation and stiffness. Abnormalities such as contracture and stiffness in even a few muscles will add yet another obstacle to smooth and efficient movements (Keenan et al., 2009; Kutch and Valero-Cuevas, 2011; Valero-Cuevas et al., 2015) which can progress very quickly in these children.

Spastic CP is characterized by a hyperexcitibility of the stretch reflex (Lance, 1980). Due to the fact that many children with spastic CP eventually develop skeletal muscle contractures, spasticity has been believed to be causative and clinical practice has up until recently been focused on spasticity reducing therapies for the preservation of motor function. However, recent reports have clearly demonstrated that despite good control of spasticity and reduction of muscle

CP. (C) Grip strength (N) in TD and CP. \*\*Denotes significantly different from TD, p < 0.01. (D) Cross-sectional area of the flexor carpi radialis muscle (FCR) in cm<sup>2</sup> in TD and CP. \*Denotes significantly different from TD, p < 0.05. For all bar graphs, TD (white) & CP (red). (E,F) Representative ultrasound images of forearm muscles in TD (E) and CP (F) children. Encircled areas represent FCR.

tone, by intramuscular botulinum toxin injections (chemical denervation) or selective dorsal rhizotomy that disrupts the reflex loop, skeletal muscle contracture formation progresses (Alhusaini et al., 2011; Tedroff et al., 2014). The poor effect of spasticity reduction on contracture development has raised the question of whether other factors such as growth disturbances and/or alterations in muscle composition are of greater mechanistic importance. Furthermore, recent findings suggest an overestimation of the contribution of spasticity to increased passive tension in muscle of young children with CP and at the same time an underestimation of the presence of contractures (Willerslev-Olsen et al., 2013; Herskind et al., 2015). In a large population study of children with CP, spasticity has been shown to peak around 4 years of age and thereafter decreases and level out at 12 years of age, whereas reductions in passive range of motion i.e., skeletal muscle contractures is progressive (Hagglund and Wagner, 2011). Therefore, we have chosen to focus on the instantaneous mechanical response of the muscle that is independent of reflex activity.

Muscle fiber size increases from 10–12µm<sup>2</sup> at birth to 40– 60µm<sup>2</sup> after puberty (Oertel, 1988). Lexell et al have shown that skeletal muscle fiber size continuously increases with age up until and shortly after puberty. The same study also found a close relationship between fiber size and muscle cross sectional area (Lexell et al., 1992). Children with CP have reduced growth in general (Day et al., 2007). Skeletal muscle growth rate is reduced, evident as early as at 15-months of age (Herskind et al., 2015) and without showing any tendency to catch up later in life (Malaiya et al., 2007; Barber et al., 2011, 2016; Noble et al., 2014b). Just as for passive stiffness of skeletal muscles, deficit in skeletal muscle size in children with CP has so far only been described for the lower limb. The cross sectional area of the FCR muscle was significantly smaller in children with CP as compared to TD children. We have shown in previous studies that wrist flexors in children with CP have a greater fiber size variability and altered myosin composition compared to TD children (Ponten and Stal, 2007). In addition, the sarcomere, the smallest contractile unit of the muscle, has altered properties in CP (Ponten et al., 2007). By using laser diffraction methodology, we have measured sarcomere lengths of wrist flexors intraoperatively during tendon transfer surgeries. With the wrist held in neutral position, the sarcomeres were longer in CP compared to control, and sarcomeres were also longer—i.e., more stretched out- the worse the wrist contracture was (Ponten et al., 2007). This means that the wrist extension movement by the NeuroFlexor <sup>R</sup> likely results in extreme stress of the sarcomeres, with less overlap of the actin and myosin filaments and more stress on the perimyseal collagen surrounding the fiber bundles (de Bruin et al., 2014). In our study population the cross sectional area of the wrist flexor FCR, as determined by ultrasound, was significantly smaller in CP as compared to TD children and did not correlate to age or body weight. This implies that the differing brain damage the children with CP have, has a greater impact on muscle growth than age and body size. This further supports the impression of great heterogeneity within CP previously reported (Handsfield et al., 2016) and underscores the need for structured, individualized longitudinal follow-up.

Increased skeletal muscle echo intensity as measured by ultrasound has previously been used to infer altered skeletal muscle composition (Heckmatt et al., 1982; Nielsen et al., 2006). Pitcher et al. recently investigated the gastrocnemius muscle of 40 children with cerebral palsy and compared their findings to typically developed age matched controls (n = 12) (Pitcher et al., 2015). They show increased skeletal muscle echo intensity in CP and suggest that the underlying cause is due to increased content of non-contractile tissue e.g., collagen. Biochemical analysis of hydroxyproline content, a measure of intramuscular collagen content, correlates to spasticity severity (muscle tone as assessed

area (cm<sup>2</sup> ) and grip strength (N) in TD (graph E) and CP (graph F). R <sup>2</sup> and significance level indicated in each graph when relevant.

by Modified Ashworth Scale, MAS) in children with CP (Booth et al., 2001). Staining muscle collagen using Sirius red has in CP shown increased amount of perimyseal collagen surrounding muscle fiber bundles (de Bruin et al., 2014). Similarity, MRI assessment of the lower limb indicates that young adults with CP have higher intramuscular fat content as compared to healthy controls (Noble et al., 2014a). Thus, it's without a doubt that skeletal muscle in individuals with CP with time contains an increased amount of non-contractile material. We interpret this as an indicator of the progressive remodeling in CP muscle. Despite that CP is defined as a non-progressive disease, skeletal muscle pathophysiology seems to continuously worsen over time.

We conclude that children with CP have weaker, thinner and stiffer forearm flexors as compared to typically developed children.

## AUTHOR CONTRIBUTIONS

FvW: Performed experiments, analyzed data, drafted manuscript (MS), designed figures, finalized MS, approved final version of MS. KJ: Analyzed data, drafted MS, designed figures, finalized MS, approved final version of MS. BE: Performed experiments, designed figures, finalized MS, approved final version of MS. JF: Performed experiments, finalized MS, approved final version of MS. FJVC: Conception of study, analyzed data, approved final version of MS. EP: Conception of study, drafted MS, finalized MS, approved final version of MS.

#### FUNDING

The research reported herein was supported by grants from Sunnerdahls handikapp Stiftelse, Norrbacka-Eugeniastiftelsen and Stiftelsen Samariten to FvW; Linnea and Josef Carlssons Stiftelse, Stiftelsen Promobilia and by grants provided by the Stockholm County Council (ALF project) to EP. This

#### REFERENCES


material is in part based upon work supported by NIH Grant R01-052345, NIH Grant R01-050520 to FJVC. This project is also supported by the Postdoctoral Research Scholarship from Fonds de Recherche du Québec- Nature et Technologies to KJ.

#### ACKNOWLEDGMENTS

We are indebted to Johan Gäverth, Påvel Lindberg, Jeanette Plantin and Ruoli Wang for their valuable comments and advice.


years after selective dorsal rhizotomy in cerebral palsy. Dev Med Child Neurol. 57, 484–490. doi: 10.1111/dmcn.12665


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 von Walden, Jalaleddini, Evertsson, Friberg, Valero-Cuevas and Pontén. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Electromyography Data Processing Impacts Muscle Synergies during Gait for Unimpaired Children and Children with Cerebral Palsy

#### Benjamin R. Shuman1, 2, Michael H. Schwartz 3, 4 and Katherine M. Steele1, 2 \*

*<sup>1</sup> Department of Mechanical Engineering, University of Washington, Seattle, WA, United States, <sup>2</sup> WRF Institute for Neuroengineering, University of Washington, Seattle, WA, United States, <sup>3</sup> James R. Gage Center for Gait and Motion Analysis, Gillette Children's Specialty Healthcare, St. Paul, MN, United States, <sup>4</sup> Department of Biomedical Engineering, University of Minnesota, Minneapolis, MN, United States*

#### Edited by:

*Francisco J. Valero-Cuevas, University of Southern California, United States*

#### Reviewed by:

*Rong Chen, University of Maryland, Baltimore, United States Radwa Khalil, Rutgers University–Newark, United States*

> \*Correspondence: *Katherine M. Steele kmsteele@uw.edu*

Received: *30 December 2016* Accepted: *19 May 2017* Published: *06 June 2017*

#### Citation:

*Shuman BR, Schwartz MH and Steele KM (2017) Electromyography Data Processing Impacts Muscle Synergies during Gait for Unimpaired Children and Children with Cerebral Palsy. Front. Comput. Neurosci. 11:50. doi: 10.3389/fncom.2017.00050* Muscle synergies calculated from electromyography (EMG) data identify weighted groups of muscles activated together during functional tasks. Research has shown that fewer synergies are required to describe EMG data of individuals with neurologic impairments. When considering potential clinical applications of synergies, understanding how EMG data processing impacts results and clinical interpretation is important. The aim of this study was to evaluate how EMG signal processing impacts synergy outputs during gait. We evaluated the impacts of two common processing steps for synergy analyses: low pass (LP) filtering and unit variance scaling. We evaluated EMG data collected during barefoot walking from five muscles of 113 children with cerebral palsy (CP) and 73 typically-developing (TD) children. We applied LP filters to the EMG data with cutoff frequencies ranging from 4 to 40 Hz (reflecting the range reported in prior synergy research). We also evaluated the impact of normalizing EMG amplitude by unit variance. We found that the total variance accounted for (tVAF) by a given number of synergies was sensitive to LP filter choice and decreased in both TD and CP groups with increasing LP cutoff frequency (e.g., 9.3 percentage points change for one synergy between 4 and 40 Hz). This change in tVAF can alter the number of synergies selected for further analyses. Normalizing tVAF to a z-score (e.g., dynamic motor control index during walking, walk-DMC) reduced sensitivity to LP cutoff. Unit variance scaling caused comparatively small changes in tVAF. Synergy weights and activations were impacted less than tVAF by LP filter choice and unit variance normalization. These results demonstrate that EMG signal processing methods impact outputs of synergy analysis and z-score based measures can assist in reporting and comparing results across studies and clinical centers.

Keywords: electromyography, muscle synergies, low pass filtering, amplitude scaling, walk-DMC, non-negative matrix factorization, cerebral palsy

## INTRODUCTION

Muscle synergies have been used to describe the low-dimensional sets of weighted muscle groups that are recruited during functional tasks (Tresch and Jarc, 2009; Bizzi and Cheung, 2013). Prior research has theorized that the nervous system uses these synergies as a simplified method of control, rather than controlling each muscle individually. Recent research has applied muscle synergies as a framework to evaluate altered neuromuscular control in individuals with neurologic disorders. Research on individuals with stroke and CP have shown that fewer synergies are required to describe EMG data during functional tasks compared to unimpaired individuals, and this reduction in activation complexity may contribute to movement impairments (Cheung et al., 2009b; Clark et al., 2010; Monaco et al., 2010; Allen et al., 2013; Roh et al., 2013; Routson et al., 2014; Steele et al., 2015). However, despite the general agreement that synergy complexity is reduced in stroke and CP, there is no consistent methodology for calculating muscle synergies. Prior to calculating muscle synergies, raw EMG data are processed to generate linear envelopes describing the activation of each muscle during a task such as walking. In general, this process consists of an initial filtering (e.g., high pass or band pass filtering), full wave rectification, low pass (LP) filtering, and amplitude scaling. As researchers investigate potential clinical applications of synergy analyses, such as in clinical gait analysis, understanding the impact of EMG preprocessing is important to compare across studies or clinical centers.

Prior synergy research has used a wide variety of EMG preprocessing methods. In particular, a wide range of LP filters have been used to smooth EMG data, with LP cutoff frequencies ranging from 1 (Muceli et al., 2010) to 40 Hz (Torres-Oviedo and Ting, 2010), and including many intermediate values including 4 (Clark et al., 2010), 10 (Steele et al., 2015), 20 (Cheung et al., 2009a), 30 (Torres-Oviedo et al., 2006), or 35 Hz (Sawers et al., 2015). Despite this, there has been little research examining how muscle synergy calculations are impacted by these LP filter choices. Kleissen (1990) showed large differences in smoothness and cycle-to-cycle variability in EMG envelopes from the gluteus medius during gait when LP filtered at 3.4 or 25 Hz. For synergies, Van der Krogt et al. (2016) showed that the total variance accounted for (tVAF) by a given number of synergies was reduced with increasing LP cutoff frequency in children with CP for EMG data LP filtered between 2 and 25 Hz. Since tVAF is commonly used to pick or choose the number of synergies for further analysis (e.g., the number of synergies required for tVAF > 90 or 95%), impacts of LP filtering on tVAF can further impact conclusions about muscles that are activated together or differences in synergies between control and clinical populations. Hug et al. (2012) noted that the number of synergies required to explain 90% of the variance in EMG data changed between 4, 10 and 15 Hz LP filters. However, it has not been shown how LP filters can affect calculated synergy weights, which describe muscles commonly activated together, or synergy activation curves, which describe how each synergy is activated over time.

After filtering, the processed EMG data amplitudes are often scaled through one of several methods. These include peak measured amplitude (Clark et al., 2010; Steele et al., 2015), maximum voluntary contractions (Berger et al., 2013; Zelik et al., 2014), or median trial maximums (Cheung et al., 2009b). Additionally, for synergy analyses, prior research has scaled the amplitude so that each muscle has unit variance (Torres-Oviedo et al., 2006; Roh et al., 2013; Sawers et al., 2015). Unit variance scaling has been applied to avoid larger representations of highvariance muscles in the output synergy weights (Cheung et al., 2009a). As with filter cutoff, the effects of amplitude scaling on synergy outputs remains unclear.

To reduce potential impacts of EMG preprocessing on synergy results and facilitate comparison across studies or clinical centers, some prior research has suggested normalizing data to a z-score. For example, the dynamic motor control index during walking (walk-DMC) provides a summary measure of synergy complexity by normalizing tVAF by one synergy to the average and standard deviation of a group of unimpaired individuals (Steele et al., 2015; Schwartz et al., 2016). By normalizing to a group of controls from a given clinic or research lab, walk-DMC may help to reduce the impacts of different equipment, muscles, or EMG preprocessing methods across institutions. walk-DMC differs between typicallydeveloping (TD) children and children with CP and is associated with treatment outcomes for children with CP (Schwartz et al., 2016). The impact of EMG processing on walk-DMC has not been investigated.

The goal of this research was to examine how EMG processing, specifically the choice of LP filter cutoff frequency and amplitude scaling, affects synergy analyses for TD children and children with CP. We evaluated how processing choices impact synergy complexity, in terms of tVAF and walk-DMC. We also evaluated how synergy weights and synergy activation curves change with processing choices. We hypothesized that walk-DMC would be more consistent across EMG processing conditions than tVAF, and that synergy weights and activations would change across processing parameters. Further, we hypothesized that both the TD and CP children's synergies would be similarly impacted by processing choices. Understanding the impact of EMG data processing on synergy outputs will help inform comparisons between studies and guide future clinical applications of synergy analyses.

## METHODS

Human subjects' approval was obtained from both the University of Washington and the University of Minnesota for this study.

We retrospectively analyzed individuals who previously received gait analysis at Gillette Children's Specialty Healthcare. For this study, we sought to identify 40 individuals with diplegic CP, belonging to each of the Gross Motor Function Classification System (GMFCS) Levels I, II, and III (120 total participants), who had EMG data collected from five muscles during routine clinical gait analysis. For GMFCS Level III, only 33 individuals met these inclusion criteria. Data for TD children were obtained from the control database at Gillette Children's Specialty Healthcare. **Table 1** summarizes the demographic data for all participants in this study.

#### TABLE 1 | Study population.


*N, number of participants; F, Female; M, Male; GMFCS, Gross Motor Function Classification System.*

#### Electromyography Data

Surface EMG data (Motion Laboratory Systems, Baton Rouge, LA, USA) were collected at 1,080 Hz for five muscles (rectus femoris, medial hamstrings, lateral hamstrings, medial gastrocnemius, and tibialis anterior) during barefoot walking at a self-selected speed. For each individual, one limb was randomly selected for analysis. We took the middle 80% of the entire gait trial to avoid transient accelerations and decelerations near the beginning and end of the trial and maximize data for analysis (Oliveira et al., 2014). Raw EMG data were band pass filtered between 35 and 500 Hz upon collection.

EMG data for each child were digitally processed with a high pass (HP) filter and a set of varying LP filters (**Figure 1**). The filter parameters were based upon prior studies of synergies during gait (Torres-Oviedo et al., 2006; Cheung et al., 2009a; Clark et al., 2010; Torres-Oviedo and Ting, 2010; Steele et al., 2015). The pipeline for processing EMG data for synergy analysis involves the following sequence: (1) HP filtering (40 Hz) to eliminate DC drift and movement artifacts, (2) full wave rectification, and (3) LP filtering to create a linear envelope of muscle activity. For both filtering steps, we used 4th order Butterworth filters, which have commonly been used in synergy analyses (Neptune et al., 2009; Clark et al., 2010; Allen et al., 2013; Routson et al., 2014). The specific LP cutoff frequencies evaluated were: 4, 6, 8, 10, 20, 30, and 40 Hz. Since maximum voluntary contractions are not collected as part of clinical care at Gillette, EMG data were scaled to the peak amplitude for each muscle. Since some prior synergy analyses scale EMG data to unit variance, we also investigated the impact of unit variance scaling with varying filter parameters. Each EMG channel was scaled to unit variance across the walking trial. After filtering and amplitude scaling, the EMG envelopes were down-sampled to 100 Hz to reduce synergy computation time.

#### Synergy Analysis

Synergies were calculated from the EMG data processed with each filtering condition using non-negative matrix factorization (NMF) (**Figure 2**). This method calculates a set of synergy weights (Wmxn) and synergy activations (Cnxt), such that EMG = W × C + error where n is the number of synergies (1–4 in this study), m is the number of muscles (5 in this study), and t is equal to the number of EMG data points. The error term is defined as the difference between the filtered EMG data and the EMG data reconstructed from the product of the synergy weights and activations. We calculated synergies with NMF in

Matlab (Statistics and Machine Learning Toolbox, MathWorks, Inc., Natick, Massachusetts, United States) using the following parameters: 50 replicates, 1,000 maximum iterations, a 1 <sup>×</sup> <sup>10</sup>−<sup>4</sup> minimum threshold for convergence, and a 1 <sup>×</sup> <sup>10</sup>−<sup>6</sup> threshold for completion. Note that specific synergies were calculated separately for each number of synergies specified. In other words, a synergy from a 2-synergy solution may be different than all of the synergies from a 3-synergy solution.

We used three measures to evaluate synergy complexity: (1) the total variance accounted for (tVAF), (2) the number of synergies required for tVAF > 90%, and (3) a z-score of tVAF (walk-DMC). The tVAF by n synergies was defined as one minus the ratio of the sum of squared errors to the sum of filtered EMG data over all muscles (Equation 1, Ting and Macpherson, 2005). Traditionally, tVAF is used to define the number of synergies to evaluate in synergy analyses. For each LP filter, we used a t-test to compare tVAF and a Mann-Whitney U-test to compare the number of synergies between CP and TD groups.

$$tVAF\_n = \left(1 - \frac{\left[\sum\_{j}^{t} \sum\_{i}^{m} \left(error\right)^{2}\right]}{\left[\sum\_{j}^{t} \sum\_{i}^{m} \left(EMG\right)^{2}\right]}\right) \times 100\% \tag{1}$$

walk-DMC is a z-score based upon tVAF by one synergy (tVAF1, Equation 2), and uses the average and standard deviation of tVAF<sup>1</sup> (tVAFAVG and tVAFSD) from unimpaired controls. Thus, the average walk-DMC score for the TD group is 100, and each 10-point deviation represents one standard deviation from the TD controls. Note that a higher tVAF<sup>1</sup> results in a lower walk-DMC score. For example, a walk-DMC of 80 indicates that an individual's tVAF<sup>1</sup> during walking is two standard deviations above the TD group, suggesting simplified control.

$$\% walk - DMC = 100 + 10 \left[ \frac{tVAF\_{AVG} - tVAF\_1}{tVAF\_{SD}} \right] \tag{2}$$

were calculated for *n* = 1–4 synergies. (C) Total variance accounted for by *n* synergies provides a measure of synergy complexity and is often used to select a number of synergies for further analysis (e.g., tVAF > 90%). RF, rectus femoris; MH, medial hamstrings; LH, lateral hamstrings; MG, medial gastrocnemius; AT, anterior tibialis.

To evaluate the effect of filter parameters on synergy weights and activations, we calculated the correlation coefficients comparing synergy weights and activations across all filter conditions. We computed the average correlation coefficients between the W matrices output by NMF for each of the LP filtering conditions. Similarly, we computed the average correlation coefficients for synergy activations between the C matrices output by NMF from each of the LP filtering conditions.

Since some prior studies (Torres-Oviedo et al., 2006; Cheung et al., 2009a) scale EMG data for each muscle to unit variance before running NMF, we also evaluated the impact of unit variance scaling on the resulting synergies. We compared the outputs of synergy analyses performed with EMG scaled by unit variance and by peak activation. We calculated the change in average tVAF<sup>1</sup> and walk-DMC with each LP filter condition to examine the impact of unit variance scaling on synergy complexity. Similarly, we calculated the correlation coefficients in synergy weights (W) and activations (C) with each LP filter condition between the unit variance and peak activation scaling methods. Note that EMG may be scaled directly to unit variance (Cheung et al., 2009a; Roh et al., 2013) or scaled to peak amplitude and then to unit variance (Torres-Oviedo et al., 2006; Hayes et al., 2014; Sawers et al., 2015) with equivalent synergy outputs. In this paper, we first scaled to peak amplitude and then to unit variance.

## RESULTS

#### Synergy Complexity

LP filter cutoff frequency impacted synergy complexity, as measured by tVAF. Varying the LP cutoff frequency from 4 to 40 Hz decreased tVAF<sup>1</sup> by 9.6 percentage points (i.e., from 72.0 to 62.4%) for the TD group, and 9.4, 8.9, and 9.1 percentage points for the GMFCS Level I, II, and III groups, respectively (**Figure 3A**). For individual participants, changes in tVAF<sup>1</sup> ranged from 2.2 to 15.1 percentage points across LP filtering conditions. For more than one synergy, tVAF2−<sup>4</sup> also decreased with increasing LP cutoff frequency with an average absolute reduction in tVAF of 6.5 percentage points for tVAF2, 3.8 percentage points for tVAF3, and 1.6 percentage points for tVAF4, across all participants. Despite changes in tVAF<sup>n</sup> with LP cutoff frequency, tVAF was still significantly greater in CP compared to TD across all LP cutoff frequencies and numbers of synergies (t-test, p < 0.01 for all comparisons).

Changes in tVAF influenced the choice of number of synergies (**Figure 3B**). When we applied a threshold of tVAF > 90% to identify the number of synergies, only 27% of individuals with CP and 52% of TD had the same the number of synergies across all LP cutoff frequencies (**Figure 4**). The average number of synergies during walking with the 90% tVAF threshold was 2.12 (0.58) and 2.89 (0.36) for CP and TD groups when we applied a 4 Hz LP cutoff frequency, vs. 2.88 (0.43) and 3.37 (0.49) with a 40 Hz LP cutoff frequency. However, the number of synergies was significantly less in CP compared to TD across all LP cutoff frequencies (Mann-Whitney U-test, p < 0.01 for all cutoff frequencies). A total of 69% of children with CP and 44% of the TD group increased the number of synergies by one with increasing LP cutoff frequency, while 4% increased by two synergies in both groups. Increasing LP cutoff frequency did not always lead to a greater number of synergies; four TD children decreased the number of synergies with increasing LP cutoff frequency.

Walk-DMC reduced the impact of LP cutoff frequency on synergy complexity. Between 4 and 40 Hz, GMFCS Levels I, II, and III average walk-DMC scores increased by 1.7, 1.6, and 3.3 points, respectively (**Figure 3C**). Since walk-DMC normalizes tVAF<sup>1</sup> based upon the mean and standard deviation of the TD group, the TD group's average walk-DMC does not change (average of 100 with a 10 point standard deviation). For individual participants, the change in walk-DMC with LP cutoff

FIGURE 3 | Average ± one standard deviation of (A) total variance accounted for by one synergy (tVAF1), (B) number of synergies for tVAF > 90%, and (C) walk-DMC across LP cutoff frequencies for TD and CP groups. As LP filter cutoff frequency increased, tVAF decreased and number of synergies increased for all groups. Average walk-DMC scores had minimal changes across LP cutoff frequencies.

frequency ranged from <0.01 to 15.4 points with an average change of 4.0 points. Some individual's walk-DMC increased with increasing LP cutoff frequency, while others decreased.

## Synergy Weights

Similar to tVAF, changes in LP cutoff frequency also impacted synergy weights. Synergy weights calculated with a 4 and 40 Hz LP filter had an average correlation coefficients of 0.68, 0.87, 0.93, and 0.92 for 1–4 synergies, respectively (**Figure 5**). The average correlation coefficients by group were 0.68, 0.89, 0.94, and 0.93 for the TD children for 1–4 synergies, respectively; 0.69, 0.86, 0.94, and 0.92 for GMFCS Level I; 0.63, 0.85, 0.92, and 0.91 for GMFCS Level II; and 0.68, 0.86, 0.90, and 0.90 for GMFCS Level III. For an individual participant, the minimum correlation coefficient of synergy weights across LP cutoff frequencies was <0.01, 0.09, 0.47, and 0.57 for 1–4 synergies. Between 4 and 40 Hz, the correlation coefficient was >0.8 for 56, 20, 11, and 17% of all individuals for 1–4 synergies.

## Synergy Activations

Synergy activations calculated with 4 or 40 Hz LP filters had an average correlation coefficient of 0.79, 0.78, 0.78, and 0.74 across all participants for 1–4 synergies, respectively (**Figure 5**). The average correlation coefficients were 0.79, 0.81, 0.81, and 0.78 for the TD children for 1–4 synergies, respectively; 0.79, 0.78, 0.79, and 0.74 for GMFCS Level I; 0.79, 0.76, 0.76, and 0.72 for GMFCS Level II; and 0.79, 0.75, 0.74, and 0.70 for GMFCS Level III. For an individual participant, the minimum correlation coefficient of synergy activations across LP cutoff frequencies was 0.51, 0.23, 0.40, and 0.45 for 1–4 synergies.

## Unit Variance

Scaling EMG data to unit variance impacted synergy complexity, structure, and activations. Scaling to unit variance had a variable impact on tVAF1, increasing tVAF<sup>1</sup> for some children and decreasing tVAF<sup>1</sup> for others when compared to peak amplitude scaling (average difference across all participants: 1.7, SD 1.6 percentage points). However, group average tVAF<sup>1</sup> changed only slightly with unit variance scaling, with a maximum change of 1.3 percentage points for TD with a 4 Hz LP filter (**Figure 6A**). Changes in walk-DMC due to unit variance scaling were largest with a 4 Hz LP filter with increases of 4.0, 5.4, and 6.2 points for GMFCS Levels I, II, and III and smallest with a 40 Hz LP with group changes of 1.0, 1.1, and -0.7 points, respectively (**Figure 6B**). The synergy weights correlation coefficients calculated with and without unit variance scaling were lowest for one synergy and decreased with greater LP cutoff frequency (**Figure 6C**). Synergy activations were similar between scaling methods and correlation coefficients slightly decreased with increasing LP filter cutoff (**Figure 6D**).

For EMG data scaled to unit variance, LP cutoff frequency caused slightly larger changes in tVAF1, with an average change of 10.5 percentage points between 4 Hz and 40 Hz LP filters. When we applied a threshold of tVAF > 90% to choose the number of synergies, 12% of children with CP and 11% of TD children had the same the number of synergies across all LP cutoff frequencies. Changes in walk-DMC were similar with changes of 1.2, 2.8, and 3.7 points for GMFCS Levels I, II, and III, respectively. The correlation coefficients of synergy weights were higher with unit variance scaling than EMG data normalized by peak amplitude, with average correlation coefficients of 0.90, 0.94, 0.95, and 0.96

for 1–4 synergies comparing 4 Hz and 40 Hz LP filters. The correlation coefficients of synergy activations were also slightly higher than EMG data normalized by peak amplitude, with average correlation coefficients of 0.81, 0.81, 0.79, and 0.77 for 1–4 synergies comparing 4 Hz and 40 Hz LP filters.

#### DISCUSSION

A z-score normalized measure of synergy complexity, walk-DMC, was more stable across LP filter parameters than tVAF or number of synergies. For both TD and CP children, tVAF decreased with increasing LP cutoff. Amplitude scaling of EMG data had smaller effects than LP filter choice on synergy complexity. These results highlight one disadvantage of using tVAF thresholds (i.e., tVAF > 90%) to identify the number of synergies for further analyses. Since tVAF is sensitive to filtering parameters, different studies may report different synergy numbers and co-activation patterns, depending on their choice of LP cutoff frequency. Despite the sensitivity of tVAF to LP cutoff, the TD and CP groups were significantly different across all LP filters for all measures of synergy complexity. These results suggest that z-score measures may be useful for comparing synergy results across studies or clinical centers. However, z-score normalization requires EMG data from TD or control participants, which may not be available at all institutions. Similarly, caution should be exercised when picking a single tVAF threshold for selecting number of synergies or comparing number of synergies across studies.

The choice of LP filter also affected individual muscle contributions (muscle weights) within each synergy. Synergy weights for solutions with fewer synergies (e.g., n = 2 synergies) were more sensitive to LP cutoff. Increasing the number of synergies increased similarity of synergy weights since fewer muscles were activated together in each synergy. In this study we used a clinical dataset with EMG data from five muscles. We anticipate that the impacts of LP filter choice on synergy weights

may be greater for datasets that have more muscles, since more muscles would be activated together in each synergy (Steele et al., 2013). Our results support work by Chvatal and Ting (2012) who demonstrated that further smoothing EMG data LP filtered at 40 Hz by subsequently averaging across bins ranging from 10 to 200 ms resulted in similar synergy weights (similarity >0.85 for the selected number of synergies with a threshold of tVAF > 85%). The correlation coefficients of the synergy activation curves also decreased with increasing LP filter cutoff. The decrease in correlation with increasing LP cutoff for the activation curves is reflective of the input EMG data, which retains additional high frequency components when processed with a higher LP filter (**Figure 2**).

The choice of amplitude scaling between unit variance and peak amplitude also impacted the individual muscle weights within each synergy. Synergy weights and activations were more similar across LP filter conditions for EMG data scaled to unit variance than peak amplitude, since scaling by unit variance reduces differences between muscles that may impact synergy weights and activations. All amplitude scaling methods involve applying a unique scaling factor to each EMG channel, which impacts the scaling of calculated synergy weights. Note that scaling to unit variance negates the effects of any previous scaling (e.g., peak amplitude or maximum voluntary contraction). As with LP filter choice, analyses that calculated fewer synergies (e.g., n = 2 synergies) were more sensitive to amplitude scaling. The stronger influence of EMG processing methods on synergy solutions with fewer synergies is especially important for evaluations of clinical populations, which typically have reduced synergy complexity compared to control populations.

Just as we found a range of LP filters used in prior research, we also found a range of HP filters used before rectification, including 40 Hz (Bowden et al., 2010; Clark et al., 2010; Routson et al., 2014), 35 Hz (Torres-Oviedo and Ting, 2010; Sawers et al., 2015), and 20 Hz (Hug et al., 2012; Van der Krogt et al., 2016). We could not explore the effects of HP filter choice with our dataset since our data were originally recorded with an on-board 35 Hz HP filter. However, the International society of Electrophysiology and Kinesiology (ISEK) currently recommends a HP filter from 5 to 10 Hz (Merletti and Torino, 1999). De Luca et al. (2010) found that a 20 Hz HP filter was the best compromise between eliminating movement artifacts and retaining EMG power. HP filters primarily act to reduce DC drift in the EMG signal due to motion artifact and other nonphysiological signals. Consequently, we do not expect large impacts from HP filters on synergies, but the precise impacts of HP filtering on synergy analyses remain an open question.

Beyond filter cutoff frequency and amplitude scaling methods, there are other EMG preprocessing choices we did not explore, including filter type and filter order. Devaprakash et al. (2016) compared a 2nd order critically damped filter to a 2nd order Butterworth filter with consistent cutoff frequencies and found only small differences in the EMG data that did not affect clinical interpretation. (De Luca et al., 2010) found >1% difference in root mean square difference between EMG profiles processed with 2nd or 3rd order Butterworth filters. Taken together, these results suggest that filter choice and order are less significant than LP cutoff frequency for EMG and synergy analyses.

Given the wide variety of EMG data processing methods used in prior research, exploring and discussing the underlying biological mechanisms that should inform the choice of filters and synergy analyses methods would be useful for future

variance.

research. Current EMG data processing methods are largely based upon technical specifications. Prior work has found that LP filters should be tailored to the specific task (Shiavi et al., 1998; Hug, 2011). However, there is a need to explore the neurophysiology underlying synergy analyses, especially considering some of the limitations of surface EMG data (Farina et al., 2004). For example, if synergies are driven by an underlying central pattern generator, what are the rates of these reflex loops, and how can these biological processes inform data preprocessing and interpretation of synergy analyses? If central pattern generators are driven by low-frequency mechanisms, then perhaps low-frequency LP cutoff frequencies are more appropriate. Future research, such as newly developed methods with direct central nervous system recordings and surface EMG data may assist in understanding these relationships (Godlove et al., 2016).

Output measures of synergy analyses including tVAF, synergies weights, and synergy activations were sensitive to EMG processing methods. We found that increasing LP filter cutoff frequency decreased synergy complexity, as measured by tVAF. Since tVAF is commonly used to identify the number of synergies, LP filter choice can impact conclusions about the number of synergies and muscle coactivation patterns from synergy analyses. Synergy weights and activations are less sensitive to LP cutoff frequency when calculated for two or more synergies. Future studies of synergy analyses and potential clinical applications should

## REFERENCES


carefully consider and report EMG processing methods to enable comparisons across studies and institutions. As synergy analysis is adopted in clinical gait analysis to inform treatment planning, these results highlight the importance of carefully considering EMG processing methods and the utility of a control database. We found that z-score based measures, such as walk-DMC, that compare to control populations can reduce sensitivity to LP filter choice and facilitate comparisons between studies and clinical centers with different EMG protocols.

#### AUTHOR CONTRIBUTIONS

MS provided the clinical EMG data. BS, MS and KS designed and crafted the research questions. BS wrote the code and analysed the data. BS, MS, and KS interpreted the results and drafted and revised the manuscript. BS, MS, KS assert that the contents of this research are their own and accurate as they have been presented.

## ACKNOWLEDGMENTS

Research reported in this publication was supported by the National Institute of Neurological Disorders and Stroke (NINDS) of the National Institutes of Health under award number R01NS091056, as well as graduate student funding from the Washington Research Foundation Funds for Innovation in Neuroengineering.


during multidirectional reaching in the horizontal plane. J. Neurophysiol. 103, 1532–1542. doi: 10.1152/jn.00559.2009


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Shuman, Schwartz and Steele. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Methodological Choices in Muscle Synergy Analysis Impact Differentiation of Physiological Characteristics Following Stroke

Caitlin L. Banks 1, 2, 3, Mihir M. Pai <sup>4</sup> , Theresa E. McGuirk <sup>1</sup> , Benjamin J. Fregly 2, 4 and Carolynn Patten1, 3 \*

<sup>1</sup> Neural Control of Movement Lab, Malcom Randall VA Medical Center, Gainesville, FL, United States, <sup>2</sup> Department of Biomedical Engineering, University of Florida, Gainesville, FL, United States, <sup>3</sup> Rehabilitation Science Doctoral Program, Department of Physical Therapy, University of Florida, Gainesville, FL, United States, <sup>4</sup> Department of Mechanical and Aerospace Engineering, University of Florida, Gainesville, FL, United States

Muscle synergy analysis (MSA) is a mathematical technique that reduces the dimensionality of electromyographic (EMG) data. Used increasingly in biomechanics research, MSA requires methodological choices at each stage of the analysis. Differences in methodological steps affect the overall outcome, making it difficult to compare results across studies. We applied MSA to EMG data collected from individuals post-stroke identified as either responders (RES) or non-responders (nRES) on the basis of a critical post-treatment increase in walking speed. Importantly, no clinical or functional indicators identified differences between the cohort of RES and nRES at baseline. For this exploratory study, we selected the five highest RES and five lowest nRES available from a larger sample. Our goal was to assess how the methodological choices made before, during, and after MSA affect the ability to differentiate two groups with intrinsic physiologic differences based on MSA results. We investigated 30 variations in MSA methodology to determine which choices allowed differentiation of RES from nRES at baseline. Trial-to-trial variability in time-independent synergy vectors (SVs) and time-varying neural commands (NCs) were measured as a function of: (1) number of synergies computed; (2) EMG normalization method before MSA; (3) whether SVs were held constant across trials or allowed to vary during MSA; and (4) synergy analysis output normalization method after MSA. MSA methodology had a strong effect on our ability to differentiate RES from nRES at baseline. Across all 10 individuals and MSA variations, two synergies were needed to reach an average of 90% variance accounted for (VAF). Based on effect sizes, differences in SV and NC variability between groups were greatest using two synergies with SVs that varied from trial-to-trial. Differences in SV variability were clearest using unit magnitude per trial EMG normalization, while NC variability was less sensitive to EMG normalization method. No outcomes were greatly impacted by output normalization method. MSA variability for some, but not all, methods successfully differentiated intrinsic physiological differences inaccessible to traditional clinical or biomechanical assessments. Our results were sensitive to methodological choices, highlighting the need for disclosure of all aspects of MSA methodology in future studies.

#### Keywords: muscle synergies, motor modules, stroke, locomotion, EMG, variability

#### Edited by:

Massimo Sartori, University of Göttingen, Germany

#### Reviewed by:

Leonardo Gizzi, University of Stuttgart, Germany Vincent C. K. Cheung, The Chinese University of Hong Kong, Hong Kong

> \*Correspondence: Carolynn Patten patten@phhp.ufl.edu

Received: 26 January 2017 Accepted: 02 August 2017 Published: 31 August 2017

#### Citation:

Banks CL, Pai MM, McGuirk TE, Fregly BJ and Patten C (2017) Methodological Choices in Muscle Synergy Analysis Impact Differentiation of Physiological Characteristics Following Stroke. Front. Comput. Neurosci. 11:78. doi: 10.3389/fncom.2017.00078

## INTRODUCTION

Muscle synergy analysis (MSA) is a mathematical strategy developed under the premise that complex patterns of muscle activity are driven by a small set of activation components termed synergies. Nikolai Bernstein first proposed the idea of synergies to explain how the nervous system simplifies control of a vast number of independent parameters (Bernstein, 1967). While the relationship between the underlying neuromuscular control strategies and the mathematical concept of synergy analysis are still under debate, MSA is arguably an effective method for reducing the dimensionality of a data set into units that represent most of the variability in the original signals. Importantly, there are numerous analytical parameters involved in MSA. A priori decisions and assumptions regarding MSA may influence the outcomes and interpretation of results. Among these choices are: number of synergies used, filtering parameters, electromyogram (EMG) normalization method, computational algorithm, output variable normalization method, and which components [i.e., synergy vectors (SVs) or neural commands (NCs)] remain constant and which can vary between trials. Many research groups base these methodological decisions on the intended application of the synergy analysis [e.g., a musculoskeletal model, a device controller, or a comparison to kinematic, kinetic, or functional variables within and across populations (Ivanenko et al., 2003; Bowden et al., 2010; Berger and d'Avella, 2014; Walter et al., 2014)]. Importantly, when comparing studies that utilize MSA, great care is required to understand the consequences of these decisions. Methodological inconsistencies in performing MSA create challenges not only for interpreting results but also replicating analyses across research groups. These inconsistencies limit our ability to build a body of evidence based on this analytical approach and detract from resolving debate regarding the physiological relevance of MSA.

MSA decomposes EMG activation patterns into a smaller dimension of time-varying signals, often referred to as neural commands, and a matrix of weights, or synergy vectors, that can be linearly combined to reconstruct the original EMG signals. Neural commands are sets of basis functions that represent the time-varying component of the signal and are also known as activation components or activation signals (Ivanenko et al., 2005; Cappellini et al., 2006; Gizzi et al., 2011). Synergy vectors are scalar values that represent activity patterns across all EMG signals and are also referred to as mmodes or weighting coefficients (Ivanenko et al., 2005; Ting and Chvatal, 2010). Collectively, one synergy vector and its corresponding neural command can be termed a synergy or module (Clark et al., 2010; Ting and Chvatal, 2010). The number of synergies selected to represent a data set typically stems from the percentage of variance (or variability) accounted for (VAF) by a combination of synergies. Variability is a key component of MSA because the synergies must be flexible enough to combine into the variable movement patterns that humans or animals employ to perform a task. Several numerical methods can be applied to perform MSA decomposition, including principal component analysis (PCA), independent component analysis (ICA), and non-negative matrix factorization (NNMF). PCA is a linear eigenvalue decomposition technique that finds a set of orthogonal components that represent the covariance of the original data set (Chau, 2001). ICA is a non-linear blind-source separation technique that identifies the statistically independent sources that can be re-combined to generate a mixed set of signals (Bell and Sejnowski, 1995; Hart and Giszter, 2004). Non-negative matrix factorization creates a parts-based representation of the final signal using only positive, additive components (Lee and Seung, 1999). Once the choice of analysis method has been made, the corresponding analysis parameters should be carefully chosen based on the intended outcomes.

Myriad methodological choices are required throughout the MSA process. First, EMG signals are processed, typically involving filtering, time normalization, and amplitude normalization. Filtering and amplitude normalization strategies vary greatly in the literature (Ivanenko et al., 2005; Hug et al., 2012; Santuz et al., 2016). Prior to or during the analysis, depending on the algorithm, a decision must be made regarding the number of synergies needed to reconstruct an intended activity. Some investigators specify a minimum percent VAF (or R 2 ) threshold across all muscles (Roh et al., 2012; Routson et al., 2013), while others include additional criteria such as the requirement that the addition of one more synergy will not increase the VAF by a considerable amount (Ting and Chvatal, 2010; Hayes et al., 2014). In some cases investigators take additional steps, such as measuring the slope of the VAF or R 2 curve and adding a synergy if doing so leads to a substantial change in either of these parameters (Gizzi et al., 2011; Frère and Hug, 2012; Berger and d'Avella, 2014). Additionally, the SVs can be held constant or allowed to vary from trial-to-trial (Frère and Hug, 2012). Finally, after the synergy algorithm is complete, the SVs or NCs are typically normalized in some fashion to facilitate comparisons in the final output.

MSA has a wide range of applications, from musculoskeletal modeling to complementing a biomechanical analysis, and more recently to studying characteristics of movement pathology. Clark et al. showed that fewer synergies could be used to account for muscle activity patterns during walking in the paretic leg of stroke survivors compared to the non-paretic leg or either leg of healthy controls (Clark et al., 2010). Based on these results, they hypothesized that some of the synergies employed by healthy individuals may be merged in the paretic leg of persons following stroke (Clark et al., 2010). This merging of modules was also described in the upper extremity (Cheung et al., 2012) and lower extremity following stroke, incomplete spinal cord injury, and Parkinson's disease, respectively (Rodriguez et al., 2013; Routson et al., 2013; Hayes et al., 2014). However, two other studies reported no difference in the number of synergies between stroke survivors and healthy controls during walking (Gizzi et al., 2011) or in studying the upper extremity (Roh et al., 2012). This variation

**Abbreviations:** MSA, Muscle synergy analysis; RES, treatment responders; nRES, treatment non-responders; SV, synergy vector; NC, neural command; VAF, variance accounted for; NNMF, non-negative matrix factorization; MagPer, magnitude per trial; MaxOver, maximum value over all trials; MaxPer, maximum value per trial; UnitPer, unit variance per trial; UnitOver, unit variance over all trials.

in results could be attributable to many factors, including: synergy analysis methods, number and choice of muscles included, task performed, chronicity of pathologic condition, and heterogeneity of deficits inherently present following stroke. The latter two details are difficult to control, but careful selection of synergy analysis methods should improve our ability to replicate or compare results across studies. In the absence of repeatable results both within and across clinical populations, it becomes a challenge to understand the significance and utility of MSA.

Recently, several studies have investigated key methodological details involved in MSA, ranging from EMG collection and processing to the chosen factorization algorithm. Selecting the largest and most dominant muscles within a movement synergy decreases the effect of experimental constraints on the outcome of synergy analysis (Steele et al., 2013). The choice of high-pass and low-pass filter cutoff frequencies impacts the number of synergies selected and the quality of reconstruction of the original signals (Hug et al., 2012; Santuz et al., 2016). The number of trials, as well as whether trials are analyzed individually, averaged, or concatenated into a single matrix, has not been found to produce a major impact on the number of synergies extracted, but averaging or concatenating smaller data sets decreases reconstruction quality (Oliveira et al., 2014). Some studies have compared factorization algorithms (Ivanenko et al., 2005; Tresch et al., 2006), while another has focused on comparing variations of the NNMF algorithm (Devarajan and Cheung, 2014). Collectively, researchers are moving toward standardizing synergy analysis methods in an effort to advance the field and allow for better validation of the utility of MSA in context.

Here our goal was to assess how variations in MSA methodology affect the quantification of trial-to-trial variability in muscle synergies. We chose to quantify variability because of Bernstein's theories regarding the importance of variability within the nervous system; while variability is always present in cyclical movements like walking, this movement variability likely arises from the same movement synergy (Bernstein, 1967). We applied MSA to EMG data collected during assessments prior to an experimental rehabilitation intervention that targeted walking recovery in chronic stroke. While the primary outcome, walking speed, improved overall following intervention, the cohort was divided equally between responders (RES) and non-responders (nRES) (Clark and Patten, 2012). Importantly, at baseline no clinical or neuromechanical gait parameter differentiated individuals who were identified as RES and nRES post-intervention. These data, having a known functional outcome, afford an ideal test-bed for analyzing the impact of variations in MSA methodology. Specifically, we sought to determine the ability of MSA to differentiate RES and nRES using only pre-treatment data. We applied MSA to the five greatest and five least treatment responders with useable datasets to evaluate differences in synergy vector and neural command variability. We hypothesized that one or more methods of MSA would detect differences in synergy variability between RES and nRES.

## METHODS

This study involves a subset analysis of subjects with chronic hemiparesis following stroke who participated in 8 weeks of rehabilitation. The intervention consisted of 5 weeks of paretic lower extremity power training and 3 weeks of traditional clinic-based gait training (Clark and Patten, 2012). At baseline and post-treatment, self-selected walking speed and EMG data were collected while participants walked over three force plates (Advanced Mechanical Technology, Inc., Watertown, MA). Gait events of heel strike and toe-off were recorded (200 Hz) using a vertical ground reaction force threshold (F>20 N) and target pattern recognition from heel marker placement using a seven-camera Qualisys motion capture system (ProReflex MCU 240, Göteborg, Sweden). Analog force signals were low-pass filtered (second order bidirectional Butterworth, 10 Hz cutoff). Marker data were low-pass filtered (second order bidirectional Butterworth, 6 Hz cutoff). Surface EMG data were sampled (1 kHz) from eight paretic leg muscles: tibialis anterior (TA), medial gastrocnemius (MG), soleus (SO), rectus femoris (RF), vastus lateralis (VL), biceps femoris (BF), semitendinosus (ST), and gluteus medius (GM) using active, pre-amplified electrodes (17 mm inter-electrode distance, input impedance>100,000,000 , CMRR>100 dB at 65 Hz, and signal bandwidth 20–3,500 Hz; MA-411, Motion Lab Systems, Baton Rouge, LA). All procedures were approved by the Stanford University Administrative Panels on Human Subjects Research and conducted in accordance with the Declaration of Helsinki.

Participants were classified as either RES or nRES based on post-treatment change in self-selected walking speed. Individuals demonstrating a post-treatment change exceeding a minimal important difference of 0.123 m/s were classified as treatment RES (n = 15). Conversely, individuals who did not produce or exceed this change were classified as nRES (n = 17). Importantly, clinical and functional measures at baseline were not different between the RES and nRES in the original cohort (Clark and Patten, 2012). The five highest RES and five lowest nRES with useable EMG data sets were selected for the primary analysis. To evaluate the validity of our findings, we compared five additional nRES to the five lowest nRES. The number of trials included for each subject ranged from 2 to 12, with a mean of 10 trials. Subject characteristics for each group can be found in **Table 1**.

## Muscle Synergy Analysis

We used non-negative matrix factorization (NNMF) to perform the MSA for all conditions (Lee and Seung, 1999; Ivanenko et al., 2005; Ting and Chvatal, 2010). All EMG data were band-pass filtered (fourth-order zero phase-lag Butterworth filter, cutoff 20–200 Hz), demeaned, rectified, low-pass filtered (fourth-order zero phase-lag Butterworth) with a variable cutoff frequency of 7/gc Hz (gc corresponds to the duration of the subject's average gait cycle) and time interpolated using the gait events to obtain 101 points per gait cycle (Clark et al., 2010; Chvatal and Ting, 2013; Routson et al., 2013). An individual's ideal low-pass filter frequency relates to the frequency of the task performed, which

#### TABLE 1 | Subject demographics.


Demographic and clinical data are presented mean ± SD and median (range), respectively. \*Indicates a significant difference from RES, p < 0.05.

is variable in this sample because subjects walked overground at self-selected speed (Shiavi et al., 1998; Hug, 2011; Meyer et al., 2016).

We performed MSA using a total of 30 methodological variations comprised of: five approaches for EMG normalization, two approaches for SV calculation, and three approaches for synergy output normalization. The five EMG normalization approaches were: unit magnitude per trial (MagPer), maximum value over all trials (MaxOver; Clark et al., 2010; Frère and Hug, 2012; Routson et al., 2013; Zelik et al., 2014), maximum value per trial (MaxPer; Gizzi et al., 2011; Walter et al., 2014), unit variance per trial (UnitPer), and unit variance over all trials (UnitOver; Roh et al., 2012; Steele et al., 2013). The SV calculation approaches either held SVs constant across all trials (Clark et al., 2010; Ting and Chvatal, 2010) or allowed them to vary (Ivanenko et al., 2005; Cappellini et al., 2006). The three synergy output normalization approaches were: SVs by unit magnitude (SV Mag)<sup>1</sup> , SVs by maximum value (SV Max; Safavynia and Ting, 2012; Chvatal and Ting, 2013; Rodriguez et al., 2013), and NCs by maximum value (NC Max; Ivanenko et al., 2005; Gonzalez-Vargas et al., 2015). If SVs were normalized, then NCs were multiplied by the same normalization values so that their product remained constant, and vice versa. Every possible combination of EMG normalization, SV calculation, and synergy output normalization was applied, thus creating 30 different methodological variations of MSA.

EMG normalization was either computed within individual trials (per trial) or across all trials within a given muscle (over all trials). MagPer normalization involves dividing each element within the vector of 101 EMG data points by its 2-norm, to create a unit vector:

$$\mathcal{Y}\_{\text{MagPer}} = \frac{1}{||\mathfrak{x}||} \mathfrak{x},\tag{1}$$

where **x** is the original EMG vector and **y** is the normalized EMG vector.

In MaxPer normalization, each vector element is divided by the vector's maximum value:

$$\mathcal{y}\_{\text{MaxPer}} = \frac{1}{\max(\mathbf{x})} \mathbf{x} \tag{2}$$

MaxOver normalization involves the same calculation as MaxPer except the denominator is replaced with the maximum EMG value for the given muscle over all walking trials. UnitPer normalization involves dividing each element of the EMG vector by the vector's standard deviation:

$$\mathcal{Y}\_{UnitPer} = \frac{1}{std(\mathfrak{x})} \mathfrak{x} \tag{3}$$

Similarly to MaxOver, UnitOver involves dividing the EMG vector elements by the standard deviation over all trials for a given muscle within a subject.

After each iteration of NNMF, the calculated synergies were sorted within each trial, since NNMF algorithms do not output synergies in any particular order. The neural commands were sorted using the maximal cosine similarity (cossim):

$$\cos\_{\sin}(a,b) = \cos \theta\_{ab} = \frac{a \cdot b}{||a|| \, ||b||},\tag{4}$$

where **a** and **b** are two neural commands within a synergy and θ is the angle between the two vectors. This step was performed to ensure that each synergy was similar across trials within each subject (d'Avella and Bizzi, 2005; Santuz et al., 2016).

Once MSA was completed for each subject, we calculated the trial-to-trial variability in the SVs and the trial-to-trial similarity in NCs as a basis for identifying differences between RES and nRES at baseline. VAF was averaged within subjects to quantify mean differences between RES and nRES. Standard errors in the SVs were averaged within and across subjects for each of the RES and nRES to quantify the variability within these outcome measures. Neural command similarity was calculated two ways: using the cosine similarity and the maximum value of the circular cross-correlation coefficient. The cosine similarity was calculated as in Equation (4), above, for every combination of trials within each synergy (d'Avella and Bizzi, 2005; Coscia et al., 2014; Santuz et al., 2016). These values were then averaged to determine the trial-to-trial similarity of each NC. The cross-correlation coefficient was also averaged across all possible combinations of trials (Ivanenko et al., 2004; Frère and Hug, 2012).

A two-way group∗ synergy analysis of variance (ANOVA) was conducted on mean VAF across all methods. Tukey's HSD was applied post-hoc to assess significant effects. Hedges' gtest was used to compare trial-to-trial differences in the SVs, NCs, and VAF in the RES and nRES across MSA methods. Hedge's g is complementary to the t-test, but due to the small sample size and the desire to determine the generalizability of results to larger data sets, we computed effect sizes rather than performing inferential statistics. We interpret results using

<sup>1</sup>MATLAB. Statistics and Machine Learning Toolbox. Natick, MA: The MathWorks, Inc.

subjects.

effect thresholds as follows: small = ±0.2, medium = ±0.5, large = ±0.8, very large = ±1.3 (Cohen, 1977). Because the present study is a test-bed for MSA, we considered medium effect sizes and larger noteworthy, with special emphasis on large and very large effects. To evaluate our results, we also calculated effect sizes of noteworthy MSA methods between

the five worst nRES and five additional nRES from the larger dataset.

EMG processing, MSA, and Hedge's g-tests were performed using Matlab's Optimization ToolboxTM (Release r2015a, The MathWorks, Natick, MA) and the Measures of Effect Size Toolbox (Hentschke and Stüttgen, 2013). Custom functions were written to analyze these data using each of the MSA methods described above. ANOVA and post-hoc analyses were conducted in JMP Pro 11 (SAS Institute, Inc., Cary, NC, USA).

#### RESULTS

A two-way ANOVA for mean VAF revealed significant main effects of group (p = 0.005) and synergy (p < 0.0001) but no interaction effect. RES had a lower average pre-treatment VAF than did nRES (**Figure 1**). Across all methods, only two synergies were needed to reach at least 90% VAF (92.72 ± 1.04% for RES, 94.44 ± 0.50% for nRES, mean ± standard error). **Figure 2** shows example synergies for one RES and one nRES (top two rows). Addition of a third synergy increased the average VAF to 95.95 ± 0.61% for RES and 96.90 ± 0.40% for nRES. Whether an absolute 90% VAF cutoff or an average 95% requirement was applied, the full EMG data set could be well-approximated using either two or three synergies. Since VAF is unaffected by output normalization, there are 10 possible combinations of MSA methods for comparing VAF between

FIGURE 2 | Example synergies for a responder (A), non-responder (B), and validation non-responder (C). The left two columns depict Synergy 1, while the right two depict Synergy 2. Within each synergy, the leftmost plot represents the synergy vectors (SVs) and the rightmost plot represents the neural commands (NCs). Individual trials are represented by thin bars (SVs) and thin lines (NCs), while the thicker bars and lines represent the average across trials for each subject. Synergies were extracted using maximum value per trial EMG normalization, varying SVs, and normalization of SVs to unit magnitude. TA, tibialis anterior; MG, medial gastrocnemius; SO, soleus; RF, rectus femoris; VL, vastus lateralis; BF, biceps femoris; ST, semitendinosus; GM, gluteus medius.

groups. With two synergies, all 10 methods produced at least medium effects, indicating that a larger sample with similar characteristics is likely to have significantly greater VAF for the non-responders than the responders (**Figure 3**). For three synergies, there was one small effect, four medium effects, and four large effects. All of the large effects occurred with varying SVs. Because of these differential effects in VAF, we examined both two and three synergies for effects in SV and NC variability.

Synergy analysis methodology greatly influenced the calculated trial-to-trial variability within the synergy vectors. With two synergies and varying SVs between trials, six methods produced medium effects and three methods produced large effects (**Table 2**). All of these effects were positive, indicating greater variability in the RES than the nRES. **Figure 4** illustrates absolute effect sizes when comparing SV variability between RES and nRES with two and three varying synergies. MagPer EMG normalization revealed large effects with two synergies regardless of SV output normalization method. For three synergies, notable differences between RES and nRES were revealed with six methods. The greatest effect sizes were revealed with MaxPer and MaxOver EMG normalization and NCs normalized to their maximum value.

The trial-to-trial variability within the neural commands differed depending on both the MSA method and the method of quantifying variability (**Table 3**). For both cosine similarity and cross-correlation, negative effect sizes indicate greater trialto-trial variability among RES, compared to nRES. For the cosine similarity metric with two synergies and constant SVs, there were three medium effects and no large or very large effects (**Figure 5**). Two synergies with varying SVs produced large effects for all fifteen combinations of EMG and output normalization. Three synergies with constant SVs revealed five medium effects and two large effects. Three synergies with varying SVs revealed five medium and no large or very large effects. For the cross-correlation comparison with constant SVs, neither two nor three synergies produced notable effects (**Figure 6**). Two synergies with varying SVs revealed six medium effects and six large effects (**Table 3**). Three synergies with varying SVs revealed three medium effects and one large effect. In general, the same patterns

#### TABLE 2 | Notable effects in synergy vector variability.


SVs, synergy vectors; MagPer, magnitude per trial; SV Mag, synergy vector magnitude; SV Max, synergy vector maximum value; NC Max, neural command maximum value; MaxPer, maximum value per trial; UnitPer, unit variance per trial; MaxOver, maximum value over all trials; UnitOver, unit variance over all trials.

<sup>a</sup>Effect sizes calculated using Hedge's g. Positive values in SV variability indicate greater variability in RES.

FIGURE 3 | Absolute effect sizes at baseline, measured by variance accounted for (VAF). Effect sizes for two synergies with constant synergy vectors (SVs, A), two synergies with varying SVs (B), three synergies with constant SVs (C), and three synergies with varying SVs (D). EMG normalization methods include: magnitude per trial (MagPer), maximum value over all trials (MaxOver), maximum value per trial (MaxPer), unit variance per trial (UnitPer), and unit variance over all trials (UnitOver). Color bar represents effect size values.

were present across output normalization methods but two synergies with varying SVs produced large effects most frequently.

represents effect size values.

To validate our findings, we compared results from the original nRES to those from five additional nRES identified from the larger data set. Example synergies for a validation nRES are shown in **Figure 2** (bottom row), which appear qualitatively similar to the example nRES in the same figure. Two synergies were sufficient to achieve 90% VAF in all validation nRES across all methods, and there were no differences in VAF between the original nRES and the validation group. Because two synergies with varying SVs produced the most consistent effects in the initial RES/nRES analysis, validation focused on large effects within this subset of results. In this case, large effects in the initial RES/nRES analysis coupled with small or no effects in the nRES validation analysis are desirable findings, indicating that the method has the capacity to differentiate between response groups. For synergy vector variability, MagPer EMG normalization produced large effects in the initial analysis and no effects when comparing nRES to nRES TABLE 3 | Notable effects in neural command variability.


SVs, synergy vectors; MagPer, magnitude per trial; SV Mag, synergy vector magnitude; SV Max, synergy vector maximum; NC Max, neural command maximum; MaxOver, maximum value over all trials; MaxPer, maximum value per trial; UnitPer, unit variance per trial; UnitOver, unit variance over all trials; –, no notable effect.

<sup>a</sup>Effect sizes calculated using Hedge's g. Negative values in NC cosine similarity and NC cross-correlation indicate greater variability in RES.

<sup>b</sup>Neural command similarity compared by trial-to-trial cosine similarity.

<sup>c</sup>Neural command similarity compared by trial-to-trial cross-correlation.

(**Table 4**). For cosine similarity within the neural commands, MagPer EMG normalization produced medium effects when comparing between nRES and validation nRES across all output normalization methods. All other EMG methods that produced large effects from the initial analysis produced small or no effects in the validation analysis. For NC cross-correlation, all comparisons within the validation analysis revealed small or no effects.

## DISCUSSION

This study demonstrates that the methodological choices made during MSA have a significant impact on the outcome. Several, but not all, MSA methods were able to differentiate between two groups of pathologic individuals with no distinguishing clinical differences, but presumed intrinsic physiologic differences. Two synergies with varying SVs across trials produced the highest frequency and greatest magnitude of effects. When assessing SV variability, MagPer EMG normalization was the most salient method within this analysis. However, when comparing NCs by cosine similarity or cross-correlation, any of the EMG normalization methods other than MagPer produced differences between RES and nRES that were validated in a secondary analysis. MSA output normalization had no notable influence on results. In general, our results were highly sensitive to changes in MSA methodology, illustrating the need for careful methodological consideration and disclosure when conducting MSA or comparing with results found in the literature.

## Methodological Considerations

We do not believe that our choice of one signal decomposition method (i.e., NNMF) limits the ability to detect group differences within our results. We chose to use NNMF for this MSA for three reasons: muscle activation signals are inherently non-negative, other methods require assumptions such as orthogonality and independence of parameters (Ting and Chvatal, 2010), and NNMF is a commonly used technique, allowing for greater generalizability of results. Previous work using MSA illustrates that NNMF produces synergy vectors and neural commands that are highly correlated with those produced by other computational methods such as PCA and ICA (Ivanenko et al., 2005; Cappellini et al., 2006). NNMF is particularly robust to differences in data distribution and noise across data sets (Tresch et al., 2006). NNMF tends to define a solution subspace where the synergies can be found, building non-negative components (i.e., SVs and NCs) together to reconstruct the original signals (Ting and Chvatal, 2010). This approach is appropriate for the physiological interpretation of the muscle synergy data. There are several variations of the NNMF algorithm and it is possible that changing the algorithm could change MSA results. However, the current consensus in the literature is that the decomposition method does not drastically change the structure of the extracted synergies (Ivanenko et al., 2005, 2006; Cappellini et al., 2006; Tresch et al., 2006). Ultimately, the choice of decomposition method should be tailored to the intended application of the results.

It is common for investigators to specify a minimum VAF cutoff of 90% for identification of the number of synergies that can be used to reconstruct the original signals

(Clark et al., 2010; Frère and Hug, 2012; Roh et al., 2012). If this common cutoff was applied to the current analysis, two synergies would have been more than adequate to reconstruct the original EMG walking signals with reasonable accuracy. If a cutoff of 95% VAF for all methods and all subjects was applied, all but one subject would have required three synergies, with the remaining subject requiring four. The decision to use either two or three synergies simplifies the analysis and prevents potential over-fitting problems. This result is also consistent with findings that individuals post-stroke require fewer synergies to reconstruct their movements (Clark et al., 2010; Cheung et al., 2012). However, three varying synergies were also able to detect differences between groups for some methods.

Decisions beyond decomposition method and number of synergies, such as fixing or varying SVs and EMG normalization method, exhibited a profound influence on the ability to detect group differences within our sample. Allowing SVs to vary from trial-to-trial, rather than fixing them across trials, produced more frequent and larger effect sizes, although this choice is less common in the literature (Ivanenko et al., 2005; Oliveira et al., 2014). MagPer EMG normalization produced large differences between groups for SV variability. When comparing NCs with cosine similarity, all methods of output normalization and EMG normalization (except MagPer) produced group differences that were validated with further analyses. When applying a cross-correlation analysis, MaxPer and UnitPer EMG normalization were the best performing methods across all output normalization approaches. EMG normalization is a common step in MSA, however one utilized without much justification for the choice of method. The impact of normalization strategies is shown when one investigator attempts to replicate the analysis of another. Two research groups performed the same experiment with one added condition, but found different results, which could be attributable in part to differences in MSA methods (de Rugy et al., 2013; Berger and d'Avella, 2014). Our analysis indicates that choice of normalization methods and the choice of constant or variable SVs across trials have a strong effect on the outcomes, thus representing a non-trivial choice among MSA parameters.

It is conceivable that variation in the selection of muscles included in the analysis could impact results (Steele et al., 2013). Addition of more muscles or muscles that could more accurately capture the differences between RES and nRES would likely increase the resolution to detect group differences. These data, and other synergy analysis data, should be interpreted within the context of the muscles included in the analysis.

## Detecting Physiological Differences

Differences in synergy vector and neural command variability between RES and nRES were greatest using two synergies with varying SVs. The largest difference between the two groups was found in the SVs when EMG was normalized to its maximum


TABLE 4 | Comparison of initial results for two synergies with varying synergy vectors with non-responder validation analysis.

RES, responders; nRES, non-responders; SV, synergy vector; NC, neural command; MagPer, magnitude per trial; SV Mag, synergy vector magnitude; SV Max, synergy vector maximum; NC Max, neural command maximum; MaxOver, maximum value over all trials; MaxPer, maximum value per trial; UnitPer, unit variance per trial; UnitOver, unit variance over all trials. <sup>a</sup>Effect sizes calculated using Hedge's g. Positive values in SV variability and negative values in NC cosine similarity and NC cross-correlation indicate greater variability in RES.

value per trial, while EMG normalization was less important when detecting differences in the neural commands. The effect sizes indicate that RES had more variability in their SVs than did nRES. Previous findings indicate that the SVs, but not the NCs, reveal differences between individuals post-stroke and healthy controls (Gizzi et al., 2011). Our results differ slightly, indicating that variability in the SVs and NCs can be used to quantify differences between groups after stroke. However, we compared two groups, both with neural pathology. Our most salient results could not have been produced if the SVs were held constant, an assumption that many investigators have used in their analyses (Clark et al., 2010; Ting and Chvatal, 2010). Differences in trial-to-trial neural command similarity between RES and nRES were also greatest when two varying synergy vectors were used. Similar to our findings in the SVs, these effects indicate that RES exhibited more trial-to-trial variability in the NCs than nRES. This characteristic could be due to a greater flexibility of commands that can be combined to produce a richer variety of movement patterns (Bernstein, 1967), which could be relevant to identifying one's capacity for recovery or treatment response. When NCs were compared using cross-correlation instead of cosine similarity, the results were similar; however, the effects were often blunted. This difference in effect magnitude provides further indication regarding the importance of all aspects of MSA with respect to the final results.

## Recommendations for Reporting Results

MSA has been popularized due to its ability to reduce the dimensionality of complex patterns of muscle activity (Ivanenko et al., 2006). However, the rapid popularization of a technique, coupled with the lack of validation of each MSA methodological choice, presents implementation challenges. In publications utilizing synergy analysis, some investigators provide more than adequate disclosure of the methods employed; however, others provide little to no detail, making comparison of results a challenge. We therefore propose a list of decisions that should be reported explicitly in manuscripts performing MSA:


The present comparison of MSA methods is not intended to suggest there is any one "best" method for all future applications of MSA. Rather, our results illustrate and emphasize the vast differences produced by variations in methodological choices. Ultimately, the choice of analysis methodology should be tailored to the application and research design. To facilitate understanding and reproducibility we recommend disclosure and justification of methods. Such openness between investigators will move research forward, improving the likelihood and timeliness of a research impact involving MSA.

#### CONCLUSIONS

The main goal of MSA, when applied to pathologic populations, is to better understand the intrinsic physiologic characteristics reflected in muscle activity. MSA, with specific focus on trial-to-trial variability, has the potential to provide insight regarding neural strategies that could be relevant to human performance and rehabilitation. Notably, our analysis revealed large differences between response groups with only 10 subjects. Clinical assessments typically require large samples, lack sensitivity, and in this case, fail to differentiate subpopulations that would later respond to an intervention. If information from MSA can be successfully employed as a predictor of recovery or intervention response on an individual basis, this information could improve outcomes of neurorehabilitation. The analysis must, however, be performed properly, with careful selection and justification of methods.

#### REFERENCES


#### AUTHOR CONTRIBUTIONS

BF, CP, and MP conceived and designed experiments; MP, TM, and BF contributed analysis tools; CB and MP performed experiments; CB analyzed data, prepared figures, and drafted manuscript; CB, BF, and CP interpreted results of experiments; CB, BF, and CP revised manuscript; CB, MP, TM, BF, and CP approved final version of manuscript and agreed to be accountable for all aspects of the work.

#### FUNDING

This research was supported by the Department of Veterans Affairs, Rehabilitation Research & Development Service [Project #B29792R (CP) and Research Career Scientist Award #F7823S (CP)] and NSF CBET #1159735 (BF, CP). CB receives support from a University of Florida Graduate School Fellowship. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

#### ACKNOWLEDGMENTS

We thank the members of the Computational Neuromechanics and Neural Control of Movement labs for their support, especially Nick Bianco and Andrew Meyer for methodological consultation, and Eric Walker and Virginia Little for helpful comments on previous versions of this manuscript. A portion of this work has been presented in abstract form at the International Society of Electrophysiology and Kinesiology XX Congress in July 2014 in Rome, Italy.


stroke survivors. J. Neurophysiol. 109, 768–781. doi: 10.1152/jn.006 70.2012


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Banks, Pai, McGuirk, Fregly and Patten. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Mobility as the Purpose of Postural Control

Charlotte Le Mouel\* and Romain Brette

Institut National de la Santé et de la Recherche Médicale, Centre National de la Recherche Scientifique, Institut de la Vision, Sorbonne Universités, Université Pierre et Marie Curie, Univ. Paris 06, Paris, France

Counteracting the destabilizing force of gravity is usually considered to be the main purpose of postural control. However, from the consideration of the mechanical requirements for movement, we argue that posture is adjusted in view of providing impetus for movement. Thus, we show that the posture that is usually adopted in quiet standing in fact allows torque for potential movement. Moreover, when performing a movement—either voluntarily or in response to an external perturbation—we show that the postural adjustments are organized both spatially and temporally so as to provide the required torque for the movement. Thus, when movement is performed skillfully, the force of gravity is not counteracted but actually used to provide impetus to movement. This ability to move one's weight so as to exploit the torque of gravity seems to be dependent on development and skill learning, and is impaired in aging.

Keywords: neuromechanics, posture, balance, motor control, movement

## INTRODUCTION

#### Edited by:

Matthew Tresch, Northwestern University, United States

#### Reviewed by:

Robert H. Lee, Emory University, United States Daya Shankar Gupta, Camden County College, United States

#### \*Correspondence: Charlotte Le Mouel charlotte.le-mouel@inserm.fr

Received: 23 February 2017 Accepted: 05 July 2017 Published: 27 July 2017

#### Citation:

Le Mouel C and Brette R (2017) Mobility as the Purpose of Postural Control. Front. Comput. Neurosci. 11:67. doi: 10.3389/fncom.2017.00067 The position of the center of mass (CoM) is adjusted by the central nervous system during quiet standing (Winter et al., 1998; Sasagawa et al., 2009), in reaction to perturbations (Horak and Nashner, 1986), and in voluntary movement (Cordo and Nashner, 1982; Pedotti et al., 1989; Lee et al., 1990). The traditional theory is that the purpose of this postural control is to immobilize the CoM despite movement and external perturbations (Nashner et al., 1989; Massion et al., 2004; Horak, 2006; Bouisset and Do, 2008). We will refer to this theory as the immobility theory. The underlying assumption is that, because of gravity, standing is unstable. Therefore, if the CoM is displaced from its equilibrium position, then the displacement must be counteracted by postural adjustments, so as to return the CoM to its equilibrium position, otherwise the person will inevitably fall. As argued by Hasan (2005), this notion stems from an analysis of how linear systems respond to perturbations: in linear systems, if deviations from the unique equilibrium position are not corrected, then they grow exponentially. Balance (the ability to prevent falling), is therefore assumed to be equivalent to stabilization, in the strict sense of immobilizing the CoM at a unique equilibrium position by counteracting any displacement away from this position. From this assumption, it follows that moving poses a threat to balance, since any voluntary movement might displace the CoM. This theory has motivated a large body of experiments, performed over the last thirty years, in which a subject is asked to perform a movement of the upper body, while their muscle activity is being recorded (Cordo and Nashner, 1982; Crenna et al., 1987; Pedotti et al., 1989; Lee et al., 1990). In these experiments, a change in the contraction of the lower leg muscles is systematically observed, and this change often precedes the contraction of the upper body muscles. This is interpreted by saying that movement of the upper body might displace the CoM, and must therefore be counteracted by the contraction of the lower leg muscles so as to immobilize the CoM despite movement.

We will argue however that the equivalence between balance and immobilization does not hold for human postural control, and that these postural responses should be understood as providing the impetus for the movement. We will indeed show that during quiet standing, voluntary movement, and in reaction to perturbations, the position of the CoM is not immobilized at a unique equilibrium position, but on the contrary adjusted so as to use the torque of one's own weight, either to counteract external forces so as to maintain balance, or to provide impetus for voluntary movement. We therefore develop an alternative to the immobility theory. We propose that the purpose of postural control is mobility, the ability to produce appropriate impetus by adjusting the position of the CoM. We will refer to this theory as the mobility theory.

We will first show that the posture which is typically adopted in quiet standing allows for one's weight to be used to provide impetus to potential movement, and that when the direction of the movement to be performed can be anticipated, the position of the CoM during stance is shifted in that direction. Secondly, we will show that, during voluntary movement, postural adjustments which are traditionally thought of as immobilizing the CoM despite movement should on the contrary be interpreted as displacing the CoM at the initiation of the movement, so that one's own weight can be used to provide impetus to the movement. Finally, we will show that this ability to use displace one's weight, rather than immobilize it, plays a crucial role when balance is upset by external forces.

#### ADJUSTMENT OF POSTURE DURING STANCE

#### The Standing Posture Allows For Mobility The Standing Posture Requires Tonic Muscular Contraction

When someone is asked to stand quietly, without further instructions, they typically maintain their CoM vertically aligned with the middle of the foot, a few centimeters forwards of the ankle joint (Schieppati et al., 1994). However, when requested to do so, a young, healthy person can maintain their CoM at positions up to 40% of their foot length forwards of its typical position, and up to 20% backwards (Schieppati et al., 1994). There is therefore no unique equilibrium position for the CoM in quiet standing, since a young, healthy person can maintain a range of standing postures without this posing a threat to balance.

If the position of the CoM were controlled only in view of counteracting the torque of one's weight, then it would be most appropriate to place it vertically above the ankles, such that weight would exert no torque (**Figure 1A**). This position can indeed be maintained with minimum lower leg muscle contraction (Schieppati et al., 1994). However, when no instructions are given, subjects maintain their CoM vertically aligned with the middle of the foot, a few centimeters forwards of the ankle joint (**Figure 1B**), so that the weight exerts a forwards torque. In order to maintain this posture, an equivalent backwards torque must be exerted by the ground reaction force (see Section 1.1 in Appendix). As developed in the Appendix (Section 1.2), the torque of the ground reaction force is determined by the contraction of the lower leg muscles. Indeed, if we consider the forces acting on the foot, the weight of the body, carried by the skeleton, is applied at the ankle and therefore exerts no torque. The ground prevents the foot from turning, therefore the ground reaction torque instantly opposes the torque exerted by the lower leg muscles onto the foot (**Figure 2**). Maintaining a standing posture with the CoM forwards of the ankles therefore requires tonic contraction of the calf muscles (**Figure 1B**, Schieppati et al., 1994). The normal standing posture is therefore not the most economical in terms of muscular contraction.

#### The Standing Posture Allows Torque for Movement

Why would subjects actively maintain their CoM forwards of the ankles in quiet standing if this is not efficient? We suggest that this allows them to use their own weight for initiating forwards movements. Forwards torque for movement can only be induced by the external forces: the person's weight and the ground reaction force. As we have shown (Appendix Section 1.2) the ground reaction torque instantly follows the torque of the lower leg muscles. However, this torque is limited. Indeed, as long as the person neither jumps up nor collapses, the ground reaction force has the same magnitude as the person's weight. Its torque is therefore the product of the weight, and the distance between the ankle and the point of application of the ground reaction force, called the center of pressure and noted CoP. Thus, contracting the calf muscles (gastrocnemius and soleus) shifts the CoP forwards of the ankle (**Figure 2A**), and contracting the shin muscle (tibialis anterior) shifts the CoP backwards of the ankle (**Figure 2C**), but the CoP can only move within the limited range of the foot (see Appendix Section 1.3 for further detail).

The net torque is proportional to the distance between the CoM and the CoP. Whereas, the CoP moves instantly when the forces exerted by the muscles change, but can only move within the limited range of the foot, the position of the CoM on the other hand, does not change instantly when the forces exerted by the muscles change. This first requires the sum of the external forces to accelerate the CoM. Displacements of the CoM therefore occur more slowly than displacements of the CoP (as seen for example in Burleigh et al., 1994). Thus, the initial

then the CoP is below the ankle. (C) The torque around the ankles exerted by the shin muscle onto the foot is instantly compensated for by a backwards

net torque that can be produced, either for opposing external perturbation forces or for voluntary movement, is limited by the initial position of the CoM (see Appendix Section 1.4 for further detail).

When initiating fast forwards movements, either starting to walk (Burleigh et al., 1994) or movements performed with the feet in place such as leaning forwards (Crenna et al., 1987) or rising onto one's toes (Nardone and Schieppati, 1988), the CoP is first brought toward the heel by inhibiting the calf muscle contraction and contracting the shin muscle (Crenna et al., 1987; Nardone and Schieppati, 1988; Burleigh et al., 1994). If the CoM were initially above the ankle, this would produce little initial forwards torque (**Figure 3A**), whereas with the CoM forwards of the ankle this produces larger torque (**Figure 3B**). Maintaining the CoM forwards of the ankle thus allows one's own weight to be used for initiating forwards movement. Maintaining the CoM in the middle of the foot allows for either forwards or backwards initial torque to be induced by changes in the forces of the lower leg muscles.

shift of the CoP.

## The Standing Posture Is Actively Maintained

This position of the CoM is precisely and actively maintained on a short timescale, with small adjustments of the CoP in quiet standing serving to immobilize the CoM at this position (Winter et al., 1998). Moreover, the tonic contraction of the calf muscles is adjusted when standing on different slopes so as to maintain the CoM aligned with the middle of the foot (**Figure 4A**, Sasagawa et al., 2009). This precise positioning is also maintained at the longer timescales of growth and aging. Indeed, the curvature of the spine and trunk increases with aging (red line in **Figure 4B**, Schwab et al., 2006), and the position of the CoM is maintained across people with different trunk curvatures by shifting the position of the pelvis relative to the heels (**Figure 4B**, Schwab et al., 2006; Lafage et al., 2008).

Moreover, this forwards position of the CoM emerges with skill learning. Thus, Clément and Rézette (1985) observed acrobats at various competitive levels performing handstands. All the acrobats were able to maintain their balance in the upside-down posture, however they did so in different ways. The acrobats at lower competitive levels maintained their mean CoP a few millimeters forwards of their wrist; they could therefore maintain their posture with very little tonic contraction in the arm muscles (**Figure 4C**, left). The acrobats at higher competitive levels maintained their mean CoP more forwards of their wrists, with the acrobat at the highest level maintaining his mean CoP 3 cm forwards of his wrists; this posture requires tonic contraction of the wrist extensors (**Figure 4C**, right).

Thus, the standing posture is actively adjusted so as maintain the CoM above the middle of the foot (and above the middle of the hand in handstands). Contrary to the immobility theory, this position is not a unique equilibrium point, since a variety of standing postures can be maintained without this leading to a loss of balance. According to the mobility theory, this position is maintained because it allows for torque of the appropriate direction to be produced at short notice, even when this direction cannot be anticipated. This may be useful both for opposing external perturbations and for initiating voluntary movements.

## The Standing Posture Is Adjusted in Anticipation of Movement

When the direction of the appropriate torque can be anticipated, the mobility theory predicts that the CoM would be displaced in that direction in anticipation of the movement. Such a shift can indeed be induced experimentally, either by challenging someone's balance in a predictable direction, or by indicating in advance the direction of a voluntary movement to be performed.

Someone's balance can be challenged by having them stand facing the edge of the platform they are on. According to the immobility theory, this should lead, if anything, to an even more stringent immobilization of the CoM at its equilibrium position,

FIGURE 4 | Adaptation of the position of the CoM. (A) The tonic calf muscle contraction decreases when going from a slope with the toes down (left panel), to a flat slope (middle panel), to a slope with the toes up (right panel) such that position of the CoM is maintained vertically aligned with the middle of the feet. (B) People of different trunk curvatures maintain their pelvis at different distances from the heel line (vertical line above the heel), such that the CoM line (vertical line passing through the CoM) is at the same distance from the heel line. (C) Left panel: Acrobats at lower competitive levels maintain their CoP and CoM aligned with their wrist without tonic contraction of their wrist extensors. Right panel: Acrobats at higher competitive levels maintain their CoP and CoM forwards of their wrist, through tonic contraction of their wrist extensors.

Le Mouel and Brette Poised for Movement

but what is observed is that the CoM is shifted slightly backwards (**Figure 5A**, Carpenter et al., 2001). This is in accordance with the mobility theory, since it increases the person's capacity for producing backwards torque, in the eventuality that they might be subjected to a forwards push. In the experiment, the person's balance was not challenged beyond placing them in front of a drop, which might explain why the shift in CoM position was rather small (less than a centimeter).

Another way of challenging someone's balance is to have them stand on a platform (**Figure 5B**) which is then translated backwards (**Figure 5C**). The person ends up with their CoM in a forward position relative to the feet. A commonly observed response to such a translation is to straighten up (Welch and Ting, 2014). This requires backwards torque, however their capacity for producing backwards torque is limited by the forwards position of their CoM (**Figure 5C**). If such a perturbation is repeated, then over a few trials, the person adjusts their quiet standing posture by shifting their CoM backwards by a few centimeters (**Figure 5D**, Welch and Ting, 2014). This is again in contradiction with the immobility theory, but in accordance with the mobility theory, since the backwards shift of the CoM increases the person's capacity to produce backwards torque for straightening up (**Figure 5E**). When the platform is repeatedly translated forwards, then the person shifts their CoM forwards (Welch and Ting, 2014).

The mobility theory predicts that the position of the CoM in quiet standing would also be shifted if the direction in which a voluntary movement to be performed could be anticipated. This occurs at the start of a race: in sprinting, the initial forwards acceleration is crucial in winning the race. Consistently with the mobility theory, the CoM in the starting position is shifted even beyond the toes by several tens of centimeters (Slawinski et al., 2010). This is achieved by placing the hands on the ground and having the hands carry some of the weight (**Figure 5F**). This ability to use one's own weight to produce torque for movement again seems to depend on skill learning. Indeed, in elite sprinters, the CoM is shifted 5 centimeters further forwards than for welltrained sprinters (Slawinski et al., 2010).

#### Summary

Thus, when the direction of the appropriate torque to be produced cannot be anticipated, the CoM is positioned at the middle of the feet, in a position which allows for both forwards and backwards torque to be produced. When the direction of the torque to be produced can be anticipated, then the standing posture is adjusted by shifting the CoM in that direction. This adaptation of the standing posture in view of movement seems to be dependent on learning.

## ADJUSTMENT OF POSTURE DURING VOLUNTARY MOVEMENT

According to the immobility theory, when a voluntary movement is being performed, postural control serves to immobilize the CoM despite the movement or the perturbation. The mobility theory predicts, on the contrary, that the position of the CoM

FIGURE 5 | Adjustment of the position of the CoM. (A) When a person stands facing a slope, they shift their CoM slightly backwards. (B,C) When someone stands normally (B) and the platform they stand on is shifted backwards, their CoM ends up far forwards of the ankle joints, which limits the net backwards torque for straightening up (C). (D,E) When a backwards perturbation is repeated, the person shifts their CoM backwards in quiet standing (D), which increases the net backwards torque for straightening up after the perturbation (E). (F) In the posture adopted before a sprint, the CoM is placed far forwards of the feet by having the arms carry some of the weight.

is adjusted so as to use the torque of weight for movement. It therefore predicts that muscular contractions are temporally organized so as to accelerate the CoM at the initiation of the movement in the appropriate direction for producing torque for movement.

## Initiation of Voluntary Movement

#### Pulling on a Handle

When someone pulls on a handle placed in front of them, the contraction of the arm muscles is preceded then accompanied by the contraction of the calf muscles (Cordo and Nashner, 1982; Lee et al., 1990). Cordo and Nashner (1982) suggest that this contraction of the calf muscles allows for the CoM to be immobilized despite the movement. However, in order for the CoM to be immobilized, the ground reaction torque would have to exactly compensate for the handle reaction torque throughout the movement, and this would notably require the calf and arm muscle contractions to be simultaneous (as in **Figure 6A**). On the contrary, the initial contraction of the calf muscles which is observed (Cordo and Nashner, 1982) accelerates the CoM backwards (**Figures 6B,C**, further details are provided in Appendix Section 2); and when the person is asked to pull harder on the handle, this initial period lasts longer, the calf muscle activation is stronger, and the initial backwards acceleration of the CoM is larger (Lee et al., 1990). This is in accordance with the mobility theory, since initially accelerating the CoM backwards allows one's own weight to be used to assist the movement (**Figure 6C**).

#### Leaning the Trunk

numbers 1 to 2.

When someone leans the trunk forwards, the contraction of the abdominal muscles is preceded then accompanied by the inhibition of calf muscle contraction and the contraction of the shin muscle (**Figures 7A–C**, Crenna et al., 1987). The CoM could in theory be immobilized if the shin and abdominal muscle contractions were simultaneous, such that the forwards acceleration of the CoM induced by the shin muscle contraction would compensate for the backwards acceleration of the CoM induced by the abdominals contraction (further details are provided in Appendix Section 2), as suggested by Alexandrov et al. (2001). However, these authors report an initial backwards displacement of the CoP (**Figure 7A**), followed by a forwards displacement of the CoM (**Figure 7B**), in accordance with the sequential muscular contraction observed by Crenna et al. (1987). This contradicts the immobility theory, but concords with the mobility theory's predictions.

Thus, postural responses should be considered as an integral part of the movement itself, since they provide the torque for the movement, first by shifting the CoP and secondly by accelerating the CoM through sequential muscle contraction (a more complete explanation can be found in Appendix Section 2).

#### Gait Initiation

Bouisset and Do (2008) distinguish between two types of anticipatory postural adjustments. For voluntary movements without a change in the basis of support, such as raising the arm, they provide a very classical interpretion for the displacement of the CoM which precedes the displacement of the arm. They present it as a counterperturbation whose purpose is to "counterbalance the disturbance to postural equilibrium due to the intentional forthcoming movement" (Bouisset and Zattara, 1981). However, for voluntary movements involving a change of the basis of support, such as walking, or rising onto one's toes,

Frontiers in Computational Neuroscience | www.frontiersin.org

CoM backwards, allowing for a larger net backwards torque to be exerted during the handle pull (C). The sequence of activation of the muscles is indicated by the

FIGURE 7 | Leaning the trunk. When control subjects perform fast forwards leaning, the initial contraction of the shin muscle (A) accelerates the CoM forwards, thus allowing for more net forwards torque during the subsequent contraction of the ventral muscles (B), which enables the person to lean the trunk (C). When control subjects perform fast backwards leaning, the dorsal muscles contract simultaneously (D), which increases backwards rotational (Continued)

#### FIGURE 7 | Continued

momentum without translating the CoM (E). When gymnasts perform fast backwards leaning, the initial contraction of the calf muscles (F) accelerates the CoM backwards, thus allowing for more net backwards torque during the subsequent contraction of the dorsal muscles (G) which enables the gymnast to lean the trunk (H). The sequence of activation of the muscles is indicated by the numbers 1 to 3.

they present anticipatory postural adjustments as a perturbation involved in "body weight transfer" (Do et al., 1991).

We propose that in movements with or without a change in the basis of support, anticipatory postural adjustments play the same role of moving the CoM in order to provide impetus for movement. Indeed, the changes in posture which precede walking are organized in the same way as those which precede pulling on a handle or leaning the trunk. Thus, when going from standing to walking, a few hundred milliseconds before the heel of the swing foot is raised, the calf muscles are silenced and the shin muscle contracts, which brings the CoP to the heels and accelerates the CoM forwards, even before the first step is taken (**Figure 3B**, Burleigh et al., 1994). This is in accordance with the mobility theory, since initially accelerating the CoM forwards allows one's own weight to be used to assist the movement. Indeed, this initial acceleration of the CoM is correlated with the speed reached at the end of the first step, and is larger if the person is asked to walk faster (Brenière et al., 1987).

#### The Ability to Use One's Weight for Movement Requires Practice

For walking, a movement which is learned very early on in life, the ability to displace the CoM at the initiation of the movement emerges over the course of development (Ledebt et al., 1998; Bril et al., 2015). The amplitude of the initial backwards shift of the CoP thus increases over the first several years of life as children learn to walk faster (Ledebt et al., 1998; Bril et al., 2015). It then decreases with age, and with certain neurological diseases such as Parkinson's disease (Halliday et al., 1998; Mancini et al., 2016).

For leaning the trunk, the sequential muscle contraction, which allows for the displacement of the CoM at the initiation of the movement, seems to be dependent on learning. Indeed, when control subjects are asked to lean backwards, a movement for which they presumably have less practice than leaning forwards, then the calf and dorsal trunk muscle contractions are simultaneous (**Figure 7D**), and the movement is performed twice as slowly as leaning forwards (Pedotti et al., 1989). This is presumably because the CoM was not displaced backwards (**Figure 7E**). However, when gymnasts are asked to lean backwards, then their calf muscles contract first, and they perform the movement faster than controls (**Figures 7F–H**, Pedotti et al., 1989). Moreover, the ability to displace one's CoM during movement seems to remain plastic throughout life, and to depend on the possibility to use one's weight to assist movement. Thus, when astronauts return from a several months journey in space (during which they could not use their weight to assist their movements), the forwards displacement of the CoM when leaning forwards is no longer observed (Baroni et al., 2001).

Finally, for movements requiring skill learning, the temporal coordination which enables using one's weight to provide impetus for movement seems to develop with skill learning. Thus, when learning a complex gymnastics skill, such as the swings under parallel bars, in bent inverted hang position (**Figure 8**), beginners swing their legs and arms in synchrony, whereas experts swing their legs out of phase with their arms, which allows them to use the work of their own weight to provide impetus to the swing (Delignières et al., 1998).

#### BALANCE REQUIRES MOBILITY RATHER THAN IMMOBILITY

According to the immobility theory, if postural control does not immobilize the CoM at a unique equilibrium position, then the person must fall (Nashner et al., 1989; Massion et al., 2004; Horak, 2006; Bouisset and Do, 2008). We have shown however that in quiet standing, people can keep their balance over a range of positions of the CoM (Schieppati et al., 1994), and actually displace their CoM when their balance is challenged in a predictable (Carpenter et al., 2001; Welch and Ting, 2014). Moreover, we have shown that in well-practiced movements, people accelerate their CoM at the initiation of the movement, without this leading to a loss of balance (Cordo and Nashner, 1982; Crenna et al., 1987; Pedotti et al., 1989; Lee et al., 1990). We will now show that the response to an external perturbation should be considered as a movement in its own right, and therefore also benefits from the ability to use one's weight for movement, rather than to immobilize it.

#### Responding to External Perturbations Straightening Up after a Platform Translation

When the platform on which someone stands is translated backwards, the CoM ends up in a forward position relative to the feet (**Figures 9A,B**), as seen in Section The standing posture is adjusted in anticipation of movement. A response which is commonly observed is to straighten up (Horak and Nashner, 1986). The backwards acceleration of the CoM is performed through a sequential contraction of the dorsal muscles, starting with the calf muscles (**Figure 9A**), then the dorsal thigh then dorsal trunk muscles (Horak and Nashner, 1986). This contraction pattern is usually not considered as an actual movement, since it moves the CoM closer to its initial position, in accordance with the immobility theory. However, we believe it should be considered as a movement in its own right. Indeed, straightening up after a platform translation requires producing the appropriate backwards torque. The sequential contraction pattern allows for the CoM to be initially accelerated backwards, which increases the net backwards torque for the movement. Further details are provided in Appendix—Horizontal acceleration of the CoM. Moreover, contrary to the immobility theory, returning the CoM to its initial position is not the only way of preventing a fall.

#### Stepping after a Platform Translation

Indeed, another response which is also commonly observed is to take a step forwards (Maki et al., 2003): the CoM is then not returned to its initial position, without this causing a loss of balance. This response takes advantage of the forwards position of the CoM, such that the CoM needs not be accelerated backwards, and indeed the initial calf muscle contraction and forwards CoP shift is much reduced (**Figure 9B**) compared to when the person straightens up (**Figure 9A**); nor does the CoM need to be accelerated forwards, and indeed the shin muscle contraction lasts much less long and the backwards shift of the CoP is much smaller (**Figure 9C**, Burleigh et al., 1994) than when the person takes a step without the platform translation (**Figure 9D**).

## Emergence Over Development and Impairment with Aging

The ability to mobilize one's weight emerges over development. Thus, when straightening up after a backwards platform translation, both the systematic recruitment of the dorsal muscles and their temporal sequencing emerge during development. They are not observed in pre-walking infants, but are seen in children with a few years' walking experience (Burtner et al., 1998).

This ability is then deteriorated with aging, and with Parkinson's disease. The elderly, and even more so Parkinsonian patients, are less capable of moving their CoM, either when asked to adjust their quiet standing posture by leaning forwards or backwards (Schieppati et al., 1994), or during voluntary movement, such as gait initiation (Halliday et al., 1998). They are however quite as capable as young healthy adults of remaining immobile in quiet standing (Schieppati et al., 1994), and adjust the position of their pelvis to compensate for trunk curvature

such that the CoM remains above the middle of the feet (Schwab et al., 2006). Nevertheless, they have a heightened risk of falling. Thus, although the elderly and Parkinsonian subjects are quite as capable as young adults of maintaining their CoM immobile during quiet standing, we suggest that their higher risk of falling is due to a limited capacity to move when this becomes necessary to prevent a fall. Therefore, not only is immobilizing the CoM unnecessary for balance, it moreover seems that balance benefits from the ability to move one's CoM. This suggests that efficient balance training for the elderly can be achieved by practicing mobility (Xu et al., 2005).

#### DISCUSSION

## Posture Is Adjusted in View of Mobility Rather than Immobility

Although the position of the CoM is adjusted by the nervous system, this postural control does not serve to immobilize the CoM. On the contrary, the position of the CoM is adjusted so as to use the torque of one's own weight both for selfinitiated movements and for responding to external perturbation forces.

Thus, in quiet standing, when the direction of the torque to be produced cannot be anticipated, the CoM is maintained above the middle of the foot (Schieppati et al., 1994), allowing for the torque of one's weight to be used both for forwards and backwards movements. This position is actively maintained despite short-term changes in slope (Sasagawa et al., 2009) or long-term changes in trunk curvature (Schwab et al., 2006). However, when the direction of the torque to be produced can be anticipated, then the CoM is shifted in that direction. There is thus a small backwards shift of the CoM when someone is placed in front of a drop (Carpenter et al., 2001), or on a platform which is repeatedly translated backwards (Welch and Ting, 2014). Skill learning leads to much larger shifts in the position of the CoM, with the CoM placed forwards of the feet in anticipation of sprinting (Slawinski et al., 2010).

Moreover, during movement, we have shown that the postural responses which were thought to immobilize the CoM despite movement are actually temporally organized so as to accelerate the CoM at the initiation of the movement, in the appropriate direction such that the torque of one's weight can be used for the movement (Cordo and Nashner, 1982; Crenna et al., 1987; Pedotti et al., 1989; Lee et al., 1990). These postural responses should therefore be understood as providing impetus to the movement.

Finally, we have shown that in order to respond effectively to external perturbation forces, the CoM need not be immobilized, since the person can take a step (Maki et al., 2003). When the person straightens up without taking a step (Horak and Nashner, 1986), this requires producing forces to counteract the external perturbation, and may benefit from the ability to mobilize one's CoM rather than immobilize it. Balance therefore requires mobility rather than immobility.

FIGURE 9 | Response to platform translation. When straightening up after a platform translation (A), the initial contraction of the calf muscle accelerates the CoM backwards which increases the potential net backwards torque. When stepping forwards in response to a platform translation (B,C), this initial calf muscle contraction is reduced (B). Then, the shin muscle contracts (C). This shin muscle contraction is smaller than the initial contraction of the shin muscle which accelerates the CoM forwards when the person steps forwards without a platform translation (D). The sequence of activation of the muscles is indicated by the numbers 1 to 2.

## Mobility Emerges Through Development and Skill Learning

The ability to use one's weight for movement emerges through development and skill learning, and remains plastic throughout life. The appropriate temporal organization of muscular contraction emerges during development both for walking and for balancing responses (Burtner et al., 1998; Ledebt et al., 1998). It is not observed for less practiced movements, such as when control subjects lean the trunk backwards (Crenna et al., 1987). The extent to which the CoM can be mobilized seems to depend on the level of skill: thus, both for sprinters at the initiation of a race (Slawinski et al., 2010) and acrobats performing handstands (Clément and Rézette, 1985), elite athletes place their CoM further forwards than well-trained athletes. Future work should address the following questions: how is this ability learned through development and practice? Does the impairment of this ability in aging result from a lack of practice, and could this

## REFERENCES


ability be maintained during aging through appropriate training regimes?

## AUTHOR CONTRIBUTIONS

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

#### FUNDING

This work was supported by Agence Nationale de la Recherche (ANR-14-CE13-0003).

## SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fncom. 2017.00067/full#supplementary-material


extensors: quiet standing on inclined surfaces. Exp. Brain Res. 196, 537–544. doi: 10.1007/s00221-009-1876-4


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Le Mouel and Brette. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# The Propagation of Movement Variability in Time: A Methodological Approach for Discrete Movements with Multiple Degrees of Freedom

Melanie Krüger 1, 2, 3 \*, Andreas Straube<sup>2</sup> and Thomas Eggert <sup>2</sup>

<sup>1</sup> Sensorimotor Neuroscience and Ageing Research Laboratory, School of Medicine, University of Tasmania, Hobart, TAS, Australia, <sup>2</sup> Department of Neurology, University Hospital Munich Großhadern, Munich, Germany, <sup>3</sup> Department of Sport and Health Sciences, Technical University of Munich, Munich, Germany

In recent years, theory-building in motor neuroscience and our understanding of the synergistic control of the redundant human motor system has significantly profited from the emergence of a range of different mathematical approaches to analyze the structure of movement variability. Approaches such as the Uncontrolled Manifold method or the Noise-Tolerance-Covariance decomposition method allow to detect and interpret changes in movement coordination due to e.g., learning, external task constraints or disease, by analyzing the structure of within-subject, inter-trial movement variability. Whereas, for cyclical movements (e.g., locomotion), mathematical approaches exist to investigate the propagation of movement variability in time (e.g., time series analysis), similar approaches are missing for discrete, goal-directed movements, such as reaching. Here, we propose canonical correlation analysis as a suitable method to analyze the propagation of within-subject variability across different time points during the execution of discrete movements. While similar analyses have already been applied for discrete movements with only one degree of freedom (DoF; e.g., Pearson's product-moment correlation), canonical correlation analysis allows to evaluate the coupling of inter-trial variability across different time points along the movement trajectory for multiple DoF-effector systems, such as the arm. The theoretical analysis is illustrated by empirical data from a study on reaching movements under normal and disturbed proprioception. The results show increased movement duration, decreased movement amplitude, as well as altered movement coordination under ischemia, which results in a reduced complexity of movement control. Movement endpoint variability is not increased under ischemia. This suggests that healthy adults are able to immediately and efficiently adjust the control of complex reaching movements to compensate for the loss of proprioceptive information. Further, it is shown that, by using canonical correlation analysis, alterations in movement coordination that indicate changes in the control strategy concerning the use of motor redundancy can be detected, which represents an important methodical advance in the context of neuromechanics.

Keywords: movement coordination, variability, canonical correlation, reaching, sensory loss

#### Edited by:

Matthew Tresch, Northwestern University, United States

#### Reviewed by:

J. Michael Herrmann, University of Edinburgh, United Kingdom Daya Shankar Gupta, Camden County College, United States

#### \*Correspondence:

Melanie Krüger melanie.krueger@tum.de

Received: 31 December 2016 Accepted: 26 September 2017 Published: 13 October 2017

#### Citation:

Krüger M, Straube A and Eggert T (2017) The Propagation of Movement Variability in Time: A Methodological Approach for Discrete Movements with Multiple Degrees of Freedom. Front. Comput. Neurosci. 11:93. doi: 10.3389/fncom.2017.00093

## INTRODUCTION

Analyzing movement variability to gain insights into movement planning and control processes has been in the focus of researchers in the field of (computational) human motor control ever since Bernstein's (1967) famous Blacksmith example, which describes that, even in highly skilled movements with high outcome stability, execution variability can be observed across repetitions. Originating from that observation, numerous studies have been conducted to investigate changes in the amount and structure of movement variability with changing external task constraints (Gera et al., 2010; van der Steen and Bongers, 2011; Krüger et al., 2012) or under manipulated availability of sensory information during movement planning and execution (Tseng et al., 2002; Krüger et al., 2011), showing that variability in movement execution is an inherent characteristic of human performance. In the last recent years, this research has significantly profited from the emergence of a range of different mathematical approaches to analyze the structure of movement variability. Approaches such as the Uncontrolled Manifold method (Scholz and Schöner, 1999) or the Noise-Tolerance-Covariance decomposition method (Müller and Sternad, 2004) allow to detect and interpret changes in movement coordination by analyzing the structure of withinsubject, inter-trial movement variability. Referring to these methods, it was shown that changes in the structure of movement variability can be related to changes in movement planning and control processes due to learning, aging, and pathology (Cirstea and Levin, 2000; Müller and Sternad, 2009; Stergiou and Decker, 2011; Krüger et al., 2013).

In that context, particularly the investigation of the time course of movement variability during movement execution has stimulated theory-building in motor neuroscience and advanced our understanding of the synergistic control of the redundant human motor system. Research on reaching and pointing movements (Domkin et al., 2002, 2005; Cohen and Sternad, 2009; Krüger et al., 2012) provided empirical support for theories of motor control which postulate that the human motor system exploits its inherent redundancy to cost-optimize movement execution, such that only variability in task-relevant dimensions is minimized, a principle referred to as "minimum intervention principle" (Todorov and Jordan, 2002; Todorov, 2004).

For time course analyses of movement variability, different time points between movement start and end, usually corresponding to either certain percent of the normalized time between movement start and end or to distinct time points in the movement, such as e.g., time point of maximum velocity, are examined. Following the definition of relevant time points between movement start and end, two principally different approaches can be followed to analyze the time course of intertrial movement variability: first, the variance structure of the effector position at a single time point during the movement can be analyzed across many movement repetitions, and conclusions can be drawn from changes of this variance structure between different time points. As an example for this approach, a range of studies using the Uncontrolled Manifold method were able to show significant differences in the control of task-relevant and—irrelevant variability in the effector space across the time course of movement execution for, e.g., sit-to-stand, shooting, or goal-directed reaching movements (Scholz and Schöner, 1999; Scholz et al., 2000; Tseng et al., 2002). However, this approach focuses on the relations between variability in the effector and task space at one or multiple points in time, but does not allow direct conclusions about statistical coupling of effector variables across time.

In contrast to this first approach, the second approach focuses on the coupling of movement variability between different time points during movement execution. Importantly, while for cyclical movements (e.g., locomotion), mathematical approaches exist to investigate the propagation of movement variability in time (e.g., time series analysis, or Lyapunov exponent, Stergiou and Decker, 2011), similar approaches are missing for discrete, goal-directed movements with a redundant effector system, such as reaching with the arm. Here, we propose canonical correlation analysis as a suitable method to analyze the propagation of within-subject variability across different time points during the execution of discrete movements. While similar analyses have already been applied for discrete movements with only one degree of freedom (DoF; Messier and Kalaska, 1999; Richardson et al., 2011; Kuang and Gail, 2015; Eggert et al., 2016), canonical correlation analysis allows to evaluate the coupling of intertrial variability across different time points along the movement trajectory for multiple DoF-effector systems, such as the arm.

This method will be illustrated by empirical data from a study on reaching movements under normal and disturbed proprioception. Proprioception about joint positions is an important source of information for the control of complex reaching movements (Ghez and Sainburg, 1995; Bagesteiro et al., 2006). Studies on chronically deafferented patients suffering from severe peripheral sensory neuropathy showed impaired motor control of arm movements, including slowed movement execution (Gentilucci et al., 1994; Hepp-Reymond et al., 2009), increased movement variability (Gentilucci et al., 1994; Medina et al., 2010) and deteriorated movement coordination (Sainburg et al., 1993, 1995; Ghez and Sainburg, 1995). Studies of temporary peripheral deafferentation of healthy humans showed immediate adjustment to the loss of proprioception on a behavioral (Moisello et al., 2008, applying limb immobilization) and cortical level (Björkman et al., 2004a,b, applying a local anesthetic cream; Ziemann et al., 1998b, applying an ischemic nerve block). However, these studies mainly requested the production of simple motor tasks with a limited range of kinematic DoF. Studies on the production of complex motor behavior, such as reaching movements, are rare. Here, we investigated the effect of temporary proprioceptive deafferentation, induced by an ischemic block at the upper arm level, on the time course of joint angle variability of complex arm movements.

## METHODS

## Participants

Fifteen healthy volunteers (mean age ± SD: 26 ± 5 years; 8 female) participated in the study. All participants were righthand dominant as determined by the Edinburgh Handedness

Inventory (Oldfield, 1971) and had normal or corrected-tonormal vision. None of the participants had any record of neurological disorder. All participants were paid for their participation and had given written informed consent prior to participation. The experimental procedure was in accordance with the Declaration of Helsinki and was approved by the Ethics Committee of the Medical Faculty of the Ludwig-Maximilians University Munich.

## Experimental Set-Up

Participants were seated on a chair in front of a table, with the trunk supported by a chair back. A linear table track was mounted on the table, with a spherical object (reaching target, diameter: 80 mm) attached to it. Due to its geometric properties, the reaching target constrained final hand position but not final hand orientation. The size of the target forced the participants to grasp it with the whole hand, and not just with two fingers, which is why single finger motion was not of interest in the current study. The reaching target could be freely moved horizontally (in the fronto-parallel plane) between the bounds of the table track. These bounds (distance: 39 cm) were the two positions at which the reaching target could be located. The sitting position of the participants was adjusted so that: (a) trunk movement was not necessary to reach the target, and (b) body midline was centered to the table track. To minimize within-subject inter-trial variability due to differences in the initial position, the starting position was defined by a wooden lever, attached to the right side of the chair, which had to be grasped with the dominant right hand before each trial (see **Figure 1A**).

Joint angle motion of the arm in its seven degrees of freedom was recorded by an ultrasonic sound-emitting system (Zebris Medical, Isny, Germany). Six sound-emitting markers were attached to the arm and hand of the participant; each marker recorded at a frequency of 33 Hz. The following marker positions were chosen and are depicted in **Figure 1B**: marker 1 and 2 were attached to the metacarpophalangeal joints of the index (1) and little finger (2). The third marker was at the center of the wrist. Marker 4 and 5 were attached to the medial (4) or lateral (5) end of a bracelet directly above the elbow. The sixth marker was attached at the acromion. From those marker positions, the individual length of the participant's upper arm, lower arm, and hand could be determined. Based on these lengths, a geometrical model of the arm was created, as described in more detail below (see section Data Analysis). Further, the signal of the first marker was used to trigger the opening and closing of shutter glasses (Translucent Technologies, Toronto, Canada) that were used to prevent visual online control of the movement. The first contact with the reaching target was detected by changes in the electrical resistance between the participant and the target (sampled at 1 kHz).

## Procedure and Design

Participants repeatedly had to reach toward and grasp the reaching target with their dominant right hand. At the beginning of each trial, participants had to adopt the starting position (see **Figure 1A**). Subsequently, participants were instructed to press a button with their non-dominant hand, after which the target changed its position. After an acoustic go-signal, participants had to perform the reaching movement in a natural manner. To provoke the most natural movement behavior, participants were informed before movement recording that movement speed and reaction time were not of interest in the study. Shutter glasses occluded as soon as the participants started their movement, thus preventing visual online control of the movement. After the participants had grasped the target, the shutter glasses opened again and the participants returned to the starting position. A new trial was initiated by pressing the button again.

All participants participated in two experimental conditions in separate sessions, the order of which was counterbalanced across participants. Experimental sessions were separated by 1 to 2 days. In the first condition ("ischemia"), a customary blood pressure cuff was applied to the upper arm of the participants and inflated up to 150–160 mmHg (i.e., slightly above systolic blood pressure) to induce a transient ischemic block. Ischemic nerve block is an established experimental technique to study sensory control of movements (e.g., Glencross and Oldfield, 1975; Ziemann et al., 1998a). It is known to first affect the large, fast conducting afferent fibers, especially Ia afferents arising from the muscle spindle afferents (Fellows et al., 1993). In contrast to acute limb ischemia, a sudden decrease in limb perfusion caused by e.g., thrombosis, transient ischemic block is an incomplete block of limb perfusion caused by externally applied pressure and is non-threatening to the limb. Glencross and Oldfield (1975) showed that 20–25 min of ischemic nerve block results in complete dropout of finger sensation, and significant sensory decrease in wrist and elbow. Decrements in the exerted force were observed only after complete sensory dropout. In the current study, the duration of inflation was in a range of 20–25 min. This timeframe included 10 min of preparation to guarantee impairment in the global sensory afference, and a subsequent 10–15 min of movement recordings. Consequently, other effects of the ischemic nerve block, such as changes in producible muscle force (Björkman et al., 2004a), can be disregarded in the currentset-up because of the brevity of the ischemic block. Before movement recording started, the proprioceptive impairment was tested indirectly by assessing participants' touch sensitivity with von-Frey filaments (Marstock, Schriesheim, Germany, Rolke et al., 2006). On the back of the participants' hands it was tested which of the 12 logarithmically scaled filaments (range: 0.25–512 mN) participants were at least able to perceive. On average, participants were able to perceive a minimum pressure of 0.5 mN before the application of the blood pressure cuff (i.e., mean filament number ± SD: 1.91 ± 0.53). Participants' touch sensitivity had to be reduced by at least one filament (i.e., increase by a factor 2) before the experiment was continued. On average, participants perceived a minimum of 1 mN at the start of the movement recordings (i.e., mean filament number ± SD: 3.09 ± 0.70). This procedure allowed us to be sure about the effectiveness of the ischemic block. At the same time, the duration of preparation was minimized, which was of importance to prevent unwanted side effects of the ischemic block, as for example ischemic pain. The second experimental condition ("control") served as a control condition, executed identically but without inflated blood pressure cuff.

Two blocks of 40 trials in each block were recorded in each session (i.e., 80 trials per session). Each experimental block consisted of 20 trials of each of the two target positions, arranged in a random order to avoid predictability of the target position. Between the blocks a break of maximally 5 min was offered to avoid fatigue. Before movement recording started, participants were allowed to perform five trials to familiarize themselves with the experimental task and apparatus.

#### Analysis Data Analysis

Data analysis was calculated using Matlab 7.9.0 (Mathworks, Natick, USA) and was in line with earlier studies by our group (Krüger et al., 2011, 2012). In a first step, the seven joint angles of the arm were computed from the marker positions using a three-segment rigid body model, and expressed as seven consecutive Cardan angles. The order of the angles was as follows: two angles for the wrist (vertical, and horizontal), two angles for the elbow (torsion, and flexion), and three angles for the shoulder (torsion, horizontal, and vertical). The zero position of the arm was defined as the arm pointing straight forward with the elbow extended and the palm facing up. Based on that, positive joint angle indicated the following directions: vertical upward, horizontal rightward, and torsion clockwise. The vector containing the seven joint angles is hereafter referred to as arm posture. The position of the hand in space (i.e., 3D) was defined by the center of the two hand markers in world fixed Cartesian coordinates. In addition, the orientation of the hand in space was defined in Helmholtz coordinates relative to the external world.

Temporal and spatial movement characteristics were analyzed separately for each condition, participant, target position and trial. Overall movement duration was defined as the time between movement initiation and movement end. To determine movement initiation, movement start was defined as the time point at which the hand velocity first exceeded 10% of its peak velocity. Movement initiation was then determined by subtracting 10% of the acceleration time (i.e., the time between movement start and reaching peak velocity) from movement start. Movement end was defined as the last sample recorded before the first contact with the reaching target, as determined by the change in electrical resistance (see section Experimental Set-Up). Subsequently, duration of acceleration and duration of deceleration were calculated. In addition, peak velocity was analyzed. Thus, temporal characteristics of the reaching movements will be described by four measures: (1) overall movement duration, (2) duration of acceleration, (3) duration of deceleration, and (4) peak velocity. To determine spatial characteristics of the reaching movement, movement amplitudes were determined by calculating the absolute value of the difference between the maximum and minimum joint angle separately for each of the seven joint angles. Subsequently, mean movement amplitude was calculated as the average movement amplitude across the seven joint angles. In addition, to evaluate the changes in the diversion from shortest trajectory between starting and end position, the total path length in the 7D-joint space of the arm was calculated.

Within-subject inter-trial movement variability during the time course of movement execution and at movement end was analyzed separately for each condition, participant, and target position. On that account, the full temporal resolution of the joint angle motion was reduced to 10 equidistant sampling points between movement initiation and movement end. To account for small inter-trial variations in the actual starting position of the arm and in movement duration, a correction of the joint angle trajectories was calculated as described in detail in Krüger et al. (2011). Briefly, the within-subject deviations of the joint angles from their mean were submitted to a linear regression analysis with the predictor initial arm position and movement duration (i.e., 7 + 1 = 8 continuous predictor variables). Separate regression analysis were conducted for each participant, experimental condition, target position, and for each of the 10 sampling points, thus, containing the data of 40 trials. Subsequently, the joint angle deviations from the mean that were predicted by this linear model were subtracted from the actual joint angles. After this correction, the covariance matrix of the starting position (first sample) reduced to zero and was not considered in further analytical steps. Thus, the covariance matrix of the joint angles was analyzed at nine equidistant sampling points during the movement.

Afterwards, movement variability was analyzed by means of the two approaches described in the Introduction: first, analyzing the amount of variability at the nine single sampling points during movement execution, and second, analyzing the coupling of movement variability across different sampling points. To achieve the first, the square-root of the mean within-subject variance, averaged across the seven joint angles of the arm (hereafter referred to as: standard deviation of arm posture), was calculated. Further, the square root of the mean within-subject variance, averaged across its three dimensions was calculated for the task variables (a) hand position (standard deviation of hand position) and (b) hand orientation (standard deviation of hand orientation). In addition, the coupling between joint angles within the arm posture at a given sampling point was analyzed by calculating a principal component analysis on the 7 × 7 covariance matrix of the arm posture. Subsequently, the variances for each of the seven eigenvalues of the covariance matrix were averaged across sampling points, and the percentage of total variance explained by the first two eigenvalues was calculated. A relative increase of this percentage is closely related to a relative decrease of the number of kinematic DoF with respect to the mechanical DoF.

To achieve the second, the coupling between the arm posture at a given sampling point and the final arm posture was examined. On that account, the coupling between the arm posture during the movement and the final arm posture was assessed by canonical correlation analysis evaluating the percentage of inter-trial variance of the final arm posture that could be explained by the variance of arm posture at a given sampling point. The canonical correlation analysis returns a coefficient of determination, which equals the mean R 2 across the multiple regressions explaining the final arm posture as linear functions of the arm posture at a given sampling point.

#### Statistical Analysis

Statistical analysis was calculated using SPSS 9.0. Pairwise comparisons were calculated for the temporal and spatial measures of the reaching movements, the percentage of total variance explained by the first two eigenvalues, as well as for the coefficient of determination. A repeated measurement ANOVA with condition (control vs. ischemia) as the between-group factor, and sampling point as the repeated factor was calculated for the following dependent variables: (1) standard deviation of arm posture, (2) standard deviation of hand position, and (3) standard deviation of hand orientation. Bonferroni corrected pairwise comparisons were calculated for post-hoc analysis of significant interactions. A Greenhouse-Geisser adjustment was made if the sphericity assumption was rejected by Mauchly's sphericity test. Standard deviations of arm posture, hand position and hand orientation were tested for normal distribution with the Lilliefors-test. Data was normally distributed for both conditions and for almost all sampling points. The critical value for significance was set at p < 0.05. Participants were excluded from single analyses in case of data corruption, i.e., if the data matrix for each participant contained <10 valid trials in each condition and for each target position.

## RESULTS

Since the influence of target position on complex reaching movements was not of interest in the current study, and was already discussed elsewhere (Krüger et al., 2011, 2012), only the results for reaching toward the left target position will be presented here. Similar results were found for reaching movements toward the right target position, though in general the observed differences were smaller for the right target position as compared to the left target position.

#### Temporal Movement Characteristics

Overall movement duration was 778 ± 167 ms (mean ± SD) for the ischemia condition and 713 ± 142 ms for the control condition (see **Figure 2**). This difference was significant [paired <sup>t</sup>-test: <sup>t</sup>(14) = −3.55, <sup>p</sup> <sup>&</sup>lt; 0.01] and based on a significantly increased duration of the acceleration phase under ischemia [403 <sup>±</sup> 83 vs. 352 <sup>±</sup> 84 ms, <sup>t</sup>(14) = −3.08, <sup>p</sup> <sup>&</sup>lt; 0.01]. Neither duration of the deceleration phase (375 ± 116 vs. 360 ± 101 ms), nor peak velocity (1,068 ± 198.57 vs. 1,095 ± 195 mm/s) differed between the ischemia and control condition.

## Spatial Movement Characteristics

When reaching toward the target, trajectories for five out of the seven joint angles of the arm (shoulder torsion, shoulder horizontal, shoulder vertical, elbow torsion, and wrist horizontal) showed a continuous increase or decrease between movement initiation and movement end, with the trajectories slightly curved. For elbow flexion and wrist vertical, joint angle trajectories showed a reversal in movement direction during the movement. Under ischemia, total path length in the 7D-joint space was decreased by 15% (control: 40.8 ± 6.8◦ vs. ischemia: 34.3 ± 5.4◦ ). Associated with that, the participants' mean movement amplitude decreased significantly under ischemia as compared to the control condition [26.3 ± 4.1◦ vs. 31.3 ± 4.7◦ , <sup>t</sup>(14) <sup>=</sup> 5.32, <sup>p</sup> <sup>&</sup>lt; 0.01, see **Figure 3A**]. This difference was especially pronounced in four of the seven joint angles: shoulder torsion [t(14) = 2.46, p = 0.03], shoulder vertical [t(14) = 2.95, p <sup>=</sup> 0.01], elbow torsion [t(14) <sup>=</sup> 3.93, <sup>p</sup> <sup>&</sup>lt; 0.01], and elbow flexion [t(14) <sup>=</sup> 5.50, <sup>p</sup> <sup>&</sup>lt; 0.01, see **Figure 3B**].

#### Movement Variability

First, movement variability at the different sampling points during movement execution was analyzed with respect to three

measures: (a) standard deviation of arm posture, (b) standard deviation of hand position, and (c) standard deviation of hand orientation. The amount of movement variability did not differ between the two experimental conditions (i.e., no significant main effect of experimental condition) for any of the three measures neither across the nine sampling points nor at movement end. However, for each of the three measures, a significant main effect of sampling point became evident: (a) <sup>F</sup>(2.39, 23.92) <sup>=</sup> 21.21, <sup>p</sup> <sup>&</sup>lt; 0.01, (b) <sup>F</sup>(2.36, 23.62) <sup>=</sup> 53.35, <sup>p</sup> <sup>&</sup>lt; 0.01, and (c) <sup>F</sup>(2.48, 24.83) <sup>=</sup> 22.93, <sup>p</sup> <sup>&</sup>lt; 0.01. In all cases, movement variability increased until the middle of the movement and decreased afterwards. Variability was smallest at movement initiation and on an intermediate level at movement end (see **Figure 4**).

The interaction of experimental condition × sampling point was significant for standard deviation of hand position. Qualitatively, this effect became evident as a weaker modulation of hand position variability across the nine sampling points in the ischemia condition (see **Figure 4B**). Post-hoc analysis revealed that, under ischemia, only the first two sampling points differed largely from the other sampling points, whereas in the control condition almost all sampling points differed significantly from each other (see **Table 1**). No other effects reached the level of significance.

The coupling of joint angles within the arm posture was analyzed using a principal component analysis applied to the inter-trial 7 × 7 covariance matrix of the arm posture. Under ischemia, the first two eigenvalues explained 88.90 ± 2.44% of total joint angle variance compared with 83.40 ± 2.27% in the control condition (see **Figure 5A** for group mean and **Figure 5B** for a representative participant). This difference was significant [t(8) = −18.43, <sup>p</sup> <sup>&</sup>lt; 0.01].

The coupling of the arm posture was analyzed using the coefficient of determination between of final arm posture with respect to the arm posture during movement execution. As a matter of course, the coefficient of determination increased toward movement end and finally reached the level of 1 (see

**Figure 5D** for a representative participant). The coefficient of determination of final arm posture with respect to the variance of the arm posture at the first sampling point was smaller in the control condition (R <sup>2</sup> <sup>=</sup> 0.4) than under ischemia (<sup>R</sup> <sup>2</sup> <sup>=</sup> 0.7). Consequently, the subsequent increase in the coefficient of determination up to the value 1 at movement end was steeper in the control condition than under ischemia. For group comparison, only the coefficient of determination with respect to the fifth sampling point, when the standard deviation of arm posture was maximal, was analyzed. Under ischemia the coefficient of determination was significantly higher than in the control condition [R 2 : 0.82 ± 0.18 vs. 0.54 ± 0.06, t(7) = −3.89, p < 0.01, see **Figure 5C**].

#### DISCUSSION

In the current study, we introduced a method to investigate the coupling of joint angle variability across the time course of discrete, goal-directed, natural reaching movements. The method

FIGURE 4 | Movement variability. (A) Standard deviation of arm posture, representing the mean across participants, is shown. Shaded areas represent the respective confidence intervals. (B) Standard deviation of mean hand position (+ confidence interval) is depicted. Hand position variability was less modulated under ischemia. (C) Standard deviation of mean hand orientation and the respective confidence interval is shown.

TABLE 1 | Post-hoc analysis for the significant interaction Condition × Sampling point for the measure: Standard deviation of hand position.


Data in the upper right half of the table represents p-values of significant pairwise comparisons of single sampling points for the ischemia-condition. Data in the lower left half of the table represents p-values of significant pairwise comparisons of single sampling points for the control condition. Bonferroni correction for multiple comparisons was applied to all calculations. Asterisks indicate close to significant interactions.

was exemplified on a dataset that was collected to study the influence of temporary proprioceptive deafferentation on the control of a complex reaching movement. In the following, the main outcomes of the study will be discussed first, followed by a discussion on the canonical correlation method introduced to analyze the propagation of movement variability across time.

#### Adjustment of Movement Duration

Movement duration was increased by the ischemia as a result of increased acceleration duration. The influence of proprioception on the duration of acceleration was already recognized by Bagesteiro et al. (2006) and associated with sensory-based online-correction of the movement. Movement's peak velocity was not increased under ischemia. Increased duration of acceleration without increased peak velocity indicates decreased peak acceleration and, consequently, decreased peak force. A reduction in total force applied during movement execution is accompanied by a reduction in signal-dependent noise (Harris and Wolpert, 1998). This may be advantageous under ischemia, as the precision of movement planning is of greater importance when movement online-control based on proprioceptive feedback is impaired. Our results suggest that healthy participants are able to immediately and efficiently adjust the precision of movement planning to the lack of proprioceptive information in elbow and wrist.

## Adjustment of Movement Amplitude

Movement amplitude was decreased under ischemia. Importantly, this was not only true for joints distal to the applied blood pressure cuff (i.e., elbow torsion and elbow flexion), which were directly affected by the ischemic block, but also for two joint angles proximal to the cuff (i.e., shoulder

torsion and shoulder vertical), which were not directly affected by the ischemia. In combination with the finding of stronger inter-joint coupling under ischemia, this suggests a more global change in the strategy of joint angle coordination involving all joints of the arm to compensate for the ischemia. A reason for planning a reaching movement with decreased mean movement amplitude may be the associated decrease in the signal-to-noise ratio (Harris and Wolpert, 1998), facilitating the control of movement endpoint variability. This assumption is also supported by Fitts' Law (Fitts, 1954), which describes the relationship between movement amplitude, movement duration and movement accuracy. According to this law, in order to keep movement endpoint variability constant in a task with increased task difficulty, movement duration and/or movement amplitude must be adjusted. Assuming that the ischemia may have increased task difficulty, as an

arm posture variance for each sampling point is shown for a representative participant for both conditions.

important source of sensory information was disabled, planning a movement with decreased movement amplitude and increased movement duration may have allowed the participants to keep movement endpoint variability constant, as observed in our study.

## Adjustment of Movement Variability

Another important finding of our study was that the modulation of hand position variability during movement execution was altered under ischemia in such a way that the initial increase and subsequent decrease of hand position variability was less pronounced. The increase-decrease pattern of movement variability was already described in earlier studies by our group (Krüger et al., 2011, 2012) and is a sign of successful minimization of variance at movement end. It indicates that signal-dependent noise (Harris and Wolpert, 1998), introduced by forces during the acceleration period, is successfully compensated by feedback control acting primarily during the deceleration phase (Elliott et al., 2001, 2010; Eggert et al., 2016). The fact that this increase-decrease pattern of hand position variability is less pronounced under ischemia (see **Figure 4**) is probably related to both reduced acceleration forces, resulting in a reduced increase of variability, and impaired proprioceptive feedback, resulting in a reduced decrease of variability. Interestingly, both of these changes compensated for each other in such a way that endpoint variability was almost identical in the control condition and under ischemia. This is in contrast to findings of studies with chronically deafferented patients (Gentilucci et al., 1994; Gordon et al., 1995; Nougier et al., 1996; Medina et al., 2009) and reflects the ability of the motor control system of healthy participants to immediately and efficiently adjust to the loss of proprioceptive information in parts of the effector.

#### Adjustments of Movement Coordination

Movement coordination was altered under ischemia, a finding similar to that observed in studies on deafferented patients (Sainburg et al., 1993; Ghez and Sainburg, 1995; Sarlegna et al., 2006). In the current study, the alterations in movement coordination became evident for the coupling between single joints of a specific arm posture as well as for the coupling of arm posture during movement execution with that at movement end: for both parameters the coupling was stronger under ischemia. Increasing the strength of joint angle coupling under ischemia, i.e., increasing the synergistic coordination of the redundant DoF at the same time point and across time points, may reflect a change in the control strategy concerning the way motor redundancy is used under impaired proprioceptive feedback. Alternatively, it may reflect the limited capacity of the brain to plan and control coordinated movement with a naturally higher number of DoF with decreased proprioceptive feedback. Similar to that assumption, Gupta (2014) highlighted the relevance of feedback information for the precise temporal control of complex motor actions. Independent of which explanation holds true, the observed effect of stronger joint angle coupling under ischemia can be interpreted as a reduction of the number of kinematic DoF of the redundant effectorsystem arm and consequently as a facilitation of its onlinecontrol.

### Methodological Considerations

In this study, canonical correlation analysis was introduced as a method to investigate the propagation of movement variability for discrete movements involving a redundant effector system. It was shown that, by using canonical correlation analysis, alterations in movement coordination that indicate changes in the control strategy concerning the use of motor redundancy could be detected. To the best of our knowledge, this is the first time such an approach is suggested for a multiple DoF-effector system in the context of discrete movements. Previous approaches to capture the propagation of movement variability for discrete movements have studied either eye movements (West et al., 2009; Richardson et al., 2011; Eggert et al., 2016) or arm movements with a limited number of DoF (Messier and Kalaska, 1999; Heath, 2005; Kuang and Gail, 2015). However, to advance our understanding of the human motor control system in its complexity, natural movements as used in this study have to be analyzed. In this context, the introduction of canonical correlation analysis as a suitable approach represents an important methodical advance in the context of neuromechanics. One aspect contributing to the importance of the introduced method is that it does not substitute or extend other approaches that have been previously suggested to identify structure in movement variability, but represents a new approach with the potential to also broaden the application of already existing methods and to increase their significance: In this study, it was analyzed how strongly final arm posture variability was determined by overall variability of arm posture during movement execution. Previous studies identified different components of overall movement variability (Müller and Sternad, 2004, 2009) and the relevance of independent components of effector variability for task variability (Scholz and Schöner, 1999; Latash et al., 2002). Following this line of thinking, future research could use canonical correlation analysis to investigate the propagation of certain components of overall effector variability in time.

## CONCLUSIONS

In this study, we introduced canonical correlation analysis as a method to investigate the temporal propagation of movement variability in reaching movements under normal or impaired proprioceptive feedback. As general findings, we found increased movement duration due to increased acceleration duration, decreased movement amplitude, as well as changes in movement coordination under reduced proprioceptive afference due to ischemia. The changes in movement coordination became evident as an increased coupling between arm postures during movement execution with final arm posture, resulting in a decreased number of kinematic DoF of the effector-system. Movement endpoint variability was not influenced by the ischemia. Thus, the canonical correlation analysis revealed that healthy participants are able to immediately and efficiently adjust their control strategy to the impaired flow of proprioceptive information. In conclusion, canonical correlation analysis provides a valuable method to advance our understanding of human movement control by offering an approach to analyze the temporal propagation of movement variability during discrete movements executed by multiple DoF-effector systems, such as the arm.

## AUTHOR CONTRIBUTIONS

All authors of this study contributed substantially to its conception, the interpretation of data, as well as drafting and revising earlier versions of the manuscript. In addition, MK and TE were responsible for data acquisition and analysis. MK, AS, and TE all approved the version to be published and agreed to be accountable for all aspects of the work.

#### FUNDING

This work was financially supported by the Research Training Group (GRK) 1091 "Orientation and motion in space" of the

#### REFERENCES


German Research Foundation (DFG) as well as by the Robert Bosch Foundation (Grant No. 32.5.G412.0007.0).

### ACKNOWLEDGMENTS

A modified version of this manuscript has been included as part of the dissertation thesis of MK (Krüger, 2013).


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Krüger, Straube and Eggert. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Predictive Simulation of Reaching Moving Targets Using Nonlinear Model Predictive Control

Naser Mehrabi\*, Reza Sharif Razavian, Borna Ghannadi and John McPhee

Systems Design Engineering, University of Waterloo, Waterloo, ON, Canada

This article investigates the application of optimal feedback control to trajectory planning in voluntary human arm movements. A nonlinear model predictive controller (NMPC) with a finite prediction horizon was used as the optimal feedback controller to predict the hand trajectory planning and execution of planar reaching tasks. The NMPC is completely predictive, and motion tracking or electromyography data are not required to obtain the limb trajectories. To present this concept, a two degree of freedom musculoskeletal planar arm model actuated by three pairs of antagonist muscles was used to simulate the human arm dynamics. This study is based on the assumption that the nervous system minimizes the muscular effort during goal-directed movements. The effects of prediction horizon length on the trajectory, velocity profile, and muscle activities of a reaching task are presented. The NMPC predictions of the hand trajectory to reach fixed and moving targets are in good agreement with the trajectories found by dynamic optimization and those from experiments. However, the hand velocity and muscle activations predicted by NMPC did not agree as well with experiments or with those found from dynamic optimization.

#### Edited by:

Massimo Sartori, University of Göttingen, Germany

#### Reviewed by:

Leonardo Abdala Elias, University of Campinas, Brazil Ton Van Den Bogert, Cleveland State University, USA

#### \*Correspondence: Naser Mehrabi

nmehrabi@uwaterloo.ca

Received: 14 September 2016 Accepted: 20 December 2016 Published: 13 January 2017

#### Citation:

Mehrabi N, Sharif Razavian R, Ghannadi B and McPhee J (2017) Predictive Simulation of Reaching Moving Targets Using Nonlinear Model Predictive Control. Front. Comput. Neurosci. 10:143. doi: 10.3389/fncom.2016.00143 Keywords: reaching, NMPC, prediction horizon, motor control

## INTRODUCTION

The human central nervous system (CNS), consisting of brain, and spinal cord, is responsible for controlling and maintaining body motions. As first formulated by Bernstein (1967), the CNS simultaneously coordinates the kinematics and kinetics of body motions, despite uncertain/unknown (future) trajectories and the redundancy in muscle actuators. As an example, in a goal-directed planar reaching task, where only the final position of the hand is specified, an infinite solution set of hand trajectories and muscle activation patterns exist to reach the final position. The early observations of reaching and pointing tasks led to the well-known "Minimum-X" models [e.g., minimum-jerk model (Flash and Hogan, 1985; Wada et al., 2001), minimum-torque-change model (Uno et al., 1989), minimum-variance model (Harris and Wolpert, 1998), and minimumwork model (Soechting et al., 1995)] to predict the hand trajectory. These models hypothesize that the CNS coordinates the body movement such that an exertion (X) is minimized. Later, this hypothesis is extended to consider physiologically-motivated exertions such as muscle activation effort (Crowninshield and Brand, 1981; Happee and Van der Helm, 1995; Ackermann and van den Bogert, 2010), metabolic energy expenditure (Anderson and Pandy, 2001; Peasgood et al., 2006), and muscle fatigue (Sharif Razavian and McPhee, 2015).

In computer simulations, the Minimum-X model has been successfully implemented using dynamic optimization (DO) to predict the average human motion for a given task. A common DO approach parameterizes the muscle activation profiles for the period of motion and searches the feasible space to find the profiles that minimize X (Davy and Audu, 1987; Yamaguchi and Zajac, 1990; Neptune and Hull, 1998; Anderson and Pandy, 2001; Kaplan and Heegaard, 2001; Sha and Thomas, 2013; Kistemaker et al., 2014). This approach provides an open-loop (feedforward) command of muscle activations to control the given task. This command can represent the descending command of a wellrepeated/well-learned task [e.g., platform diving (Koschorreck and Mombaur, 2011)]. In this model, the CNS only recalls the learned information, and does not intelligently adjust the commands in real-time. However, during conscious voluntary movements, the CNS has to continuously update the motor commands to correct for errors (Todorov, 2004). For example, previous studies (Sarlegna and Pratik, 2015) on pointing and reaching have shown that the CNS constantly updates the hand trajectory based on sensory (feedback) information. This sensory information can be received from vision, proprioception, audition, the vestibular system, and internal models that can predict the motion (Desmurget and Grafton, 2000).

Dynamic optimization implementation of minimum-X models raises an interesting question: does the CNS predict the trajectory at the beginning of the motion? Or does it constantly readjust the trajectory? If the latter is true, how far in advance does the CNS predict the motion, and how does that affect the motion? This article focuses on these questions and provides a computational platform to study the effects of the prediction horizon using optimal feedback control theory. Optimal control methods have been previously used to find a unique solution for motor coordination (Meyer et al., 1988; Loeb et al., 1990; Sporns and Edelman, 1993; Kuo, 1995; Anderson and Pandy, 2001; Todorov and Jordan, 2002b; Liu and Todorov, 2007); however, there are few applications of optimal feedback control to a nonlinear redundantly-actuated musculoskeletal model. The LQR (linear quadratic regulator) and LQG (linear quadratic Gaussian) control methods have been applied to a linear arm model to describe the hand trajectory (Harris and Wolpert, 1998; Todorov and Jordan, 2002a; Liu and Todorov, 2007). Later, to control the nonlinear dynamics of the neuromuscular system, an iterative LQG (iLQG) controller has been developed, in which the nonlinear model is iteratively linearized (Todorov and Li, 2005).

In the present research, a nonlinear model predictive controller (NMPC) with a finite prediction horizon is employed. Predicting infinitely into the future is highly improbable in humans, and a finite prediction horizon allows more realistic simulation. The NMPC allows us to consider the complexity and nonlinearity of the musculoskeletal system without compromising the accuracy and optimality, as occurs using model linearization. It can be formulated to simulate trajectory tracking and goal-oriented tasks with both fixed and moving targets where it only corrects the deviations from the task goal (Todorov and Jordan, 2002a). The NMPC is a simultaneous control method because the optimal trajectory and its required muscular activities are calculated at the same time. To the best of the authors' knowledge, this work is the first use of NMPC for fully predictive simulation of human reaching tasks.

In this research, we are not focused on the source of the sensory information; we assume that the current biomechanical states (posture and velocities) are available to the CNS when necessary. This assumption seems to be valid for healthy individuals, as a wide range of sensory organs is available to sense and transmit information to the CNS. However, a pathological condition might limit the CNS access to this available sensory information. For instance, in a deafferented patient, the sense of position (and therefore the motor skills) is largely lost due to the loss of somatosensory inputs (Bringoux et al., 2016).

This paper is organized as follows. In the Method section, the experimental procedure, the planar arm model, the nonlinear model predictive controller, and forward dynamic simulation framework are provided. Next, in the Results and Discussion section, the use of NMPC as the motor control unit in human reaching tasks is presented and discussed. This study investigates the use of anticipatory planning with continuous error correction by the CNS during reaching tasks. The first goal of this study is to study the effects of varying the prediction horizon on the hand trajectory and muscle activities in a reaching task. Therefore, we ran a number of NMPC simulations with various prediction horizons, as well as a DO simulation to obtain a "gold standard" for comparison. Secondly, the capability of the NMPC as an optimal feedback controller for tracking predefined trajectories has been investigated. This ability is useful when an expected/desired trajectory is available. Lastly, the effectiveness of the proposed NMPC for the simulation of reaching to moving targets is studied. We hypothesize that the anticipatory behavior of the CNS can be modeled by NMPC and verified by comparing the hand trajectory predicted by NMPC to those collected in experiments. Finally, Conclusions and Future Work are presented.

## METHODS

## Experiments

To examine the accuracy of the NMPC predictions, a 27 year old male subject was selected to perform reaching tasks. An Optotrak Certus motion capture system (Northern Digital Inc., NDI) was used to measure the arm trajectory at 30 Hz. In these experiments, the subject was seated with the arm elevated at the shoulder level. An active marker attached to the back of the hand (as shown in **Figure 1A**) has been used to capture centerout hand motion trajectories. The subject was asked to move his hand from an initial central position to one of eight final targets spread evenly on a circle of 20 cm radius at a self-selected convenient speed. The experiment was repeated 10 times for each target with 2 min rest intervals between each set. The subject also performed reaching to moving targets. He was instructed to reach to a target, which was relocated to another position midway through the movement. The subject was instructed to adjust his motion to reach to the moving target. The subject was also given 5 min to rest before performing the reaching moving targets experiment.

During the reaching trails, the electromyography (EMG) activity of seven muscles (anterior/middle/posterior deltoid, long/lateral triceps brachii, biceps brachii, and brachioradialis) were collected at 2000 Hz using a Trigno portable EMG system (Delsys Inc.). The EMGs were band-pass filtered (5–800 Hz cutoff), rectified, low-pass filtered (4 Hz cut-off), and normalized to maximum voluntary contractions (MVCs). We performed a Pearson correlation analysis to investigate the correlation between the EMGs and muscle activity predicted by NMPC and DO simulations. We resampled both the captured EMGs and simulation results with a sampling rate of 100 Hz, and performed the Pearson correlation analysis using the "Corr" command in MATLAB. The experiments have been approved by the Office of Research Ethics at the University of Waterloo and carried out with written informed consent from the subject. The subject gave written informed consent in accordance with the Declaration of Helsinki.

#### Planar Arm Model

In this research, a planar arm model similar to the one developed by Ghannadi et al. (2015) was used. The model consisted of torso, upper arm, and forearm to simulate the hand motion. The torso was fixed and the shoulder and elbow were modeled using revolute joints. Six muscle groups including shoulder and elbow mono-articular flexors/extensors and two bi-articular flexors/extensors were used to actuate the arm as shown in **Figure 1B**. A modified Hill-type muscle model with muscle excitation-to-activation dynamics was used to simulate the skeletal muscle contraction dynamics (see Appendix A for details). The muscle parameters of the planar arm model (i.e., insertion and origin positions, maximum isometric force, fiber optimal length, slack length, and pennation angle) were tuned to represent the dynamics of the upper extremity in the experimental condition (reaching targets in a horizontal plane elevated at the shoulder level). These parameters were tuned through a series of optimizations so that the planar model provides the same joint torques as a high-fidelity three-dimensional upper extremity model (Ghannadi et al., 2015). Kinematic and dynamic parameters of the arm model and the Hill muscle model parameters used here can be found in **Tables 1**, **2**, respectively.

## Principles of Nonlinear Model Predictive Control

The arm motion is controlled by complex commands descending from the CNS, which are the combination of the motion prediction (feedforward control) from an internal representation


\*About the center of mass, the mechanical y axis is assumed to be perpendicular to the plane of movement.

#### TABLE 2 | Hill-type muscle model parameters.


The details of how the upper-extremity muscle groups are lumped into these representative muscles can be found in Ghannadi et al. (2015).

of the body and environment (or so-called internal model; Desmurget and Grafton, 2000), and the corrective command from the sensory organs to correct any errors due to uncertainty or unknown environment (feedback control). This complexity is captured here by a model-based NMPC with a receding horizon. The NMPC uses a control-oriented model (COM) representing the human's internal model to predict the optimal trajectory, and feedback information to correct the prediction errors. The NMPC predicts the optimal dynamics of the system (x¯, u¯) over a prediction horizon as shown in **Figure 2A** by minimizing the following cost function:

$$J = \Psi \left( t\_0 + t\_{ph} \right) + \int\_{t\_0}^{t\_0 + t\_{ph}} \psi \left( \varkappa \left( t \right), \mu \left( t \right) \right) dt \tag{1}$$

$$\text{subject to: } 0 < \mu(t) < 1 \tag{2}$$

where 9 is the cost evaluated at the end of prediction horizon, ψ is the cost evaluated during the prediction horizon, and tph is the length of prediction horizon. The state variables at the current time (to) are obtained from the current sensory information. The input (u¯) is an optimal open-loop solution over the prediction horizon. If there are no external disturbances and no model uncertainty in the system, with infinitely long prediction horizon, the open-loop solution can be applied to the system for all time t > to. However, for the finite horizon case and in the presence of noise and uncertainty, the open-loop solution should only be applied until the next sampling time (t<sup>0</sup> + δ). At the new time step, the optimal solution is re-evaluated with the new initial conditions for the receding horizon and iteratively applied to the system. By incorporating the feedback information, the NMPC is converted from a completely open-loop controller to an optimal closed-loop controller. The NMPC can handle constraints on both the states and the inputs. In musculoskeletal models, the muscle activation command must be non-negative and less than one, and constraints on states can be added to avoid unphysiological movements.

#### Mathematical Formulation of NMPC

In this article, the optimal dynamics over the prediction horizon were calculated using the GPOPS-II optimal control package that utilizes an orthogonal collocation method (Patterson and Rao, 2014). This method is a direct (simultaneous) optimization method in which both states (x) and inputs (u) are parameterized using a series of connected Legendre polynomials and become part of a Nonlinear Programing (NLP) problem. Here, the arm model described in the previous section was used as the COM of the NMPC and the simulation model for planar reaching and pointing tasks. The dynamic equation of the arm model can be described by:

$$\dot{\boldsymbol{x}} = \boldsymbol{f}\left(\boldsymbol{\varkappa}\left(\boldsymbol{t}\right), \boldsymbol{\mu}\left(\boldsymbol{t}\right)\right), \quad \boldsymbol{\varkappa}\left(\boldsymbol{0}\right) = \boldsymbol{\varkappa}\_{0} \tag{3}$$

where x ǫ R 10×1 are the arm model state variables consisting of shoulder angle and angular velocity plus elbow angle and angular velocity, and muscle activation states, and x<sup>0</sup> is the vector of the initial states. The muscle excitation inputs u ǫ R 6×1 represent the ratio of excited motor units to the maximum number of motor units in that muscle.

In this research, for the particular case of a goal-directed reaching task, the terminal cost of the NMPC cost function (9) was removed, and the integral part (ψ) is computed from the summation of two terms: (i) a choice of specific physiological cost function, and (ii) a trajectory tracking error. Therefore, the NMPC cost function shown in (1) is converted to:

$$J = \int\_{t\_0}^{t\_0 + t\_{ph}} \left( p(\zeta(t) - \zeta\_{des})^2 + q \, G^M(u \,\left(t\right)) \right) \, dt \tag{4}$$

where p and q are cost function weightings, and ζ and ζdes are the hand position and its desired final value (in the Cartesian coordinate system), respectively. The simulated hand position ζ varies on the prediction horizon, while the desired final value

position trajectory. The hand moves from its original position to the marked position on its left.

ζdes was kept constant. The hand position is calculated from.

$$\begin{aligned} \xi &= \left[ L\_1 \cos(\theta\_1) + L\_2 \cos(\theta\_1 + \theta\_2) \right. \left. L\_1 \sin(\theta\_1) \right. \\ &+ L\_2 \sin(\theta\_1 + \theta\_2) \right]^T \end{aligned} \tag{5}$$

where θ<sup>1</sup> and θ<sup>2</sup> are shoulder and elbow angles, and L<sup>1</sup> and L<sup>2</sup> are upper arm and forearm lengths, respectively. The physiological cost G <sup>M</sup> (u (t)) is defined as:

$$G^{\mathcal{M}} = \mathfrak{u}^2 \tag{6}$$

The term G <sup>M</sup> in the cost function represents the neural excitation effort to perform the reaching tasks.

GPOPS-II finds the optimal dynamics of each given horizon by minimizing Equation (4) while satisfying inequality constraints related to muscle excitation (2) and equations of motion (3) using the Sparse Nonlinear Optimization (SNOPT) solver (Gill et al., 2005). An hp-adaptive mesh refinement method (Liu et al., 2015) has been used within GPOPS-II to refine the individual interval widths and the polynomial degree to reach a final optimal solution. Then, the first five-percent (e.g., 50 for 1000 ms prediction horizon) of optimal activations are applied to the muscles, and the arm motion is simulated. The new position and orientation of the arm are measured and sent back to the NMPC as initial conditions of the next iteration. In this research, we have assumed that the sensory organs can measure the exact joint angles and angular velocities. Uncertainty can be added to the measurements to account for the noise within the sensory organs, and to simulate the variability in the movement repetition. However, this has not been included in the scope of this work. The optimal muscle activations are shifted and considered as the initial guess of the next iteration.

#### Dynamic Optimization

In addition to the NMPC simulations, a dynamic optimization (DO) using GPOPS-II was performed to simulate the same task. Unlike the NMPC simulations, which continue until the position tracking error passes a certain threshold, the final simulation time and the final position of the hand are explicitly specified in the DO simulations. The DO cost function is:

$$J = \int\_0^{t\_f} G^M(\mu\_\cdot(\mathbf{t})) \, dt \tag{7}$$

subject to : 0 < u(t) < 1, and x tf = x<sup>f</sup> (8)

where t<sup>f</sup> is the final simulation time, and x<sup>f</sup> is the state vector corresponding to the target position. The same physiological cost function as in the NMPC simulations (Equation 6) was used in DO to compute the optimal hand position trajectory and muscle activations.

#### RESULTS

#### Effects of the Prediction Horizon Length

In this section, a goal-directed reaching task is simulated and the effect of prediction horizon length variation on the hand trajectory and muscle activation is studied. Here, the hand is initially at rest and in a natural position (θ<sup>1</sup> <sup>=</sup> <sup>44</sup><sup>o</sup> and <sup>θ</sup><sup>2</sup> <sup>=</sup> <sup>58</sup><sup>o</sup> ) and moves toward a target 20 cm to the left of its initial position as shown in **Figure 2B**. This task was simulated using NMPC with 0.2, 0.3, 0.4, 0.5, and 0.8 s prediction horizons, and DO with a fixed time duration. In the NMPC simulations, the cost function weightings (p = 20 and q = 1) were kept the same. The DO final simulation time was chosen to be 1.5 s, in accordance with the experimental reach duration (1.431 ± 0.176 s in 10 repetitions).

**Figure 2B** demonstrates the hand trajectories for the aforementioned prediction horizons (tph). Despite the slight differences between the trajectories, the solutions with 0.4, 0.5, and 0.8 s prediction horizons closely correlate with the experimental results. The Pearson correlation factors between NMPC simulations and the experimental trajectories are 0.9351, 0.9758, 0.9947, 0.9991, 0.9994 respectively for 0.2, 0.3, 0.4, 0.5, 0.8 s. The Pearson factor between the DO and experiment is 0.867.

As shown in **Figure 3**, the elbow and shoulder angle variations with 0.2 s prediction horizon are less than those with longer prediction horizons. This signifies the importance of prediction horizon; with short prediction horizons, the controller is more cautious and takes longer to reach a desired position. Simultaneously, this results in smaller muscle activations for shorter horizons since the transient time to reach the final position is longer (see **Figure 4**). The reaching error at 1.5 s of the simulation is about 23% for 0.2 s prediction horizon, and reduces to 0.36% for 0.8 s prediction horizon. As expected for this motion, the shoulder mono-articular flexor (muscle 1), elbow mono-articular flexor (muscle 3) and bi-articular flexor (muscle 5) are activated at the beginning to accelerate the body; then, the antagonistic muscles are activated to reach a full stop at the desired position.

In the DO, similar to NMPC, flexor muscles are active at the beginning of the motion to accelerate the hand toward the target, then the extensor muscles are activated to stop the hand movement (a bang-bang control strategy). Since DO has to stop at the specified final time (1.5 s) the extensor muscle activities are larger than NMPC predictions at the decelerating phase of motion. On the other hand, the flexor muscle activities at the accelerating phase of the simulation are larger for NMPC than DO because the trajectory error at the beginning of the motion is large and exponentially reducing when it gets closer to target.

As shown in **Figure 4**, the muscle activations predicted by NMPC and DO simulations can capture the general trends of the experimental measurements. It can be observed that as the prediction horizon increases, the NMPC predicts larger muscle activities at the beginning of the motion. A Pearson correlation analysis was performed between the flexor muscle activities predicted by the NMPC and DO simulations and the EMGs from experimental measurements; the correlation coefficients are presented in **Table 3**. The correlation coefficient for extensor muscles are not reported since the EMG activity of these muscles were minimal in the experiments. As shown in **Table 3**, the correlation coefficient of flexor muscles reduces when the prediction horizon increases in the NMPC simulations, while the DO predictions correlate better with the experiments.

FIGURE 4 | Optimal activations of flexor and extensor muscles: (A) Optimal activation of shoulder mono-articular flexor (muscle 1), (B) Optimal activation of elbow mono-articular flexor (muscle 3), (C) Optimal activation of bi-articular flexor (muscle 5), (D) Optimal activation of shoulder mono-articular extensor (muscle 2), (E) Optimal activation of elbow mono-articular extensor (muscle 4), (F) Optimal activation of bi-articular extensor (muscle 6)

TABLE 3 | Pearson correlation analysis of flexor muscle activations predicted by NMPC and DO vs. experimental measurements.


As suggested by Morasso (1981), subjects tend to move in straight lines with bell-shaped tangential-velocity profiles when reaching for a target. **Figure 2B** shows that the NMPC with long-enough prediction horizon can realistically predict the hand trajectory, while **Figure 5** shows the differences in the velocity profiles of the NMPC and DO predictions. In these simulations, the NMPC tends to accelerate the hand more quickly than the DO; in DO, the optimization knows the final time and can distribute the acceleration over a longer time. On the contrary, in the receding horizon NMPC, the controller can only predict the motion as far as the prediction horizon. This results in the fast acceleration at the beginning the motion due to large tracking errors and slow deceleration at the end of motion due to small tracking errors.

## Reaching Targets Using a Predefined Trajectory

Reaching to eight different directions was also simulated using NMPC with 0.8 s prediction horizon. The target positions of the hand are located on a circle centered at the initial position of the hand with a radius of 20 cm (see **Figure 1**). In the NMPC simulations, a smooth 5th-order polynomial with zero initial and final velocities and accelerations is used as the desired straight-line hand trajectory. These trajectories begin at the initial position of the hand and end at the target positions. As shown in **Figure 6**, the NMPC is able to follow the desired trajectories, which are qualitatively correlated with the measured trajectories in experiments.

## Reaching a Moving Target

We have assumed that the CNS plans a trajectory to reach a target and constantly monitors the deviations from this trajectory and the target position. In this section, we study the case where the target position is suddenly relocated. Here, the hand is initially at rest at point O (**Figure 7A**) and moves toward the target at point A. Then 1 s later, the target position suddenly moves to the point B. This protocol was achieved in the lab by manually moving the target from its initial point A to B when the subject reached half the way to A. In this simulation, the time delay related to the visual cognition of this change [about 150 ms, (Jeannerod, 2006)] has not been considered.

**Figure 7** depicts that the NMPC controller can track the location of the target and correct the hand trajectory to reach the new target. As shown in **Figure 7A**, it seems that when the target moves, the subject over-compensates by moving the hand to the right, while the NMPC finds a trajectory that minimizes both position error and control effort. It is not possible to simulate this scenario with DO; it is one of its disadvantages compared to NMPC, which is able to make online adjustments to the hand trajectory.

## DISCUSSION

In this research, we presented a NMPC to mimic the human motor control system. The results showed that it can successfully replicate certain features of human motor control such as path planning and target tracking. This controller is a fully predictive

left of its initial position.

optimal controller that simultaneously solves the kinematic redundancy and muscle-sharing problem.

In hierarchical models (Menegaldo et al., 2006; Guigon et al., 2007; Mehrabi et al., 2015), the computational burden is reduced by separating the computation into two steps. In the first step, an optimal trajectory is generated based on a kinematic criterion; then, in the second step, the muscle sharing problem is solved based on another criterion (kinetic criterion). The main drawback of hierarchical models is that they follow a preplanned trajectory (output of the first step); therefore the online movement corrections are in favor of trajectory tracking. However, the NMPC controller simultaneously takes into account both kinetic and kinematic exertions to determine an optimal path along with the optimal muscle activations to achieve it. Therefore, it can be argued that this controller is more similar to the human CNS, as it receives proprioceptive information to adjust the predicted trajectory that satisfies the new condition and in favor of the end goal.

Thelen et al. (2003) developed a feedback/feedforward controller (computed muscle control, CMC) that uses inverse dynamics and static optimization to find muscle activities that track a set of desired kinematics. However, such an approach is applicable only if the kinematics is known or if there is a desired kinematics. One advantage of NMPC is its ability to control the motion with and without a prescribed motion, or when the target position moves.

In our study and for the first time, the effect of varying the prediction horizon on the path planning ability in reaching tasks has been investigated. As expected, increasing the prediction horizon improves the tracking performance, but makes the solution computationally more expensive. The prediction horizon length can be adjusted to capture the characteristic of a desired motion. Here, simulation results showed that the resultant hand trajectory with long enough prediction horizons resembles those found from the experiments. However, NMPC accelerates the hand faster and decelerates it slower than the bell-shaped speed trajectories reported by Morasso (1981) and observed in our experiments. This can be due to the fact that the reach time is not specified in the NMPC, or due to the selection of minimum control effort as the physiological cost function. The proposed NMPC is not limited to the suggested cost function; various cost functions can be implemented. For example, Kistemaker et al. (2014) studied the effect of different cost functions on the trajectory of the hand while performing a reaching task using a DO approach.

The Pearson correlation coefficients reported in **Table 3** show that muscle activities predicted by an NMPC with a short horizon can better predict (in a temporal sense) the experimental measurements. In contrast, the hand trajectory predictions (in a spatial sense) of an NMPC with larger prediction horizons can more closely replicate those from the experiments as shown in **Figure 2B**. Finally yet importantly, the NMPC simulations can be used to reproduce the experimental hand trajectories with a moving target as shown in **Figure 7**. The differences between the experiments and simulations may be due to the subject anticipation of another movement of the target point, while the final target position is known to the controller immediately following the shift. This unique feature of NMPC simulations can advance our theoretical understanding of hand movements and enables the next generation of assistive.

## Online Implementation of NMPC

The focus of this paper has been on the proof-of-concept of the NMPC as a possible model for CNS control of human movement. The current implementation of this approach is computationally expensive and is not real-time. For instance, with a prediction horizon of 0.5 s, the NMPC takes 0.45 ± 0.24 s to find the optimal dynamics at each time step and re-plan the movement. These simulations were performed on a computer with an Intel CoreTM i7-4790 processor and CPU 3.60 GHZ and RAM 16 GB. However, online NMPC methods such as the Continuation/GMRES method (Ohtsuka, 2004), advanced-step NMPC (Zavala and Biegler, 2009), and explicit MPC (Kouramas et al., 2013) can be used to achieve real-time performance. As an example, Mehrabi et al. (2016) developed a Newton/GMRES NMPC controller to control the functional electrical stimulation of knee extension. This controller by discretising the system dynamics and employing a fast online optimization method (GMRES), significantly reduced the computational time and allowed the real-time implementation of the NMPC. Furthermore, the COM can be further simplified using muscle synergy theory, in which the CNS coordinates human body movements by bundling individual muscles into groups. This allows a low-dimensional control input that significantly reduces of the size of the control problem (Sharif Razavian et al., 2015).

#### Conclusions and Future Work

In this research, the first use of NMPC to simulate human motor control in reaching movements was presented. It was shown that NMPC can replicate certain properties of the human motor control system (i.e., path-planning, prediction, and target tracking), and can be used to realistically simulate reaching movements. Due to its feedback nature, it can correct the tracking errors for static targets or can follow a moving target seamlessly. The NMPC prediction horizon can represent the time horizon for which the CNS minimizes a physiological cost function. It should be noted that the NMPC conclusions from this research are specific to the cost function used here; stronger conclusions can only be made if more diverse cost functions are investigated. Nonetheless, this method opens up new opportunities to study challenging problems such as predictive forward dynamic simulation of biomechanics and biomechatronic systems.

#### REFERENCES


As a possible future research direction, an online NMPC can be used to represent a user/patient in an assistive devices controller to facilitate the shared control between the device and user. This shared control allows the device to perform some tasks independently of the user by sensing information about the environment (Millán et al., 2010). By predicting the motion of the user and adjusting the trajectory online, the NMPC can reduce the cognitive workload imposed on the user, who does not need to consider low-level executions in the presence of external disturbances or obstacles (Tucker et al., 2015). The variability in limb movement is another known characteristic of reaching movements. This characteristic can be incorporated in the NMPC assistive device by accounting for noisy sensory information and sending noisy motor commands to musculotendon units.

### AUTHOR CONTRIBUTIONS

All authors listed, have made substantial, direct and intellectual contribution to the work, and approved it for publication.

#### ACKNOWLEDGMENTS

The authors would like to thank the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Canada Research Chairs program for financial support of this research.


reduction. J. Franklin Inst. 352, 4081–4106. doi: 10.1016/j.jfranklin.2015. 05.028


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Mehrabi, Sharif Razavian, Ghannadi and McPhee. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

### APPENDIX A

#### Hill-Type Muscle Model

In this research, a Hill-type muscle model is used to simulate the muscle contraction dynamics. The Hill muscle model shown in **Figure A1A** has three elements: contractile element (CE), parallel elastic element (PE), and series elastic element (SE) (Thelen et al., 2003). In this work, we assume that the SE length is constant during the motion; therefore, the SE is replaced with an inextensible string (see **Figure A1B**). To justify this assumption, we performed an analysis (in Appendix B) that shows that the SE length variation during an NMPC simulation of a straight-line reaching task is negligibly small.

With this assumption, the musculotendon force is simplified to

$$F\_{TM} = F\_0^{\max} \left( F\_{PE}(t, L\_M) + F\_{CE}(t, a, L\_M, V\_M) \cdot \cos \left( \alpha\_p \right) \right) \tag{A1}$$

where F max 0 , FCE, and FPE are the maximum isometric force, CE and PE forces, and α<sup>p</sup> is the muscle pennation angle. Here, L<sup>M</sup> and V<sup>M</sup> represent muscle fiber length and velocity. Muscle fiber length is defined as L<sup>M</sup> = (LTM – LSE) cos(αp) where LTM and LSE are total length of musculotendon unit and slack length of tendon, respectively. The force generated by FCE can be separated into force-length and force-velocity relations scaled by the muscle activation command (a):

$$F\_{\rm CE} = a(t)F\_{\rm CE}^L(t, L\_M)F\_{\rm CE}^V(t, a, L\_M, V\_M) \tag{A2}$$

where the force-length (F L CE) and force-velocity (F V CE) relations are:

$$F\_{CE}^L = \text{e} \begin{pmatrix} L\_M \\ L\_M^{opt} \end{pmatrix}^2 / \nu \tag{A3}$$
 
$$\begin{cases} \qquad V\_M \end{cases}\_{\text{MAX}} \tag{A3}$$

$$F\_{CE}^V = \begin{cases} \frac{V\_M}{V\_M^{max} L\_M^{opt}} + A V\_M^{max} \\ - \frac{V\_M}{V\_M} \\ \frac{V\_M^{max} L\_M^{opt} A\_f + A V\_M^{max}}{V\_M B \dot{F}\_{max}^{len}} \\ \frac{V\_M B \dot{F}\_{max}^{len}}{V\_M^{max} L\_M^{opt}} + A C V\_M^{max} \\ - \frac{V\_M B}{V\_M^{max} L\_M^{opt} A\_f + A C V\_M^{max}} \end{cases} \quad \text{(A4)}$$

where γ, A, B, and C are shape factors, V max <sup>M</sup> is the maximum fiber velocity, L opt <sup>M</sup> is the optimal length of fiber at which FCE is a maximum, and F¯ len max is the maximum normalized muscle force during lengthening. The numerical values of the muscle parameters are taken from (Thelen, 2003).

The Parallel Elastic force of muscle (FPE) is represented by an exponential function:

$$F\_{PE} = \frac{\mathbf{e}^{k\_{\rm p\epsilon}} \left(\frac{L\_M}{L\_M^{\rm opt}} - 1\right) / \epsilon\_0^m}{\mathbf{e}^{k\_{\rm p\epsilon}} - 1} - 1\tag{A5}$$

where kpe (= 0.5) is a shape factor and ǫ m 0 is passive muscle strain due to maximum isometric force.

In this research, a first-order differential equation based on He et al. (1991) is used to simulate muscle excitation-to-activation dynamics. In this case, muscle activation (a) is related to the excitation (u) as follows:

$$
\dot{a} = (u - a)(t\_1 u + t\_2) \tag{A6}
$$

where u and a are muscle excitation and activation respectively, and t<sup>1</sup> and t<sup>2</sup> are defined as follows:

$$t\_2 = \frac{1}{\mathfrak{r}\_{\rm fall}} \quad \text{and} \quad t\_1 = \frac{1}{\mathfrak{r}\_{\rm rise}} - t\_2 \tag{A7}$$

where τfall is the deactivation time constant (=50 ms), and τrise is the activation time constant (=15 ms).

#### APPENDIX B

In this section, we have computed the SE strain using the optimal muscle activations from an NMPC simulation. Since the SE element (representing the tendon) is in series with the PE and CE elements of the Hill muscle model, the tendon force is equal to the muscle fiber force, and to that of the musculotendon unit. Therefore, we used the musculotendon force computed in NMPC simulations (with the prediction horizon of 0.8 s) as the tendon force. Then, based on the tendon force-length relation described in Thelen (2003), the tendon

strain was calculated using following equation:

$$F^T = F\_0^{\max} \left\{ \frac{\vec{F}\_{\text{tot}}^T}{\epsilon\_{\text{tot}}^{k\_{\text{tot}}} - 1} \left( \epsilon\_{\text{tot}}^{k\_{\text{tot}} \cdot \epsilon^T / \epsilon\_{\text{tot}}^T} - 1 \right); \ \epsilon^T \le \epsilon\_{\text{tot}}^T \right. \tag{A8}$$

where F T is the tendon force, ktoe (=3) is an exponential shape factor, <sup>K</sup>lin (<sup>=</sup> 1.712/ǫ<sup>T</sup> 0 ) is a linear scale factor, F¯ <sup>T</sup> toe (=0.33), ǫ T 0 (=0.033) is tendon strain due to maximum isometric force, and ǫ T toe(= 0.609ǫ T 0 ) is the tendon strain above which the tendon exhibits linear behavior. The tendon strain ǫ T is defined as ǫ <sup>T</sup> <sup>=</sup> LT−LSE LSE . To find the tendon strain, the first term of the piecewise equation (B.1) was used. If the calculated strain was within the linear region (less than ǫ T toe) the strain value is valid; otherwise the second term of (B.1) was used to calculate the tendon strain.

**Figure A2** shows the tendon strain variations during the NMPC simulation of reaching with a 0.8 s prediction horizon. The tendon strains are less than 0.5%; therefore, the SE element of the Hill muscle model can be neglected in the planar arm model.

# Evaluation of a Neuromechanical Walking Control Model Using Disturbance Experiments

Seungmoon Song\* and Hartmut Geyer

*Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA*

Neuromechanical simulations have been used to study the spinal control of human locomotion which involves complex mechanical dynamics. So far, most neuromechanical simulation studies have focused on demonstrating the capability of a proposed control model in generating normal walking. As many of these models with competing control hypotheses can generate human-like normal walking behaviors, a more in-depth evaluation is required. Here, we conduct the more in-depth evaluation on a spinal-reflex-based control model using five representative gait disturbances, ranging from electrical stimulation to mechanical perturbation at individual leg joints and at the whole body. The immediate changes in muscle activations of the model are compared to those of humans across different gait phases and disturbance magnitudes. Remarkably similar response trends for the majority of investigated muscles and experimental conditions reinforce the plausibility of the reflex circuits of the model. However, the model's responses lack in amplitude for two experiments with whole body disturbances suggesting that in these cases the proposed reflex circuits need to be amplified by additional control structures such as location-specific cutaneous reflexes. A model that captures these selective amplifications would be able to explain both steady and reactive spinal control of human locomotion. Neuromechanical simulations that investigate hypothesized control models are complementary to gait experiments in better understanding the control of human locomotion.

#### Edited by:

*Manish Sreenivasa, Heidelberg University, Germany*

#### Reviewed by:

*Boris Prilutsky, Georgia Institute of Technology, USA Jacques Duysens, KU Leuven, Belgium*

#### \*Correspondence:

*Seungmoon Song smsong@cs.cmu.edu*

Received: *23 December 2016* Accepted: *28 February 2017* Published: *14 March 2017*

#### Citation:

*Song S and Geyer H (2017) Evaluation of a Neuromechanical Walking Control Model Using Disturbance Experiments. Front. Comput. Neurosci. 11:15. doi: 10.3389/fncom.2017.00015* Keywords: neuromechanical simulation, human locomotion, spinal control, model evaluation, spinal reflex, central pattern generator

## 1. INTRODUCTION

Understanding the control that underlies human locomotion remains a challenging problem. One reason for this is that many experimental techniques provide only incomplete access to the control circuits, making it impossible to directly probe the entire control involving millions of neurons in complex animals (Vogelstein et al., 2014). Another reason is that the control mechanism seems to vary across species (Orlovski˘ı et al., 1999; Capaday, 2002), which limits our ability to extrapolate control circuits identified with direct methods in other animals to humans (Arshavsky et al., 1985; Zehr and Stein, 1999; Moraud et al., 2016). Yet a third reason is that theoretical results from modeling studies of the control circuitry remain inconclusive (Ijspeert, 2014; Sartori et al., 2016).

Neuromechanical simulations are used as a theoretical tool to study human locomotion control. Since bipedal locomotion emerges from the interaction between the legs and the ground by utilizing and resisting gravitational force (Mochon and McMahon, 1980; McGeer, 1990; Perry and Burnfield, 1992), accounting for the mechanical dynamics as well as the neural control is essential. This integrative approach of simulating the neural control with the biomechanical dynamics allowed researchers to investigate the spinal control layer where a large portion of locomotion control is conducted (Enoka, 2008; Dietz, 2010; Kiehn, 2016). Previously proposed spinal control models range from central pattern generators (CPGs; Aoi et al., 2010) to reflexes (Günther and Ruder, 2003; Geyer and Herr, 2010; Song and Geyer, 2015a) and to a mix of both (Taga et al., 1991; Ogihara and Yamazaki, 2001; Hase and Yamazaki, 2002; Jo and Massaquoi, 2007; Dzeladini et al., 2014). Many of these models with competing control structures are plausible candidates for human control, since they produce locomotion with kinematics, kinetics, or muscle activations similar to the ones observed in humans. Therefore, to genuinely evaluate the plausibility of these models a more in-depth comparison to experimental results is required.

Disturbance reactions provide such a more in-depth comparison. Studying the reaction to disturbances is a common approach to establish system models and to identify controllers (Ogata and Yang, 1970). Specifically for human locomotion, several walking experiments have been conducted that report on the immediate responses of the human spinal control to different types of unexpected disturbances including electrical stimulation (Simonsen and Dyhre-Poulsen, 1999; Courtine et al., 2007), mechanical perturbation at individual leg joints (Dietz et al., 1990; Sinkjaer et al., 1996; Faist et al., 1999), and more natural mechanical perturbation of the whole body (Schillings et al., 1999; Sloot et al., 2015). Although external disturbances have been used in neuromechanical human walking models to either test the robustness of control models (Aoi et al., 2010; Kim et al., 2011; Song and Geyer, 2015a) or to study specific high-level recovery strategies (Jo, 2007; Murai and Yamane, 2011), comparisons of the reference data on the reactions of the human spinal control to the reactions predicted by the different walking models have so far not been performed.

Here, we perform the in-depth comparison of disturbance reactions for one neuromechanical spinal control model of human locomotion (Song and Geyer, 2015a). In previous work, we have shown that this model, which consists of primarily proprioceptive spinal reflexes (**Figure 1**), can explain undisturbed locomotion behaviors. The model not only produces kinematics, dynamics, and muscle activations similar to humans during normal walking (**Figure 1** and **Video S1**) but also generates other locomotion behaviors such as running, walking on slopes and stairs, and avoiding obstacles. We investigate the plausibility of the model by comparing its reactions against disturbances to those of humans and discuss its implications in better understanding the control of human locomotion.

## 2. METHODS

We select a range of unexpected disturbances used in human gait studies from the literature, replicate them in simulation with the neuromechanical model, and compare the models reactions to the reported human experimental data.

## 2.1. Experiment Selection

Five disturbance experiments are selected from the literature: electrical stimulation of the lumbar spinal cord to evoke multisegmental monosynaptic responses (MMR; Courtine et al., 2007), mechanical tap of tendons to induce tendon tap reflexes (TR; Dietz et al., 1990; Faist et al., 1999), actuation of the ankle joint to induce stretch reflexes (SR; Sinkjaer et al., 1996), and tripping (TRIP) of the swing leg (Schillings et al., 1999), and slipping (SLIP) of the stance leg (Sloot et al., 2015; refer **Video S2** for visual guidance). In these experiments, the reactions of the spinal control are assessed through the changes that occur in the leg muscle activations within a short time after the disturbances. Specifically, the activation changes are measured by surface electromyograms (EMGs) and their trend with respect to gait phase or disturbance magnitude is used to estimate the activity of spinal reflexes.

The five experiments are selected to cover a broad range of disturbances and responses. For instance, from several reports of studies using similar types of disturbances, the ones that include the EMG changes for more leg muscles and across more conditions are selected. Specifically, while both MMR (Courtine et al., 2007) and H-reflex (Capaday and Stein, 1986; Simonsen and Dyhre-Poulsen, 1999) experiments disturb afferent signals using electrical stimulations, the former was selected since MMR disturbs multiple afferents and, as a result, induces responses in more muscles. Similarly, the SR (Sinkjaer et al., 1996), TRIP (Schillings et al., 1999), SLIP (Sloot et al., 2015), and TR (Dietz et al., 1990; Faist et al., 1999) experiments were chosen over similar ones that apply disturbances for fewer conditions (Berger et al., 1984; Yang et al., 1991; Eng et al., 1994; Van de Crommert et al., 1996; Cronin et al., 2009; Chvatal and Ting, 2012; Villarreal et al., 2016). Note that the SLIP experiment by Sloot et al. (2015) reports on muscle responses with latencies of about 150 ms, which are longer than usual for spinal reflexes. Although it is acknowledged that one cannot completely exclude that these responses are long-latency reflexes, we still included the study, as the authors clarify that these apparent latencies are in part an outcome of their experimental protocol for detecting disturbances, and as we could not find an alternative study reporting responses against a range of disturbance intensities. However, to further support our analysis on the response amplitudes in the SLIP experiment (compare Section 3.2), we have verified the consistency of our model results for a similar experiment by Berger et al. (1984), in which the reported responses are clearly within the time window of spinal reflexes.

## 2.2. Replication in Simulation

We adapt the original neuromechanical model (Song and Geyer, 2015a) for each of the five experiments (**Table 1**). Since all the experiments reported on sagittal plane disturbances, the model is first reduced to its sagittal plane musculoskeletal architecture and spinal control. Then, the musculoskeletal properties are scaled (Winter, 2009) to match the average height and weight of the subjects in each experiment (Courtine et al., 2007; Sloot et al., 2015). If this information is not reported (Dietz et al., 1990; Sinkjaer et al., 1996; Faist et al., 1999; Schillings et al., 1999), the height and weight are set to 1.8 m and 80 kg. Finally, the model's


control parameters are optimized with the cost function

$$J = \mathcal{C}\_E + \mathcal{c}\_\nu \left\| \nu\_{\text{avg}} - \nu\_{\text{tgt}} \right\|, \tag{1}$$

which encourages energy efficient walking at a target walking speed. In this equation, C<sup>E</sup> is the metabolic energy consumed by the muscles, c<sup>v</sup> = 100 is a weighting factor, and vavg and vtgt are the average and target walking speeds. The target walking speed, vtgt, is set to the reported speed in each experiment. A demonstration of the simulation model can be found in **Video S2**. More details about the original model and the optimization procedure to obtain control parameters for stable and steady walking are given in (Song and Geyer, 2015a).

The disturbances were simulated for the reported conditions in each experiment, which either included different gait phases (for MMR, TR, SR, and TRIP) or different disturbance intensities (for SLIP). The mechanical disturbances of the SR, TRIP, and SLIP experiments were directly replicated in the simulation by modeling an unexpected ankle flexion, the encounter of the tripping obstacle, and the shift of the supporting ground with the same parameters as reported in each experiment, respectively.

The MMR and TR experiments were less straightforward to replicate in simulation, as the neuromechanical model does not include the corresponding physiological detail. In the MMR experiment (Courtine et al., 2007), muscle responses (spikes with about 20 ms durations) are induced by percutaneous electrical stimulation (1 ms square pulses) at the lumbar spinal cord, which disturbs the afferent pathways from the legs. Instead of modeling the electrophysiological dynamics such as the filtering effects of the skin layer, the MMR disturbance was simulated as 10 ms square pulses that were simultaneously added to the afferent signals from all muscles. The duration of 10 ms was chosen because it created similar muscle responses (spikes with about 20 ms durations) in the model. The amplitudes of the square pulses were set to be arbitrarily large (maximum isometric forces, Fm, for force afferents; optimum length, lce, for length afferents; and maximumcontraction-velocity value, |vmax|, for velocity afferents) to evoke responses much larger than the normal activations seen during walking, as reported in the MMR experiment (Courtine et al., 2007).

For the TR experiment, it is generally observed that the tendon tap reflex amplitude is proportional to the tapping force (Mildren et al., 2016), although the neurophysiological process behind this observation is not well understood (Zhang et al., 1999). The effect of tendon taps was modeled by simulating the length changes in the muscle tendon unit affected by the tapping. Specifically, we simulated the length change based on the tension of the muscle and the kinetic energy of the tapping hammer. As a result, the effect of the taps on length change varied over the gait cycle according to the variation of the muscle tension.

#### 2.3. Reaction Comparisons

The response trends and amplitudes were compared separately for each experiment and muscle. While the model has nine muscles per leg, data for only six muscles was available in the literature (compare **Figure 1**). Similarities of the response trends were quantified as the % of the model responses that lie within ±1 standard deviation (s.d.) of human responses when linearly scaled to maximize overlap. For example, 12 out of 16 of the model's SOL responses in the MMR experiment lie within ±1 s.d. of the corresponding human responses and thus the similarity is 12/16 = 75% (**Figure 2**).

The response amplitudes are only compared for the SR, TRIP, and SLIP experiments. The MMR and TR disturbances induce synchronous and artificially exaggerated muscle activation responses, which is not observed in normal voluntary activations (Yang et al., 1991). As the model does not include these artificially synchronized muscle activations, the response amplitudes are not meaningful to compare for these studies.

#### 3. RESULTS

#### 3.1. Response Trends

The neuromechanical control model and humans react to disturbances with a similar trend for the majority of investigated muscles and experimental conditions. **Figure 2** summarizes the changes in muscle activation organized by disturbance experiment and leg muscle. The changes observed in humans (gray lines and shaded areas indicating ±1 s.d.) are normalized to their peak value and overlaid by the corresponding changes of the model (scaled to maximize overlap and compare trends as described in Section 2.3, black lines). While some of the response trends do not match well (≤50% overlap within one s.d., comparisons marked with ∗), for the majority of the investigated muscles and experimental conditions the scaled model responses lie within one s.d. of the human responses (78% average overlap for unmarked comparisons).

For several of the marked comparisons, simple modifications of either the reflex control or the model tuning could improve the overlap. First, in the model, the rectus femoris muscle (RF) is used mainly for sensing but not actuation. As a result, it cannot change activation except during swing. In the human experiments, by contrast, RF shows response trends similar to the synergistic vasti muscle group (VAS) throughout stride, although careful interpretation of these RF responses is needed, since surface EMGs of RF, which are used in the disturbance experiments, are prone to crosstalk from VAS (Nene et al., 2004). If fine wire EMG of RF reveal response trends similar to those of VAS, these trends can be reproduced by modifying the model to control RF with the same reflex pathways as VAS. Such modification is tenable in the functional point of view, since RF and VAS share a common role of knee extension.

Second, the difference between human and model responses of the vasti and the gastrocnemius muscles (GAS) during late swing may be an artifact of the model tuning process, which only considered undisturbed walking. The late swing reflexes that control VAS and GAS in the model do not engage during undisturbed locomotion (Song and Geyer, 2015a), and thus the

normalized with respect to their maximum value in each experiment and for each muscle. The model responses (black lines) are linearly scaled to place as many of the responses as possible within ±1 *s.d.* of the human responses (gray shaded area). The % of the model responses within ±1 *s.d.* of human responses are shown at the top of each graph, and those which are ≤50% are marked with \*.

optimization process sets their parameters to arbitrary values as far as they do not effect normal walking. In other words, these control parameters could be further tuned to improve the overlap with human responses for the two muscles without changing the undisturbed walking behavior.

Finally, the weak overlap for the soleus muscle (SOL) in the late swing phase of the SR experiment may be the result of natural variability in humans. It is known from human experiments using the H-reflex, the electrically elicited equivalent of the stretch reflex, that the swing phase responses in SOL vary among subjects between no responses (similar to the trend predicted by the model) and the responses shown in the SR experiment (Simonsen and Dyhre-Poulsen, 1999).

#### 3.2. Response Amplitudes

Whereas, the model captures the majority of the human response trends, it clearly underestimates the response amplitudes for the more natural, whole body disturbances. In the SR experiments, the model reacts with amplitudes in the muscle activation changes that are similar to the ones reported for humans (about 90% of human amplitudes). Yet in the more natural TRIP and SLIP experiments, the response amplitudes are very small in the model (about 20 and 4%, respectively, and 8% for the experiment in Berger et al., 1984 as noted in Section 2.1). The difference occurs as the reflexes of the model only respond to changes in the muscle lengths, velocities and forces, and the SR disturbance induces much larger changes (up to about 100 times) in these proprioceptive signals than the TRIP and SLIP disturbances, which act on the muscles through the entire body and its mechanical inertia.

One explanation for the shortfall in the model's response amplitudes could be the missing integration of reflex pathways from skin receptors. Experimental studies have shown that cutaneous reflexes evoke muscle responses with different trends across the gait cycle depending on the location of the skin receptors (Van Wezel et al., 1997; Duysens et al., 2000; Nakajima et al., 2016). Additional modulation of the model's current proprioceptive reflexes by location-specific cutaneous reflexes (**Figure 3**), which have been observed in cat experiments (Lundberg et al., 1987), could produce human-like muscle response amplitudes in all experiments without altering the response trends. Such additional modulation against specific disturbances, such as those in SLIP and TRIP experiments, is also in agreement with previous observations that cutaneous stimulations are not accountable for the responses against certain

proprioceptive reflex control during tripping. Location specific skin sensors at the foot detect an obstacle encounter. Cutaneous reflex pathways return this information to the spinal cord and amplify the proprioceptive reflex control of locomotion.

joint specific disturbances (for example, in SR experiment; Grey et al., 2001) but do evoke muscle responses during human walking (Nakajima et al., 2016). However, the functional relevance of this amplification remains open for speculation. For instance, it could promote the recovery strategies seen during human tripping (elevating and lowering strategies in early and late swing; Eng et al., 1994) and slipping (ankle and hip strategies for anterior-posterior and medial-lateral perturbations; Oliveira et al., 2012).

## 4. DISCUSSION

A neuromechanical model of human locomotion has been evaluated by comparing its reactions to disturbances with those of humans during walking. The comparison of the response trends reinforces the plausibility of the majority of the model's reflex circuits. However, the observation of smaller response amplitudes of the model for the whole body disturbances suggests that these circuits are selectively amplified in humans.

An extension of the current control model with additional circuits that modulate the current reflex gains would likely be able to better reproduce both the human response trends and amplitudes (**Figure 4A**). For example, instead of the abrupt switches in the reflex gains in the current model, either the supraspinal control (Jo and Massaquoi, 2007; Song and Geyer, 2015a) or CPGs can gradually change these reflex gains (**Figures 4A-a,b**) and shape the response trends closer to humans (for example, during the transitions between stance and swing phases in VAS, GAS, and SOL, **Figure 2**). In addition, selective amplifications of response amplitudes for particular disturbances can be realized through additional reflex pathways that modulate the reflex gains based on the detection of those particular disturbances (**Figure 4A-c**). These additional reflex gain modulations would be able to reproduce the human control during steady walking as well as its reactions against unexpected disturbances.

On the other hand, it remains open whether other types of models, where CPGs generate motor outputs, can reproduce steady and reactive human walking behaviors with a similar level of agreement. It is often hypothesized that CPGs generate some portion or most of the normal (background) muscle activations while reflexes in parallel generate the remaining portion (Duysens and Van de Crommert, 1998; Dominici et al., 2011; Kiehn, 2016; **Figure 4B**). However, it is less likely that the previously proposed human walking models based on this hypothesis (Ogihara and Yamazaki, 2001; Jo and Massaquoi, 2007; Dzeladini et al., 2014) can explain human responses observed in the disturbance experiments, because the more of the normal activations is generated in a feed-forward manner by CPGs the smaller the response amplitudes will be, which stands in contrast to the large reactions observed in humans. For example, in a model that generates 90% of the normal activations with CPGs and the remaining 10% with the reflex pathways of the reflex-based model (Dzeladini et al., 2014), the response trends will remain the same but the response amplitudes will only be a tenth of the reflex-based model. Alternatively, the responsive activations could also be partially generated by CPGs as they get

activations. Each block diagram represents a spinal mechanism of generating muscle activations, where the spinal control can potentially consist of serial and parallel combinations of the each mechanism. Outputs of reflex circuits and CPGs are marked in blue and green, respectively, afferent signals *(Continued)*

#### FIGURE 4 | Continued

during normal walking are marked in yellow, and those signals in response to disturbances are marked in red. (A-a) Responses through the supraspinal system appears with larger time delays than the spinal responses. This holds true for supraspinal modulations of any spinal control (not shown for B,C). (A-b) Modulation of reflex circuits by pure CPGs does not change the responsive activations. (A-c) Response activations of reflexes can be selectively modulated by additional reflex circuits. (B) If muscle activations are generated mostly by CPGs, in other words, if the reflex circuits generate only a small portion of the activation signals, the response to the change in afferent signals would be small as well. (C-a) Phase resetting of CPGs results in persistent phase shift of the muscle activation signals. (C-b) If CPGs are continuously modulated by sensory feedback, all afferent signals, including the disturbance signals, get modulated by CPG dynamics.

modulated by sensory feedback (**Figure 4C**). For example, phase shifts in CPG activations in response to perturbations, which is called phase resetting (**Figure 4C-a**), have been observed in cats (Conway et al., 1987; Schomburg et al., 1998) and have been proposed to increase the robustness of human walking (Yamasaki et al., 2003; Aoi et al., 2010). However, the responses observed in the disturbance experiments considered in this study do not seem to originate from phase resetting of CPGs since they are transient responses rather than persistent phase shifts. Finally, CPGs have also been proposed to be continuously modulated by sensory feedback in many models, where the muscle responses result from more complicated CPG dynamics (Taga et al., 1991; Ogihara and Yamazaki, 2001; Hase and Yamazaki, 2002; McCrea and Rybak, 2008; **Figure 4C-b**). CPGs are usually modeled to consist of mutually inhibiting neurons with internal dynamics (Matsuoka, 1985), and many human walking models (Taga et al., 1991; Ogihara and Yamazaki, 2001; Hase and Yamazaki, 2002) incorporate continuous sensory feedback modulation of CPGs by adding afferent signals to this internal dynamics (for example, in the form of τu˙ = −u + other-terms + feedback, where τ is a time constant and u is the neural output). In this case, the muscle responses are likely to be slower and smaller, since the disturbance signals need to be integrated to appear in the neural outputs of the CPGs. Therefore, in order to explain both steady and reactive behaviors during human walking with control structures in which CPGs generate muscle activations, more complicated reflex circuits may be necessary that selectively amplify the responses not only for the whole body disturbances but also for the other disturbances.

Still, there is clear evidence that CPGs are highly involved in locomotion of many animals including mammals, and it is reasonable to expect human locomotion involves a similar control structure if the functional role of CPGs remained valid in the course of evolution to upright bipedal locomotion (Capaday, 2002; MacKay-Lyons, 2002; Ijspeert, 2008). One functional role that has been proposed to be realized by CPGs is the generation of transitional behaviors such as changing gait, as well as locomotion speed and direction. This view is supported by observations on decerebrate animals, where simple supraspinal stimulations control locomotion by modulating the frequency and amplitude of CPGs (Armstrong, 1988; Stein et al., 1997; Sirota et al., 2000). It has been shown with a neuromechanical model that human locomotion speed can be controlled in a similar way by modulating CPGs of the hip muscles (Van der Noot et al., 2015). On the other hand, transitional behaviors including speed and directional changes also can be realized in the absence of CPGs by changing the reflex gains directly through the supraspinal control (Song and Geyer, 2012, 2015a,b). Therefore, the role of CPGs in transitional locomotion behaviors of humans calls for further experimental studies. To this end, investigating the responses of the hip muscles (Hof and Duysens, 2013), which lack in previous gait disturbance experiments, can be crucial.

Our results also show that solely relying on indirect experimental observations can be misleading when assessing the role of reflexes. First, the changes in muscle responses do not necessarily indicate modulation of reflex gains. For example, in the TR experiment the changes in the model's HAM and VAS responses during stance (**Figure 2**) result from the changes in muscle configurations while the reflex gains remain constant. Second, the correlation between the muscle states and muscle responses is not sufficient to explain the underlying muscle reflexes. For instance, in a gait experiment similar to the SR experiment, Yang et al. (1991) suggested velocity feedback to contribute about 45% in the generation of SOL activations during the stance phase. The suggested contribution is based on the correlation between the changes in ankle velocity and the responses in SOL activation. However, as noted by the authors of the study, this quantification neglects the potential contributions of different afferent pathways. Performing the same correlationbased analysis in our model suggests a contribution of about 40% of velocity feedback in the stance control of SOL, even though the model uses no velocity feedback but 100% force feedback.

Although, the findings of our study may help to construct a model that can explain the steady and reactive spinal control of human walking, it will take further research to settle the actual circuitries in humans. First, neuromechanical simulations with more physiological details will be needed to incorporate other types of experimental studies in the evaluation of control models. For instance, we would be able to compare the response amplitudes of our control model to human responses in MMR and TR experiments if our simulations could more faithfully describe the relationship between cutaneous electrical stimulation and synchronous muscle activation as well as the related neurophysiology. Second, other models which can

#### REFERENCES


explain normal human walking should also be subjected to gait disturbance experiments to genuinely evaluate their plausibility and arrive at a consensus about what the human circuitry might be. Finally, the resulting control model should be verified by direct probing of the proposed neural circuits in human experiments. Although, it is currently impossible to probe the entire control of humans that involves millions of neurons, a control model that is thoroughly evaluated and specified may substantially reduce the search space. Evaluation beyond steady behavior will play an important role in this quest.

## AUTHOR CONTRIBUTIONS

SS and HG designed the research. SS developed the computational model, conducted the simulation studies, and analyzed the data. SS and HG drafted the manuscript and approved the final version of the manuscript.

## FUNDING

This work is supported in part by the National Science Foundation (grant no. 1527140), and the Richard King Mellon Foundation Presidential Fellowship in the Life Sciences at Carnegie Mellon University.

### ACKNOWLEDGMENTS

The authors thank R. Gregg, D. Villarreal, and B. M. Oh for the discussion in the early stage of this study and L. Sloot for the clarifications on the SLIP experiment. The authors would also like to thank the reviewers whose comments and suggestions lead to substantial improvements of the paper.

## SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fncom. 2017.00015/full#supplementary-material

#### Video S1 | Spinal-reflex-based neuromuscular model of human locomotion.

Video S2 | Simulation replications of the human disturbance experiments.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Song and Geyer. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Linear Parameter Varying Identification of Dynamic Joint Stiffness during Time-Varying Voluntary Contractions

#### Mahsa A. Golkar\*, Ehsan Sobhani Tehrani and Robert E. Kearney

Department of Biomedical Engineering, McGill University, Montréal, QC, Canada

Dynamic joint stiffness is a dynamic, nonlinear relationship between the position of a joint and the torque acting about it, which can be used to describe the biomechanics of the joint and associated limb(s). This paper models and quantifies changes in ankle dynamic stiffness and its individual elements, intrinsic and reflex stiffness, in healthy human subjects during isometric, time-varying (TV) contractions of the ankle plantarflexor muscles. A subspace, linear parameter varying, parallel-cascade (LPV-PC) algorithm was used to identify the model from measured input position perturbations and output torque data using voluntary torque as the LPV scheduling variable (SV). Monte-Carlo simulations demonstrated that the algorithm is accurate, precise, and robust to colored measurement noise. The algorithm was then used to examine stiffness changes associated with TV isometric contractions. The SV was estimated from the Soleus EMG using a Hammerstein model of EMG-torque dynamics identified from unperturbed trials. The LPV-PC algorithm identified (i) a non-parametric LPV impulse response function (LPV IRF) for intrinsic stiffness and (ii) a LPV-Hammerstein model for reflex stiffness consisting of a LPV static nonlinearity followed by a time-invariant state-space model of reflex dynamics. The results demonstrated that: (a) intrinsic stiffness, in particular ankle elasticity, increased significantly and monotonically with activation level; (b) the gain of the reflex pathway increased from rest to around 10–20% of subject's MVC and then declined; and (c) the reflex dynamics were second order. These findings suggest that in healthy human ankle, reflex stiffness contributes most at low muscle contraction levels, whereas, intrinsic contributions monotonically increase with activation level.

Keywords: joint stiffness, ankle biomechanics, system identification, time-varying, linear parameter varying

## 1. INTRODUCTION

Ankle joint biomechanics can be described by the relationship between the joint position and the torque acting about it, defined as dynamic joint stiffness. It describes the properties of the human actuator and determines (a) the internal load that the central nervous system (CNS) must control and (b) the joint behavior in response to external loads or perturbations. Consequently, a quantitative knowledge of joint stiffness is essential for understanding the normal control of posture and movement and the nature of motor function disorders such as spasticity, rigidity, hypertonia, hypotonia, and flaccidity (Amato and Ponziani, 1999; Bar-On et al., 2014). Also, a good model of

#### Edited by:

Manish Sreenivasa, Heidelberg University, Germany

#### Reviewed by:

Eric Jon Perreault, Northwestern University, USA Mo Rastgaar, Michigan Technological University, USA

#### \*Correspondence:

Mahsa A. Golkar mahsa.aliakbargolkar@mail.mcgill.ca

> Received: 31 January 2017 Accepted: 21 April 2017 Published: 19 May 2017

#### Citation:

Golkar MA, Sobhani Tehrani E and Kearney RE (2017) Linear Parameter Varying Identification of Dynamic Joint Stiffness during Time-Varying Voluntary Contractions. Front. Comput. Neurosci. 11:35. doi: 10.3389/fncom.2017.00035 joint stiffness is invaluable for the design and control of ankle prostheses and orthoses (Palazzolo et al., 2007).

Joint stiffness modeling has been extensively investigated in the literature (e.g., Kearney et al., 1997; Mirbagheri et al., 2000; Jalaleddini and Kearney, 2011; Sobhani Tehrani et al., 2014). Two distinct physiological mechanisms contribute to joint stiffness: (i) Limb inertia, viscoelasticity of muscle-tendon complex, and active properties of muscle contraction that together define intrinsic stiffness; and (ii) Stretch reflex feedback that changes muscle activation in response to changes in muscle length leading to reflex stiffness. At the human ankle, this has been efficiently modeled with a Parallel-Cascade (PC) structure having separate pathways for intrinsic and reflex stiffness (Kearney et al., 1997). This study showed that under quasi-stationary conditions, where the joint is perturbed around an operating point (OP) defined by joint position and activation level, the intrinsic stiffness can be modeled by an impulse response function (IRF) and the nonlinear reflex stiffness can be modeled by a Hammerstein system consisting of a static nonlinearity followed by a linear dynamics.

However, numerous quasi-stationary studies, using system identification techniques, demonstrated that both intrinsic and reflex stiffness parameters change drastically and systematically with ankle position and activation level (Weiss et al., 1986; Sinkjaer et al., 1988; Carter et al., 1990; Mirbagheri et al., 2000; Van der Helm et al., 2002; Bar-On et al., 2014; Jalaleddini et al., 2016). Thus, in many functional tasks, like normal gait, where joint position and neural activation continuously change to control movement and counteract external perturbations, joint stiffness will exhibit time-varying (TV) behavior. Furthermore, there is evidence that this TV behavior cannot be predicted simply by interpolating local TI models identified under quasistationary conditions (Kirsch and Kearney, 1997). Therefore, more advanced methodologies are required to identify and characterize joint stiffness during movement or functional tasks.

To this end, a number of approaches have been proposed and used over the years. These include intramuscular mechanism modeling using optimization that minimizes a predefined cost function (Sartori et al., 2015), system identification techniques, or a combination of both (de Vlugt et al., 2010). Methods for identification of TV systems can be divided into four main categories: (i) short segment, (ii) ensemble-based, (iii) timevarying, and (iv) linear parameter varying (LPV).

Short segment methods (Ludvig and Perreault, 2012; Rouse et al., 2014; Jalaleddini et al., 2017) divide non-stationary data into a number of segments with quasi-stationary behavior and identify a time-invariant model for each segment. The segmentation is not always trivial and often requires the TV behavior to be very slow. Ensemble-based methods (MacNeil et al., 1992; Kirsch et al., 1993; Ludvig et al., 2011; Lee and Hogan, 2015) are effective but require many trials with identical TV behavior, which is hard to achieve in many experimental conditions. Moreover, repeating the same task many times may result in fatigue and affect the reliability of estimates. Timevarying identification techniques (Sanyal et al., 2005; Ikharia and Westwick, 2006, 2007; Guarin and Kearney, 2015) use temporal expansion to estimate how the system parameters change continuously with time using data from a single trial; thus simplifying data requirements significantly. However, selecting proper basis functions for temporal expansion is often difficult and the number of model parameters increases significantly if the time-dependent changes are fast; thus reducing the quality of the estimates. Moreover, none of the models identified by these methods can predict the system response to novel trajectories.

LPV models have a structure resembling that of linear systems whose parameters change as functions of one or more timedependent signal called scheduling variables (SV). As such, the LPV structure is an excellent candidate for modeling joint stiffness during functional tasks where the TV behavior is mostly due to dependency on neuromuscular variables that vary with time. Also, by relating TV behavior to SVs rather than time, LPV models model the nonlinear mechanisms that generate the TV behavior and thus have the ability to predict the response to novel trajectories. Finally, control theory is well developed for LPV systems (Mohammadpour and Scherer, 2012), which makes LPV models suitable for prostheses and orthoses control.

Despite the significant advantages of LPV models, methods for LPV identification of nonlinear physiological systems have not been studied much. Examples include the LPV modeling of glucose-insulin dynamics in type I diabetes (Cerone et al., 2012) and of the hemodynamic response to profiled hemodialysis (Javed et al., 2010). Our lab has pioneered the use of LPV methods for the identification of joint stiffness. Specifically, Sobhani Tehrani et al. (2013a) identified a LPV mass-springdamper (LPV IBK) model of intrinsic ankle joint stiffness for imposed movements at rest. Soon after, Van Eesbeek et al. (2013) used a LPV subspace method to identify time-variant intrinsic impedance of the human wrist joint. Subsequently, Sobhani Tehrani et al. (2014) developed subspace LPV parallelcascade (LPV-PC) method for the identification of both intrinsic and reflex stiffness during large passive ankle movements. However, these studies were conducted under passive (i.e., at rest) conditions and quantified position dependent changes in stiffness. The study of joint stiffness changes during large timevarying muscle contractions is challenging since neither the muscle activation level nor the voluntary torque are directly measurable as scheduling variable.

In this work, we used the subspace LPV-PC algorithm (Sobhani Tehrani et al., 2014) to characterize changes in both intrinsic and reflex stiffness during isometric, time-varying contractions of the ankle plantarflexors of healthy human subjects. This algorithm, models the intrinsic pathway as a non-parametric LPV impulse response function (LPV IRF) and reflex stiffness as a LPV-Hammerstein cascade of a LPV static nonlinearity and a time invariant (TIV) linear dynamics. The reflex linear dynamic was assumed TIV, similar to previous works (Sinkjaer et al., 1996, 1988; Ludvig et al., 2011). The scheduling variable, the joint voluntary torque, was estimated from EMG signals using a time-invariant Hammerstein model of EMG-Torque dynamics, which was previously identified using an error-in-variable subspace algorithm. In addition to the experimental examination of the subspace LPV-PC identification method, we also performed Monte-Carlo simulations to demonstrate its accuracy and precision.

## 2. METHODS

#### 2.1. Problem Formulation

**Figure 1** shows a block diagram of the subspace LPV-PC model with joint angle as input (θ), total torque as output (TQtot), and voluntary torque as scheduling variable (µ). The total torque is the sum of intrinsic (TQI), reflex (TQR), and voluntary torques (TQV), and the colored measurement noise (n). This can be written as:

$$TQ\_{tot}(k) = TQ\_I(k) + TQ\_R(k) + TQ\_V(k) + n(k)\tag{1}$$

and the stiffness torque is:

$$TQ\_{\mathfrak{s}}(k) = TQ\_{I}(k) + TQ\_{\mathbb{R}}(k) \tag{2}$$

where,

$$TQ\_s = \begin{bmatrix} TQ\_s(0) \dots \ TQ\_s(N-1) \end{bmatrix}^T$$

$$TQ\_I = \begin{bmatrix} TQ\_I(0) \dots \ TQ\_I(N-1) \end{bmatrix}^T\_{\dots} \tag{3}$$

$$TQ\_R = \begin{bmatrix} TQ\_R(0) \dots \ TQ\_R(N-1) \end{bmatrix}^T$$

$$E = \begin{bmatrix} n(0) \dots n(N-1) \end{bmatrix}^T \tag{4}$$

and N represents the total number of samples. The intrinsic stiffness is represented by a LPV IRF model:

$$TQ\_l(k) = \sum\_{l=-L}^{l=L} h\_l(\mu(k))\theta(k-l) \tag{5}$$

where h<sup>l</sup> are the IRF weights that are functions of SV (µ(k)) represented by a basis expansion on the SV:

$$h\_l \triangleq \sum\_{j=0}^{n\_l} h\_{lj} \mathfrak{g}\_{\mathfrak{f}}(\mu(k)) \tag{6}$$

where hij is the (i, j)-th coefficient for the i-th lag of IRF, g<sup>j</sup> represents the j-th basis expansion of the SV and n<sup>i</sup> is the expansion order. Now, rewrite this equation in matrix form to obtain a data equation for the intrinsic pathway; the unknown intrinsic stiffness parameters are:

$$\beta \mathbf{l} = \begin{bmatrix} H - \mathbf{L} \ \dots \ H \end{bmatrix} \dots \mathbf{H} + \mathbf{L} \begin{bmatrix} \end{bmatrix}^T \tag{7}$$

where H<sup>l</sup> contains the LPV IRF weights for lag l,

$$H\_l = \begin{bmatrix} h\_{l\_0} \ \dots \ h\_{l n\_i} \end{bmatrix}^T \tag{8}$$

The basis expansion of the SV can be represented in vector form:

$$G\_i(k) = \begin{bmatrix} \mathbb{g}\_0(\mu(k)) \dots \mathbb{g}\_{n\_i}(\mu(k)) \end{bmatrix}^T \tag{9}$$

and the lagged position inputs with the vector:

$$\Theta(k) = \begin{bmatrix} \theta(k+l) \dots \theta(k) \dots \theta(k-l) \end{bmatrix}^T \tag{10}$$

Then, the input to the intrinsic pathway is constructed by the Kronecker product of Equations (9, 10):

$$U\_l(k) = \Theta(k) \otimes G\_l(k) \tag{11}$$

Now, rewriting Equation (5) in vector form, the data equation for the intrinsic pathway is:

$$TQ\_I = \Psi\_I \beta\_I \tag{12}$$

with the regressor:

$$\Psi\_I = \begin{bmatrix} U\_I(L) \dots \ U\_I(N-1-L) \end{bmatrix}^T \tag{13}$$

The reflex stiffness is modeled by a differentiator, a delay, and a Hammerstein system comprising a LPV static nonlinearity followed by a time-invariant linear state-space model. The input to the Hammerstein system is the delayed joint velocity (due to reflex delay) denoted by dvel in the equations. The output of

the static nonlinearity is approximated by an orthonormal basis function expansion of the Hammerstein system input, dvel:

$$z(k) = f(d\nu l(k), \mu(k)) \simeq \sum\_{i=0}^{n\_p} \alpha\_i(\mu(k)) g\_i(d\nu l(k))$$
 
$$\text{where,}$$
 
$$\omega\_i = \sum^{n\_r} \omega\_{i\bar{j}\bar{g}\_{\bar{j}}}(\mu(k))$$

and gi(dvel(k)) is the i-th basis expansion of reflex input (dvel), gj(µ(k)) is the j-th basis expansion of the SV, and ωij is the coefficient of their products; n<sup>p</sup> and n<sup>r</sup> are the expansion orders of the input (dvel) and the SV, respectively. Thus, using basis expansions of the input, the static nonlinearity is converted to n<sup>p</sup> parallel linear functions, where the expansion weights are dependent on the SV. The vectors of input and SV basis expansions, for reflex pathway, can be written as:

$$\begin{aligned} G\_r(k) &= \begin{bmatrix} \operatorname{g}\_0(\mu(k)) \dots \operatorname{g}\_{n\_r}(\mu(k)) \end{bmatrix}^T \\ DV(k) &= \begin{bmatrix} \operatorname{g}\_0(d\nu l(k)) \dots \operatorname{g}\_{\nu\_\mathbb{P}}(d\nu l(k)) \end{bmatrix}^T \end{aligned} \tag{15}$$

with unknown parameters:

j = 0

$$\begin{aligned} \Omega &= \begin{bmatrix} \Omega\_0 \dots \Omega\_{n\_p} \end{bmatrix}^T \\ \Omega\_i &= \begin{bmatrix} \omega\_{i0} \dots \omega\_{in\_r} \end{bmatrix}^T \end{aligned} \tag{16}$$

Thus, the input to reflex linear dynamics becomes:

$$U\_R(k) = DV(k) \otimes G\_r(k) \tag{17}$$

The linear system is modeled using a discrete-time state-space representation of order m:

$$\begin{aligned} X(k+1) &= AX(k) + Bz(k) \\ TQ\_R(k) &= CX(k) + Dz(k) \end{aligned} \tag{18}$$

where X(k) is the state vector, z(k) is the input to reflex linear dynamics, and A, B, C, and D are the state-space matrices and:

$$B = \begin{bmatrix} b\_1 \ \dots \ b\_m \end{bmatrix}^T, \qquad D = \begin{bmatrix} d \end{bmatrix} \tag{19}$$

Substituting Equation (17) in Equation (18) yields:

$$\begin{aligned} X(k+1) &= A\_R X(k) + B\_{\Omega} U\_R(k) \\ T Q\_R(k) &= C\_R X(k) + D\_{\Omega} U\_R(k) \end{aligned} \tag{20}$$

where,

$$B\_{\Omega} = B \otimes \Omega = \begin{bmatrix} b\_1 \Omega\_0^T & \dots & b\_1 \Omega\_{n\_\rho}^T \\ \vdots & \ddots & \vdots \\ b\_m \Omega\_0^T & \dots & b\_m \Omega\_{n\_\rho}^T \end{bmatrix},$$

$$D\_{\Omega} = D \otimes \Omega = \left[ d\Omega\_0^T \dots d\Omega\_{n\_\rho}^T \right] \tag{21}$$

Combining the data equations for intrinsic and reflex pathways (Equations 12, 20), the total joint stiffness can be represented with a Multi-Input-Single-Output (MISO) state-space model:

$$\begin{aligned} X(k+1) &= A\_R X(k) + B\_T U\_T(k) \\ \hat{T} \hat{Q}\_s(k) &= C\_R X(k) + D\_T U\_T(k) + n(k) \end{aligned} \tag{22}$$

where,

$$U\_T(k) = \begin{bmatrix} U\_R(k) \ U\_I(k) \end{bmatrix} \tag{23}$$

$$\begin{aligned} B\_T &= \begin{bmatrix} B\_{\Omega} & \underbrace{0 \dots 0}\_{(2L+1)n\_i \text{columns}} \end{bmatrix} \\ D\_T &= \begin{bmatrix} D\_{\Omega} & \beta\_I \end{bmatrix} \end{aligned} \tag{24}$$

#### 2.2. Subspace LPV-PC Identification Algorithm

An orthogonal projection algorithm (Sobhani Tehrani et al., 2014; Jalaleddini et al., 2016) was used to first decompose intrinsic and reflex torque components and subsequently estimate the unknown model parameters. The unknown parameters to estimate are (i) the intrinsic IRF parameters (β<sup>I</sup> in Equation 7); (ii) the reflex non-linearity coefficients ( in Equation 16); and (iii) the reflex linear system matrices A, B, C, and D in Equation (18). This can be achieved through the following steps:


$$T\ddot{Q}\_{\delta} = TQ\_{I} + TQ\_{R} + E = \Psi\_{I}\beta\_{I} + \Psi\_{R}\beta\_{R} + E \tag{25}$$

where, 9<sup>I</sup> and β<sup>I</sup> are defined in Equations (7, 13), respectively, and:

$$\begin{aligned} \Psi\_R &= \begin{bmatrix} 0 & \mathcal{U}\_R^T(\mathbf{0}) \\ \vdots & \vdots \\ \sum\_{\tau=0}^{N-2} \mathcal{U}\_R^T(\mathbf{\tau}) \otimes \hat{\mathcal{C}}\_R \hat{\mathcal{A}}\_R^{N-2-\tau} & \mathcal{U}\_R^T(N-1) \end{bmatrix}, \\ \beta\_R &= \begin{bmatrix} \mathcal{B}^T d \end{bmatrix}^T \otimes \Omega \end{aligned}$$

5. Use orthogonal projection to decompose the total torque into its intrinsic and reflex components:

$$\begin{aligned} \hat{T}\hat{Q}\_I &= (I - \Psi\_I^\dagger \Psi\_R \Psi\_R^\dagger \Psi\_I)^\dagger \Psi\_I^\dagger (I - \Psi\_R \Psi\_R^\dagger) \hat{T} \hat{Q}\_S \\ \hat{T}\hat{Q}\_R &= \hat{T}\hat{Q}\_s - \Psi\_I \hat{\beta}\_I \end{aligned}$$

6. Use the subspace Hammerstein method described in Sobhani Tehrani et al. (2013b) to estimate the reflex pathway model using dvel(k) as input and TQˆ <sup>R</sup>(k) as output.

## 3. SIMULATION STUDY

#### 3.1. Methods

We evaluated the performance of the subspace LPV-PC identification algorithm using a simulation study of the LPV-PC model of human's ankle stiffness dynamics (**Figure 1**). All parameter and nominal values of simulation model were selected based on experimental results reported in literature (Mirbagheri et al., 2000; Jalaleddini et al., 2016).

#### 3.1.1. Model

The intrinsic stiffness was simulated as the LPV IBK model:

$$TQ\_I(k) = I\ddot{\theta}(k) + B\dot{\theta}(k) + K(\mu(k))\theta(k) \tag{26}$$

The inertia (I) and viscosity (B) were set to 0.015 Nm.s<sup>2</sup> /rad and 1.1 Nm.s/rad. The intrinsic elastic parameter (K) and reflex gain and threshold were simulated to have a non-linear behavior with changes in voluntary torque (SV). The linear dynamics of reflex pathway was assumed TIV. **Figure 2** demonstrates the simulated parameters. Elasticity was modeled as a polynomial of order 3 for SV. The reflex gain (represented as NL slope in **Figure 2C**) and threshold (NL threshold, **Figure 2D**) of reflex Hammerstein system were modeled as polynomial of order 6 for input and a polynomial of order 4 for the SV.

The linear dynamic element of the reflex pathway was assumed to be a second-order low-pass filter with the dynamics:

$$H(\mathbf{s}) = \frac{G\boldsymbol{\alpha}\_n^2}{s^2 + 2s\boldsymbol{\zeta}\,\boldsymbol{\alpha}\_n + \boldsymbol{\alpha}\_n^2} \tag{27}$$

where G = 1 is the system gain, ω<sup>n</sup> = 25 rad/s is the natural frequency and ζ = 0.9 rad/s is the damping factor. The reflex delay was assumed to be 40 ms. This system was simulated using MATLAB Simulink at 1 kHz for 120 s.

#### 3.1.2. Input and Noise

The input signal was a pseudo random arbitrary level distributed signal (PRALDS) with random switching time uniformly distributed over [250, 350] ms, and maximum amplitude equal to 0.05 rad. This input signal was then filtered with a second order Butterworth low-pass filter with cutoff frequency of 30 Hz to represent the actuator dynamics.

Output noise was modeled as a white Gaussian signal filtered with a second order Butterworth low-pass filter with cutoff frequency equal to 15 Hz. The noise amplitude was adjusted to produce an average signal-to-noise ratio (SNR) of 10 dB. SNR was calculated as:

$$\text{SNR(dB)} = 20 \log\_{10} \left( \frac{\text{RMS}\_{\text{signal}}}{\text{RMS}\_{\text{noise}}} \right) \tag{28}$$

**Figure 3** shows a 4s segment of the position input and noise free and noisy output data, and **Figure 4** shows the simulated input (position), scheduling variable (voluntary torque), and torques.

#### 3.1.3. Analysis

To avoid aliasing, all simulation data were filtered with an eighth-order low-pass filter with cutoff frequency of 45 Hz and decimated to 100 Hz before analysis. The intrinsic pathway was identified using a LPV IRF model as described by Equation (5).

FIGURE 4 | Typical simulation data: (A) position input, (B) voluntary torque (SV), (C) total torque (TQtot) and SV (TQv), 20s segments of: (D) intrinsic stiffness, (E) reflex stiffness, (F) stiffness torque.

We calculated the equivalent elasticity of the identified model as the low-frequency (or DC) gain of the LPV IRFs at each SV snapshot. This gain is the steady state value of the integral of identified intrinsic LPV IRF at each SV snapshot.

We assessed the quality of fit by calculating the variance accounted for (VAF):

$$\%VAF = \left[1 - \frac{\sum\_{i=1}^{N} (TQ\_i - \hat{T}Q\_i)^2}{\sum\_{i=1}^{N} TQ\_i^2} \right] \times 100 \tag{29}$$

where TQ<sup>i</sup> represents the noise free simulated torque at time interval i and TQˆ represented the estimated value; N is the number of samples.

We quantified the quality of identification estimates by using 200 Monte-Carlo trials, each having a new realization of input and noise. The bias and random errors for reflex static nonlinearity estimates were calculated as:

$$\begin{aligned} \text{Bias Error} &= \rho - E(\hat{\rho})\\ \text{Random Error} &= E(\hat{\rho} - E(\hat{\rho}))^2 \end{aligned} \tag{30}$$

where ρ and ρˆ represent true and estimated parameter respectively. Note that both Bias Error and Random Error are also functions of delayed velocity and SV.

#### 3.2. Results

**Figure 5** shows the torque prediction profiles for a typical trial. The subspace LPV-PC identification algorithm used, identified the simulated model very accurately as confirmed by high VAFs calculated for each pathway. **Figure 6** summarizes the torque prediction accuracy for each pathway as well as the stiffness torque, for 200 Monte-Carlo trials identified, in boxplot representation. The VAFs were always above 98% for the high

noise level tested in this simulation study, confirming the efficiency of method in decomposing the total torque into intrinsic and reflex contributions.

**Figure 7A** shows the simulated values of intrinsic pathway elasticity (K) as a function of SV in blue and the mean of 200 Monte-Carlo identification estimates bracketed by two standard deviations of the estimates in red. It is evident that mean of estimates were very close to true value with small variance.

**Figures 7B,C** show the simulated (blue) and estimated (red) slope and threshold of the estimated nonlinearity extracted from 3D nonlinearity. These values were obtained by finding the best half-wave rectifier (HWR) fit to estimated nonlinearity at each SV using Levenberg-Marquardt method in MATLAB curve fitting toolbox. The red curve shows the mean of 200 Monte-Carlo identification estimates bracketed by two standard deviations of the estimates. The mean of the estimates for slope was very close to simulated values showing that we can accurately retrieve the reflex gain. The estimates of thresholds at some SVs were subject to a maximum of 25% error. There are two explanations for this: (1) the simulated model was different from the identified model, i.e., HWR was simulated and Chebyshev polynomials were used for identification. (2) The distribution of input (velocity for reflex pathway) affects the estimation of threshold. The estimates are expected to be more accurate for an input with rectangular probability distribution. However, these choices were made intentionally in this work to evaluate the performance of the algorithm for a practical case, i.e., true nonlinearity may not be the same as that used for identification for physiological systems, and the actuator dynamics affects the input distribution. Nevertheless, the overall estimated threshold variation trend is very close to the true simulated value. Note that since torque has little power at thresholds, the bias in threshold estimate has little effect on torque prediction.

The LPV nonlinear block of reflex pathway is plotted in **Figure 8** in 3D representation; **Figure 8A** shows the true simulated nonlinearity whereas the average of 200 estimated nonlinear block is plotted in **Figure 8B** of this figure. The

FIGURE 7 | True (blue) and the mean of estimated (red) (A) intrinsic elasticity, (B) reflex LPV-static nonlinearity slope and (C) threshold for 200 Monte-Carlo simulations, bracketed by 2× standard deviation, SNR = 10 dB. Parameters of static nonlinearity were estimated by fitting a half-wave rectifier to nonlinearity at each SV.

lower two panels show the bias and random errors for static nonlinearity estimate for 200 simulation trials from top view; both errors were small with maximum bias error occurring around nonlinearity threshold. This is consistent with our estimation of threshold demonstrated in **Figure 7C**. The maximum bias error was around 10 Nm/rad and the maximum random error was 1 Nm/rad, while the nonlinearity has a maximum gain of 160 Nm/rad. This confirms the efficiency of the proposed algorithm for estimating the LPV static non-linearity. The frequency response of reflex linear dynamic estimate is demonstrated in **Figure 9**. The linear system was calculated as a subspace system; the frequency response representation is used for better visualization of accuracy at different frequencies. Both the gain and phase estimates were close to true simulated values.

## 4. EXPERIMENTAL STUDY

## 4.1. Methods

The new algorithm was used to characterize the modulation of joint stiffness with activation level in healthy humans performing an isometric torque tracking task of the ankle plantarflexors.

#### 4.1.1. Apparatus

**Figure 10** shows a schematic of the experimental setup which is described in details in Morier et al. (1990). Subjects lay supine on an experimental table with the left foot attached to a hydraulic actuator using a costume-made fiberglass boot. The neutral position was defined as a 90 degree angle between the foot and shank. Dorsiflexing rotations were taken as positive. The mean ankle angle was set to 0.2 rad.

## 4.1.2. Subjects

Five healthy subject (one female and four males) aged 26– 33 with no history of neuromuscular disorders participated.

for 200 Monte-Carlo simulations, bracketed by 2× standard deviation, SNR = 10 dB. The bode plot is presented up to 50 Hz where the input has enough power for identifications.

Subjects gave informed consent to the experimental procedures, which had been reviewed and approved by McGill University Research Ethics Board. **Table 1** summarizes the subjects' demographics.

#### 4.1.3. Data Acquisition

EMG signals from tibialis anterior (TA) and triceps surae (TS) including lateral and medial Gastrocnemius muscles were recorded separately using differential surface electrodes. EMGs were amplified and band-pass filtered with a gain of 1,000 and cutoff frequencies 20–2,000 Hz. Ankle torque was low-pass filtered with an eighth-order Bessel filter with cut-off frequency equal to 0.7 Hz in real time and provided to the subject as visual feedback signal. Position, torque and EMG signals were filtered

TABLE 1 | Subject characteristics: gender, age, Maximum Voluntary Contraction (MVC) torque in Plantarflexion (PF), and the normalization factors.


with an anti-aliasing filter at 486.3 Hz, sampled at 1 kHz, and recorded.

#### 4.1.4. Trials

Subjects were instructed to modulate their ankle torque by tracking a visual command signal. The command signal comprised of a sine-wave with a period of 60 s and peak-to-peak amplitude equal to 40% of their maximum voluntary contraction (MVC). Two conditions were examined:


Data were recorded for 120 s at sampling frequency of 1kHz and then decimated to 100 Hz for analysis. Data were examined for evidence of fatigue or co-activation; there was no evidence of either in any of the trial.

#### 4.1.5. Analysis

Identification was performed in three steps:


estimated voluntary torque (TQˆ v ) from total measured torque (TQtot).

## 4.2. Results

#### 4.2.1. EMG-Torque Dynamics Estimation

**Figure 11** shows the joint position, visual command, full-wave rectified Soleus EMG and measured and predicted voluntary torque from a typical UT trial. The model estimated between Soleus EMG and torque, predicted the torque extremely well; the variance accounted for was 93% for this subject and 92 ± 3% for all subjects. **Figure 11E** shows the measured and estimated transient torques. These were obtained by filtering the torques with a moving average Butterworth low-pass filter to remove the slow time-varying torques (sine-wave). The VAF for transient response was 81% for this subject.

#### 4.2.2. Joint Stiffness

**Figure 12** shows the position perturbation, the visual command, and the resulting torque from a typical PT trial. The voluntary torque, estimated from the UT EMG-Torque model is shown in magenta in **Figure 12C**, superimposed on the total measure torque in blue. The three lower panels show the intrinsic, reflex, and stiffness torques estimated using LPV-PC identification algorithm for the trial segment with largest variation in voluntary torque (SV). Comparing the stiffness torque and that predicted using LPV identification algorithm, it is evident that the LPV method captured the TV behavior of the system well with a VAF of 82% for stiffness torque and 95% for total torque (stiffness + voluntary torque). The total VAF was never <90% in any trial. **Figure 13** shows the LPV-PC model estimate for a typical subject. **Figure 13A** shows the TV behavior of the intrinsic dynamics and how it varies with voluntary torque. **Figure 13B** shows that the static nonlinearity has a strong uni-directional sensitivity to velocity; the slope varies with voluntary activation increasing from rest to 5 Nm and then decreased. **Figures 13C,D** show the bode diagram of estimated TIV reflex linear dynamics resembling a second-order low-pass filter system.

**Figure 14** shows the variation of estimated parameters with voluntary torque for the five subjects. The estimates of intrinsic elasticity and reflex gain (nonlinearity slope) were normalized to their maximum value for the contraction range studied for each subject to allow inter-subject comparison. The original values corresponding to data points in the Figure, can be calculated by multiplying the x-axis value by subject's MVC and y-axis value by their corresponding normalization factor. The MVC and normalization factors for each subject are given in **Table 1**. The intrinsic elasticity (K) (**Figure 14A**), monotonically increased with contraction level in all subjects. The reflex gain (**Figure 14B**) and threshold (**Figure 14C**) of the static nonlinearity systematically changed with voluntary contraction. The reflex gain increased with voluntary torque up to 10–30% MVC in different subjects and then decreased. The variation in reflex gain was higher than 50%. The reflex nonlinearity threshold also varied with voluntary torque and was not always zero as assumed in most quasi-stationary studies. Given the results of the simulation study, the estimates of threshold values may be biased but the overall trends are expected to be informative. The reflex linear block was estimated to be a second-order

predicted the voluntary torque with a VAF equal to 93%, (E) the transient torque prediction after removing the large slow-varying torque from both measured and predicted torques. The VAF for transient response was 81%.

low-pass filter with delay varying between 40 and 45 ms (see **Table 1**). **Figures 14D,E** show the gain and phase of reflex linear dynamics represented in frequency domain. The bandwidth of reflex pathway varies between 1.65 and 2.9 Hz in subjects examined in this work.

## 5. DISCUSSION

This paper investigated and quantified the effects of voluntary contractions on ankle joint dynamic stiffness and its intrinsic and reflex components. Previous work has demonstrated that voluntary muscle activation causes substantial changes of stiffness during functional tasks (Ludvig and Perreault, 2014). Thus, studying this system during large, continuous variations in voluntary contraction will lead to better understanding of the control of movement. We used a subspace LPV-PC identification algorithm to track stiffness changes during large, isometric voluntary torque contractions. We first validated the method using a Monte-Carlo simulation study. These demonstrated that the method yielded estimates that were accurate, precise (thus reliable) and capable of capturing time-varying stiffness changes similar to those expected from quasi-stationary results, efficiently. We then applied the method to experimental data acquired while healthy human subjects made large, transient voluntary contractions. Our analysis of these data showed that the stiffness dynamics varied significantly with the contraction. We believe that the system identification algorithm used in this study provides an accurate description of intrinsic and reflex stiffness dynamics throughout a voluntary contraction and so can be used to asses the contribution of each pathway to joint mechanics in functional tasks.

## 5.1. Simulation Study

We used simulations of a realistic stiffness model to validate the performance of the subspace LPV-PC identification algorithm when torque varied sinusoidally. The variation in stiffness

82% and total VAF (stiffness + voluntary torque) of 95%.

parameters with torque was obtained by interpolating the results of quasi-stationary experiments with normal human subjects. We used colored output noise with its amplitude adjusted to give an average SNR of 10 dB for each simulation trial. The true experimental noise is expected to be lower than this value (Ludvig et al., 2011). Thus, we evaluated the identification algorithm under condition that is more challenging than that actually seen experimentally. There are two main differences between our simulation study and the experimental conditions: (i) SV estimation: In the simulations we assumed that the voluntary torque could be measured and completely removed from total torque. However, in the experiments, the SV must be estimated from the recoded EMG signal. Any errors in estimating the SV will result in identification performance to be lower than that predicted from the simulations. (ii) Identification model structure: We made two assumptions about the model structure: (1) Stiffness dynamics at the ankle can be represented using a PC model structure; this model has been widely used and shown to be successful in predicting the stiffness torque for both quasi-stationary and TV conditions (Mirbagheri et al., 2000; Sobhani Tehrani et al., 2014; Jalaleddini et al., 2017), (2) The reflex pathway has a delay of 40–45ms; this is shown to be true in many studies (Stein and Kearney, 1995; Kearney et al., 1997; Mirbagheri et al., 2000). There were few assumptions about structures of the elements of the PC model. Thus, for the intrinsic pathway the linear dynamics were modeled as a nonparametric IRF whose length was limited to be less than the reflex delay. For the reflex pathway, the nonlinearity is modeled with an orthonormal expansion whose order minimize the prediction error; the linear dynamics were modeled with a parametric model whose order is determined as part of the identification. The excellent prediction ability of the resulting model demonstrates that it accurately reproduces the observed behavior. It is possible that the true structure is more complex than the PC model (i.e., involve more pathways or have complex pathways such as nonlinear-linear-nonlinear cascade). If so, the model is still useful as an approximation since an arbitrary nonlinear system can be represented by a parallel cascade of block structured elements. However, in such a case, there would no longer be a direct relation between the structure of the model and that of the underlying physiological system; this possibility must be taken into account in the interpretation of the results.

### 5.2. Dynamic Stiffness

Our experimental results showed that stiffness increased with contraction level suggesting that system became more stiff at high contraction levels. The increase in stiffness may be justified by increase in the number of cross-bridges occurring at higher contraction levels. Reflex gain increased going from rest to lowest active level (occurring between 10 and 20% MVC) and then started to decrease. The variation in reflex gain can be explained by recruitment of more muscle fibers at higher contraction levels and existence of an upper-limit in motoneuron pool excitation. The changes in the nonlinearity threshold suggest changes in motoneuron pool excitation threshold with torque levels. These results indicate that contribution of reflex stiffness is highest at low contractions and decreases as contraction level increase, whereas, intrinsic stiffness monotonically increases with contraction level. Note that we did not attempt to parameterize the LPV IRFs for the intrinsic pathway as a second-order system because: (1) intrinsic dynamics may be more complex than the I,B,K model as demonstrated recently in Sobhani et al. (2017) (2) the fitting procedure would involve nonlinear minimization that would introduce an additional source of error.

These findings are essential in understanding the role of stretch reflexes during a motor task particularly those involving low contraction levels such as the control of posture and balance. Other works suggested that intrinsic stiffness is not sufficient to maintain stable upright posture (Morasso and Sanguineti, 2002; Moorhouse and Granata, 2007). Our results show that the range of activation where reflex stiffness is significant, varies among subjects and the reflex contribution was substantial in all subjects examined in this study. Comparing our results to those reported in quasi-stationary condition, the reflex maximum contribution was found to occur around 10% MVC and above whereas this was reported to occur at 5% MVC (Mirbagheri et al., 2000). However, it is not clear whether this is due to the dynamics changes due to task or simply because of differences between the subjects who participated in these studies.

In a separate work, we used a similar approach as that used here to estimate the Hammerstein system of reflex pathway, and evaluated the variation in position-reflex EMG dynamics with contraction levels, in isometric condition (Golkar et al., 2015). It was demonstrated that both gain and threshold of static nonlinearity changed with contraction levels. The results presented in this work combined with that study gives us a comprehensive understanding of how stiffness modulates during isometric TV contractions in plantarflexors of healthy human subjects.

Given the limited dataset required for the subspace LPV-PC identification algorithm used in this study, this can be used toward exploring the effect of some other factors such as contraction history, contraction rate, and contraction trajectory on dynamics of joint stiffness. This can be achieved by repeating the experiment when: (i) the TV torque-matching task starts after a constant activation level is maintained for a short period of

representation of Reflex Linear Dynamics (D) gain, (E) phase; reflex linear dynamics was a second-order low-pass filter and cutoff frequency between 1.65 and 2.9 Hz for different subjects.

time, (ii) use torque-tracking trajectory with different bandwidths (e.g., different periods for sine-wave) as command signal, (iii) use different torque trajectories, e.g., multi-level, and compare the estimated models for each case.

#### 5.3. Comparison to Previous Works

The overall trends in our findings agree with the results of quasi-stationary studies. For example, we found that the intrinsic elasticity increased with activation level, similar to the results of Mirbagheri et al. (2000). Also, for reflex gain, We observed a behavior similar to that reported in Jalaleddini et al. (2016). Nonetheless, the magnitudes of the changes were different. We observed 50% increase in stiffness whereas Mirbagheri et al. (2000) reported this to be around 90% for the same range of contraction. Our estimates of reflex gain were similar to those of Mirbagheri et al. (2000), except that we observed a persistence of reflex contribution for a wider range of contraction levels (up to 30% for some subjects). Some other quasi-stationary works reported the maximum reflex contribution to occur around 50% MVC in dorsiflexors (Sinkjaer et al., 1988; Cathers et al., 2004). Based on our experience, this level of activation is very likely to cause fatigue which affects the reliability of results from such experiments. Also, the nominal values reported for maximum reflex contribution based on %MVC might vary among different works due to the differences in measuring the MVCs or the muscle studied.

Van Eesbeek et al. (2013) also used the LPV identification to study wrist stiffness in an activation varying task. However, their method was limited to intrinsic estimates and did not decouple the effects of reflex contribution on the total torque variations. Reflex contributions were reported to be minimal in the upper arm (Bennett et al., 1992) but found to be significant in the ankle (Kearney et al., 1997), wrist (Sinkjær and Hayashi, 1989), and knee (Ludvig and Perreault, 2014). Consequently, the results of Van Eesbeek et al. (2013) cannot be directly compared to ours. Also, the range of activation is very different in the wrist compared to the ankle. Nevertheless, they showed that the main variation in intrinsic parameters at human wrist was in the elastic parameter, variations in viscosity were small and the inertia was found invariant. This is consistent with our results.

Other studies have used ensemble-based method to evaluate the effect of activation level on joint stiffness. Visser (2010) studied ankle joint stiffness during a sinusoidal torque matching task, where a monotonic increase in elastic parameter with voluntary torque was observed similar to the observation of this study. The main difference with our results was that Visser (2010) found two peaks in the reflex gain at the lowest and highest activation levels. Also, Ludvig and Perreault (2014) used a similar ensemble-based method to study knee stiffness during rapid activation and reported similar results for the elastic parameter. Nonetheless, using ensemble-based methods for activation-varying experiments have a number of drawbacks. It requires the exact same time-varying behavior to be repeated many times while (i) it is extremely difficult to match muscle activation levels between trials, (ii) the muscle recruitment strategy might change to avoid fatigue, (iii) antagonist muscle(s) might be activated in some trials to assist the tracking task, (iv) occurrence of fatigue is inevitable especially if activation levels above 30% are used in the study, (v) the desired torque trajectory needs to be slow enough so that subject can repeat the same task many times, and (vi) system behavior may change from the first experiment to the last one considering the large number of trials required.

The LPV identification algorithm described here, models the underlying dependency of system parameters on torque mean and thus should predict the response to novel trajectories for similar conditions. This predictive ability is a strong asset for studying physiological systems. The experiments described here were not designed to demonstrate this ability but are an important next step. In addition, it is not yet known how this predictive ability depends on the temporal and amplitude properties of the SV. This is an important topic for future work.

## 5.4. Limitations of the study

In this study, we used the subspace LPV-PC algorithm and identified a nonlinear model of both intrinsic and reflex ankle stiffness during isometric, time-varying contractions. The model accurately predicted non-stationary torques recorded from experiments with five healthy subjects. In the identified subspace LPV-PC model, the time-varying behavior of the joint was related to background voluntary torque, instead of time, defined as the scheduling variable. Consequently, it provided insight into functional relationships underlying biomechanics of the joint. Also, the model is expected to predict joint response to novel time trajectories of isometric muscle contractions. However, this study has some limitations too, including:

• It assumed that the time-varying behavior of the joint is a function of an a priori known scheduling variable. This assumption was valid for the slow isometric contraction experiments of this study. However, may not hold for other situations such as muscle fatigue, rapid contractions, or neuromuscular disorders where the SV is not well known. Similarly, it will almost certainly not hold in functional tasks where stiffness parameters depend on multiple variables. For example, during most movements both torque and position change; stiffness parameters are known to depend strongly on both, so it is to be expected that modeling this behavior would require at least two SVs.


Finally, note that this study was performed under open-loop experimental conditions, where the perturbing actuator was many times more stiffness than the ankle. Consequently, the torque generated at the ankle could not change the position of the actuator. This is not the case when subjects interact with compliant loads, where closed-loop conditions may arise. The subspace family of identification algorithms are believed to work with data acquired in closed-loop conditions (Van Wingerden and Verhaegen, 2009); however, validating this with experimental data acquired specifically for LPV-PC modeling of joint stiffness is a subject of future work.

#### 5.5. Clinical Significance

The subspace LPV-PC method would be an invaluable tool for objective and quantitative assessment of neuromuscular performance (or impairment) and motor function (or dysfunction). In fact, the early signs of recognizing the clinical benefits of exploiting system identification and modeling approach have recently appeared in the literature (Meskers et al., 2015; Sloot, 2016), where, for example, system identification was used to assess motor dysfunction in children with cerebral palsy. The subspace LPV-method can actually enable and expedite this shift from conventional scoring techniques to model-based clinical assessment, diagnosis, and treatment recommendation. Few of the reasons are:


## REFERENCES


the mechanical and reflex contributions to the abnormal joint mechanics, which is very important from a clinical standpoint. Thus, the method will have significant clinical benefits for diagnosis and treatment monitoring of patients suffering from neuromuscular diseases such as cerebral palsy, spinal cord injury, stroke, and Parkinson's disease.

#### ETHICS STATEMENT

This study was carried out in accordance with the recommendations of McGill University Research Ethics Board with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the McGill University Research Ethics Board.

## AUTHOR CONTRIBUTIONS

MAG implemented the simulation study, collected the experimental data, and performed analysis on experimental data. EST developed the identification algorithm. MAG and EST contributed to the execution and drafting of this paper and the work was supervised, reviewed, and approved by REK.

## ACKNOWLEDGMENTS

This paper was made possible by NPRP grant #6-463-2- 189 from the Qatar National Research Fund (a member of Qatar Foundation). The statements made herein are solely the responsibility of the authors. This work was also supported by a FQRNT doctorate scholarship to MAG.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Golkar, Sobhani Tehrani and Kearney. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Sensor-Motor Maps for Describing Linear Reflex Composition in Hopping

Christian Schumacher\* and André Seyfarth

Lauflabor Locomotion Laboratory, Centre for Cognitive Science, Institute of Sport Science, Technische Universität Darmstadt, Darmstadt, Germany

In human and animal motor control several sensory organs contribute to a network of sensory pathways modulating the motion depending on the task and the phase of execution to generate daily motor tasks such as locomotion. To better understand the individual and joint contribution of reflex pathways in locomotor tasks, we developed a neuromuscular model that describes hopping movements. In this model, we consider the influence of proprioceptive length (LFB), velocity (VFB) and force feedback (FFB) pathways of a leg extensor muscle on hopping stability, performance and efficiency (metabolic effort). Therefore, we explore the space describing the blending of the monosynaptic reflex pathway gains. We call this reflex parameter space a sensor-motor map. The sensor-motor maps are used to visualize the functional contribution of sensory pathways in multisensory integration. We further evaluate the robustness of these sensor-motor maps to changes in tendon elasticity, body mass, segment length and ground compliance. The model predicted that different reflex pathway compositions selectively optimize specific hopping characteristics (e.g., performance and efficiency). Both FFB and LFB were pathways that enable hopping. FFB resulted in the largest hopping heights, LFB enhanced hopping efficiency and VFB had the ability to disable hopping. For the tested case, the topology of the sensor-motor maps as well as the location of functionally optimal compositions were invariant to changes in system designs (tendon elasticity, body mass, segment length) or environmental parameters (ground compliance). Our results indicate that different feedback pathway compositions may serve different functional roles. The topology of the sensor-motor map was predicted to be robust against changes in the mechanical system design indicating that the reflex system can use different morphological designs, which does not apply for most robotic systems (for which the control often follows a specific design). Consequently, variations in body mechanics are permitted with consistent compositions of sensory feedback pathways. Given the variability in human body morphology, such variations are highly relevant for human motor control.

Keywords: feedback pathways, hopping, motor control, functional decomposition, neuromechanics, multisensory integration, muscle-tendon function, sensor-motor map

#### Edited by:

Massimo Sartori, University of Twente, Netherlands

#### Reviewed by:

Sergiy Yakovenko, West Virginia University, United States Virginia Ruiz Garate, Fondazione Istituto Italiano di Technologia, Italy

#### \*Correspondence:

Christian Schumacher schumach@sport.tu-darmstadt.de

Received: 31 December 2016 Accepted: 13 November 2017 Published: 27 November 2017

#### Citation:

Schumacher C and Seyfarth A (2017) Sensor-Motor Maps for Describing Linear Reflex Composition in Hopping. Front. Comput. Neurosci. 11:108. doi: 10.3389/fncom.2017.00108

## 1. INTRODUCTION

The redundancy of the musculoskeletal and neural systems poses a major challenge in human locomotion research. For instance, the motor control system may utilize different strategies for performing specific motions with redundancy in the body's physiology (e.g., many involved muscles), kinematics (e.g., redundant motion trajectories) (Bernstein, 1967), and neuromuscular control (e.g., recruitment of motor units) (Henneman et al., 1965) including neural networks in the spinal cord that contribute substantially to controlling rhythmic and repetitive motions. To date, it is unknown how the neuromuscular system explores and exploits the redundancy and how the different levels are organized and interconnected to achieve functionally relevant activation patterns (Donelan and Pearson, 2004b). Proprioceptive feedback and central pattern generators (CPGs) presumably generates appropriate motor control commands depending on the tasks and the phase of the motion (Dietz, 1992; Taube et al., 2012).

Similarly, computational approaches aiming at mimicking human activation patterns and motion trajectories must address the "redundancy problem" in motor control. Most commonly, these approaches reduce the degrees of freedom by specific neuromuscular structures or hierarchies (e.g., specific combinations of CPGs, sensory pathways etc.) that follow certain control policies or rules. For instance, Song and Geyer (2015) used multiple "spinal modules" (decentralized feedback control) coordinated by a supra-spinal layer to predict several gaits and generate robust behavior even after perturbations (Song and Geyer, 2017). Other studies used combinations of CPGs and proprioceptive feedback (modifying the central patterns) to generate appropriate activation patterns (Taga, 1998; Ogihara and Yamazaki, 2001; Hase et al., 2003; Paul et al., 2005). Moreover, muscle synergies (groups of synchronized cocontracting muscles during a motion) are used to reduce the dimensionality and thus the redundancy of the neuromuscular system (D'Avella et al., 2003; Bizzi et al., 2008). For instance, Ting et al. (2012) used a neuronal network for generating a muscle synergy driven balancing task based on center of mass (COM) kinematics.

In contrast to previous studies with a detailed representation of the neural networks (including their hierarchies), this study focused on integrating multiple sensory pathways at the elementary sensor-motor-level (Loeb, 1995) to determine how individual reflex pathways of muscle force (FFB), fiber length (LFB) and velocity (VFB) can support—in isolation and in combination—the repulsive leg function (Sharbafi and Seyfarth, 2017) during the stance phase of hopping (Haeufle et al., 2012). By blending individual sensory pathways, we investigated the capacity of the neuromuscular feedback system to generate goaldirected motions. We visualized and evaluated the space in which the monosynaptic reflex system can operate to generate functional motions (for generating stable, performant or efficient hopping). We call these reflex parameter spaces sensor-motor maps and suggest that studying their topology can be used to explore the redundancy of multisensory integration. This approach differs from previous approaches because the general concept of such sensor-motor maps only relies on a few primitive assumptions on the neuromuscular structure. The topologies of these sensor-motor maps reflect the task-specific contributions of the sensory pathways that are moderated by the mechanical interaction of the locomotor system with the environment. Our overall goal was to identify enabling and disabling pathways for individual locomotor functions. We expected that several different pathways may generate stable hopping, but that pathway-specific features determine hopping performance and efficiency.

To show the general validity of our approach we varied parameters of the environment (ground compliance) and the body morphology: compliance (tendon elasticity), geometry (segment lengths) and inertia (body mass). Furthermore, we explored the sensitivity of the model to variations of feedback and model parameters.

## 2. MATERIALS AND METHODS

To focus on the integration of different sensory pathways, we considered a highly simplified muscle-driven model allowing the evaluation of motion execution with respect to stability, performance and efficiency. Therefore, we used the hopping model by Geyer et al. (2003) with idealized sensory receptors and motorneurons capturing the basic neural control principles with the least possible system complexity (Full and Koditschek, 1999; Brown and Loeb, 2000; Pearson et al., 2006). We chose the signals of three muscle receptors (muscle force of Golgi tendon organs, fibre length and fibre velocity of muscle spindles) to focus on local proprioceptive circuits.

## 2.1. Mechanical Hopping Model

The model of Geyer et al. (2003) consists of a point mass m (center of mass, COM) and two massless segments (length lS) representing the thigh and shank (**Figure 1**). The leg length during flight (l<sup>f</sup> ) is held constant until the vertical COM height equals the flight leg length (touch-down). During stance, a muscle-tendon-complex (MTC) modeling the knee extensors counteracts the gravitational force (gravitational constant g). The MTC consists of a contractile element (CE) and a serial elastic element (SE) (Equations 1, 2). Take-off occurs when the leg force vanishes or when the vertical displacement of the point mass exceeds the flight leg length.

$$l\_{\rm MTC} = l\_{\rm CE} + l\_{\rm SE} \tag{1}$$

$$F\_{MTC} = F\_{CE} = F\_{SE} \tag{2}$$

The length of the MTC is defined by a reference length (lMTC,ref), a corresponding reference knee angle (ϕref) and the knee lever arm (d): lMTC = lMTC,ref − d (ϕ − ϕref). The force of the CE is calculated as FCE = Fmax ∗ f<sup>l</sup> ∗ f<sup>v</sup> ∗ ACT using the maximum isometric force (Fmax), force-lengthrelationship (f<sup>l</sup> ), force-velocity-relationship (fv) and activation state of the contractile element (ACT, see Equation 11). The force-length-relationship and force-velocity-relationship

are implemented by non-linear approximations (Geyer et al., 2003):

$$f\_l(l\_{\rm CE}) = \exp\left(c \left| \frac{l\_{\rm CE} - l\_{\rm opt}}{l\_{\rm opt} \,\, \omega} \right|^3 \right) \tag{3}$$

$$f\_{\boldsymbol{\nu}}(\boldsymbol{\nu\_{CE}}) = \begin{cases} N + (N - 1) \frac{\nu\_{\text{max}} - \nu\_{\text{CE}}}{7.56 \text{ K } \boldsymbol{\nu\_{CE}} - \nu\_{\text{max}}} & \boldsymbol{\nu\_{CE}} \ge \mathbf{0} \\ \frac{\nu\_{\text{max}} - \nu\_{\text{CE}}}{\nu\_{\text{max}} + K} & \boldsymbol{\nu\_{CE}} < \mathbf{0} \end{cases} \tag{4}$$

These equations use a width (w) and a curvature constant (c) of the force-length-curve as well as optimum length of the CE (lopt), eccentric force enhancement (N), maximum shortening velocity (vmax) and a second curvature constant (K). The force-length-relationship values can range from 0 to 1. The force-velocity-relationship values can range from 0 to 1 for concentric contractions and from 1 to 1.5 for eccentric contractions (because of the eccentric force enhancement N). To define the serial elastic element in the MTC a progressive force-length dependency (Equation 5) was used (van Ingen Schenau, 1984). Therefore, the reference strain (εref ) determines the relation of the force acting on the serial element and its corresponding stretch in relation to its rest length (lrest) (Equation 6).

$$f\_{\rm SE}(\varepsilon) = \begin{cases} (\varepsilon/\varepsilon\_{\rm ref})^2 & \varepsilon > 0\\ 0 & \varepsilon \le 0 \end{cases} \tag{5}$$

$$\varepsilon = \left(\frac{l\_{SE}}{l\_{rest}}\right) - 1\tag{6}$$

#### 2.2. Extension of the Neuromuscular Model

To consider fused feedback pathways, we extended the neuromuscular feedback model by a linear combination of muscle force (FFB), fibre length (LFB) and fibre velocity feedback (VFB) pathways (**Figure 2**). All three afferent pathway signals are multiplied by a blending factor λ<sup>i</sup> weighting the individual pathways resulting in the summation signal S(t) (Equation 7) where G<sup>i</sup> , Fmax, Loff and Voff denote the individual gains, maximum isometric force and offsets of length and velocity pathways, respectively. By restricting the sum of all blending factors (Equation 8), one weight can always be calculated from the other two (Seyfarth et al., 2001).

$$S(t) = \lambda\_F \* G\_F \* F\_{CE}/F\_{max}$$

$$+ \lambda\_L \* G\_L \* (l\_{CE} - L\_{off})\tag{7}$$

$$+\lambda\_V\*G\_V\*\left(\nu\_{CE}-V\_{\mathcal{off}}\right)$$

$$\lambda\_F+\lambda\_L+\lambda\_V=1,\quad 0\le\lambda\_{F,L,V}\le 1\tag{8}$$

This normalizes the blending of individual contributions and reduces the dimensionality by projection onto a two-dimensional space (of independent blending factors). Triangles visualize all possible (projected) feedback compositions (**Figure 3**). The corners of the triangle represent the isolated individual feedback pathways (purely FFB, LFB, or VFB). Every point within the triangle represents a blending of the individual feedback pathways and refers to two-dimensional Cartesian coordinates (space V, x, and y between 0 and 1). The projection to the blending factors (space W) is described by: <sup>f</sup> :<sup>V</sup> <sup>→</sup> <sup>W</sup> (Equation 9). Hence, the larger the distance of a point to a corner (e.g., VFB) the smaller is the contribution of that specific feedback pathway in the blended signal S(t). Individual feedback pathways in isolation (corners) are parameterized by optimization (see section 2.3).

$$f = \begin{cases} \varkappa \to \lambda\_L & 0 \le \varkappa \le 1 \\ \varkappa \to \lambda\_F & 0 \le \varkappa \le 1 \end{cases} \tag{9}$$

After blending, the proprioceptive signal (S(t)) is delayed by 1S, gained by G and added to the stimulation bias (PreStim)

FIGURE 2 | Neuromuscular reflex model which fuses Ia afferent signals (lCE , vCE and offsets Loff , Voff ) and normalized Ib afferent (FCE ) pathways: All three sensory signals are gained (GL , G<sup>V</sup> , G<sup>F</sup> ≥ 0) and weighted (λ<sup>i</sup> ). The resulting summation signal S(t) is then delayed (1S) and gained (G ≥ 0). This signal is then added to a constant pre-stimulation value to mimic a positive excitatoric postsynaptic potential at the α-motoneuron. The stimulation signal STIM(t) is confined to values between 0 and 1 and delayed by the excitation-contraction-coupling (ECC) resulting in the activation signal ACT(t) of the CE.

FIGURE 3 | Schematic explanation of sensory pathway blending through parameter reduction (Equation 8). Corners of the triangle represent a full contribution of one single feedback (e.g., 0% VFB, 0% FFB, 100% LFB). Every point within the triangle (coordinates x and y) represents a unique combination of the three feedback pathways. Arrows (thick) and contour lines (thin) explain the blending of feedback pathways. The larger the distance of a point to a corner (e.g., LFB) the smaller the contribution of that feedback pathway in the blended signal. Exemplary, the middle point describes an equal composition of all feedback pathways (1/3 VFB, 1/3 FFB, 1/3 LFB). Optimal tuning of the individual feedback pathways is done by optimization for full contribution of only one sensory reflex (corners). These optimal reflex pathways are used for all compositions (see section 2.3). The blue circles (S1(F) to S7(F, L, V)) denote the specific compositions of feedback pathways used for the sensitivity analysis (see section 2.3.8).

(Equation 10). Then, this signal is confined to values between 0 and 1 and input into the excitation-contraction-coupling (Equation 11) described by a first-order differential equation resulting in the activation signal ACT(t) (Geyer et al., 2003; Haeufle et al., 2012).

$$\text{STIM}(t) = \begin{cases} \text{PreStime} & t < \Delta\_S \\ \text{PreStime} + \text{G} \* \text{S} (t - \Delta\_S) & t \ge \Delta\_S \end{cases} \tag{10}$$

$$\frac{\text{d}ACT(t)}{\text{d}t} = \text{STIM}(t) - \text{ACT}(t) \tag{11}$$

#### 2.3. Model Parameter and Optimization 2.3.1. Model Parameters

τ

Parameters of the mechanical model (**Table 1**) were taken from Geyer et al. (2003). The initial position of the point mass was set to 1.05 m and its initial velocity to 0 m/s.

#### 2.3.2. Optimization of Feedback Parameters

To identify feedback parameters of the extended neuromuscular model (optimal tuning part in **Figure 2**), a pattern search optimization algorithm was used (Torczon, 1997). The pattern search algorithm was implemented to search for parameters sets that result in stable hopping patterns (more than n = 50 steps, first criterion). As second optimization criterion, the maximum height of the body mass hmax = ymax,apex for steady-state hopping (n = 49 step) was chosen. Simulations were checked for steady-state motion. Optimizations of all individual feedback pathways (in isolation) were done for "stiff tendon" and "rigid ground" and repeated five times each with random starting points to avoid finding local maxima. The limits of parameter values were 0.1 ≤ G<sup>F</sup> ≤ 3 (FFB), 0.1 ≤ G<sup>L</sup> ≤ 200 and 0 ≤ Loff ≤ 3 (LFB) as well as 0.1 ≤ G<sup>V</sup> ≤ 3 and −1 ≤ Voff ≤ 0 (VFB) aligned to results from Geyer et al. (2003). The best performing solution was used for further simulation and analysis.

#### 2.3.3. Simulation and Optimization Environment

Simulations and optimizations were implemented in Matlab Simulink (release 2016b, Mathworks, Natick, Massachusetts,



USA). For the simulations, the variable-step solver "ode45" with relative and absolute tolerances of 10−<sup>8</sup> was used. Optimization was done using the Global Optimization Toolbox (Version 3.4.1).

#### 2.3.4. Tendon Elasticity Changes

To change the SE elasticity, three configurations for the reference strain were used: (1) "stiff tendon" (εstiff = 0.01), (2) "moderate tendon" (εmoderate = 0.03), and (3) "compliant tendon" (εcompliant = 0.05). For equal forces, smaller reference strain values indicated less associated stretch and thus a stiffer lengthforce dependency of the SE. These SE elasticity levels are in a range used by other simulation studies (Pandy et al., 1990; Bobbert, 2001; Nagano et al., 2004).

#### 2.3.5. Body Mass Changes

The body mass of the model (m) was varied to 80 and 120% of the original body mass (80 kg): (1) "light mass" (mlight = 64 kg), (2) "moderate mass" (mmoderate = 80 kg), and (3) "heavy mass" (mcompliant = 96 kg).

#### 2.3.6. Segment Length Changes

The leg geometry was altered by changing the length of both segments (lS) to 80 and 120% of the original segment length (0.5 m): (1) "short segments" (lS,short = 0.4 m), (2) "moderate segments" (lS,moderate = 0.5 m), and (3) "long segments" (lS,long = 0.6 m). To keep the take-off conditions and energy level of the system consistent for all segment length configurations, the initial body mass height (y<sup>0</sup> = 2 ∗ l<sup>S</sup> + 0.05 m) and the flight leg length (l<sup>f</sup> = 2 ∗ l<sup>S</sup> − 0.01 m) were adjusted accordingly.

#### 2.3.7. Ground Compliance Changes

To modulate the vertical ground stiffness, the model was slightly adapted. During stance, a linear spring constant (kground) and the leg force or vertical ground reaction force (Fleg ) define the foot position (yFP):

$$\mathcal{y}\_{FP}(F\_{\log}) = \begin{cases} \mathcal{y}\_{COM} - \mathcal{l}\_f & \text{during flight} \\ -\frac{F\_{\log}}{k\_{\text{ground}}} & \text{during state} \end{cases} \tag{12}$$

The foot position during stance can only reach values ≤ 0 because the take-off condition is met for vanishing leg force (Fleg < 0). To change the ground compliance, three configurations for the spring constant were chosen: (1) "compliant ground" (kcompliant = 100 kN/m), (2) "moderate ground" (kmoderate = 500 kN/m), and (3) "stiff ground" (kstiff = 9, 999 kN/m). These ground stiffness values are in a range used by other computational or experimental studies (Farley et al. (1998): 20–35,000 kN/m, Moritz and Farley (2004): 27– 411 kN/m, van der Krogt et al. (2009): 75–3,100 kN/m).

#### 2.3.8. Sensitivity Analysis

To evaluate the performance of the model for different parameter settings, we analyzed its parametric sensitivity. We randomly altered feedback parameters (GF, GL, Loff , GV, Voff , PreStim, 1S) as well as model parameters (εref , lS, m). Physiological parameters were normally distributed (lS: µ = 0.5 m, σ <sup>2</sup> <sup>=</sup> 0.02 m; m: µ = 80 kg, σ <sup>2</sup> <sup>=</sup> 5 kg) whereas other parameters were uniformly distributed (1 ≤ G<sup>F</sup> ≤ 5; 100 ≤ G<sup>L</sup> ≤ 160; 0.06 m ≤ Loff ≤ 0.1 m; 1 ≤ G<sup>V</sup> ≤ 5; −1 m/s ≤ Voff ≤ 0 m/s; 0.01 ≤ PreStim ≤ 0.2; 0.01 s ≤ 1<sup>S</sup> ≤ 0.05 s; 0.01 ≤ εref ≤ 0.05). For the sensitivity analysis, the ground stiffness remained unchanged (no compliance). Because parametric influences differ depending on the feedback blending, we considered seven reflex pathway compositions for our sensitivity analysis. **Figure 3** shows the location of these seven compositions (S1(F) to S7(F, L,V)). For each composition, n = 1,000 simulations with randomized parameters were performed. Maximum hopping height (1hmax) and hopping efficiency (η) were calculated as performance measures (see section 2.4.2). The sensitivity of these variables was further tested with SPSS 24.0. (IBM Corporation, Armonk, New York, USA). Spearman's rho correlation coefficients (r) with significance values (two-sided test) and standardized regression coefficients (β) were calculated for simulations that resulted in stable hopping. Correlations were considered to be moderate for 0.5 ≤ r < 0.7 (−0.5 ≥ r > −0.7) or high for r ≥ 0.7 (r ≤ −0.7) if p-values were significant (p < 0.01).

#### 2.4. Performance Metrics

Depending on the force generated during stance, the predicted motion will result in continuous and stable hopping or in a bound motion (leg remains in contact to the ground) where the model lands but does not lift off. In case of hopping, the blending compositions were evaluated by calculating the following metrics.

#### 2.4.1. Stability Criterion

To determine if the extended neuromuscular reflex model will result in stable hopping or bound motion, we examined the number of steps to fall, and simulations resulting in at least 50 steps were considered stable.

#### 2.4.2. Hopping Metrics

The following criteria were used to evaluate the performance of the hopping model with respect to energetics and motion dynamics for the last step (n = 49) of the simulation. Simulations were checked for steady-state motion.


$$\alpha = \frac{W\_{\text{CE,max}}}{W\_{\text{MTC,max}}} = \frac{F\_{\text{MTC,max}} \ast \Delta l\_{\text{CE,max}}}{F\_{\text{MTC,max}} \ast \Delta l\_{\text{MTC,max}}} = \frac{\Delta l\_{\text{CE,max}}}{\Delta l\_{\text{MTC,max}}} \tag{13}$$

This factor describes the maximum amount of work produced in the muscular element relative to the overall maximum contribution of the MTC.

4. To evaluate the metabolic effort of the CE, we used the velocity-dependent metabolic cost model by Minetti and Alexander (1997) and Robertson and Sawicki (2014) favoring eccentric contractions with reduced metabolic effort (Equation 15). This is scaled by the activation signal during ground contact (ACT(t)), maximum isometric Force (Fmax) and maximum shortening velocity (vmax) to calculate the metabolic rate (Meff (t)) (Krishnaswamy et al., 2011; Robertson and Sawicki, 2014):

$$M\_{\rm eff}(t) = \Phi(\nu\_{\rm CE}) \ast \, ACT(t) \ast \, | \, F\_{\rm max} \, \ast \, \nu\_{\rm max} \, | \,\tag{14}$$

$$\nu\_{\rm{CF}} = \begin{cases} 0.23 - 0.16 \ast e^{( -8 \ast \frac{\rm{vCE}}{V\_{\rm{max}}})} & \nu\_{\rm{CE}} \ge \mathbf{0} \\ & \cdot \end{cases} \quad \nu\_{\rm{CE}} \ge \mathbf{0}$$

$$\Phi(\nu\_{\rm CE}) = \begin{cases} \frac{\nu\_{\rm CE}}{\nu\_{\rm CE}} & \text{even} \cdot \frac{\nu\_{\rm CE}}{\nu\_{\rm max}} + 0.06 \ast e^{\left(8 \ast \frac{\nu\_{\rm CE}}{\nu\_{\rm max}}\right)} & \nu\_{\rm CE} < 0 \end{cases}$$

$$\begin{array}{cc} \text{(15)} & \text{f} \\ \end{array}$$

TABLE 2 | Optimization results of individual feedback parameters (y<sup>0</sup> = 1.05 m, G = 1, PreStim = 0.01, εstiff = 0.01, rigid ground).


We derived the averaged metabolic effort (Meff ) per hopping cycle by an integration of the metabolic rate (Meff (t)) during ground contact and a normalisation with the body mass (m) and the contact time (Tcontact) (Robertson and Sawicki, 2014):

$$\overline{M}\_{\rm eff} = \int\_0^{T\_{\rm contact}} M\_{\rm eff}(t) \, dt/(m\*T\_{\rm contact}) \tag{16}$$

5. Hopping efficiency was quantified as the ratio of hopping height (1hmax) (output) to averaged metabolic effort of the CE (input): η = 1hmax Meff .

#### 3. RESULTS

#### 3.1. Individual Hopping Patterns

The optimization of the individual feedback pathways (with εstiff = 0.01) resulted in neuromuscular model parameters that produced a maximum hopping height for stable hopping patterns (**Table 2**). For these feedback parameters, FFB was the best performing optimization with a hopping height of 0.126 m. The maximum hopping height of LFB was 0.063 m. VFB did not produce a high performance with a maximum hopping height of 0.002 m just above the flight leg length. The predicted leg forces and activation signals are shown in **Figure 4**. The activation profile and subsequently the leg force profiles of FFB showed an increasing amplification. Compared to leg forces of FFB, LFB produced higher peak leg forces but shorter contact times. The rise of the LFB activation signal was delayed to the instance of touch-down by about 50 ms. Here, the length offset Loff suppressed the early activation signal (also reported by Geyer et al., 2003). VFB produced half the leg force and half the contact time compared to FFB and LFB reflected in the small hopping height (0.002 m). While FFB and LFB showed delayed increase in the leg force (more than 50 ms after touch-down), the VFB caused an almost instantaneous response in the activation signal resulting in high (eccentric) force generation and thus high energy losses during leg compression. The CE remained less stretched and started to shorten before reaching optimal fibre length (f<sup>l</sup> < 0.2) limiting positive (concentric) muscle work during leg extension.

#### 3.2. Sensor-Motor Maps

The following section describes the results of the blended feedback pathways and the Sensor-motor maps for different

motion characteristics (e.g., hopping stability, performance and efficiency).

#### 3.2.1. Hopping Stability

The blended feedback pathways produced both stable and unstable motions (**Figure 5A)**. Motions of stable hopping (more than 50 hops) were found for compositions of FFB and LFB with small proportions of VFB. Here, a balanced composition of FFB and LFB resulted in greater stability (with respect to higher VFB proportion) compared to predominant FFB. A thin envelope of transitions (between 1 and 49 steps) was observed representing a distinct margin of stable and unstable areas.

#### 3.2.2. Hopping Performance

The performance map (**Figure 5B**) shows the maximum hopping height (1hmax) for all feedback compositions of steady-state motions where only stable predictions were considered. The contours show greater hopping heights for smaller proportions of VFB. In areas close to unstable solutions, the maximum hopping height (maximum vertical displacement of COM) was just above the leg length leading to smooth transitions from unstable (no hopping) to slight hopping patterns. Thus, the energy level of the system (Esystem) gradually increased when VFB was reduced. Compared to LFB, high proportions of FFB performed better, and higher hopping heights occurred closer to pure FFB. A composition of FFB and LFB (but not VFB) produced maximum performance (see red point in **Figure 5B**). Although, compared to individual contributions (e.g., pure FFB or LFB) hopping performance was amplified by blending multiple sensory pathways, the pathway-specific feature enabling motion performance (Geyer et al., 2003) was found for dominant FFB.

#### 3.2.3. Hopping Efficiency

To identify feedback compositions resulting in hopping patterns that required less metabolic resources than others, the energetic relation of output and input: η = 1hmax Meff was used. In the topology of the efficiency map (**Figure 5C**), efficient motions were predicted in areas with dominant LFB and only small proportions of VFB (below 0.2), and a small band of efficient hopping patterns evolved. The spectrum of this band ranged from small proportions of FFB to pure LFB and gradually spread toward pure LFB. The most efficient feedback blending was found for a combination of small FFB, dominant LFB and no proportion of VFB (see maximum). Because VFB resulted in lower hopping heights (**Figure 5B**), VFB also reduced the hopping efficiency (η). Moreover, only moderate hopping heights led to most efficient hopping (**Figure 5C**). For the used metabolic model, hopping efficiency increased if the amount of positive work used for propulsion (and consequently hopping height) was reduced. Higher proportions of LFB led to lower force (and also work) production during late stance caused by a reduced activation signal in late stance due to the length offset (**Figure 4B**). In areas of higher hopping heights (dominant FFB and minor VFB), the metabolic model predicted a high metabolic effort leading to low efficiency values. As observed for hopping performance, the most efficient hopping pattern was found for a fusion of sensory pathways.

#### 3.2.4. Hopping Motion

To evaluate and compare the predicted hopping motions with human hopping data (where possible), we calculated biomechanical parameters of the motions. Hopping frequencies ranged from 1.5 to 3.0 Hz (**Figure 6A**). Higher frequencies were found for higher VFB and hence in areas of smaller hopping heights (**Figure 5B**). The effective leg stiffness of our hopping model ranged from 15 to 30 kN/m (**Figure 6B**). This parameter depended mostly on the relation of VFB and LFB but was slightly influenced by increases in FFB (see vertical contour lines). Thus, the model produced motions of the best performing and most efficient compositions at small hopping frequencies (around 1.5 Hz) and low leg stiffness (around 15 kN/m).

#### 3.3. Robustness of Sensor-Motor Maps

To explore the robustness of the sensor-motor maps, the effects of parameter variations of the model configuration

explanation of triangles). (A) Stability map: steps to fall (max = 50), (B) Performance map: maximum hopping height (1hmax) and (C) efficiency map: hopping efficiency (η).

(tendon elasticity εref , body mass m, segment lengths lS) and the environment (ground compliance kground) were analyzed. Moreover, we investigated the parametric sensitivity of the model to variations of feedback and model parameters.

#### 3.3.1. Tendon Elasticity Changes

The three performance maps of altered elasticity of the serial elastic element showed only slight differences (**Figure 7A**). For all three tendon elasticity configurations, the size and location of stable hopping patterns were consistent, and smooth transitions from unstable to stable hopping patterns (with only small hopping heights) were predicted. While the topology of the performance maps remained similar (compared to the "moderate tendon"), the level of the predicted hopping height changed. The greatest hopping heights were found for a more compliant tendon and gradually decreased for stiffer configurations. Maximum hopping heights for each configuration ranged from 0.18 m ("compliant tendon") to 0.15 m ("moderate tendon") to 0.13 m ("stiff tendon") and were found for consistent feedback compositions. For the stiffest elasticity (εstiff = 0.01), a second margin of stable solutions for high proportions of the VFB evolved. However, these feedback compositions resulted in hopping patterns just above the leg length (see also **Figure 4C**).

#### 3.3.2. Body Mass Changes

For all body mass configurations, performance map regions of stable hopping emerging for FFB and LFB remained similar (**Figure 7B**). A reduction of the body mass was predicted to result in higher maximum hopping height while the blending location of the most performant hopping patterns did not change. For the light mass, stable hopping patterns were found for dominant VFB.

#### 3.3.3. Segment Length Changes

The sensor-motor map topology remained similar for changes in the leg geometry (**Figure 7C**). Only for short segment lengths, VFB resulted in stable hopping patterns (with small hopping heights). Motions with the highest performance were found for a consistent sensory pathway blending, and the performance level increased with decreasing segment lengths.

#### 3.3.4. Ground Compliance Changes

Similar to the other parameter variations, changes in ground stiffness only minimally influenced regions of stable solutions (**Figure 7D**). Steady-state hopping heights decreased with increasing proportions of VFB (for all three ground configurations). Thus, maximum hopping heights were found for no proportions of VFB and dominant FFB, and decreased with decreasing ground compliance. The location of the maxima was consistent for different ground compliance and changes in the other parameter variations. The topology of the performance maps remained similar.

#### 3.3.5. Sensitivity of the Model

We evaluated the sensitivity of specific model and feedback parameters to predicted hopping performance (1hmax) and hopping efficiency (η) for seven feedback compositions (S1(F) to S7(F, L,V), **Figure 3**). Correlation coefficients (r), p-values and the standardized regression coefficients (β) are shown in **Table 3**. While pure VFB (S3(V)) did not generate any stable hopping pattern, for FFB (S1(F)) and LFB (S2(L)) 958 out of 1,000 simulations resulted in stable hopping. For both individual feedback pathways, hopping height was moderately influenced by the feedback signal time delay (1S): β = 0.637 (r = 0.585, p < 0.01, FFB) and β = 0.531 (r = 0.521, p < 0.01, LFB). Also, moderate correlations between hopping efficiency and reference strain (εref ) were found for FFB (β = 0.628, r = 0.643, p < 0.01) and LFB (β = 0.431, r = 0.494, p < 0.01). For all feedback compositions, the model was most sensitive to the feedback signal time delay and the reference strain of the serial elastic element. Other feedback parameters such as gains (Gi), offsets (Loff , Voff ), the pre-stimulation bias (PreStim) or the model parameters segment length (lS) and the body mass (m) did not result in moderate or high correlations.

#### 3.3.6. Muscle-Tendon Interaction

To further explore the robustness of the sensor-motor maps we investigated the muscle-tendon interaction because the elasticity of the SE also influenced the interplay of the CE and the SE. The muscle interaction maps in **Figure 8** show the calculated index α for each tendon configuration describing the relation of maximum work generated by the CE to the whole MTC. While values of α were mostly determined by the relation of VFB and LFB (see vertical contours), the map topologies were only slightly influenced by changes in serial elasticity. α values decreased with increasing tendon compliance, and α values at the location of maximum hopping heights ranged from 0.9 ("stiff tendon") to 0.75 ("moderate tendon") to 0.57 ("compliant tendon"). The related work loops show the detailed interplay of CE and SE for simulations that predicted the highest hopping heights (red points). At touch-down, the MTC was stretched by a low force. The MTC generated forces (feedback response) while being stretched (eccentric contraction) which led to negative work loops. During leg extension, the MTC shortened during force generation (concentric contraction) and produced a positive work loop. More compliant tendon resulted in slightly higher MTC deflections and less lengthening of the CE. Because maximum MTC forces did not change with different elasticity, a stiffer configuration caused less deflection and reduced the energy recoil. The energy stored in the SE decreased from 728 J ("compliant tendon") to 437 J ("moderate tendon"). For a maximal hopping height of 0.13 m, the "stiff tendon" stored the least amount of energy with 142 J.

## 4. DISCUSSION

This simulation study investigated the composition of afferent feedback pathways for generating a repulsive leg response in hopping. Therefore, a neuromuscular reflex model (Geyer et al., 2003) was extended by blending length (LFB), force (FFB) and velocity feedback pathways (VFB) of one anti-gravitational leg extensor muscle. Sensor-motor maps were derived to evaluate the predicted motion with respect to stability, performance and efficiency. The topology of the sensor-motor maps was further evaluated for different tendon elasticity, body mass, segment lengths and ground compliances. Below, we first highlight the key insights gained by our hopping model, which will then be discussed in more detail.


(y<sup>0</sup> = 1.05 m, G = 1, PreStim = 0.01). Global maxima are visualized by red points. Every point within the triangle represents a unique combination of the three feedback pathways. The larger the distance of a point to a corner (e.g., LFB) the smaller the contribution of that feedback pathway in the blended signal (see Figure 3 for explanation of triangles). (A) Tendon elasticity changes: "compliant tendon" (εcompliant = 0.05), "moderate tendon" (εmoderate = 0.03), "stiff tendon" (εstiff = 0.01); (B) body mass changes: "light mass" (mlight = 64 kg), "moderate mass" (mmoderate = 80 kg), "heavy mass" (mheavy = 96 kg); (C) segment length changes: "short segments" (lS,short = 0.4 m), "moderate segments" (lS,moderate = 0.5 m), "long segments" (lS,long = 0.6 m); and (D) ground compliance changes: "compliant ground" (kcompliant = 100 kN/m), "moderate ground" (kmoderate = 500 kN/m), "stiff ground" (kstiff = 9, 999 kN/m).

changes (ground compliance, **Figure 7D**). Thus, in our model the neuromuscular feedback system relied on a consistent topology of feedback compositions.

The modeling framework presented here can be used to establish relations to biomechanical (loco-)motion concepts (e.g., preflex Loeb, 1995; Brown and Loeb, 2000) and template models (Full and Koditschek, 1999) and to explore the capacity and physiological limitations of the biological neuromuscular system (Pearson et al., 2006), e.g., due to signal delays or muscle dynamics. Moreover, neuromuscular simulation models can be validated, improved and used for different applications, for instance to derive model-based experimental designs for investigating human or animal motor control. The results of our study should be confirmed by experimental studies in biological systems.


TABLE 3 | Results of the sensitivity analysis.

For each specific composition, n depicts the number of stable simulations (max(n) = 1, 000). Correlations r (Spearman-Rho) and standardized regression coefficients β (in parentheses) of predictors on the maximum hopping height (1hmax ) and the hopping efficiency (η). Composition S3(V) (pure VFB) did not result in any stable hopping simulations (n = 0). High (r ≥ 0.7 or r ≤ −0.7) and very significant (p < 0.01) correlations are highlighted. (y<sup>0</sup> = 1.05 m, G = 1, PreStim = 0.01, εstiff = 0.03, rigid ground). \*p < 0.05; \*\*p < 0.01. The italic font corresponds to the standartized regression coefficients, the bold font highlights high correlation and very significant results.

## 4.1. Different Feedback Pathways Have Different Functional Contributions

We found pathway-specific features that resulted in different characteristics of the hopping motion. Firstly, FFB was the dominant feedback pathway to produce high hopping heights and thus high hopping performance. Previous studies reported that the combination of the muscle force-velocity-relationship and positive force feedback (FFB) produced stable and highperforming motions (Prochazka et al., 1997b,a; Geyer et al., 2003; Haeufle et al., 2012). In contrast to a combination of the forcelength-relationship (f<sup>l</sup> ) and negative LFB (which would result in similar behavior compared to FFB Prochazka and Yakovenko, 2002), negative LFB did not provide self-stabilizing behavior (Haeufle et al., 2010). An elastic structure or the f<sup>l</sup> helpsto achieve energy efficient and spring-like behavior, but does not generate energetically stable hopping on its own (Haeufle et al., 2010). However, experimental evidence for functional force feedback of ankle extensor activity to generate a repulsive leg function has been found for walking cats (Donelan and Pearson, 2004a,b) and humans (Grey et al., 2007; af Klint et al., 2010), but may be different for hopping motions.

Secondly, efficient hopping was found for areas of moderate hopping height in which LFB is dominant. This occurred because the feedback suppression of the length offset (e.g., through fusimotor drive) reduced the positive work production of the CE in late stance phase also reported by Geyer et al. (2003) resulting in high metabolic effort. In our simulations, this suppression could be used to fine-tune the hopping efficiency by determining the length-dependent activation of the CE during stretch-shortening cycle (SSC) (Gollhofer, 2003). This influenced the force-lengthening characteristic of the MTC and thus the energy recoil of the serial elastic element (with the used tendon elasticity). Raburn et al. suggested that proprioceptive information is used to adjust a periodic bouncing motion pattern

(y<sup>0</sup> = 1.05 m, G = 1, PreStim = 0.01, rigid ground). Global maxima of hopping heights are visualized by red points. Every point within the triangle represents a unique combination of the three feedback pathways. The larger the distance of a point to a corner (e.g., LFB) the smaller the contribution of that feedback pathway in the blended signal (see Figure 3 for explanation of triangles). (A) "Compliant tendon" (εcompliant = 0.05), (B) "moderate tendon" (εmoderate = 0.03) and (C) "stiff tendon" (εstiff = 0.01). Right side: Influence of serial elasticity on individual work loops of CE (red), SE (black) and MTC (blue) for maximum hopping heights. Positive (energy generation) and negative (energy dissipation) work loops are indicated by positive and negative signs.

to achieve energetically optimal patterns (Raburn et al., 2011). If afferent pathways of muscle spindles are blocked by an ischemia blockage, the patterns are less adapted (Raburn et al., 2011). It was argued, that the observed effect was most likely caused by an update of the internal model for planning the motion. Nonetheless, implications of direct activation pattern changes as considered in our model could not be ruled out (Dean, 2012). It seems reasonable that reflex contribution directly influences the SSC (e.g., through generating "muscular stiffness" Nichols and Houk, 1976; Gollhofer, 2003) and thus the hopping efficiency (Komi, 2003).

Surprisingly, blending dominant VFB and other feedback pathways did not result in a higher performance, but lead to substantial losses in hopping height resulting from an early rise of the activation signal. Next to this extensive force generation (during leg compression) the CE remained less stretched during stance (force-length-relationship below 0.2) leading to lower push-off forces and hopping performance. This shift of the CE operation point was also found in the simulation study by Robertson and Sawicki (2014) where higher muscle stimulation frequencies (earlier stimulation onset) and magnitudes resulted in less CE lengthening. In our model, this effect was stronger than the resultant performance increase associated with an increased proportion of FFB. Thus, we found a function disabling behavior for compositions with dominant VFB and small feedback signal time delays. These results indicate that VFB might function as the primary regulator of the system energy in hopping complementing the previous suggestion that FFB might be used to control the energy state of the system (Geyer et al., 2003). McDonagh and Duncan (McDonagh and Duncan, 2002) provided evidence for the contribution of velocitysensitive afferent signals in their experimental study of landing motions. Increased electromyographic (EMG) amplitudes of gastrocnemius and rectus femoris muscles were found for increased ankle and knee joint velocities at touchdown (due to different landing heights) during false floor landings compared to expected grounds (McDonagh and Duncan, 2002). For hopping, it might be necessary to delay or inhibit the VFB for example by presynaptic inhibition to functionally enable FFB or LFB pathways. Such inhibition of VFB in hopping were indicated by results by Voigt et al. (1998) who reported a negative correlation between peak stretch velocity and EMG amplitude of the soleus muscle. However, the disabling function of the VFB may not be transferable to other motion tasks such as walking. Because such feedback mechanisms are strongly task and phase dependent the contribution of a feedback pathway may have opposite effects for different motion tasks, e.g., in walking and standing (Pearson and Collins, 1993; Donelan and Pearson, 2004b). In a biological system this mechanism might be superimposed or modulated by fusimotor action (Prochazka and Ellaway, 2012) or descending signals (see section 4.3). Thus, further research is warranted to identify the task-specific role of VFB in generating appropriate leg extensor muscle activation in humans and animals.

## 4.2. Robustness of Sensor-Motor Maps

Regions of stable and unstable hopping motions were only minimally influenced by the changes in tendon stiffness, body mass, segment lengths and ground compliance. Stable solutions for pure VFB were found for "light mass," "short segments" or "stiff tendon" and resulted in very low hopping performance. This result is in agreement with previous studies where VFB produced stable hopping only in the absence of serial compliance (Geyer et al., 2003; Haeufle et al., 2012). More compliance resulted—as expected—in greater hopping heights and more efficient hopping patterns. Similar results were found in experimental and computational studies (Anderson and Pandy, 1993; Kubo et al., 1999; Bobbert, 2001; Nagano et al., 2004). While the level of hopping height was influenced by all altered configurations, the topology of these maps was not affected. Interestingly, we found that for all parameter variations a consistent pathway composition resulted in the maximum hopping performance suggesting a unified sensormotor map topology with consistent optimal solutions. Apart from this isolated observation, the (multidimensional) sensitivity analysis also revealed only small to moderate dependencies of the predicted hopping height and hopping efficiency for changes in tendon elasticity, body mass or segment lengths.

Our results indicate that sensor-motor maps are robust against these morphological changes. Previous studies showed that the elastic leg function is highly determined by the interplay of the compliant tendomuscular system and the neuronal control system (Nichols and Houk, 1976; Lin and Crago, 2002; Gollhofer, 2003; Taube et al., 2012; Robertson and Sawicki, 2014). In our simulations, a more compliant tendon was able to store and release more energy during hopping agreeing with other studies (Anderson and Pandy, 1993; Bobbert, 2001). Moreover, the model predicted decreasing α values with increasing tendon compliance (increased tendon lengthening) revealing a compensatory behavior of the CE by stiffening (producing equal forces with less deflection). Such behavior was observed by experimental studies, where changes in ground stiffness lead to adaptations of the leg stiffness, such that the total stiffness, consisting of leg and ground, remains similar (Ferris and Farley, 1997; Moritz and Farley, 2004; van der Krogt et al., 2009). A similar stiffness adaptation was found for hopping with a passive ankle joint orthesis acting in parallel to the ankle (Ferris et al., 2006). For our model, this effect may be accomplished by the stabilizing function of the non-linear muscle properties (van Soest and Bobbert, 1993; Moritz and Farley, 2004; van der Krogt et al., 2009; Haeufle et al., 2010) as the feedback pathway composition remained the same for all three conditions. These "exploit mechanics" or "preflex" function (Loeb et al., 1999) may serve as a selector (functional filter) to offer favorable solutions for the neural motor control system (sensor-motor maps) that allow to learn its simple (Loeb, 1995) and consistent topology.

## 4.3. Integration of Spinal Reflexes and Feed-Forward Control

Although we did not consider supra-spinal motor commands, we would like to reflect our results on their functional integration with feed-forward commands. Our model suggests that the neuromuscular feedback system alone could generate appropriate adjustments of the muscle activation to permit a fine-tuning of hopping motions. In addition to our results, previous experimental studies revealed the importance of preplanned feed-forward commands in hopping and drop jumps. Descending commands may contribute during the early contact phase (Zuur et al., 2010) and toward push-off (Taube et al., 2008). Accordingly, the EMG activity of ankle extensor muscles (such as the soleus) was found to be pre-programmed and adjusted dependent on the task (e.g., jumping or landing) (Leukel et al., 2008a, 2012) and with respect to the timing of the touch-down (Santello and McDonagh, 1998; McDonagh and Duncan, 2002). Integrating feed-forward commands would certainly influence the results presented in this study. For example, pre-planned motor commands could compensate the repressive behavior following dominant VFB. Such descending commands could be superposed to our blended feedback signal (as PreStim in our model) (Taube et al., 2012) or adapt and suppress the afferent gating (G in our model), for instance by presynaptic inhibition (McDonagh and Duncan, 2002; Leukel et al., 2008b,a).

The motor control system is likely to rely more on afferent feedback in the case of misplanned motions (if perturbations occur) or if knowledge about the environment is uncertain and planning is difficult (Donelan and Pearson, 2004b). The contribution of afferent feedback pathways may increase under these conditions (McDonagh and Duncan, 2002). A function specific fine-tuning of hopping motions with respect to different movement targets or cost functions (e.g., performance, efficiency), as found for our model, would support the generation of appropriate activation patterns in such conditions. Therefore, supra-spinal centers might plan an appropriate blending of the afferents (setting of λF,L,V) before touch-down. For example, in our model shifting from targeting hopping performance to hopping efficiency could be moderated by fading from dominant FFB to dominant LFB (see Video in Supplementary Material). By comparing predicted and actual afferents (Wolpert et al., 1995; McDonagh and Duncan, 2002), the overall feedback gain (in our model G) might increase if deviations and errors are detected. If so, the pre-setting of the afferent blending permits a fast and function-oriented contribution (performance, efficiency) of feedback responses. Another advantage would be the reduced control effort due to the low-dimensionality of the blending (setting of λF,L,V). However, higher centers must be able to learn such feedback blending even in the case of sensorial and mechanical perturbations. Thus, in order to be functionally useful, the solution space of possible feedback compositions (as shown in the sensor-motor maps) must follow a simple and consistent topology (Loeb, 1995). Indeed, we found compact and robust topologies that were invariant to changes of morphological design or environmental parameters with respect to motion stability and optimal compositions. Nonetheless, we can only derive hypotheses about a potential integration of our feedback model and feed-forward commands. The discussion presented here warrants validation and support from experimental studies.

## 4.4. Comparison of the Model to Human Hopping

The neuromechanical hopping model (with a reference strain of εmoderate = 0.03, "moderate" mass, "moderate" segment lengths and rigid ground) produced similar hopping motions as those of human hopping. A high effective stiffness of the leg was found for high hopping frequencies. While similar results have been reported in experimental studies (Farley and Morgenroth, 1999; Riese et al., 2013), other studies reported higher leg stiffness values ranging from 30 to 55 kN/m (Farley et al., 1991; Hobara et al., 2011; Kuitunen et al., 2011). However, hopping frequency and leg stiffness were in reasonable ranges and changed accordingly (Rapoport et al., 2003). Moreover, our model predicted increasing hopping performance with increasing tendon elasticity agreeing with results of experimental studies (Kubo, 2005; Fukashiro et al., 2006) and other simulation studies (Anderson and Pandy, 1993; Bobbert, 2001; Nagano et al., 2004). Although this study utilises a highly simplified model structure the model predicted the basic dynamics (e.g., hopping frequency and leg stiffness) of human hopping. We thus consider this model a valid simplification for the scope of this study.

#### 4.5. Model Limitations

We chose a rather simplistic model to integrate multiple sensory pathways at the elementary sensor-motor-level to determine how individual reflex pathways of muscle force (FFB), fiber length (LFB) and velocity (VFB) can support—in isolation and in combination—the repulsive leg function during the stance phase of hopping. By blending individual sensory pathways, we investigated the capacity of the neuromuscular feedback system to generate goal-directed motions. These simplifications may have influenced our results.

We selected a bouncing task (one dimensional hopping) as primary locomotor function. The functional contribution of the different feedback pathways predicted for hopping will most likely be different for other motion tasks. The hopping model of Geyer et al. (2003) utilized one anti-gravitational leg extensor muscle representing all involved muscles during the stance phase of hopping. The foot segment and ankle joint were neglected possibly influencing the overall leg stiffness of the model because the ankle joint was found to be the main contributor for leg stiffness modulations at higher hopping frequencies (above 2.2 Hz; Farley et al., 1998; Farley and Morgenroth, 1999; Hobara et al., 2011). Nonetheless, the model of Geyer et al. (2003) and our simulations generated biomechanically reasonable results. In addition, we simplified the distributed mass of the human body to a point mass neglecting the effects of wobbling masses and their influence on impact dynamics (Seyfarth et al., 1999; Schmitt and Günther, 2011). This helped to focus on the muscletendon interaction and the functional contribution of different feedback pathways. The model of the MTC did not consist of a parallel elastic element that could (additionally to the SE) store and release energy in SSC (Anderson and Pandy, 1993; Lindstedt et al., 2002; Robertson and Sawicki, 2014) when the CE is stretched beyond its optimal length (van Soest and Bobbert, 1993). However, in our simulations, the CE did not operate above its optimal length. Hence, we argue that it is tenable to neglect the parallel elastic element in this particular case. In addition, jumping simulation models by Anderson and Pandy (1993) and Seyfarth et al. (2000) showed only insignificant contribution of the parallel elastic elements. Moreover, while in our study damping within the MTC was neglected, a damping element has also been found to be negligible for hopping (Rapoport et al., 2003).

The used neuromuscular feedback model is a highly simplified representation of the complex biological network. Sensory signals were handled as ideal, averaged and analogue physical quantities without frequency modulation and sensory or signal noise. By using simple delays, offsets and gains, we only considered a highly simplified neural processing of the monosynaptic feedback pathways. In particular, no time-variant feedback gains (Pearson, 1995; Pearson et al., 1998) or other sensory signals such as joint position and velocity, mechanoreceptors or cutaneous receptors were considered. Because we found moderate correlations for the feedback time delay, feedback-specific delays (Prochazka et al.,

1997b) will certainly influence the performance and efficiency of the model. Possible causes of afferent gain changes as discussed in Sreenivasa et al. (2015) were not investigated here. All these factors were simplified and neglected for the sake of simplicity and comprehensibility (Full and Koditschek, 1999; Brown and Loeb, 2000; Pearson et al., 2006). For this study, it was important to separate and isolate the pathway and task-specific effects in the frame of the mechanical structure and muscle mechanics. Previous studies of similar model complexity (Kuo, 2002; Geyer et al., 2003; Haeufle et al., 2010, 2012) not only demonstrated realistic motions but also elucidated the functional roles of the different contributors of feedback, feed-forward or muscle properties.

#### 4.6. Outlook and Future Directions

Based on the results of this study, the robustness of the system against mechanical or sensory perturbations can be investigated. From a control point of view, combinations of multiple sensory pathways or information channels about the system state will result in more robust and more precise estimations of the system state (Donelan and Pearson, 2004b; Green and Angelaki, 2010). Comparing effect sizes resulting from muscle properties (Haeufle et al., 2010), feedback blending and an integration of feed-forward controls (Kuo, 2002; Haeufle et al., 2012) is of high interest. In a next step, we will expand our models to other motions tasks (e.g., running and walking) and underlying locomotor subfunctions (Sharbafi and Seyfarth, 2017) such as swing and balancing (Seyfarth et al., 2012; Sharbafi et al., 2017). For these scenarios, other feedback pathways, e.g., from vestibular organs or as suggested by Song and Geyer (2015), may be used to examine the generalisability of the sensor-motor maps.

Moreover, we would like to explore the use of this type of neuromuscular model as non-invasive distinguishing tool for two purposes: (1) to further explore the mechanisms and interactions of mechanical, neuromuscular and sensory templates and (2) for a model-based design of experimental protocols and settings (e.g., perturbation profiles). While computational modeling approaches help to investigate underlying principles of locomotion, they could potentially help to predict the value and usefulness of experimental settings. Such a model-based

## REFERENCES


identification of experimental settings might improve future experimental designs.

## 5. CONCLUSION

The novel sensor-motor maps provide a tool for analysing human (and animal) motor control strategies and investigating how the biological neural control system recruits function-specific sensormotor pathways. The maps of muscle force, fibre length and velocity pathways are predicted to be robust with respect to changes in body and environment mechanics (e.g., compliance).

In addition to central (or spinal) pattern generators, musclereflex based control circuits are able to generate adjustable cyclic motions by exploiting the musculoskeletal dynamics and gravity. We call these neuromechanical pattern generators (nmPG's). Accordingly, the sensory feedback pathways (e.g., positive force feedback) operate as an antagonist system to the mechanics of the body (muscles, segments) and gravity. The mechanical system is consequently not only the target of (neuronal) control but at the same time an essential part of pattern generating networks.

## AUTHOR CONTRIBUTIONS

CS and AS contributed to the design, execution and drafting of this work, and approved the final manuscript. CS implemented the computational model and conducted the simulation studies.

#### ACKNOWLEDGMENTS

The authors thank Alexandra Voloshina, Guoping Zhao and Maziar Sharbafi for interesting discussions and helpful comments on the study. We also acknowledge support by the German Research Foundation and the Open Access Publishing Fund of Technische Universität Darmstadt.

## SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fncom. 2017.00108/full#supplementary-material


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Schumacher and Seyfarth. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Hammering Does Not Fit Fitts' Law

#### Tadej Petricˇ 1, 2 \* † , Cole S. Simpson1, 3, 4 †, Aleš Ude<sup>2</sup> and Auke J. Ijspeert <sup>1</sup>

<sup>1</sup> Biorobotics Laboratory, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland, <sup>2</sup> Department of Automatics, Biocybernetics and Robotics, Jožef Stean Institute, Ljubljana, Slovenia, <sup>3</sup> George W. Woodruff School of Mechanical Engineering, Georgia Institute of Technology, Atlanta, GA, United States, <sup>4</sup> Mechanical Engineering Department, Stanford University, Stanford, CA, United States

While movement is essential to human wellbeing, we are still unable to reproduce the deftness and robustness of human movement in automatons or completely restore function to individuals with many types of motor impairment. To better understand how the human nervous system plans and controls movements, neuromechanists employ simple tasks such as upper extremity reaches and isometric force tasks. However, these simple tasks rarely consider impacts and may not capture aspects of motor control that arise from real-world complexity. Here we compared existing models of motor control with the results of a periodic targeted impact task extended from Bernstein's seminal work: hammering a nail into wood. We recorded impact forces and kinematics from 10 subjects hammering at different frequencies and with hammers with different physical properties (mass and face area). We found few statistical differences in most measures between different types of hammer, demonstrating human robustness to minor changes in dynamics. Because human motor control is thought to obey optimality principles, we also developed a feedforward optimal simulation with a neuromechanically inspired cost function that reproduces the experimental data. However, Fitts' Law, which relates movement time to distance traveled and target size, did not match our experimental data. We therefore propose a new model in which the distance moved is a logarithmic function of the time to move that yields better results (R <sup>2</sup> <sup>≥</sup> 0.99 compared to <sup>R</sup> <sup>2</sup> <sup>≥</sup> 0.88). These results support the argument that humans control movement in an optimal way, but suggest that Fitts' Law may not generalize to periodic impact tasks.

Keywords: motor control, biomechanics, upper extremity, optimal control, arm movement, impact, Fitts' Law

## INTRODUCTION

Movement is essential to human wellbeing. However, the control of movement is a very difficult problem. To produce deft and robust movements, the human nervous system must continuously control over 600 muscles while handling nonlinearities, nonstationarities, delays, noise, and uncertainties (Franklin and Wolpert, 2011). Despite these difficulties, humans move with apparent ease. However, human motor capability may become impaired due to age, illness, or injury. Robotic systems are also faced with many of the same challenges (Egeland et al., 1991; Park, 2002; Guigon et al., 2007; Peters et al., 2009), but meet with much less success than their healthy human counterparts (Yang et al., 2011; Vanderborght et al., 2013). A better understanding of the roles that the nervous and musculoskeletal systems play in producing movement will likely lead to advances in rehabilitation and robotic control.

Many neuromechanists employ simple tasks to study the nervous system in action under controlled conditions. Isometric tasks in which subjects interact with an immoble force sensor and

#### Edited by:

Francisco J. Valero-Cuevas, University of Southern California, United States

#### Reviewed by:

Bastien Berret, University of Paris-Sud, Universit Paris-Saclay, France J. Michael Herrmann, University of Edinburgh, United Kingdom

#### \*Correspondence:

Tadej Petricˇ tadej.petric@ijs.si

† These authors have contributed equally to this work.

> Received: 19 October 2016 Accepted: 15 May 2017 Published: 29 May 2017

#### Citation:

Petric T, Simpson CS, Ude A and ˇ Ijspeert AJ (2017) Hammering Does Not Fit Fitts' Law. Front. Comput. Neurosci. 11:45. doi: 10.3389/fncom.2017.00045 reaching tasks in which the hand is moved from one point to another are commonly used to study sensorimotor learning (Rotella et al., 2015), movement control (Fitts, 1954), and neurophysiology (Shadmehr and Krakauer, 2008) in the upper extremity. Subjects may also be asked to interact with robotic co-workers that can record reaching dynamics (Burdet et al., 2001), generate disturbances, or create force fields (Shadmehr and Mussa-Ivaldi, 1994) during these tasks. When carefully considered, these experiments can provide a wealth of information on how the nervous system controls movement. However, these tasks are greatly simplified from real-world tasks. To study more complex tasks, some researchers have developed simple games, such as conkers, to study sensoriomotor learning (Sternad et al., 2011). However, even these studies simplify realworld tasks and rarely consider certain features of real-world tasks such as impacts.

Despite many possible ways to perform most tasks (Bernstein, 1967), upper extremity movements are highly stereotyped. Researchers note consistent characteristics such as bell-shaped velocity curves (Hollerbach and Atkeson, 1987; Berardelli et al., 1996) and speed-accuracy tradeoffs characterized by Fitts' Law (Fitts, 1954; Bootsma et al., 2004; Zhai et al., 2004). Fitts' Law expresses the time to complete a reach as a logarithmic function of the size of the target and the distance to the target (see Equation 1). In experiments relating to Fitts' Law, the kinematics (the beginning and final position of the arm or cursor) are prescribed and the subject is left to determine the time to reach. In certain periodic movements however, the time to complete an upper extremity movement can be specified and the subject left to determine the kinematics.

Movement is constantly refined by biological processes such as learning and evolution (Todorov, 2004). Because of this constant refinement, many researchers note that optimal control models utilizing cost functions such as minimum variance (Harris and Wolpert, 1998), minimum effort (Crowninshield and Brand, 1981), minimum jerk (Flash and Hogan, 1985), and minimum torque change (Uno et al., 1989) can be excellent models for the nervous system. In fact, many of the observed stereotypical behaviors discussed in the previous paragraph can be explained by optimality principles. Optimal control models have been used to reproduce human-like behaviors such as reaches (Todorov and Li, 2005), walking (Anderson and Pandy, 2001), and jumps (Anderson and Pandy, 1999; Ong et al., 2016). Though occasionally studied (Côté et al., 2008; Müller and Sternad, 2009), one activity that remains conspicuously unmodeled is Bernstein's hammering task (Bernstein, 1967; Müller and Sternad, 2009) that inspired much research into motor control and learning.

Here we extend Bernstein's hammering task into a targeted periodic impact task. We recorded impact forces and upper extremity kinematics in hammering. In order to examine how hammering strategies might change with different conditions, we used a set of hammers with different physical properties (hammer face area and mass) and prescribe different hammering frequencies. We hypothesized that hammering impact velocity and maximal height attained are the result of a tradeoff between maximizing task performance (quantified here as a maximal impact velocity) and minimizing effort (Crowninshield and Brand, 1981; Nelson, 1983). In order to test whether the mechanics of this task adhere to current theories in optimal human motor control, we implemented a feedforward optimal controller (Todorov, 2004) on a planar torque-driven 3-segment dynamical model of the upper extremity holding a hammer (**Figure 8**) using model parameters from Winter (2009). Our results show that humans appear to select optimal impact velocities that reflect a tradeoff between accomplishing the task and minimizing effort that do not adhere to Fitts' Law.

## METHODS

#### Subjects

Ten healthy male volunteers (age = 27.6 ± 3.6 years, height = 176.9 ± 5 cm, weight = 77.7 ± 11.2 kg) participated in the study. All subjects were right-handed and had no known neuromotor or sensory disorders (self-reported). Prior to their participation, subjects were informed of the course of study and gave their written informed consent in accordance with the code for ethical conduct in research at the Swiss Federal Institute of Technology (EPFL). This study was approved by the EPFL Human Research Ethics Committee (HREC No.: 008- 2015/17.08.2015).

## Experimental Protocol

Each subject was asked to step in front of a table on top of which was a wooden board mounted on a force plate (Kistler Instrument AG, Winterhur, Switzerland) as shown in **Figure 1**. Subjects were given one of four differently sized and weighted hammers (**Table 1**) and asked to drive a pre-started nail, i.e., a nail that had previously been driven to the point at which it would stand on its own, into the wooden board while matching their hammer strikes to the clicks of a metronome. Please note that subjects were not explicitly instructed to strike with their maximum impact speed but were allowed to self-select the best impact speed for their skill level. The metronome was set to one of five frequencies: 1, 2, 3, 4, or 5 Hz. The hammer used and metronome frequency for each trial were randomized. Note that subjects were not allowed to do a training trial first, but we assume that the random trial order cancels any learning effects. Subjects were allowed to use their nondominant hands to stabilize the wooden board. In each trial, the forces on the wooden board and the kinematic motion (14 Prime Series cameras, OptiTrac, USA) of the upper extremity and hammer were recorded at 1 kHz and 250 Hz, respectively. After completing the experimental trials, subjects were asked to subjectively rank each of the hammers in order from most to least preferred.

## Data Processing

Statistical analyses were performed using the Statistics and Machine Learning Toolbox in Matlab. We calculated the hammer velocity, average maximal heights of the hammer, average times required for the hammer to go from maximal height to impact, and average maximal impact forces during hammering for each subject. We then used these average values from

TABLE 1 | Specifications of the hammers used in these experiments.


each subject for statistical analyses. We investigated the effects of time to impact, maximum height of the movement, and maximal force normalized with the hammer weight using twoway repeated-measures ANOVA with independent variables [hammers(4)×(frequency(5)]. The effect of maximum height of the movement, and maximal force normalized with the hammer weight for each combination of hammers and frequency was further determined using one-way repeated measures ANOVA. The differences between maximal heights and the differences between the normalized maximum forces at impact were tested with post-hoc t-tests with Bonferroni correction. The level of statistical significance used was 0.05 for all statistical tests.

### Modeling

In order to determine whether human hammering strategies adhere to Fitts' Law (Fitts, 1954; Bootsma et al., 2004; Zhai et al., 2004), we attempted to fit Fitts' model,

$$T\_f = a + b \cdot \log\_2(2D/W),\tag{1}$$

with data collected in our experiment. In this formulation, the movement time, T<sup>f</sup> , is a function of the distance from the hammer at peak height to the nail, D, and the face width of the hammer, W. The values of a and b were selected using a least squares difference regression.

In order to examine whether the human nervous system uses optimality principles to control hammering movements, we employed a feedforward optimal controller on two joints (shoulder and elbow) while the wrist was maintained at a desired position with an impedance controller. The human arm holding a hammer was modeled as a 3 link torque-driven robot operating in the saggital plane (**Figure 8**, right-hand column) whose parameters were computed based on data from Winter (2009) (see Appendix for more details) and whose dynamics are given by

$$\mathbf{r} + \mathbf{J}^T \mathbf{F}\_\mathbf{e} = \mathbf{H}(q)\ddot{q} + \mathbf{h}(q, \dot{q}) + \mathbf{g},\tag{2}$$

where τ is a vector of joint torques, q, q˙, and q¨ are vectors describing the joint angular position, velocity, and acceleration respectively, **H**(q) is the inertia matrix, **h**(q, q˙) consists of the Coriolis, centrifugal, and viscous friction force vectors, g is the gravity force vector, F<sup>e</sup> is a vector representing external forces (zero throughout the simulation), and **J** T is the transpose of the Jacobian matrix. The model was simulated in Matlab using a time step of 0.001 s beginning at the instant after one impact and terminating at the time of the next impact.

Human hammering is a difficult control task due to the need to balance energy transfer to the nail with accuracy. We hypothesize that the human nervous system determines an optimal tradeoff between maximal impact velocity (complete the task in the most effective manner) and minimal effort (Crowninshield and Brand, 1981; Nelson, 1983; Missenard and Fernandez, 2011). We thus determine the optimal joint torques by minimizing the cost function,

$$\text{Cost} = (1 - \alpha) \frac{\sum\_{i=1}^{n} \sum\_{j=1}^{T} \tau\_{i,j}^{2}}{C\_{\tau\_{\text{max}}}} - \alpha \frac{\mathcal{Y}\_{T-1}^{\ast} - \mathcal{Y}\_{T}^{\ast}}{C\_{\dot{\mathcal{Y}}\_{\text{max}}}},\tag{3}$$

where τi,<sup>j</sup> represents joint torques for i = 1, . . . , n joints over j = 1, . . . , T discretized time points, y<sup>T</sup> and yT−<sup>1</sup> are the vertical positions of the hammer head at the last and second-to-last time points of the simulation, 0 ≤ α ≤ 1 was designed as an expertise factor to represent the tradeoff in relative emphasis between impact velocity and effort (large α places more emphasis on energy transfer to the nail and a small α places more emphasis on effort conservation), Cτmax is a scaling factor representing maximal effort (i.e., if maximal torque is applied for the duration of the simulation), and Cy˙max is a scaling factor representing the maximum achievable impact velocity. We compute Cτmax as the discrete integral of the joint torque limits (whichever direction has the larger magnitude) over the length of the simulation and Cy˙max by simulating a hammer trajectory in which α = 1 and Cy˙max = 1. Because maximum effort and final velocity depend on the length of the simulation, we computed unique values of Cy˙max and Cτmax for each hammering frequency. We constrain the model so that the hammer hits the same place in subsequent impacts ((x0, y0) = (xT, yT)) and there is no initial velocity ((x˙0, y˙0) = (0, 0)). We match the initial posture (location of (x0, y0) relative to the simulated shoulder) to the average posture used by our subjects determined by inverse kinematics. The terms, Cτmax and Cy˙max , are scaling factors included to facilitate direct comparison of the two terms making up the cost function, minimum effort and maximum final impact velocity. In order to determine whether the parameter, α, is constant within or across individuals, contours of constant α were generated and compared with experimental results. The optimal joint torques were determined using the interior point method implemented with the Matlab Optimization Toolbox.

#### RESULTS

#### Experimental Results

Subjects were adept at matching hammering frequency with most of those dictated by the metronome. The hammering frequencies achieved by the subjects for metronome frequencies 1, 2, 3, 4, and 5 Hz were 0.99 ± 0.01, 2.02 ± 0.01, 3.01 ± 0.03, 4.01 ± 0.01, and 4.71±0.04 Hz respectively (mean ± standard error). Hammering frequencies of 5 Hz were too fast for our subjects to reliably match. A hammering frequency of 1 Hz was uncomfortably slow for most subjects. To compensate, many subjects developed a strategy of pausing after each impact before initating an upand-down hammering motion at a more comfortable frequency (**Figures 2**, **3**).

Vertical trajectories (**Figure 2**) and speeds (**Figure 3**) exhibited by the subjects in hammering showed very few differences between the different hammers. However, decreasing the hammering frequency increased the variability in these movements. Rather than the single bell-shaped speed profile characteristic of reaching movements, subjects showed a bell-shaped speed profile for raising the hammer and another truncated bell-shaped speed profile for the descending motion (**Figure 3**).

Analysis of variance showed significant effects of both hammers [F(1.61, 14.5) = 4.95, p = 0.03] and frequencies [F(1.14, 20.25) = 22.35, p < 0.01] on the time to impact from maximum height. There was no significant interaction [F(1.73, 16.05) = 2.56, p = 0.11] between the effects of hammers and frequencies on the time to impact from maximum height. The diagram in **Figure 4** shows the means and standard errors (SEM) of time to impact for all hammers and frequencies.

Analysis of variance showed significant effects of hammers and frequencies on the normalized maximal heights. Significant effects of both hammers [F(2.91, 26.2) = 22.8, p < 0.01], frequencies [F(1.77, 15.99) = 53.09, p < 0.01] and interaction between hammers and frequencies [F(4.14, 37.26) = 2.71, p = 0.04] were observed. Further analysis of the effects of hammers on normalized maximal heights showed significant effects of hammers [F(3, 27) = 6.46 − 11.08, p < 0.01] in all frequencies. Post-hoc t-tests showed that Big Light and Small Heavy maximal heights were statistically different from any of the others [t(9) = 2.97 − 5.75, p < 0.01]. The diagram in **Figure 5** shows the means and standard errors (SEM) of normalized maximal heights for all hammers and frequencies.

Similarly, analysis of variance showed significant effects of hammers and frequencies on the impact forces normalized by hammer mass. Significant effects of both hammers [F(2.06, 23.42) = 32.07, p < 0.01], frequencies [F(1.34, 12.07) = 6.84], but no significant effect of interaction between hammers and frequencies [F(4.39, 39.55) = 0.69, p = 0.63] were observed. Further analysis of the effects of hammers on the impact forces normalized by hammer mass showed significant effects of hammers [F(3, 27) = 11.8 − 17.97, p < 0.01] in all frequencies. Post-hoc t-tests showed that Small Heavy was statistically different than Small Light [t(9) = 4.71 − 5.65, p < 0.01] and Big Heavy [t(9) = 5.4 − 6.61, p < 0.01], and Big Heavy was statistically different than Big Light [t(9) = 3.04 − 3.91, p < 0.01] for all frequencies. The diagram in **Figure 6** shows the means and standard errors (SEM) of impact forces normalized by hammer mass for all hammers and frequencies. The number of impacts needed to totally drive in the nail under each condition–a function of impact velocity–are reported in the appendix (**Table A2**).

#### Modeling Results

Fitts' Law accurately predicted the movement time from the maximum height to impact (R <sup>2</sup> <sup>≥</sup> 0.88). However, despite the high value of R 2 , the accepted formulation for Fitts' Law does not appear to follow the contours of the experimental data (**Figure 7**, light gray traces). Therefore, we propose a slightly altered model that reverses the relationship between movement time and distance to move and was able to improve upon Fitts' predictions (R <sup>2</sup> <sup>≥</sup> 0.99),

$$D = W/2\left[a + b \cdot \log\_2(T\_f)\right],\tag{4}$$

where D is the maximal height of the hammer, T<sup>f</sup> is the time from the maximal height to impact in milliseconds, W is the minimum width of the hammer face (our hammers were square, so both face length and width were the same), and a and b are parameters fit to the data using a least squares difference regression (**Table 2**).

The optimal feedforward model was able to accurately reproduce the motions of the arm during hammering (**Figures 2**, **3**, dashed lines, RMSE ≤ 0.1) using the cost function given by Equation (3). This model allows for the generation of optimal hammering trajectories by selecting just one parameter, α. This model also shows that subjects use roughly the same value of α for each hammer, despite the different properties of the different hammers (**Figures 2**, **3**, α values in each row are very similar). The superpositioning of experimental data with computed contours of constant α values (**Figure 8**) showed that in practice subjects do not use a constant value of α for all hammering frequencies, but rather emphasize lower effort at

FIGURE 3 | Speeds for all hammers and frequencies. The speed (magnitude of the velocity vector) of the hammer head was plotted with respect to time. Solid lines indicate the average speeds while shading represents standard deviation. The black dashed line indicates the optimal behavior of the model using an estimated α parameter for frequencies 1, 2, 3, 4, and 5 Hz. The root mean squared error (RMSE) of the model for each case has a root mean squared error of RMSE < 0.1 m/s.

FIGURE 4 | Means and standard errors (SEM) of time to impact for all hammers and frequencies. The time from maximal height to impact was statistically the same for all hammers at each hammering frequency despite some statistically different maximal heights (Figure 5). The time to impact decreases and becomes less variable as the hammering frequency increases.

height of all hammers and frequencies. The normalized vertical height of the hammer head decreases as hammering frequency increases. Big Light and Small Heavy maximal heights were statistically different from any of the others (\*p < 0.05).

slower hammering frequencies and energy transfer to the nail at faster hammering frequencies.

The Big Heavy hammer was the most preferred hammer followed by the Big Light, Small Heavy, and Small Light hammers in that order based on subject ratings (**Table 3**).

FIGURE 6 | Means and standard errors (SEM) of the normalized maximal impact forces for all hammers and frequencies. Impact forces normalized by hammer mass varied between hammers. The heavy hammers generally had lower impact forces per unit mass than the lighter hammers across hammering frequencies. The Small Heavy hammer had the lowest normalized impact forces across conditions. The Big Light hammer produced the highest normalized impact forces, though the Big Heavy hammer produced the largest absolute impact forces. The Small Light and Big Heavy hammers were similar with statistical differences only found at 2 Hz (\*p < 0.05).

FIGURE 7 | Relationship between time to impact and maximal height for all four hammers. The average normalized height was plotted with respect to average time to impact for each hammer. Fitts' Law was fit to the experimental data and overlayed on the experimental data (gray curves, R <sup>2</sup> <sup>≥</sup> 0.88, RMSE <sup>≤</sup> 0.24). Because Fitts' Law appears to have opposite curvature to the experimental data, a modified model was developed (Equation 4) and overlayed on the experimental results (colored traces, R <sup>2</sup> <sup>≥</sup> 0.99, RMSE <sup>≤</sup> 0.02).

Petric et al. ˇ Hammering Doesn't Fit Fitts' Law

TABLE 2 | Parameter estimations for Equation (4) and the original Fitts' Law Equation (1).


## DISCUSSION

The goal of this study was to examine the mechanics of a human upper extremity impact task and determine whether existing models of upper limb movement can explain the data. We found that subjects plan optimal trajectories that are a tradeoff between maximum impact velocity and minimal effort reminiscent of Fitts' Law and that are robust to different hammer conditions. However, we found that an altered version of Fitts' Law was able to better match the data than the typical formulation. We also found that end-effector speeds follow a "bell curve and a half " trajectory in hammering in which the hammer head moves upwards with a bell-shaped speed profile and then downwards with a bell-shaped profile before being truncated before the zenith of the curve (**Figure 3**).

Our analyses showed that Fitts' Law can be applied to human hammering (R <sup>2</sup> <sup>≥</sup> 0.88). However, the large <sup>R</sup> 2 values belie an apparent discrepancy between the curves generated using Fitts' Law and the experimental results (**Figure 7**, gray lines). Therefore, we identified a relationship between movement time and target distance that better reproduces the experimental data (Equation 4, R <sup>2</sup> <sup>≥</sup> 0.99). In most Fitts' Law experiments, subjects are prescribed a reaching distance and are asked to move as fast as possible (Fitts, 1954). However, in our experiments, we constrain permitted movement time using the metronome and subjects were allowed to select the distance to reach. This difference may account for the relative effectiveness of our inverted formulation of Fitts' Law. However, other previous studies have reported violations of Fitts' Law (Adam et al., 2006; Glazebrook et al., 2015). Glazebrook et al. (2015) determined that these Fitts' Law violations are the result of pre-planning of movements. This explanation is also certainly plausible in the context of a cyclical task such as hammering. Finally, several studies have noted that Fitts' Law does not hold for movements in which subjects were not asked to move as quickly and as accurately as possible (Young et al., 2009). We do not explicitly instruct our subjects to move as quickly and accurately as possible. Instead, we instructed them to accomplish a task that is directly dependent on the speed of the movement and allow them to balance that movement speed with their motor capability, which we believe to be an approximation of the instructions to move as quickly and as accurately as possible. In hammering frequencies above 1 Hz, the computed values of α indicate that subjects weight movement speed very highly (**Figures 2**, **3**), and thus likely approach a fastas-possible movement for which Fitts' Law is presumed to be valid.

Our feedforward optimal hammering simulation was able to reproduce many of the features of human hammering (**Figures 2**, **3**). Our simulations also allow us to show that humans prefer to emphasize energy transfer to the nail (larger values of α) when task constraints are high (high hammering frequencies) and minimal efforts (smaller values of α) when task constraints are low (low hammering frequencies; **Figure 8**). Our cost function was formulated to minimize the sum squared actuator effort, which serves to keep commanded joint torques small. These small actuation signals prevent excessive energy expenditure during the task (Crowninshield and Brand, 1981; Missenard and Fernandez, 2011), but this quadratic formulation might also serve to keep disturbances from motor noise whose effects are multiplicative with actuator effort small (Harris and Wolpert, 1998; Todorov and Li, 2005; Franklin and Wolpert, 2011). In this context, the adaptive prioritization that we observed (changing values of α) might be due to fewer task constraints permitting higher peak heights to be attained at slow hammering frequencies, thus increasing the potential for errors to accrue and increasing the relative importance of accuracy. While the exact cost function used by the nervous system cannot be known exactly, the current formulation reproduces many of the features observed in the experimental results including maximum heights attained, the general trajectories followed, and the robustness to different hammers (similar values of α for different hammers at the same hammering frequencies). However, this model failed to capture the latency after impact before initiating the upward movement of the hammer. This discrepancy may be due to compliance in the musculoskeletal system (e.g., series-elastic muscle-tendon units, Hill, 1938; Fung, 2013) that was not captured by our model.

Despite different hammer dynamics (**Table 1**), hammering kinematics were fairly uniform across many different cases with few statistical differences found between the different hammers in the time from maximal height to impact, maximal hammer height, and impact velocity. Previous studies have suggested that the redundancy of the human musculoskeletal system (Bernstein, 1967) may contribute to considerable robustness to slight changes in dynamics (Martelli et al., 2015; Simpson et al., 2015) or to dysfunction (Arnold et al., 2005; Hicks et al., 2008; Correa et al., 2012; Steele et al., 2012). While these studies rely on highly redundant lower body musculoskeletal models, other studies examining less redundant body parts have shown limited ability to compensate for dysfunction (Valero-Cuevas and Hentz, 2002; Kutch and Valero-Cuevas, 2011). However, detailed models of the upper extremity indicate muscular redundancy on the same level as detailed models of the lower body (**Table 4**) suggesting that similar robustness to perturbations might be expected. The human nervous system may also select control strategies that are purposefully robust (Mitrovic et al., 2010; Franklin and Wolpert, 2011), but our formulation does not

hammering frequencies (greater time between impacts) and energy transfer to the nail (high values of α) at high hammering frequencies (less time between impacts) rather than a constant relationship for all hammering speeds. The plots on the right hand side show examples of arm trajectories using different values of α. The specific values used are marked on the left hand plot by a white square, rhombus, and triangle for the top, middle, and bottom plots, respectively.

TABLE 3 | Results of subjects' ranking of the hammers, e.g., S-H, Small Heavy; S-L, Small Light; B-H, Big Heavy; and B-L, Big Light.


TABLE 4 | Musculoskeletal models used in examinations of robustness.


include any such criteria, suggesting that consistent movement patterns across conditions might be due to embodied intelligence (e.g., redundant actuators and compliance).

Stiffness, or impedance, is a crucial parameter modulated by humans to stably interact with their environment (Burdet et al., 2001; Franklin and Wolpert, 2011). Impedance is difficult to record experimentally, but previous studies have attempted to estimate joint stiffnesses based on muscle properties (Hu et al., 2011), through simulation (Thelen et al., 2003), or by experimentally recording endpoint stiffnesses (Burdet et al., 2000, 2001). Because of practical limitations, measurements of muscle activity or impedance were not included in this study, but likely play an important role in impact tasks and should be considered in future works.

Despite the difficulty of controlling a highly nonlinear plant using noisy control signals and noisy sensors with variable delays in an uncertain environment, biological movement appears to be highly robust. However, robustness has not been well addressed in robot learning (Schaal and Atkeson, 2010; Nguyen-Tuong and Peters, 2011) primarily because it is difficult to design controllers that are robust to the model structure or parameter errors. One possible solution is to use control policies with optimization criteria based on biological models. For example, the tradeoff between maximizing task performance and accuracy could potentially serve as an optimization criteria for robot hammering.

In this paper, we have extracted the mechanics involved in a targeted upper extremity impact task and demonstrated that the human motor control strategies involved are robust to many different conditions including hammer mass, hammer face area, and timing constraints. We have shown that while many traditional models of human reaching hold for this novel task (bell-shaped speed profiles and Fitts' Law), an altered version of Fitts' Law can better match experimental results. We have also demonstrated that optimality principles previously demonstrated for reaching movements can be generalized to targeted impact tasks and thus lay a framework that can be used for the planning of targeted impact tasks in robots.

#### AUTHOR CONTRIBUTIONS

TP, CS, AU, and AI contributed to the design, execution and drafting of this work, and approved the final manuscript. Experimental data was collected and analyzed by TP and CS.

#### REFERENCES


#### FUNDING

This material is based upon work supported by Sciex-NMSCH project 14.069 to TP and a Fulbright Scholarship, Swiss Government Excellence Scholarship, and National Science Foundation Graduate Research Fellowship to CSS. The research leading to these results has received funding from the Slovenian Research Agency under grant agreement no. J2-7360.

#### ACKNOWLEDGMENTS

The authors thank the members of the Biorobotics laboratory at EPFL for their assistance with data collection and participant recruitment.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Petriˇc, Simpson, Ude and Ijspeert. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

## APPENDIX

#### Model Parameters

In this paper, we have modeled the human arm holding a hammer as a torque driven 3 degree of freedom (DOF) robot operating in the sagittal plane. Each DOF is driven by an independently controlled torque generator capable of producing both positive and negative torques. The robot parameters (link lengths, center of mass locations, etc.) were computed based on data from Winter (2009) using the mean height (176.9 cm) and

TABLE A1 | Parametes of the dynamical model.


Note that all units are SI (m, kg). Note also that the center of mass location is relative to the proximal end of the relevant link.

weight (77 kg) of subjects that participated in this study. The hammers were simulated by adding the relevant mass to the end effector. Parameters of the hammers are in **Table 1**. The robot parameters are given in **Table A1**.

## Hammer Hits

The table reports how many cycles were necessary to totally drive in the nail with respect to the hammer and frequency.

TABLE A2 | Number of impacts required to drive nail by hammer and frequency (mean ± standard error).


# Using a System Identification Approach to Investigate Subtask Control during Human Locomotion

David Logan<sup>1</sup> \*, Tim Kiemel <sup>1</sup> and John J. Jeka2, 3

<sup>1</sup> Department of Kinesiology, University of Maryland, College Park, MD, USA, <sup>2</sup> Department of Kinesiology, Temple University, Philadelphia, PA, USA, <sup>3</sup> Department of Bioengineering, Temple University, Philadelphia, PA, USA

Here we apply a control theoretic view of movement to the behavior of human locomotion with the goal of using perturbations to learn about subtask control. Controlling one's speed and maintaining upright posture are two critical subtasks, or underlying functions, of human locomotion. How the nervous system simultaneously controls these two subtasks was investigated in this study. Continuous visual and mechanical perturbations were applied concurrently to subjects (n = 20) as probes to investigate these two subtasks during treadmill walking. Novel application of harmonic transfer function (HTF) analysis to human motor behavior was used, and these HTFs were converted to the time-domain based representation of phase-dependent impulse response functions (φIRFs). These φIRFs were used to identify the mapping from perturbation inputs to kinematic and electromyographic (EMG) outputs throughout the phases of the gait cycle. Mechanical perturbations caused an initial, passive change in trunk orientation and, at some phases of stimulus presentation, a corrective trunk EMG and orientation response. Visual perturbations elicited a trunk EMG response prior to a trunk orientation response, which was subsequently followed by an anterior-posterior displacement response. This finding supports the notion that there is a temporal hierarchy of functional subtasks during locomotion in which the control of upper-body posture precedes other subtasks. Moreover, the novel analysis we apply has the potential to probe a broad range of rhythmic behaviors to better understand their neural control.

#### Edited by:

Manish Sreenivasa, Heidelberg University, Germany

#### Reviewed by:

David W. Franklin, Technische Universität München, Germany Robert Edward Kearney, McGill University, Canada

#### \*Correspondence: David Logan

dvidlgn@gmail.com

Received: 07 July 2016 Accepted: 27 December 2016 Published: 11 January 2017

#### Citation:

Logan D, Kiemel T and Jeka JJ (2017) Using a System Identification Approach to Investigate Subtask Control during Human Locomotion. Front. Comput. Neurosci. 10:146. doi: 10.3389/fncom.2016.00146 Keywords: human locomotion, sensorimotor control, harmonic transfer functions, phase-dependent impulse response functions, subtask control

## INTRODUCTION

Treadmill walking is very useful to study the neural control of locomotion as it constrains locomotive behavior, at a minimum, to two requirements. First, treadmill walking requires subjects adjust their speed so that they do not fall off the front or back of the treadmill. Second, as in any walking task unaided by weight support, subjects must maintain orientation relative to vertical and not allow the proportionally massive trunk to topple over the legs. What is less clear is how the nervous system simultaneously adjusts speed for maintaining position and trunk orientation for upright posture, which is the focus of this study.

Here we use visual and mechanical perturbations, as both have been used separately to successfully learn about subtasks during walking. Changes in virtual visual scene motion have been previously used to alter speed (Konczak, 1994), trunk orientation to vertical (Logan et al., 2010), stride length (Prokop et al., 1997), translation of the body on the treadmill (Warren et al., 1996; Logan et al., 2010), speed of the walk-run transition (Mohler et al., 2007) and its kinematic/energetic features (Guerin and Bardy, 2008).

Mechanical perturbations during walking have also been used to investigate many subtasks of walking. An early investigation by Nashner made use of support surface perturbations to show that stabilizing muscle activations during walking mimicked those occurring during standing posture (Nashner, 1980), reflecting postural control within locomotion. Further investigation into postural control during walking revealed that subjects will first stabilize posture prior to performing an additional, planned lever pulling task (Nashner and Forssberg, 1986). Mechanical perturbations have also been used to study the subtask of obstacle avoidance/ accommodation during walking, and have revealed an elevating or lowering strategy (Eng et al., 1994) or mixture of the two (Forner Cordero et al., 2003) depending on phase of the gait cycle. More recently, Ahn and Hogan (2012) used torque perturbations at the ankle and found that the gait period will entrain to the perturbation when advantageous for propulsion, supporting a neuro-mechanical oscillator for propulsion control. The authors interpreted these findings as a separation in control of low level propulsion and higher level "episodic supervisory control of a semi-autonomous periphery" when needed for cases, such as irregular footholds or obstacle avoidance, compatible with a subtask-dependent control scheme. In sum, visual and mechanical perturbations have been previously used in isolation to provide insight into human walking control.

Here we used simultaneous virtual scene motion and distributed pulling at the back of the trunk to probe the control of treadmill walking. Using the control theoretic view of movement shown in **Figure 1** (Kiemel et al., 2008, 2011; Logan et al., 2010), we sought to perturb treadmill walking at distinct points in the control loop to investigate whether the nervous system changes the priority of different subtasks. Our assumption is that scene motion in an immersive virtual environment perturbs the sensorimotor feedback portion of the control loop and a motor attached to the upper trunk through a spring mechanically perturbs the musculoskeletal plant (see **Figure 1**). The mechanical perturbation first moves the body, which then elicits active (neurally-driven) electromyographic (EMG) responses. In contrast, a visual perturbation first elicits muscle activation, which then moves the body. Using these perturbations simultaneously in this investigation is a step toward understanding both the control problem (musculoskeletal plant) that the nervous system faces and its solution (neural feedback) during bipedal locomotion.

To do so we used small, continuous perturbations, which are considered probes of the control structure and are less likely to change the control structure (e.g., increased effective stiffness). We sought to probe walking with perturbations that yield small, significant deviations of response variables (kinematics, EMG) from mean behavior for insight into the closed-loop control system. Perturbations across gait cycle phases were used as the effects of visual and mechanical perturbations during walking

will, in general, depend on the phase of the gait cycle at which they are applied (Nashner, 1980; Nashner and Forssberg, 1986; Eng et al., 1994; Forner Cordero et al., 2003; Logan et al., 2014). The effects of continuous perturbations on response variables were characterized with a novel application of phase-dependent impulse response functions (φIRFs, where we use "φ" to denote phrase-dependence) to the study of human walking (Kiemel et al., 2016, pre-print available at http://arxiv.org/abs/1607.01746). For a linear time periodic (LTP) system with input u(t) and output y(t), a φIRF h(t<sup>r</sup> , ts) describes the response at time t<sup>r</sup> to an impulse applied at time t<sup>s</sup> (Möllerstedt and Bernhardsson, 2000). For a nonlinear system with a stable limit cycle, a φIRF approximates its response to any small transient perturbation:

$$\chi(t\_{\mathbf{r}}) = \chi\_0(t\_{\mathbf{r}}) + \int\_{-\infty}^{t\_{\mathbf{r}}} h(t\_{\mathbf{r}}, t\_s) u(t\_s) dt\_s,\tag{1}$$

where y0(tr) is the unperturbed periodic output.

The φIRF of an LTP system can be computed directly in the time domain using ensemble methods for general linear time-varying systems (Soechting et al., 1981; Lacquaniti et al., 1982; MacNeil et al., 1992). Ludvig and Perreault (2012) noted that these methods may require many experimental trials (realizations) and proposed a more efficient method that is applicable for an LTP system in which φIRF responses decay quickly relative to the system's cycle period. The φIRF can be computed efficiently without this constraint by first computing a harmonic transfer function (HTF) in the frequency domain (Wereley and Hall, 1990) and then converting the HTF to a φIRF in the time domain (Möllerstedt and Bernhardsson, 2000). However, methods used to compute the φIRF of an LTP system are not necessarily valid for limitcycle systems, because perturbations can reset the phase of the oscillator, violating the assumption of periodicity. Much of the theory for LTP systems assumes that a transient perturbation produces a transient response (Sandberg et al., 2005), which is not true when the perturbation resets phase. The novelty of the method used in this study is that it accounts for phase resetting and, thus, can be applied to walking. Our method is a modification of the HTF-to-φIRF method for LTP systems and retains its advantage of experimental efficiency.

As seen in **Figure 2**, presenting the data as the φIRF allows a characterization of the input perturbation and output response variable throughout the phases of the gait cycle with respect to stimulus phase and normalized response time. Stimuli and impulse response functions of hypothetical walking data at three stimulus phases are observed in **Figure 2A** with corresponding visualization as a φIRF in **Figure 2B**. The φIRF in **Figure 2B** would quickly tell us in a single picture that perturbations occurring solely during swing phase yield responses in the stance phase of the following gait cycle. A φIRF describes the response to a small brief discrete perturbation at any phase of the gait cycle. However, it is methodologically inefficient to experimentally use discrete perturbations to determine the φIRF (as in Logan et al., 2014). Instead, responses to continuous perturbations are analyzed in the frequency domain and then converted to the time domain to compute the φIRF (see Methods and Kiemel et al., 2016).

Working within the theoretical framework shown in **Figure 1**, mechanical and sensory perturbations have been successfully applied to non-parametrically identify both the musculoskeletal plant (Kiemel et al., 2008) and the sensorimotor feedback (Kiemel et al., 2011) portions of the control loop during standing postural control. Here we attempt a similar identification scheme aimed at walking while simultaneously probing subtask control. Supported by the finding that postural corrections are initiated prior to performance of an additional, mechanically destabilizing task (Nashner and Forssberg, 1986), we hypothesized that both perturbations would elicit a control strategy that prioritized control of trunk orientation for staying upright over adjustments in speed to maintain position on the treadmill.

## MATERIALS AND METHODS

#### Subjects

Twenty healthy subjects [8 males and 12 females, between 19 and 30 years. of age, 67.9 ± 12.9 kg (mean ± SD)] participated in this study. All subjects were self-reported to have normal (or corrected to normal) vision. The studies conformed to the Declaration of Helsinki, and all participants provided informed, written consent to the experimental procedures detailed in this manuscript. These experimental procedures and consent process were approved by the Institutional Review Board of the University of Maryland, College Park.

## Apparatus

#### Virtual Reality Environment

Subjects walked at 5 km h−<sup>1</sup> on a treadmill (Cybex Trotter 900T, Cybex International, Inc., USA) surrounded by three screens (width, 3.05 m; height, 2.44 m; Fakespace, USA), one in front of the subject and one on either side. Subjects wore goggles with the top shield occluded to prevent them from seeing motion capture cameras mounted above the screen in front of them. Visual displays were rear projected to the screens at a frame rate of 60 Hz by JVC projectors (model DLA-M15U; Victor Company of Japan). CaveLib software (Mechdyne, USA) was used to generate a virtual moving visual scene consisting of three walls attached at right angles that coincide with the screens when the visual scene is not moving. Each wall consisted of 500 non-overlapping white small triangles (3.4 × 3.4 × 3.0 cm) with random positions and orientations on a black background. To reduce aliasing effects in the fovea region, no triangles were displayed on the front wall within a 30-cm-radius circular region directly in front of the participant's eyes. The display on each screen was varied in time to simulate rotation of the visual scene about the mediallateral axis located at the subject's ankle height at 1 m from the screen, assuming a fixed perspective point at the participant's eye height 1 m from the screen. The signals specifying scene-rotation angle were created offline (Matlab, Mathworks, USA) and were generated via Labview (National Instruments, USA) on a desktop computer (Precision T5500, Dell, USA).

#### Mechanical Perturbation

As seen in **Figure 3**, a weak continuous mechanical perturbation was applied to the subject from behind as a spring with one end attached to a modified trunk harness worn by the subject and the other end attached to a linear motor (LX80L; Parker Hannifin Corporation). The spring was attached in series with a 45.7 cm rigid cable fixed to the back of the harness. The harness was adjusted for each subject so that the point of attachment was at mid-scapula height centered on the midline of the upper trunk. The actual displacement of the motor in the anterior posterior (A-P) direction, as indicated by a VICON reflective marker on the motor, was used as the mechanical perturbation signal. The force on the body was F(t) = k(u(t) − y(t) − u0), where k is the spring constant, u(t) is the perturbation signal, y(t) is the A-P position of the point on the body at which the perturbation is applied, and u<sup>0</sup> is a constant such that F(t) < 0 (force in the backward direction) throughout each trial. We used a weak spring (k = 0.0175 N/mm) so that the effect of the mechanical perturbation on gait kinematics and EMG signals would be small. Since k is small, the φIRF for our mechanical perturbation is approximately equal to k times the φIRF that would be measured if, instead of specifying motor position, we would have specified the force applied to the body.

#### Perturbation Signals

Both visual and motor signals were filtered white noise signals. For each trial of each subject and each perturbation type, a different seed was used to generate a white noise signal using a random number generator. To create a signal specifying the angle of the visual scene, white noise with a one-sided spectral density

data presented in this manuscript, normalized time in this hypothetical case is in gait cycle units. The gray horizontal bars below indicate stance phase with times of double support indicated with a lighter shade.

within a three panel virtual "cave" providing rotating visual scene motion in the sagittal plane. Subjects were also attached to a motor through a spring and rigid cable in series.

of 150 deg<sup>2</sup> /Hz was filtered using a first-order low-pass filter with a cutoff frequency of 0.02 Hz and a second-order Butterworth low-pass filter with a cutoff frequency of 5 Hz. Across subjects, these visual signals had an average root mean square (RMS) value of 2.13 deg. In our analysis (described below), visual-scene angular velocity was used as the perturbation signal. The RMS velocity of visual signals, averaged across subjects, was 3.62 deg/s. A positive/negative signal corresponded to a forward rotation into the screen/backward rotation toward the subject.

To create a signal specifying the position of the motor, white noise with a one-sided spectral density of 1.1 cm<sup>2</sup> /Hz was filtered using an eighth-order Butterworth low-pass filter with a cutoff frequency of 4 Hz. Across subjects, these driving signals had an average RMS position of 1.30 cm and RMS velocity of 19.40 cm/s. These parameters were used for the motor signal as a balance between ensuring a flat power spectrum up to highest frequency possible and staying within traveling distance and velocity limits of the motor. Visual display generation, motor motion, and data collection software were synchronized via an external trigger. Furthermore, EMG data were synchronized in time with rest of the experimental setup by correcting for a 48 ms group delay occurring when analog output is used by TRIGNO (DELSYS, USA) EMG system.

#### Kinematics

Body kinematics were measured using a 10 camera VICON-MX motion analysis system (VICON, Inc, Oxford, UK). Reflective markers (diameter, 1.4 cm) were placed on the right and left sides of the body at external landmarks corresponding to: base of the 5th metatarsal, posterior calcaneus (heel), lateral malleolus (ankle), lateral femoral condyle (knee), greater trochanter (hip), anterior superior iliac spine (ASIS), posterior superior iliac spine (PSIS), iliac crest, superior acromion process (shoulder), mastoid process (head) and frontal eminence (head). Additionally, markers were placed at the medio-lateral center of the back of the head and the midline of the spine at the level of C6, T10, and L1 vertebrae. All markers were attached at the skin of these bony prominences except those placed on the shoe at the 5th metatarsal and heel. All kinematic data were collected at 120 Hz.

Our analysis focuses on the trunk segment in the sagittal plane as well as whole-body displacements in the A-P direction. Trunk orientation relative to the vertical in the sagittal plane was computed as the angle formed by the L1 to T1 markers. Wholebody displacement in the A-P direction was measured as the displacement of L1 in the A-P direction.

#### Muscle Activity (sEMG)

Muscular activity of the right leg and trunk was measured using surface electromyographic (sEMG) recordings. Recordings of the following 16 muscles were made: tibialis anterior, gastrocnemius lateralis, gastrocnemius medialis, soleus, vastus medialis, vastus lateralis, rectus femoris, tensor fascia latae, biceps femoris, semitendinosus, gluteus maximus, gluteus medius, rectus abdominus, lumbar erector spinae, thoracic erector spinae (EST, recorded at T9), and posterior deltoid. Electrodes were positioned at the muscle belly with placement carefully chosen to minimize cross-talk (Cappellini et al., 2006). Recording sites were shaved, lightly abraded, and cleaned with isopropyl alcohol prior to electrode application. The sEMG data were recorded at 2160 Hz using the wireless TRIGNO system (DELSYS, USA). This recording system has built in bandwidth of 20–450 Hz and gain of 909 V/V. Using Matlab, these signals were highpass filtered using a zero-lag forward-backward cascade of a 4th order Butterworth filter with a 20-Hz cutoff frequency, full-wave rectified, and then low-pass filtered with a zero-lag forwardbackward cascade of a 4th order Butterworth filter with a 10- Hz cutoff frequency. Although consistent sEMG responses were observed in many muscles to the visual perturbation, we focus on an erector spinae muscle (EST) in the results presented below as consistent responses were observed solely in this muscle for both perturbations.

#### Procedures

Prior to experimentation, subjects experienced a static visual display at the experimental locomotion speed. An experimenter was always behind the treadmill in close proximity to the subject to ensure safety in case of falling (never occurred). Subjects began each experimental trial by looking straight ahead at the static visual display at the experimental treadmill speed (5 km/h) for approximately 30 s to reach steady-state treadmill walking. At this point, the subject would declare if he or she was ready for the trial to begin. The experimenter then initiated data acquisition, scene motion and the motor simultaneously with variable delays on each trial to avoid start-up effects. Each trial was 250 s in duration with a rest of at least 60 s between trials. The initial and final 5 s of each 250 s signal were multiplied by increasing and decreasing ramps, respectively, to insure that the value of the signal at the beginning and end of the trial would be 0. Only the middle 240 s of each trial was analyzed. The experimental design consisted of 10 trials of visual scene and motor motion. Upon inspection of trajectories of the kinematic marker on the spring attached to the motor there were instances where the spring clearly went slack during the trial. These instances were removed from analysis, resulting in shorter trials in 13 of the 200 trials recorded across subjects.

### Data Analysis

#### Phase-Dependent Impulse Response Functions

Here we describe the analysis steps used to compute (φIRFs). A fuller description with equations and expanded motivation can be found in Kiemel et al. (2016, pre-print available at http://arxiv.org/abs/1607.01746). Our method is based on existing theory for linear time-periodic systems (e.g., Wereley and Hall, 1990; Möllerstedt and Bernhardsson, 2000; Sandberg et al., 2005) extended for general limit-cycle systems in which perturbations can reset the phase of the oscillator. Our method assumes that the system has smooth dynamics (see Ankarali and Cowan, 2014 for a method designed for hybrid LTP systems). The goal of the analysis is to describe the effect of u(t), a visual scene velocity or motor position perturbation, on y(t), a kinematic or sEMG response variable. **The majority of results presented are full** φIRF**s, and are calculated in step 6**. Computing the full φIRF consists of six steps:

1. **Approximate phase**. First we compute heel-strike times tk (k = 1, ..., K) for a reference leg. Then we compute T¯, the mean of the stride times tk+<sup>1</sup> − t<sup>k</sup> (k = 1, ..., K − 1), and compute the estimated gait frequency as f<sup>0</sup> = 1/T¯. Next we define a discontinuous approximation of phase as θd(t) = k + f0(t − t<sup>k</sup> ) for t<sup>k</sup> ≤ t < tk+<sup>1</sup> . Approximate phase θd(t) is designed to be causal, that is, to only depend on data up to and including time t. To obtain a continuously-differentiable causal approximation of phase, θ(t), we apply a second-order low-pass filter to θd(t):

$$
\ddot{\theta}(t) + 2d(\dot{\theta}(t) - f\_0) + d^2 \theta(t) = d^2 \theta\_\mathbf{d}(t).
$$

Here d represents the filter rate constant for estimating phase, which was 2. Note that for strictly periodic gait, approximate phase θ(t) matches the usual definition of the phase of the gait cycle.


and Piersol, 2000). The double-frequency CSD describes the relationship between the input signal u˜(ϑ) at input frequency f<sup>1</sup> and the output signal z(ϑ) at output frequency f2. The PSD and CSD are computed using Welch's method with 40-cycle Hanning windows (aligned to start at an integer value of ϑ) and 50% overlap. The k-th mode of the HTF H<sup>z</sup> from u˜(ϑ) to z(ϑ) is computed as Hz,<sup>k</sup> (f1) = puz˜ (f1, f<sup>1</sup> + kf0)/pu˜u˜ (f1). Note that H<sup>z</sup> is a function of both the mode index k and the input frequency f1.

5. **Compute transient and phase** φ**IRFs**. For a (LTP) mapping from u˜(ϑ) to z(ϑ), its HTF H<sup>z</sup> can be converted to its φIRF h<sup>z</sup> by a two-dimensional inverse Fourier transform. The φIRF h<sup>z</sup> is a function of response phase ϑ<sup>r</sup> and stimulus phase ϑ<sup>s</sup> and can be used to represent the LTP mapping from u˜(ϑ) to z(ϑ) as

$$z(\vartheta\_{\mathbf{r}}) = \int\_{-\infty}^{\vartheta\_{\mathbf{r}}} h\_z(\vartheta\_{\mathbf{r}}, \vartheta\_{\mathbf{s}}) \, \tilde{u}(\vartheta\_{\mathbf{s}}) d\vartheta\_{\mathbf{s}}.$$

Using this procedure, compute the transient φIRF ˜h<sup>y</sup> and phase-derivative φIRF ˜h<sup>q</sup> from H˜ <sup>y</sup> and H˜ <sup>q</sup>, respectively. Then compute the phase φIRF by integrating the phase-derivative φIRF:

$$h\_{\theta}(\vartheta\_{\mathbf{r}}, \vartheta\_{\mathbf{s}}) = \int\_{\vartheta\_{\mathbf{s}}}^{\vartheta\_{\mathbf{r}}} \tilde{h}\_{\mathbf{q}}(\vartheta, \vartheta\_{\mathbf{s}}) d\vartheta \downarrow$$

6. **Compute** φ**IRF**. Up to now, IRFs have been functions of response phase ϑ<sup>r</sup> and stimulus phase ϑ<sup>s</sup> . The φIRFs ˜h<sup>y</sup> and h<sup>θ</sup> can be combined to obtain the φIRF from (u(t)) to (y(t)) that is a function of response time t<sup>r</sup> = T¯ ϑ<sup>r</sup> and stimulus time t<sup>s</sup> = T¯ ϑ<sup>s</sup> :

$$h\_{\mathcal{Y}}(\mathbf{t}\_{\mathbf{r}},\mathbf{t}\_{\mathbf{s}}) = f\_0 \tilde{h}\_{\mathcal{Y}}(\mathbf{t}\_{\mathbf{r}}/\bar{T},\mathbf{t}\_{\mathbf{s}}/\bar{T}) + \tilde{\mathcal{Y}}\_0^{\prime}(\mathbf{t}\_{\mathbf{r}}/\bar{T}) h\_{\theta}(\mathbf{t}\_{\mathbf{r}}/\bar{T},\mathbf{t}\_{\mathbf{s}}/\bar{T}) .$$

The φIRF h<sup>y</sup> (t<sup>r</sup> , ts) resulting from this procedure describes for each t<sup>r</sup> and t<sup>s</sup> the response measured at time t<sup>r</sup> due to a small brief perturbation applied at time t<sup>s</sup> . Specifically, hy(t<sup>r</sup> , ts) is the change in y divided by the integral of the perturbation. It follows that hy(t<sup>r</sup> , ts) = 0 for t<sup>r</sup> < t<sup>s</sup> and h<sup>y</sup> t<sup>r</sup> + T¯, t<sup>s</sup> + T¯ = hy(t<sup>r</sup> , ts). The usefulness of the φIRF lies in the fact that it describes the response for any small transient perturbation u(t), as described by (Equation 1) in Introduction, where y0(tr) = ˜y0(tr/T¯). We plot a φIRF hy(t<sup>r</sup> , ts) as a function of stimulus phase ts/T¯ and normalized response time tr/T¯.

Steps 1–4 were computed on a trial-by-trial basis with averages of PSDs and CSDs taken across trials for each subject for completion of the HTF analysis and to compute the full φIRFs in step 6. Full φIRFs are shown in **Figures 4**–**6**, with vertical slices in **Figures 7**, **8** showing the impulse response function at specific stimulus phases. Full φIRFs defined above are now termed φIRFs in the following text.

The φIRF for mechanical perturbations is a response to an impulse in motor position while the φIRF for visual

FIGURE 4 | Trunk orientation φIRFs. φIRFs from visual scene velocity (A) and motor displacement (B) to trunk orientation. Intensity of colors indicate magnitude and direction at the plotted combination of stimulus phase and normalized response time. The diagonal black line is where stimulus phase is equal to the normalized response time, which indicates stimulus onset. The horizontal bar below indicates either double limb or single limb support phases in gray and white, respectively.

perturbations is a response to an impulse in visual scene velocity, which is equivalent to the response to a step in visual-scene position. A positive impulse response (i.e., a positive response) indicates that the variable's response is in the same direction as the perturbation and a negative impulse response (i.e., a negative response) indicates that the variable's response is in the opposite direction as the perturbation.

#### Statistics

Statistical tests of the φIRFs of all response variables were performed at each stimulus phase. For illustration, confidence intervals computed based upon the sample mean using the Matlab function "normfit" are plotted in **Figures 7**, **8**. Permutation tests (1000, Manly, 1997) based on the t-statistic (null hypothesis mean = 0) at all normalized response times up to three cycles post stimulus onset were tested simultaneously and family-wise error rate (FWER) was controlled at each stimulus phase for each response variable. The tmax method (Blair and Karniski, 1993) was used to adjust the p-value for each value at values of normalized response time within each stimulus phase (alpha = 0.05). These tests were performed in functions written by Groppe (Groppe et al., 2011). These tests are non-parametric and suited for this study as FWER control is strong compared to other methods (e.g., clusterbased permutation testing, false discovery rate) allowing determination of reliable effects in the φIRFs (Groppe et al., 2011).

## RESULTS

Phase-dependent impulse response functions (φIRFs) presented in **Figure 4** show responses of trunk orientation to mechanical perturbations (input is motor position) and visual perturbations (input is visual-scene velocity). Although φIRFs were computed based on responses to continuous perturbations, they predict the response to a small brief perturbation applied at any phase of the gait cycle and, by extension, the response to any small transient perturbation (Equation 1). Color represents impulse response value and responses have been plotted as a function of both stimulus phase and normalized response time, the time at which the response is measured in units of cycles. A φIRF value is the amount of change in the response variable divided by the integral of the perturbation. For the visual perturbation, a small brief perturbation in visual-scene velocity is equivalent to a small step in visual scene position, so the φIRF value is the change in the response variable divided by the change in visual-scene position.

Normalized response time is time divided by the mean gait cycle period T¯ of the given trial (1.04 ± 0.05 s, mean ± s.d. across subjects). Doing so allowed a gait cycle-based representation of responses when the perturbation occurred (stimulus phase) and when the response did or did not occur (normalized response time). For example, if T¯ = 1.1 s, a heel strike occurs at time 0 s, a perturbation is applied at time 0.55 s, and the response is measured at time 1.1 s, then stimulus phase is 0.5 and normalized response time is 1. For readability, we describe responses to positive perturbations: a brief increase in visual scene velocity or a brief transient forward movement of the motor. From the definition of a φIRF (Equation 1), it follows that a negative perturbation would produce the opposite response.

For both perturbations, initial trunk orientation responses were observed as forward rotations at all stimulus phases, as indicated by the diagonal red band observed in both **Figures 4A,B** which notes positive responses across phases. Put simply, the trunk rotates forward in response to either a brief increase in visual scene velocity or a brief transient forward movement of the motor.

The red band in both figures is approximately parallel to the black line noting stimulus onset, indicating that onset of the response occurs with similar time delay across all phases in which the stimulus occurs. On average across stimulus phases, peaks of the initial forward trunk rotation to vision observed in **Figure 4A** occur at 0.68 ± 0.06 (mean ± s.d.) cycles (normalized response time) after stimulus onset. As indicated by the black diagonal line in **Figure 4**, stimulus onset shifts based on stimulus phase, which means that these peak responses are occurring on average 0.68 cycles (normalized response time) in **Figure 4A** from the black diagonal line at each stimulus phase with small variability across stimulus phases. These initial peaks observed as darker red regions in **Figure 4A** have an average peak response value of 0.40 <sup>±</sup> 0.05 deg/(degs−<sup>1</sup> ), indicating a consistent response across stimulus phases. **Figure 4B** shows that initial peaks in forward trunk rotation to the motor displacement occur with comparatively shorter latency than responses to vision,

FIGURE 7 | Responses to visual and mechanical perturbation at 28% stimulus phase. Impulse response functions of trunk orientation (A,B), L1 AP displacement (C,D) and normalized erector spinae activations (E,F) to motor position and visual scene velocity. Mean waveforms are plotted below impulse response functions. Shaded blue error bars represent confidence intervals at increment of normalized response time. Asterisks at base of subplots indicate significant difference from zero at increment of normalized response time, corrected for the multiple comparisons made within the stimulus phase (p < 0.05).

FIGURE 8 | Responses to visual and mechanical perturbation at 42% stimulus phase. Impulse response functions of trunk orientation (A,B), L1 AP displacement (C,D) and normalized erector spinae activations (E,F) to motor position and visual scene velocity. Mean waveforms are plotted below impulse response functions. Shaded blue error bars represent confidence intervals at increment of normalized response time. Asterisks at base of subplots indicate significant difference from zero at increment of normalized response time, corrected for the multiple comparisons made within the stimulus phase (p < 0.05).

with average peak responses occurring at 0.17 ± 0.01 cycles (normalized response time), or 0.18 ± 0.01 s, after stimulus onset. These initial peaks in **Figure 4B** have average peak response value of 0.11 ± 0.02 deg/cm. Interestingly, vertical blue bands indicating a backward trunk rotation to the mechanical perturbation are observed at four stimulus phase ranges in **Figure 4B**. However, these negative responses are significant (p < 0.05 with FWER control, see Methods) only when stimuli are presented at 0.38–0.46 and 0.88–0.96 ("phase of stimulus") of the gait cycle, which correspond to single limb support phases.

As observed in **Figure 5**, initial forward responses were also observed in L1 displacement responses to both visual and mechanical perturbations. Forward L1 displacement responses due to visual scene velocity occurred at all stimulus phases and persisted through the 3rd gait cycle of normalized response time. On average across stimulus phases, peaks of the forward L1 displacement due to vision observed in **Figure 5A** occur at 1.89 ± 0.14 s.d. cycles (normalized response time), or 1.97 ± 0.15 s, after stimulus onset. These initial peaks observed as darker red regions in **Figure 5A** have an average peak response value of 1.80 ± 0.17 cm/(degs−<sup>1</sup> ). Initial, forward displacements due to changes in motor position, on the other hand, were not consistently observed across stimulus phases as seen in **Figure 5B**. When tested at each stimulus phase, significant responses were observed before and after heel strike at 0–0.22, 0.40–0.68, and 0.96–1 ranges of stimulus phase. Since phase is a circular variable, these values correspond to two ranges of stimulus phase which differ by roughly half a cycle: 0.40–0.68 and 0.96–1.22. Within these ranges, mean peak of the positive response occurred at 0.87 ± 0.30 cycles (normalized response time), or 0.90 ± 0.31 s, after stimulus onset and had average peak response value of 0.21 ± 0.05 cm/cm. Although backward L1 displacements due to changes in motor position were observed in **Figure 5B**, these were not significant when tested (with FWER control) at each stimulus phase.

**Figure 6** demonstrates that erector spinae (EST) responses were dependent on both phase of stimulus and normalized response time for both perturbations. A typical pattern of response in EST to increased visual scene motion is an initial decrease in activation within a cycle after perturbation which is observed as the blue band parallel to the stimulus onset line in **Figure 6A**. These initial responses are then followed by increased (red) to decreased (blue) bands of activation following at 1.5 and 2.5 normalized response time. This pattern of responses was found to be significant (p < 0.05 with FWER control) at the majority of stimulus phases (0.16–0.48 and 0.56–0.82). Also clear from **Figure 6A**, increased activation does occur after the initial decrease in activation, which was found to be significant at a subset of these stimulus phases (0.16–0.34, 0.76– 0.82). **Figure 6B** shows a comparatively less organized response to the mechanical perturbation, with few of these responses actually being significant. In all, increased activation of EST to the mechanical perturbation was observed in a limited range of stimulus phases including 0.42–0.48, 0.82–0.84, and 0.90–0.92. On average across these stimulus phases, significant responses were observed 0.04 ± 0.02 s.d. cycles (normalized response time), or 0.04 ± 0.02 s, after stimulus onset, and are seen as the red regions which run parallel to stimulus onset in **Figure 6B**.

To investigate the relationship of the kinematics and muscular activity where significant responses were observed, we focus on specific stimulus phases of the φIRFs in **Figures 4**–**6**. In **Figure 6**, clear responses of EST to either the visual scene velocity, motor position or both are seen at the 0.28 and 0.42 stimulus phases. **Figures 7**, **8** simultaneously show trunk orientation, body displacement and EST at these specific stimulus phases.

As noted in **Figures 7A,C** with asterisks, significant trunk orientation responses to the visual perturbation occurred prior to L1 displacement responses. At this stimulus phase of 0.28, forward trunk rotations began at 0.54 normalized response time while forward L1 displacements began at 0.64 normalized response time. In **Figure 7E**, an initial decreased activation at 0.46 normalized response time is followed by an increased activation at 0.54 response time in the EST muscle. This initial decrease in EST activation when virtual scene motion increases velocity occurs prior to forward rotation of the trunk (trunk flexion). Thus, EST decreases its activation prior to trunk flexion when scene motion increases velocity. For the mechanical perturbation, as seen in **Figures 7D,F**, there are no significant effects of the mechanical perturbation on L1 displacement or EST at this stimulus phase. However, there is a significant forward rotation of the trunk due to the mechanical perturbation occurring at 0.3–0.66 normalized response time, as observed in **Figure 7B** and observed previously in **Figure 4**.

At the stimulus phase of 0.42 shown in **Figures 8A,C,E**, a decreased activation of EST to visual scene motion occurs from 0.52 to 0.56 normalized response time just prior to the initiation of a forward trunk rotation response at 0.66 response time. Once again, a decrease in EST activation occurs with increased virtual scene motion velocity. Trunk orientation responses were initiated prior to L1 displacement responses at this stimulus phase, and at the majority (44/50 observed) of stimulus phases. The pattern of significant EST response followed by trunk orientation responses and then L1 displacement occurred at 28 of 50 stimulus phases, with the specific stimulus phases eliciting this pattern at 0.24– 0.44, 0.56–0.82, and 0.92–0.96 of the gait cycle. In all, the combination of responses illustrated in **Figures 7**, **8** suggests that the EST muscle typically facilitates the response of trunk orientation to visual scene motion.

Responses of the trunk to the mechanical perturbation shown in **Figures 8B,D,F** also show perturbation induced deviations in trunk orientation occurring prior to deviations in L1 displacement. Noted with asterisks at the stimulus phase of 0.42 shown in **Figure 8B**, significant forward trunk rotations are initiated at 0.44 normalized response time while forward L1 displacements are first observed at 0.7 normalized response time. As the motor perturbation will first cause responses observed in kinematics which reflect passive responses of the body to decreased pull of the motor-spring apparatus, sEMG responses to the mechanical perturbation are a critical indicator that an active, neural driven response to the mechanical perturbation has occurred. Significant, increased activations of EST were first observed at 0.46 normalized response time at the 0.42 stimulus phase observed in **Figure 8F**. This occurs prior to initiation of the downward trend of the trunk response at 0.62 normalized response time. At this stimulus phase, the downward trend in trunk orientation results in a significant backward trunk rotation from 0.92 to 1.2 normalized response time. The positive response of the trunk extensor indicates an increased EST activation when the motor is moved forward. A forward motion of the motor decreases the backward force of pulling at the trunk to cause trunk flexion, which results in an increased activation of EST, a trunk extensor, to initiate trunk extension. Significant increases in EST activation due to change in motor position were also observed at 0.44–0.48, 0.82–0.84, and 0.90–0.92 stimulus phases, and were always observed after an initial trunk flexion and prior to the decrease from peak of the trunk flexion response. In sum, the EST response observed in **Figure 8**, in addition to that observed at other stimulus phases, indicates an active response which resists the mechanical effects of changing the motor position.

## DISCUSSION

Continuous, probing visual and mechanical perturbations to treadmill walking were used in this study to learn about the neural control of human locomotion. Coupled with the novel use of phase-dependent impulse response functions to describe locomotor responses to perturbations, these continuous perturbations allowed an efficient investigation of walking control throughout phases of the gait cycle. Modifications of both sagittal plane trunk orientation and L1 A-P displacement due to visual scene motion were observed at all phases in which the perturbation was applied (stimulus phase). This phase-dependent methodology, however, revealed that additional modifications in these kinematic response variables due to mechanical perturbations occurred at different stimulus phases. Responses of the trunk musculature occurred in conjunction with responses of trunk orientation kinematics to each perturbation, and reflect an active, neural-driven response for control of trunk orientation occurring prior to modifications initiated for wholebody displacement. These findings suggest that control for the subtask of trunk orientation is enacted prior to control of the subtask of positional maintenance.

## Subtask Timing Suggests Prioritization

Responses in the trunk resulting from both perturbations showed the initiation of an active response for sagittal plane trunk orientation control prior to onset of responses of L1 displacement, which is an indicator of A-P whole body motion on the treadmill. Decreased responses in EST to changing visual scene motion were observed prior to increased responses in trunk orientation, indicating that EST responses facilitated the observed trunk orientation responses to vision. In the case of increased visual scene velocity, the visual system sensed changes in visual scene motion leading to the perception that the trunk was orienting backwards, or extending, and relayed to spinal centers for proximal musculature to decrease activation and promote trunk extension. For the mechanical perturbation at some stimulus phases, an EST response occurs just prior to the trunk orientation's decrease from peak response. In the case of a forward motion of the motor, the mechanical perturbation decreases force applied to the upper trunk to cause an increased trunk flexion. Proprioceptive afferents in trunk musculature relay this change to the spinal cord and higher for an increase in trunk extensor muscle activation for maintaining trunk orientation upright. The combination of these results suggests both an active resistance to the mechanical perturbation and use of visual scene motion information for maintenance of orientation upright which occurs before active use of vision for positional maintenance on the treadmill.

The notion that one function, or subtask, of locomotion can be prioritized over another is certainly not a new idea. An early example observed in cats found that animals will alter their strategy for responding to electrical stimuli placed at the dorsum of their paw in a phase-dependent manner (Forssberg et al., 1975). So-called "reflex reversals" whereby stimuli used during an animal's support phase increase extensor activation and delay a flexor withdrawal show that the animal prioritizes the subtask of upright stability at the expense of completing the withdrawal task. More recently, this prioritization of subtask has been observed in human walking as the lowering strategy for obstacle avoidance has been shown to decrease step length of the perturbed limb on the treadmill with increased speed needed in ensuing recovery steps (Forner Cordero et al., 2003). Thus, subjects delay how they maintain speed on the treadmill in order to avoid hitting the obstacle, indicating a subtask prioritization that is ultimately related to upright postural maintenance.

The prioritization of subtask in such studies and suggested here is in terms of time. Both the trunk toppling over the moving legs and being too forward or backward on the treadmill would have dire consequences for walking. However, responses in trunk orientation to the visual perturbation were observed before responses in whole body position on the treadmill. One interpretation of this result is that maintaining upright orientation (postural control) within locomotion is a greater concern to the nervous system than maintaining position on the treadmill (positional control).

This subtask prioritization was observed solely in terms of time, however, without clear decrement in quality of positional control at the expense of postural control that would further support the claim that postural control is more important than positional control. There are two factors other than importance that may influence the relative timing of postural and positional responses. First, postural adjustments may occur before positional adjustments because the nervous system can act to change trunk orientation at any phase of the gait cycle (for example, by modulating the activity of the erector spinae muscles), whereas the nervous system can only effectively act to change position on the treadmill at certain phases of the gait cycle (for example, by modulating the activiation of plantarflexor muscles during push-off). Second, trunk orientation may respond before whole body position due to the way walking speed is controlled. That is, the initial changes in trunk orientation are anticipatory changes, required to counteract expected trunk movement that would result from a self-induced speed change. This would be in line with the notion of anticipatory postural adjustments (Massion, 1992) suggested to occur prior to expected perturbations to standing posture or the initiation of stepping.

In sum, a temporal ordering of trunk orientation prior to AP displacement suggests the nervous system's prioritization of trunk orientation control over that for altering speed to maintain position on the treadmill. Whether this temporal prioritization of trunk orientation observed during walking is driven by importance of the postural control subtask to the nervous system, biomechanical constraints of the walking behavior, or anticipatory postural adjustment for changing speed is not yet clear. Teasing these alternatives apart will take further experimentation including increased task constraints, such as limiting trunk motion and/or use of a self-paced treadmill that does not require subjects to adjust position on the treadmill.

Interestingly, if we apply a similar impulse response function analysis used here on data collected in a previous posture experiment (Kiemel et al., 2011) where subjects stood upright ("quiet stance") in the same visual cave, we also observe a response of trunk orientation prior to hip AP displacement, a similar indicator of whole body displacement. As seen in **Figure 9**, when the visual scene rotates forward, the trunk starts to rotate forward before the hip moves forward. Thus, the same temporal ordering of responses occurs in both standing and walking, suggesting an alternative interpretation that the reason for this temporal ordering in walking is not a subtask prioritization during walking, but stems from the general mechanics of interactions between lower- and upper-body motion and how the nervous system takes these interactions into account to more efficiently control movement.

## A Phase-Dependence for Mechanical Perturbations

From **Figures 4**, **5** in combination with the report of significant responses found above, it is clear that active (neurally-driven) responses to the mechanical perturbation occurred in a phasedependent manner. These phase-dependent active responses to the mechanical perturbation suggest that the nervous system corrects for mechanical disturbances occurring at critical, destabilizing phases in a reactive manner. Winter and colleagues have shown that the proximal musculature (erector spinae and others) activates prior to heel strike to counteract a destabilizing flexion of the head, arms and trunk (HAT) segment due to posterior hip acceleration occurring at heel strike (Winter et al., 1990; Winter, 1995). The moment of force produced by CNS with combined activations of proximal musculature has been deemed the "balancing moment" while the destabilizing force has been deemed the "unbalancing moment" (Winter, 1995). Tang and colleagues have noted that these results by Winter and colleagues (Winter et al., 1990) were found during unperturbed walking, and suggested they reflect a phase-dependent proactive control when walking is not perturbed (Tang et al., 1998). Using perturbations at the support surface they found that proximal muscles of the trunk (rectus abdominus and erector spinae) are not sufficiently modulated during reactions to such stimuli, and do not play a role in active balance responses.

Here we observe a counteracting erector spinae response to a mechanical perturbation which is applied at the trunk, providing a reactive, active balance response. Interestingly, common stimulus phases of both the responses in the erector spinae and the eventual "overshoot" responses in trunk orientation are observed at terminal swing phases in either foot, and these are phases in which Winter's "balancing moment" at the hip is ramping up to its peak to counteract the peak "imbalancing moment" of heel strike. Thus, the reactive response observed here occurs simultaneous with the proactive ramping up of muscular activations for the "balancing moment," and we can speculate the nervous system's control strategy is to diminish any (internal or external) destabilizing mechanical threats to upright trunk orientation at these critical phases of the gait cycle. In sum, both the site (limb level) of application and gait cycle phase will dictate if the nervous system needs to correct for deviations to a mechanical perturbation during walking.

Clearly, active control in response to the mechanical perturbation must involve sensing the change in trunk orientation at some phase prior to initiating the phasedependent active response. Phase-dependent stimulation of sensory afferents through perturbations, such as vibration of trunk muscles could likely inform about the role of trunk muscle afferents for these phase-dependent modifications for trunk orientation. Vibration of erector spinae has been successfully performed during walking and has shown that continuous vibration can elicit deviations in walking trajectory (Schmid et al., 2005; Courtine et al., 2007). As phase-dependence in somatosensory inputs of the lower limbs has been welldocumented (Duysens et al., 1990; Sinkjær et al., 1996), it is surprising that trunk vibration dependent on gait cycle phase was not tested in those studies (Schmid et al., 2005; Courtine et al., 2007) and has not yet, to our knowledge, been tested in other studies. The question of whether or not somatosensory information regarding trunk motion is available to the nervous system on a phase-dependent basis is an open one.

Somatosensory input may inform that trunk motion has been altered at all phases, yet this input is only used at specific phases. As seen from the impulse responses and mean waveforms in **Figure 8**, modulation of EST muscle activity to the mechanical perturbation occurs during the phase of the gait cycle that EST is typically most active. The EST activations occurring at early stance observed here counteract the potentially increased "unbalancing moment" at the trunk due to the mechanical perturbation, and prevent inappropriately large flexion of the trunk after heel strike. It is most likely that the observation of active trunk responses to the mechanical perturbation are facilitated by a phase-dependent change in activation, and we suggest that it takes place because the phase of perturbation where the mechanical perturbation occurs is a known preparatory phase for balance adjustments.

#### Limitations

This study assumes that walking is the output of a system with a stable limit cycle. We also assume that both intrinsic and external perturbations are small, yielding a local limit cycle (LLC) approximation of the system in which the only nonlinearities are periodic functions of the system's phase (Ermentrout and Kopell, 1984). If the system has a "clock" that prevents phase resetting (for example, walking in sync with a metronome), then the nonlinear functions are periodic functions of time and the system is approximately (LTP) (Möllerstedt and Bernhardsson, 2000). The method used in this study extends the computation of φIRFs from LTP systems to LLC systems. However, not all LTP analyses can be extended to LLC systems. For example, for stable linear time varying systems, including LTP systems, one can compute variance accounted for (VAF), the percentage of a system's variance due to its response to a specific perturbation (e.g., MacNeil et al., 1992). This definition of VAF depends on the system's linearity and, therefore, cannot be applied to LLC systems. Phase in a LLC system is a neutrally stable direction, so that phase variability due to perturbations will, in general, grow with time until it is affected by the phase nonlinearities of the LLC approximation (Demir et al., 2000).

## Implications for Locomotive Control and Future Directions

A mechanistic extension of the experimental setup used here would be to work within the control theoretic framework of **Figure 1** with the long term goal of closed loop system identification (Roth et al., 2014) using the joint input-output (JIO) approach (Katayama, 2005; van der Kooij et al., 2005; Kiemel et al., 2011). Doing so relies on the observation of both kinematic and EMG responses to sensory and mechanical perturbations (Kiemel et al., 2011), and could lead to the nonparametric identification of the musculoskeletal plant and neural feedback for walking, such as that revealed in standing postural control (Kiemel et al., 2008, 2011). This would require a scaling of the analytical tools used for postural control already begun in the HTFs and φIRFs used here (Kiemel et al., 2016), and also require considerable advances in experimental methods used for perturbation.

Prior to full identification with use of the JIO, however, one can learn about a system with careful manipulation of experimental conditions. For example, a mechanical perturbation that produces the same kinematic responses but different EMG responses in an experiment with two conditions indicates that properties of the neural feedback change between the two conditions. As we have emphasized trunk orientation control in this experiment, it is expected that an experiment with conditions which require varying needed corrections of trunk orientation, such as use of a backboard or not would elicit changes in EST, and potentially other muscles, contributing to the trunk orientation subtask. We expect that simultaneous mechanical and visual perturbations used during experimental conditions which subjects perform a specific function will inform about how that specific function is controlled during walking. Such experiments offer a novel way to distill out how control differs between subtasks, and offers great promise for distinguishing differences in locomotive control between those with neural deficits and healthy controls.

Our present focus is to work within a system identification framework to investigate the neural control of human walking. However, these tools could be applied to study the neural control of other forms of locomotion approximated as a limit cycle, such as running, cycling, or swimming. Additionally, these techniques are ideal for the study of rhythmic motor behaviors, such as juggling and have already shown promise for application in animal models, such as the isolated lamprey spinal cord (Massarelli et al., 2014, 2015).

## AUTHOR CONTRIBUTIONS

DL, TK, and JJ designed and planned the experiment. DL collected the data. DL and TK analyzed the data. DL and TK wrote the manuscript. DL, TK, and JJ edited and readied the manuscript for submission.

## FUNDING

Support for this research provided by: NSF grants 0924883 and BCS-1230311 (JJ, TK PIs). Partial funding for open access provided by the UMD Libraries' Open Access Publishing Fund.

## REFERENCES


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Logan, Kiemel and Jeka. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Estimation of Time-Varying, Intrinsic and Reflex Dynamic Joint Stiffness during Movement. Application to the Ankle Joint

Diego L. Guarín\* and Robert E. Kearney

Biomedical Engineering Department, McGill University, Montréal, QC, Canada

#### Edited by:

Massimo Sartori, University of Göttingen, Germany

#### Reviewed by:

Hyunglae Lee, Rehabilitation Institute of Chicago, United States Steven K. Charles, Brigham Young University, United States Alfred C. Schouten, Delft University of Technology, Netherlands

#### \*Correspondence:

Diego L. Guarín diego.guarinlopez@mail.mcgill.ca

> Received: 30 January 2017 Accepted: 26 May 2017 Published: 09 June 2017

#### Citation:

Guarín DL and Kearney RE (2017) Estimation of Time-Varying, Intrinsic and Reflex Dynamic Joint Stiffness during Movement. Application to the Ankle Joint.

Front. Comput. Neurosci. 11:51. doi: 10.3389/fncom.2017.00051 Dynamic joint stiffness determines the relation between joint position and torque, and plays a vital role in the control of posture and movement. Dynamic joint stiffness can be quantified during quasi-stationary conditions using disturbance experiments, where small position perturbations are applied to the joint and the torque response is recorded. Dynamic joint stiffness is composed of intrinsic and reflex mechanisms that act and change together, so that nonlinear, mathematical models and specialized system identification techniques are necessary to estimate their relative contributions to overall joint stiffness. Quasi-stationary experiments have demonstrated that dynamic joint stiffness is heavily modulated by joint position and voluntary torque. Consequently, during movement, when joint position and torque change rapidly, dynamic joint stiffness will be Time-Varying (TV). This paper introduces a new method to quantify the TV intrinsic and reflex components of dynamic joint stiffness during movement. The algorithm combines ensemble and deterministic approaches for estimation of TV systems; and uses a TV, parallel-cascade, nonlinear system identification technique to separate overall dynamic joint stiffness into intrinsic and reflex components from position and torque records. Simulation studies of a stiffness model, whose parameters varied with time as is expected during walking, demonstrated that the new algorithm accurately tracked the changes in dynamic joint stiffness using as little as 40 gait cycles. The method was also used to estimate the intrinsic and reflex dynamic ankle stiffness from an experiment with a healthy subject during which ankle movements were imposed while the subject maintained a constant muscle contraction. The method identified TV stiffness model parameters that predicted the measured torque very well, accounting for more than 95% of its variance. Moreover, both intrinsic and reflex dynamic stiffness were heavily modulated through the movement in a manner that could not be predicted from quasi-stationary experiments. The new method provides the tool needed to explore the role of dynamic stiffness in the control of movement.

Keywords: biological system modeling, nonlinear system identification, time-varying systems, dynamic joint stiffness, joint neuromechanics

## 1. INTRODUCTION

The role of the short-latency stretch reflex during movement remains controversial (Dietz et al., 1979; Sinkjaer et al., 1996; Zehr and Stein, 1999). While some studies suggest that reflex response serves to facilitate all voluntary movements (Dufresne et al., 1980; Gottlieb and Agarwal, 1980), others have proposed that the reflex response plays a role only in extreme or pathological cases (Dietz et al., 1980), or during early adaptation to new tasks or conditions (Burdet et al., 2013).

EMG is often used to study the functional role of reflexes (Dietz et al., 1979; Stein and Capaday, 1988; Zehr and Stein, 1999; Burdet et al., 2013). However, EMG is influenced by factors other than reflexes, such as voluntary activity, and it is difficult to separate the reflex EMG response from the overall EMG activity. In addition, the relation between EMG and joint torque is influenced by muscle length and contraction velocity, so that is difficult to estimate the mechanical contributions of stretch reflex from EMG alone (Toft et al., 1991; Stein and Kearney, 1995; Kearney et al., 1999).

H-reflexes have also been used to quantify the reflex activity (Sinkjaer et al., 1993). However, H-reflexes bypass the response of muscle spindles to joint position changes, which can be heavily modulated during function via γ -motor neurons (Sinkjaer et al., 1996). In addition, direct stimulation of the nerve may excite a range of afferent mechanisms that project to α−motorneurons (e.g., skin sensors, Golgi tendon organs) so that the resultant response will be generated by unphysiological combination of afferent activity (Van der Helm et al., 2002). Consequently, the functional relevance of these H-reflex studies is not completely clear.

A better approach would be to directly measure the mechanical consequences of reflex activity. However, it is difficult to separate reflex torques from those due to the mechanical or intrinsic properties of the muscle and connective tissue. Experimentally this has been achieved by comparing the mechanical behavior of a joint before and after deafferentation using surgery (Kirsch et al., 1994), or some other manipulation to suppress the reflex response (Dietz et al., 1980; Allum and Mauritz, 1984). However, it is not possible to be sure that the deafferentation process affects only the stretch reflex, and to what extent. This process will likely also affect the intrinsic properties of the joint (Kearney et al., 1997; Van der Helm et al., 2002).

An alternative approach is to perform the "deafferentation" by using mathematical models and system identification techniques to separate the mechanical effects of intrinsic and reflex mechanisms. System identification techniques, using small, random position or torque perturbations to excite the intrinsic and reflex dynamics, have been successfully applied to multiple joints with different model types (Gottlieb and Agarwal, 1978; Zhang and Rymer, 1997; Van der Helm et al., 2002; Klomp et al., 2014). These models have typically been linear; however, the mechanical response produced by stretch reflexes are highly nonlinear (Stein and Kearney, 1995), so that these models fail to completely characterize the stretch reflex mechanisms or simply ignore it. The parallel-cascade model, proposed by Kearney et al. (1997), describes the intrinsic and stretch reflex mechanisms in terms of dynamic joint stiffness, that determines the dynamic relation between joint position and torque. Intrinsic dynamic stiffness, also referred to as joint impedance, arises from the inertial and visco-elastic properties of the joint, passive tissue, and active muscle fibers, and is described by a linear model relating joint position and torque. Reflex dynamic stiffness arises from changes in muscle activation due to the short-latency stretch reflex, and is described by a nonlinear, Hammerstein model relating joint velocity and torque (Kearney and Hunter, 1990).

Successful applications of these analytical techniques have been typically limited to stationary conditions, where the dynamic properties of the joint remain constant for the duration of the experiment. Such experiments have shown that the parallel-cascade model parameters change with joint position and voluntary torque (Mirbagheri et al., 2000; Guarin et al., 2013). Consequently, during most functional activities when the joint position and voluntary torque change rapidly and continuously, the dynamic joint stiffness model parameters will be time-varying (TV).

Several studies have characterized dynamic joint stiffness during TV conditions by modeling the intrinsic and reflex response together using a single linear model (Bennett et al., 1992; MacNeil et al., 1992; Kirsch and Kearney, 1997; Rouse et al., 2014; Lee and Hogan, 2015). These type of models cannot provide any information regarding the modulation of reflex mechanisms and likely overestimate the contribution of intrinsic mechanisms to the overall dynamic joint stiffness. We have introduced methods to estimate intrinsic and stretch reflex mechanisms using the parallel-cascade model structure during TV conditions; however, these methods require very large data sets for parameter estimation, which severely limits their application (Giesbrecht et al., 2006; Ludvig et al., 2011; Guarin and Kearney, 2012, 2015b); or make the strong assumption that there is a static-nonlinear relation between the parallel-cascade model parameters and joint position or torque (Sobhani Tehrani et al., 2013; Jalaleddini et al., 2015). Despite their limitations, these studies have shown that the interpolation of parameter values obtained from stationary experiments does not describe dynamic joint stiffness during TV conditions. Therefore, methods able to track the fast, large changes in the model parameters using short data records are required to characterize the modulation of the dynamic joint stiffness during function.

This paper develops and validates a novel method to estimate the intrinsic and reflex components of dynamic joint stiffness during periodic movements. This method improves over previous algorithms in several ways: (i) it reduces the size of the data set required for accurate parameter estimation; and (ii) it parametrizes the system and noise plants independently, which eliminates biases in parameter estimates due to the colored noise present in measurements of joint torque.

This paper is organized as follows: Section 2 presents the TV, parallel-cascade model of dynamic joint stiffness and introduces a novel re-parameterization that approximates, the non-linear, TV model with a set of linear, time-invariant models. It then introduces an algorithm to estimate the parameters of this model using data acquired during periodic, TV conditions. Section 3 describes a simulation study that evaluated the performance of the new model parameterization and identification algorithm. Section 4 demonstrates the practical application of the algorithm by using it to estimate intrinsic and reflex dynamic ankle stiffness during experiments where movements were imposed on subjects while they exerted a constant voluntary torque. Section 5 summarizes the contributions and discusses some important aspects underlying the method and its application.

## 2. MODEL FORMULATION AND PARAMETER IDENTIFICATION

#### 2.1. Joint position perturbations and torque

Estimation of dynamic joint stiffness requires the application of small position perturbations that do not modify joint position and have power over a wide enough range of frequencies to excite the system adequately (Kearney and Hunter, 1990). Consequently, to estimate dynamic joint stiffness during movement, small position perturbations must be superimposed on the movement trajectory, producing an overall, perturbed joint position given by

$$
\theta(t\_k) = \theta\_0(t\_k) + \theta\_p(t\_k), \tag{1}
$$

where θ0(t<sup>k</sup> ) is the movement trajectory and θp(t<sup>k</sup> ) is the position perturbation.

Under stationary conditions, when the joint trajectory and voluntary torque are almost constant, the net moment at the joint is

$$TQ(t\_k) = TQ\_0 + TQ\_p(t\_k),$$

where TQ<sup>0</sup> is a constant torque, produced by passive mechanisms due to θ<sup>0</sup> (which might be equal to zero if the joint is at its neutral position), and by active mechanisms due to the constant muscle activation; and TQp(t<sup>k</sup> ) is a perturbation torque, produced by the excitation of intrinsic and reflex mechanisms given by

$$TQ\_{\mathcal{P}}(t\_k) = TQ\_I(t\_k) + TQ\_R(t\_k)$$

where TQI(t<sup>k</sup> ) and TQR(t<sup>k</sup> ) are the torques produced by the intrinsic and reflex mechanisms, which cannot be measured directly. Under stationary conditions, an estimate of the perturbation torque can be retrieved from measurements of total joint torque by removing the constant offset TQ0. The perturbation position and torque can then be used to estimate the intrinsic and reflex contributions to the total torque.

In contrast, under TV conditions, when the joint trajectory (θ0(t<sup>k</sup> )) and/or the muscle activation level vary, the torque produced by passive and voluntary mechanisms (TQ0(t<sup>k</sup> )), is no longer constant. Consequently, estimating the perturbation torque from measurements of total joint torque requires three steps: First, a perturbed joint trajectory is applied and the total joint torque, given by

$$TQ(t\_k) = TQ\_0(t\_k) + TQ\_p(t\_k),$$

is recorded. Second, an unperturbed joint trajectory is applied and the joint torque TQ∗ 0 (tk ) is recorded. Finally, the difference between the net joint torque in the two experiments is computed to estimate the torque due to the perturbations. However, it is not realistic to expect that the joint will follow exactly the same trajectory and/or that the subject will exert exactly the same voluntary torque in the perturbed and unperturbed experiments. Therefore, under TV conditions, the perturbation torque will be given by

$$TQ\_{\mathcal{P}}(t\_k) = TQ\_I(t\_k) + TQ\_R(t\_k) + TQ\_\Delta(t\_k),\tag{2}$$

where TQ1(t<sup>k</sup> ) is an additional torque due to difference in passive and voluntary torques during the perturbed and unperturbed experiments.

#### 2.2. Time-Varying Dynamic Joint Stiffness

Once the perturbation position and torque are available, system identification can be used to separate the intrinsic and reflex components analytically. Under stationary conditions, this can be achieved by modeling the overall dynamic joint stiffness with a parallel-cascade model, where intrinsic stiffness is described by a linear system relating joint position and intrinsic torque, and reflex stiffness by a Hammerstein system relating joint velocity and reflex torque (Kearney et al., 1997; Guarin et al., 2013; Jalaleddini et al., 2016).

Under TV conditions, a TV version of the parallel-cascade structure, shown in **Figure 1**, has been successfully applied to describe the overall dynamic joint stiffness (Giesbrecht et al., 2006; Ludvig et al., 2011; Guarin and Kearney, 2012, 2015b; Jalaleddini et al., 2017). However, the identification algorithms used to estimated the TV model parameters require very large data sets and so are difficult to use in practice.

Here, we introduce an alternative parameterization of the TV, nonlinear, parallel-cascade model of dynamic joint stiffness that transforms it into a set of pseudo-linear, time-invariant models. Next, we will introduce an identification algorithm that uses a small data set to estimate the TV model parameters.

#### 2.2.1. Intrinsic Dynamic Stiffness

Intrinsic dynamic stiffness is usually described by a second order, linear model with limb inertia, joint viscosity, and static stiffness relating perturbation position and torque (Kearney and Hunter, 1990)

$$TQ\_l(t\_k) = K(t\_k)\theta\_p(t\_k) + B(t\_k)\frac{d[\theta\_p(t\_k)]}{dt\_k} + I\frac{d^2[\theta\_p(t\_k)]}{d^2t\_k},\tag{3}$$

where K(t<sup>k</sup> ), B(t<sup>k</sup> ) and I are the intrinsic static stiffness, viscosity and inertia. However, recent experimental evidence suggests that the intrinsic dynamics stiffness is more complex than secondorder (Sobhani Tehrani et al., 2017). Therefore, we choose to describe intrinsic stiffness with the TV, non-parametric model

$$TQ\_I(t\_k) = \sum\_{\mathfrak{r}=-L}^{\mathfrak{r}=L} h\_I(\mathfrak{r}, t\_k) \theta\_p(t\_k - \mathfrak{r}),\tag{4}$$

where hI(τ , t<sup>k</sup> ) is a TV, impulse response function (IRF) that requires no a priori assumption of model order. The length of

the system memory must be specified a priori, and there is much evidence than a memory of 40 ms is adequate for the ankle joint (Kearney et al., 1997). Therefore, intrinsic dynamic stiffness is represented by a two sided IRF with a memory from −40 ms to 40 ms.

#### **2.2.1.1. Model re-parameterization**

The TV parameters in Equation (4) will be approximated by a linear combination of basis functions as

$$h\_I(\boldsymbol{\tau}, t\_k) = \sum\_{j=0}^{j=n\_k} \lambda\_{\boldsymbol{\tau}, j} \Lambda\_j(t\_k),$$

where {3j(t<sup>k</sup> )} j=n<sup>λ</sup> j = 0 are a set of time-varying basis functions and λ<sup>τ</sup> ,<sup>j</sup> their coefficients. Intrinsic dynamic stiffness can then be approximated by the linear, time-invariant (LTI) model

$$TQ\_{I}(t\_{k}) = \sum\_{\mathfrak{r}=-L}^{\mathfrak{r}=L} \sum\_{j=0}^{j=n\_{k}} \lambda\_{\mathfrak{r},j} \Lambda\_{j} \left(t\_{k}\right) \theta\_{p}(t\_{k}-\mathfrak{r}).\tag{5}$$

#### 2.2.2. Reflex Dynamic Stiffness

Reflex dynamic stiffness can be described by a series connection of a differentiator, a delay of 40 ms and a Hammerstein system, comprising the series combination of a static-nonlinearity and a second-order, linear dynamic system, relating joint velocity and reflex torque (Kearney et al., 1997; Guarin et al., 2013; Guarin and Kearney, 2015b). The input-output relation is given by

$$\stackrel{\circ}{\partial}\_{\mathcal{P}}(t\_k) = \mathcal{g}\left(\dot{\theta}\_{\mathcal{P}}(t\_k), t\_k\right), \tag{6a}$$

$$\frac{d^2[T Q\_R(t\_k)]}{dt\_k^2} + 2\zeta(t\_k)\omega(t\_k)\frac{d[T Q\_R(t\_k)]}{dt\_k} + \omega^2(t\_k)T Q\_R(t\_k)}{dt\_k}$$

$$= G(t\_k)\alpha^2(t\_k)\dot{\bar{\partial}}\_{\mathcal{P}}(t\_k), \tag{6b}$$

where θ˙ <sup>p</sup>(t<sup>k</sup> ) is the delayed joint velocity, and g(•, t<sup>k</sup> ) is a TV, static non-linearity. G(t<sup>k</sup> ), ω(t<sup>k</sup> ), and ζ (t<sup>k</sup> ) are the gain, natural frequency and damping of the reflex linear dynamics.

This TV, continuous-time model can be approximated by the set of discrete-time, transfer function models

$$TQ\_{\mathbb{R}}(t\_k) = \frac{b\_0(t\_k)\left(1 + 2q^{-1} + q^{-1}\right)}{1 + a\_1(t\_k)q^{-1} + a\_2(t\_k)q^{-2}}\overset{\mathbb{P}}{\theta}\_{\mathbb{P}}(t\_k),\tag{7}$$

where b0(t<sup>k</sup> ), a1(t<sup>k</sup> ) and a2(t<sup>k</sup> ) are discrete-time, TV parameters and q −1 is the backward shift operator. The continuous-time and discrete-time parameters are related to each other by

$$\begin{aligned} G(t\_k) &= 4 \left[ \frac{b\_0(t\_k)}{1 + a\_1(t\_k) + a\_2(t\_k)} \right], \\ \omega(t\_k) &= \frac{2}{T\_s} \left[ \frac{1 + a\_1(t\_k) + a\_2(t\_k)}{1 - a\_1(t\_k) + a\_2(t\_k)} \right]^{1/2}, \\ \xi(t\_k) &= \frac{1 - a\_2(t\_k)}{\left[ \left( 1 + a\_1(t\_k) + a\_2(t\_k) \right) \left( 1 - a\_1(t\_k) + a\_2(t\_k) \right) \right]^{1/2}}. \end{aligned}$$

where T<sup>s</sup> is the sampling time in seconds.

#### **2.2.2.1. Model re-parameterization**

The TV, static non-linearity will be approximated by

$$\bar{\boldsymbol{\partial}}\_{\mathcal{P}}(\boldsymbol{t}\_k) = \mathcal{g}\left(\boldsymbol{\vartheta}\_{\mathcal{P}}(\boldsymbol{t}\_k), \boldsymbol{t}\_k\right) \approx \sum\_{i=0}^{i=n\_k} \boldsymbol{c}\_i\left(\boldsymbol{t}\_k\right) \mathbf{C}\_i\left(\boldsymbol{\vartheta}\_{\mathcal{P}}\left(\boldsymbol{t}\_k\right)\right) \mathbf{J}$$

where Ci(•) are a set of pre-defined basis functions (e.g., polynomials, radial basis) and c<sup>i</sup> are their TV coefficients. Following the same procedure as before, these are approximated by a linear combination of basis functions as

$$c\_i(t\_k) = \sum\_{j=0}^{j=n\_\gamma} \mathcal{V}\_{i,j} \Gamma\_j \left( t\_k \right),$$

where {Ŵj(t<sup>k</sup> )} j=n<sup>γ</sup> j=0 are a set of time-varying basis functions and γi,<sup>j</sup> their coefficients.

Similarly, the TV parameters of the linear dynamic element will be approximated by a linear combination of basis functions

$$\begin{aligned} b\_0 \left( t\_k \right) &= \sum\_{j=0}^{j=n\_{\beta}} \beta\_{0,j} \Psi\_j \left( t\_k \right), \\ a\_i \left( t\_k \right) &= \alpha\_{i,0} + \sum\_{j=1}^{j=n\_{\alpha}} \alpha\_{i,j} \Pi\_j \left( t\_k \right), \quad i = 0, \ldots, n\_{\alpha}. \end{aligned}$$

where αi,0 6= 0; {9j(t<sup>k</sup> )} j=n<sup>β</sup> j=0 and {5j(t<sup>k</sup> )} j=n<sup>α</sup> j=0 are sets of timevarying basis functions with 50(t<sup>k</sup> ) = 1, ∀t<sup>k</sup> ; β0,<sup>j</sup> , and αi,<sup>j</sup> their coefficients.

Using the approximations with the basis functions, the relation between the reflex torque and joint velocity is now time-invariant and can be described by the discrete-time, timeinvariant, Hammerstein system

$$\stackrel{\circ}{\partial}\_{\mathcal{P}}(t\_k) = \sum\_{i=0}^{i=n\_c} \sum\_{j=0}^{j=n\_V} \mathcal{Y}\_{i,j} \Gamma\_j(t\_k) \, \mathcal{C}\_i \left( \dot{\theta}\_{\mathcal{P}}(t\_k) \right), \tag{8a}$$

$$\begin{split} TQ\_{R}\left(t\_{k}\right) &= \frac{1}{F\left(q^{-1}\right)} \left[ -\sum\_{j=1}^{n\_{a}} \alpha\_{1j} \Pi\_{\dot{j}}\left(t\_{k}\right) TQ\_{R}\left(t\_{k} - 1\right) \\ -\sum\_{j=1}^{n\_{a}} \alpha\_{2j} \Pi\_{\dot{j}}\left(t\_{k}\right) TQ\_{R}\left(t\_{k} - 2\right) + \sum\_{j=0}^{j=n\_{\beta}} \beta\_{0,j} \Psi\_{\dot{j}}\left(t\_{k}\right) \vec{\dot{\partial}}\_{p}\left(t\_{k}\right) \right], \end{split} \tag{8b}$$

where F q −1 is the polynomial

$$F\left(q^{-1}\right) = 1 + \alpha\_{1,0}q^{-1} + \alpha\_{2,0}q^{-2},$$

#### 2.2.3. Other Components

TQ1(t<sup>k</sup> ) is expected to be a stochastic, low-frequency signal that will be described by a linear combination of basis functions

$$TQ\_\Delta(t\_k) = \sum\_{j=0}^{j=n\_p} p\_i P\_i(t\_k),\tag{9}$$

where {Pi(t<sup>k</sup> )} j=n<sup>p</sup> j=0 are a set of time-varying basis functions and p<sup>i</sup> their coefficients.

#### 2.2.4. Overall Joint Stiffness

Using these re-parameterizations, the overall relation between joint position and torque, shown in **Figure 1**, can be approximated by the LTI models shown in Equations (5), (8a), (8b), and (9) the unknown parameters

$$\begin{aligned} \rho\_I &= \begin{bmatrix} \lambda\_{-L,0} \cdots \lambda\_{-L,n\_\lambda} \cdots \lambda\_{L,0} \cdots \lambda\_{L,n\_\lambda} \end{bmatrix}, \\ \rho\_R &= \begin{bmatrix} \alpha\_{1,0} \cdots \alpha\_{1,n\_\alpha} \cdots \alpha\_{2,0} \cdots \alpha\_{2,n\_\alpha} \beta\_{0,0} \cdots \beta\_{0,n\_\beta} \end{bmatrix} \end{aligned} \tag{10a}$$

$$\boldsymbol{\rho}\_{\mathcal{R}} = \left[ \alpha\_{1,0} \cdot \cdots \alpha\_{1,n\_{\mathcal{U}}} \cdot \cdots \alpha\_{2,0} \cdot \cdots \alpha\_{2,n\_{\mathcal{U}}} \, \beta\_{0,0} \cdot \cdots \, \rho\_{0,n\_{\mathcal{R}}} \right. \tag{10b}$$

$$\boldsymbol{\gg}\_{0,0} \cdot \cdots \cdot \boldsymbol{\gg}\_{0,n\_{\mathcal{V}}} \cdot \cdots \cdot \boldsymbol{\gg}\_{n\_{\mathcal{L}},0} \cdot \cdots \, \boldsymbol{\gg}\_{n\_{\mathcal{L}},n\_{\mathcal{V}}} \right],\tag{10b}$$

$$\boldsymbol{\rho}\_{\Delta} = \left[\boldsymbol{\rho}\_{0} \cdots \boldsymbol{\rho}\_{n\_{\rho}}\right]^{\cdot},\tag{10c}$$

where ρ<sup>I</sup> , ρ<sup>R</sup> , and ρ<sup>1</sup> are vectors containing the unknown parameters used to describe the intrinsic, reflex and additional torques, respectively.

#### 2.3. Identification of TV, Dynamic Joint Stiffness

We now describe an algorithm for the identification of the reparametrized models of TQI(t<sup>k</sup> ), TQR(t<sup>k</sup> ), and TQ1(t<sup>k</sup> ). There are four key elements to the algorithm: First, as **Figure 1** illustrates, these torques cannot be measured directly so the models describing each component cannot be estimated directly from measured data. Consequently, the intrinsic and reflex components will be estimated using an iterative algorithm that estimates the parameters of each pathway sequentially, removing the influence of the other pathways in the total torque before estimating the parameters of each component (Kearney et al., 1997; Guarin and Kearney, 2015b).

Second, the parameters of each element of the Hammerstein system that represents the reflex component will be estimated using a second iterative algorithm. This method estimates the coefficients of the static nonlinearity and reflex dynamics iteratively using a coordinate ascent approach. The algorithm is guaranteed to converge to the true values under general conditions (Bai and Li, 2004; Guarin and Kearney, 2015a).

Third, an instrumental variable approach, that provides unbiased estimates of the model parameters even in the presence of non-white noise, will be used to estimate the reflex linear dynamic element (Laurain et al., 2010; Guarin and Kearney, 2016).

Finally, the identification algorithm assumes that there are available multiple input-output trials presenting the same timevarying behavior. The algorithm exploits this to estimate the parameters' time-course from multiple realization of inputoutput data. Moreover, the algorithm assumes that the timevarying behavior is periodic and it automatically estimates the initial conditions at each trial.

The identification algorithm combines two TV identification methodologies: temporal expansion and ensemble approaches. We recently introduced this hybrid identification approach and showed that it can track faster parameters changes than the temporal expansion method while requiring less data than classical ensemble approaches (Guarin and Kearney, 2016).

#### 2.3.1. Identification Algorithm

Assume that n cycles, each with N data points, of joint position and torque were measured for both the unperturbed and perturbed joint movements. The position perturbation and torque signals are computed by aligning and subtracting the unperturbed from the perturbed measurements. Following Equation (2), the noise-free perturbation torque for n cycles can be organized in matrix form as

$$
\begin{bmatrix} T Q\_{\mathcal{P}} \{ 1 \} \\ \vdots \\ T Q\_{\mathcal{P}} \{ n \} \end{bmatrix} = \begin{bmatrix} T Q\_{I} \{ 1 \} \\ \vdots \\ T Q\_{I} \{ n \} \end{bmatrix} + \begin{bmatrix} T Q\_{\mathcal{R}} \{ 1 \} \\ \vdots \\ T Q\_{\mathcal{R}} \{ n \} \end{bmatrix} + \begin{bmatrix} T Q\_{\mathcal{A}} \{ 1 \} \\ \vdots \\ T Q\_{\mathcal{A}} \{ n \} \end{bmatrix}, \tag{11}
$$

where

$$T\mathcal{Q}\_{\mathcal{P}}\{j\} = \left[T\mathcal{Q}\_{\mathcal{P}}(1)\{j\}\cdots \mathcal{T}\mathcal{Q}\_{\mathcal{P}}(N)\{j\}\right]^T,$$

is the perturbation torque for the j-th cycle. The identification algorithm assumes that intrinsic and reflex dynamics have the same TV behavior in each cycle and that the TV model parameters are periodic. In contrast, TQ1(t<sup>k</sup> ) is assumed to be different for each cycle, so that the parameters describing it are different for each cycle.

The identification algorithm proceeds as follows:

1. Initialize

$$
\widehat{TQ}\_I|j\rangle = \widehat{TQ}\_R|j\rangle = O\_\bullet \quad j = 1, \cdots, n.
$$

2. Estimate TQ1 for each cycles as

$$\widetilde{T}\widetilde{Q}\_{\Delta}\{j\} = TQ\_{\mathcal{P}}\{j\} - \left(\widehat{TQ}\_{I}\{j\} + \widehat{TQ}\_{R}\{j\}\right)$$



3. Estimate the intrinsic torque as

$$\widetilde{T}\widetilde{Q}\_I\{j\} = TQ\_{\mathcal{P}}\{j\} - \left(\widetilde{T}\widehat{Q}\_R\{j\} + \widetilde{T}\widehat{Q}\_\mathbf{A}\{j\}\right)$$



4. Estimate the reflex torque as

$$
\widetilde{T}\widetilde{Q}\_R\{j\} = TQ\_{\mathfrak{P}}\{j\} - \left(\widehat{T}\widehat{Q}\_I\{j\} + \widehat{T}\widehat{Q}\_\mathbf{\mathbf{A}}\{j\}\right)
$$


of TQcR{j} for each cycle. 5. Compute the net predicted torque for all cycles as

$$
\widehat{TQ}\_{\mathcal{P}}\{j\} = \widehat{TQ}\_{I}\{j\} + \widehat{TQ}\_{R}\{j\} + \widehat{TQ}\_{\mathbf{A}}\{j\}
$$

and calculate the variance accounted for (%VAF) between the predicted and measured torque signals as

$$\%VAF = \left[1 - \frac{\sum\_{t\_k=1}^{t\_k=N\*n} \left(TQ\_p(t\_k) - \widehat{TQ}\_p(t\_k)\right)^2}{\sum\_{t\_k=N\*n}^{t\_k=N\*n} \left(TQ\_p(t\_k)\right)^2}\right] \times 100\%$$

where N ∗ n is the total number of samples.

6. Repeat the procedure from step 2 until successive iterations fail to improve the %VAF.

The identification algorithm predicts the intrinsic (TQc<sup>I</sup> (tk )), reflex (TQcR(t<sup>k</sup> )), and additional (TQc1(t<sup>k</sup> )) torques, as well as the model parameters <sup>ρ</sup>ˆ<sup>I</sup> and <sup>ρ</sup>ˆ<sup>R</sup> . A Matlab implementation of this algorithm and an application example are provided by DLG in GitHub<sup>1</sup>

#### 3. SIMULATION STUDY

#### 3.1. Methods

The accuracy of the new algorithm was evaluated using simulations of TV, dynamic ankle stiffness throughout a periodic movement resembling the ankle movement during gait.

#### 3.1.1. Simulated Model

**Figure 2** shows the TV, dynamic joint stiffness model used in the simulation. Intrinsic stiffness was simulated as a TV, continuoustime, second-order system. Reflex stiffness was modeled as the series connection of a 40 ms delay, a differentiator, and a Hammerstein system whose static-nonlinear element was a half-wave rectifier with a TV threshold (th(t<sup>k</sup> )), and whose linear dynamic element was a TV, continuous-time, second-order system. The model was simulated in Simulink (the MathWorks) using a third order solver with a sampling rate of 1 kHz. Each simulated cycle lasted 1.4 s, which is equivalent to slow walking Sinkjaer et al. (1996); 40 cycles were simulated so that each trial lasted for 56 s. Perturbation position and torque were filtered and decimated to 100 Hz for analysis. The 56 s trial was repeated 100 times with a different input and noise realizations to compute statistical properties for the parameter estimates.

#### **3.1.1.1. Model parameters**

**Figure 3** shows how the simulated parameters were varied periodically in the simulations. The variation of the intrinsic stiffness parameters, shown in **Figures 3A–C**, was based on results reported by Lee et al. (2016).

<sup>1</sup>https://github.com/dguari1/Frontiers2017.

The variations of the parameters of the linear, reflex dynamics are shown in **Figures 3D–F**. The reflex gain changes were based on those reported by Sinkjaer et al. (1996), while those of the natural frequency and damping were generated by interpolating results from stationary experiments at different joint positions (Guarin et al., 2013). The threshold of the reflex static nonlinearity changed during the portion of the cycle where the reflex gain was largest, and remained constant at zero during the remainder of the cycle.

#### 3.1.2. Typical Trial

#### **3.1.2.1. Input**

**Figure 4A** shows the position input perturbation sequence which was a Pseudo Random Arbitrary Level Distributed Signal (PRALDS) with a random switching rate drawn from a uniform distribution between 250 and 350 ms, and a peak-to-peak amplitude of 0.06 rad. PRALDS signals have velocities distributed over the entire range of possible values and so it provides a rich set of values with which to estimate the reflex staticnonlinearity (Jalaleddini and Kearney, 2013).

#### **3.1.2.2. Experimental noise**

**Figure 4B** shows a realization of the noise used in the simulations. This was obtained from a library of ankle torque records acquired while subjects maintained a constant torque at a fixed ankle position (Ranjbaran et al., 2013). The library comprised 100 records each lasting 60 s, from six subjects generating dorsiflexing torques corresponding to 5, 10, and 15% of their maximum voluntary torque. The experimental noise signal is composed of a low-frequency trend (corresponding to TQ1(t<sup>k</sup> )), physiological tremor, 60 Hz noise, and white-Gaussian measurement noise (Bezrukov et al., 2003; Ranjbaran et al., 2013). For each simulation trial, a 56 s section of the recorded torque noise was chosen at random from the library, its mean removed, and its amplitude adjusted to give an average signal-to-noise ratio (SNR) of 15 dB across the trial. This SNR is lower than that expected experimentally; (Ludvig and Kearney, 2007) reported it to be around 40 dB.

#### **3.1.2.3. Output**

**Figure 4C** shows the noise-free output-torque, the sum of the simulated intrinsic and reflex torques.

#### 3.1.3. Basis functions

Cubic B-splines were selected as the basis functions to represent the TV coefficients of the intrinsic TV-IRF ({3j(t<sup>k</sup> )} j=n<sup>λ</sup> j=0 ); these basis functions were selected because they describe smoothly changing signals, such as the simulated TV parameters, very well. A total of 10 B-splines were used to represent each TV parameter since this was found to be the minimum order necessary to account for 99% of the variability of the true TV parameters. The B-splines knots were uniformly distributed along the cycle.

B-splines were also used to represent the TV, reflex staticnonlinearity ({Ŵj(t<sup>k</sup> )} j=n<sup>γ</sup> j=0 ), and the numerator of the TV, reflex linear dynamics ({9j(t<sup>k</sup> )} j=n<sup>β</sup> j=0 ). However, they could not be used to represent the parameters of the denominator due to technical limitations associated with the identification algorithm

as described in Guarin and Kearney (2016). Consequently, Chebyshev polynomials of order 0–7 were selected as the basis functions to represent the coefficients in the denominator of the TV, reflex linear dynamics ({5j(t<sup>k</sup> )} j=n<sup>α</sup> j=0 ).

Chebyshev polynomials of order 0–4 were used to represent TQ1(t<sup>k</sup> ), since we found that they provided a more parsimonious representation of the low-frequency component, TQ1(t<sup>k</sup> ), than cubic B-splines.

Moreover, Chebyshev polynomials were used to parametrize the reflex, static-nonlinearity ({Cj(θ˙ <sup>p</sup>(t<sup>k</sup> ))} j=n<sup>c</sup> j=0 ). There are some advantages of using this polynomial representation: (i) the firstorder polynomial is linear, C1(θ˙ <sup>p</sup>(t<sup>k</sup> )) = θ˙ <sup>p</sup>(t<sup>k</sup> ), so that the estimated parameters can be used to validate whether a nonlinear model is needed or not; and (ii) the variance of the output is finite in its support, which guarantees the numerical stability of the estimation process. Polynomials of order 0–4 were used to approximate the TV static-nonlinearity.

As the gain of the Hammerstein system can be arbitrarily assigned to the static-nonlinearity or the linear dynamic element without affecting the output, we assigned the gain of the reflex pathway to the static-nonlinearity and fixed the gain of the linear dynamics to unity.

#### 3.1.4. Validation

The predictive ability of the estimated model parameters were quantified in terms of the Variance Accounted For (VAF) between the predicted and simulated torques. An average-VAF was computed for the entire simulation trial as described in step 5 of the identification algorithm. In addition, a TV-VAF was computed by dividing each gait cycle in 20 segments of equal length and computing the VAF between predicted and simulated signals for each segment.

The joint intrinsic static stiffness (K(t<sup>k</sup> )), viscosity (B(t<sup>k</sup> )) and inertia (I) were computed directly from the estimated TV-IRF by using a non-linear least-squares fit algorithm (Kearney et al., 1997), and compare to the simulated parameter.

The shape of the estimated reflex, TV, static-nonlinearity (which includes the reflex gain) was compared to that of the simulated TV, half-wave rectifier. The TV, reflex natural frequency (ω) and damping (ζ ) were computed directly from the estimated, discrete-time parameters and compared to the simulated values.

#### **3.1.4.1. Ensemble identification algorithm**

For comparison purposes we estimated the model parameters using the ensemble identification algorithm for estimation of the parallel-cascade model structure previously introduced by our group (Ludvig et al., 2011). This algorithm uses an ensemble only identification approach for estimating the TV parameters of the intrinsic and reflex dynamic joint stiffness.

#### **3.1.4.2. Time-invariant, dynamic joint stiffness model**

Furthermore, a time-invariant (TI), dynamic joint stiffness model was estimated between the perturbation position and noisy torque signals using the entire record. The TI, intrinsic and reflex model parameters were estimated using the new algorithm with the orders of the basis functions used to represent the TV intrinsic and reflex model parameters set to one, forcing them to be a constant, all-ones vector.

## 3.2. Results

#### 3.2.1. Time-Invariant Results

The TI model did not predict the simulated torque well, the average-VAF was always less than 70% for both intrinsic and reflex torques. Furthermore, **Figure 5** shows that the TV-VAF varied greatly across the cycle; it ranged between 0 and 99% for the intrinsic and between 0 and 90% for the reflex torque, indicating that the TI models did not captured the simulated system dynamics.

#### 3.2.2. Ensemble identification algorithm

The ensemble only identification algorithm required at least 400 input-output realizations to produced acceptable results. With this large data set, the average-VAF was larger than 90% for both intrinsic and reflex torques. Furthermore, **Figure 5** shows that the TV-VAF for the intrinsic torque was greater than 98% at all points in the cycle, indicating that ensemble identification algorithm accurately captured the linear, intrinsic dynamics. However, the TV-VAF for the reflex models varied greatly across the cycle; it ranged between 55 and 99%, indicating that the ensemble identification algorithm did not captured the non-linear, reflex dynamics.

#### 3.2.3. Hybrid Identification Algorithm

The TV model predicted the output extremely well, the average-VAF was always larger than 99% for both intrinsic and reflex torques. Furthermore, **Figure 5** shows that the TV-VAF for both the intrinsic and reflex torques was greater than 97% at all points in the cycle; the lowest values were observed around the portion of the cycle where the gain of the intrinsic and reflex pathways were smallest.

#### **3.2.3.1. TV intrinsic dynamic stiffness**

**Figures 6A–C** compares the simulated (red) and estimated (blue) intrinsic static stiffness, viscosity and inertia, demonstrating that the estimated parameter tracked the true value very closely and with little variability in all 100 simulation trials.

#### **3.2.3.2. TV reflex dynamic stiffness**

**Figures 7A–D** present snapshots of the estimated and simulated TV static-nonlinearity at different points of the cycle. It is evident that the estimated, polynomial static-nonlinearity accurately tracked both the TV threshold and slope of the simulated halfwave rectifier. It can also be observed that the variability of the polynomial nonlinearity was smaller for velocities around zero. The bottom panels of the figure show the simulated and estimated natural frequency and damping of the reflex, linear dynamic element, demonstrating that the estimated parameters tracked the true values closely. The reflex damping was slightly underestimated; but this did no affect the VAF, indicating that the model output is less sensitive to the damping than to the other elements.

## 4. EXPERIMENTAL STUDY

The practical utility of the new TV, identification algorithm was evaluated by using it to estimate the dynamic ankle stiffness from experimental data acquired during an imposed movement with constant voluntary torque. Data was acquired from one healthy subject who provided written informed consent. The experiment was approved by the McGill University Research Ethics Office.

## 4.1. Experimental Methods

The subject lay supine with his left foot attached to the pedal of a stiff electrohydraulic actuator operating as a position servo, which prevented the subject from voluntarily moving its ankle, by means of a custom made fiberglass boot (Kearney et al., 1997). Ankle movement was restricted to dorsiflexion and plantarflexion, defined as positive and negative angles respectively with respected to a zero-position reference, taken 90◦ between the foot and shank.

Ankle position, torque, and surface EMG from the medial and lateral gastrocnemius (GM and GL), soleus (SOL) and tibialis anterior (TA) were measured, filtered at 400 Hz to prevent aliasing and sampled at 1 kHz by a 16-bit A/D converter. Data were low-pass filtered and decimated to 100 Hz for analysis. Surface EMG electrodes were placed following the SENIAM recommendations (Hermens et al., 2000).

During each experimental trial the actuator moved the ankle to zero position and held it there for a 1 min. Then, an unperturbed trajectory, consisting of the angle of the ankle joint during walking, with a duration of 2 s, was applied; this trajectory was extracted from Lee and Hogan (2015). The trajectory was repeated periodically 30 times; the trial was repeated twice to obtain a total of 60 cycles.

The unperturbed trials were performed two times. The first time the subject was instructed to (i) be relaxed, and (ii) not react to the imposed movement. The second time, the subject was instructed to: (i) maintain a constant plantarflexion torque corresponding to 10% of its maximum torque at zero position (recorded previously at 70 Nm); and (ii) not react to the imposed movement. To assist with this task, the subject was presented with a visual feedback of a low-pass filtered (0.7 Hz) version of the measured torque minus the passive torque recorded in the previous experiment. The subject was allowed to train for several minutes before the beginning of the trial.

The trials were then repeated using a perturbed ankle trajectory by adding a PRALDS signal, similar to that used in the simulation study, to the walking trajectory. In the perturbed

trials the subject was instructed to: (i) maintain a constant plantarflexion torque corresponding to 10% of his maximum torque; and (ii) not react to the imposed movement and perturbations.

Perturbed and unperturbed position and torque records were subtracted to give the perturbation position (θp(t<sup>k</sup> )) and torque (TQp(t<sup>k</sup> )). Each trial was then divided into identification and validation segments; 40 cycles were used for parameters estimation and the remaining 20 cycles for model validation; validation data was not used for parameter estimation, only for model validation. The model was validated by computing the average-VAF between the measured and predicted torques for the validation data.

The identification procedure was started with the same number of basis functions used in simulations; a subset of basis functions was then selected automatically by using a sparse identification algorithm, which forces the weights associated to basis function that do not contribute to the reduction of the prediction error to zero so that they can be discarded Guarin and Kearney (Submitted).

Finally, joint velocity was computed by numerically differentiating the perturbation position signal. Then, the reflex delay was computed by finding the time difference between the positive peaks in the joint velocity signal and the corresponding peaks in the soleus EMG signals associated with the reflex response.

## 4.2. Results

#### 4.2.1. Typical Trial

**Figure 8** shows two cycles of the perturbed and unperturbed position, where the dorsiflexion and plantarflexion directions are indicated with black arrows; soleus EMG, and torque records. The blue lines in **Figures 9A,B** show the corresponding perturbation position (θp(t<sup>k</sup> )) and torque (TQp(t<sup>k</sup> )).

#### 4.2.2. Time-Invariant Results

The output of the TI model estimated from these data did not predict the ankle torque very well (data not shown). The average-VAF never exceeded 75%, demonstrating that a TV model is required to capture the system dynamics.

#### 4.2.3. Time-Varying Model

In contrast, the estimated TV model predicted the measured torque very well; the average-VAF for the validation trials was never less than 95%. The brown line in **Figure 9B** shows the predicted total torque (the sum of intrinsic, reflex and additional torques) for two validation trials, whose average-VAF was 95%. This excellent agreement between measured and predicted torques indicates that the TV model estimates

accurately captured the system dynamics. **Figures 9C–E** also show the predicted TQc<sup>I</sup> (tk ), TQcR(t<sup>k</sup> ), and TQc1(t<sup>k</sup> ) as a function of time. The intrinsic torque accounted for most of the variance of the measured data for this experiment; however, both the reflex torque and TQ1(t<sup>k</sup> ) were non-zero for all the cycle.

#### **4.2.3.1. TV intrinsic dynamic stiffness**

The parameters of the TV-IRF describing the intrinsic dynamic stiffness underwent large, fast changes throughout the cycle. **Figure 10A** presents the variation in the intrinsic static stiffness along with the 95% confidence interval, computed by a bootstrap analysis with 100 repetitions (Press, 2007). The intrinsic elasticity increased three fold (from 35 Nm/rad to 100 Nm/rad) in the first half of the cycle, it then decreased sharply and stayed nearly constant during the remainder of the cycle.

**Figure 10C** shows the static intrinsic stiffness as a function of ankle position, demonstrating that: (i) static intrinsic stiffness is larger in plantarflexion than dorsiflexion; (ii) the relation between joint position and intrinsic elasticity is nonlinear and is influenced by the immediate history of the movement, as different values of the static stiffness were observed for the same joint position during different parts of the cycle.

Furthermore, the upper pathway of **Figure 11** shows the time-frequency response of the TV-IRF as a function of the cycle with the purple line indicating the static stiffness. The intrinsic dynamic stiffness showed a highpass behavior, typically observed during stationary experiments (Kearney et al., 1999), and underwent large, fast changes in the low and mid-frequency components, related to the joint visco-elastic properties, throughout the gait cycle. The high-frequency components, related to the joint inertial properties, did not change much throughout the cycle.

Attempts to fit a second order model to the estimated TV-IRF provided inaccurate parametric models unable to properly describe the intrinsic joint dynamics. This is consistent with recent evidence that joint mechanical properties are more complex than second order (Sobhani Tehrani et al., 2017).

#### **4.2.3.2. TV reflex dynamic stiffness**

The parameters of the polynomial nonlinearity representing the reflex, static-nonlinearity underwent large, fast changes throughout the gait cycle. **Figure 10B** shows the variation in the reflex gain, computed as the slope of the static nonlinearity, along with the 95% confidence interval. The reflex gain increased six fold (from -2 Nm/rad/s to -12 Nm/rad/s) during the first half of the cycle, and then decreased rapidly to an almost constant value for the remainder of the cycle.

**Figure 10D** shows the reflex gain as a function of ankle position. This plot indicates that: (i) the reflex gain is larger in plantarflexion than dorsiflexion; (ii) the relation between joint position and reflex gain is nonlinear and is influenced by the immediate history of the movement.

The parameters of the second-order, linear system representing the reflex, linear dynamics did not vary much. The lower pathway of **Figure 11** summaries the TV reflex behavior. It shows the TV, static-nonlinearity as a function of cycle and the frequency response of the linear dynamics. The estimated static-nonlinearity resembles a half-wave rectifier; whose gain underwent large, fast changes throughout the cycle. The linear dynamics are low-pass in nature and did not vary throughout the cycle. The shape of the static nonlinearity and the cut-off frequency of the linear dynamic element are similar to what has been observed in stationary experiments (Kearney et al., 1999).

## 5. DISCUSSION AND CONCLUSIONS

This paper presents a new model parameterization and identification algorithm for the accurate estimation of the intrinsic and stretch reflex components of dynamic joint stiffness during movement. The algorithm combines ensemble and deterministic approaches to estimate TV model parameters from position and torque records. Simulations demonstrated that the new algorithm successfully decomposed the dynamic joint stiffness into its intrinsic and reflex components, and accurately tracked the fast, large changes in the parameters of each pathway using only 40 cycles in the presence of complex, experimental noise. This represents a much-needed improvement over ensemble only algorithms, which were not able to accurately track the changes in intrinsic and reflex dynamics even after using 400 cycles. Furthermore, the practical application of the method was successfully demonstrated by using it to track the changes in ankle stiffness in a human subject in an experiment that involved an imposed walking movement with constant muscle activation. The excellent agreement between the predicted and experimental torques demonstrated that the new methodology accurately describes the modulation of dynamic ankle stiffness during the movement.

#### 5.1. Methodological Issues and Limitations

Methods that estimate TV, dynamic joint stiffness make three underlying assumptions: (i) The small perturbations applied to the joint do not change much the operating point (Kearney et al., 1997); (ii) the mechanical response of the joint to small perturbations and to large changes in the operating point are linearly superimposed (Gottlieb and Agarwal, 1978); and (iii) changes in the system dynamics with joint position and torque can be described by a set of local models at each point in time (Bennett et al., 1992). The excellent agreement between the predicted and measured torques suggests that these assumptions hold for the slow ankle trajectory used in our experiments, which resembles slow walking. However, it remains to be determined if these assumptions are valid during faster joint movements.

Our methodology leverages these assumptions and introduces a novel parameterization of the parallel-cascade model where the time-course of the local models parameters are approximated by a linear combination of basis functions. These approximations transform the TV model into a set of TI models at the cost of increasing the number of free parameters. This raises a number of issues with the new algorithm:

First, the number of free-parameters increased by the re-parameterization procedure; therefore, the identification algorithm requires large data sets for accurate parameter estimation. This limitation was addressed here by combining the basis function expansion with an ensemble identification approach, which uses multiple, input-output trials with the same TV behavior. However, compared with ensemble-only identification methods, our algorithm requires a lot less repetitions, which translates into much shorter experiments

making it easier to acquire enough trials with the same TV behavior.

Second, the type and number of basis functions used to parametrize the TV coefficients must be known a priori; the quality of the parameter estimates will depend on selecting a set of basis functions capable of efficiently describing the TV parameter changes. This study uses B-splines and Chebyshev polynomials, both of these basis functions are well suited to describe smooth parameter changes. B-splines are useful when the changes are rapid, polynomial basis are adequate to approximate lowfrequency trends (Zou et al., 2003; He et al., 2013).

Third, this method was designed to work with data measured in open-loop. This is the case in experiments, such as ours, where a very stiff actuator, acting as a position servo, imposes a desired joint trajectory so that any torques produced by the joint in response to the perturbation do not result in position changes. That is, the relation between joint position and torque is open-loop. In contrast, during most natural movements, the joint interacts with a compliant load so that torques generated in response to position changes will in turn modify the joint position, resulting in closed-loop measurement of joint position and torque. Most methods for identification of dynamic join stiffness have been designed to work with open-loop data, and using these methods with data measured in closed-loop will lead to biased parameter estimates (Kearney and Hunter, 1990).

The method presented is an open-loop method; however, it can be reformulated to work with data measured in closed-loop. This would require adopting a new model of dynamic joint stiffness, with the feedforward and feedback pathways comprising the intrinsic and reflex components respectively (Van der Helm et al., 2002). The algorithm described here for identification of intrinsic dynamics cannot be used with closed-loop data as it will provide biased results. However, an instrumental variable algorithm for parameter identification can be used directly to estimate the intrinsic component from closed-loop data as described in Guarin and Kearney (2016). The method presented here for estimation of reflex dynamic stiffness uses instrumental variables and so can be applied directly to estimate the nonlinear, Hammerstein system representing the reflex dynamics from data measured in closed-loop (Young, 2011).

Moreover, our implementation of the identification algorithm assumes that the time-varying behavior is periodic, so that the initial conditions of each trial in the ensemble will be the same facilitating their estimation. However, the algorithm could be modified to work with non-periodic data; this would require estimating the initial conditions of each trial in the ensemble as part of the identification problem as done in Jalaleddini et al. (2017).

Finally, the algorithm relies on knowledge of the reflex response delay to accurately separate the intrinsic and reflex components from the measured position and torque data. It assumes that the delay remains constant throughout the cycle. We measured the reflex delay from joint velocity and soleus EMG signals, and found that it remained constant across the cycle.

## 5.2. Simulation study

System identification methods are often validated using idealistic input and noise sequences. However, the performance of these algorithms often degrades when applied to experimental data, where inputs are non-ideal and the noise is neither zeromean, nor white. Our simulation was intended to mimic real experiments; model parameters were based on those reported in the literature; inputs signals had limited bandwidth; and the noise was extracted from experimental observations. Consequently, we believe that our simulation results are more relevant to experimental conditions.

As **Figures 6**,**7** show, the simulated, intrinsic and reflex stiffness model parameters were accurately estimated by the identification algorithm. The large variability in the polynomial nonlinearity at large velocities is likely related to the amplitude distribution of the velocity signal, which despite having velocities distributed over the entire sets of values, is highly concentrated around zero (Jalaleddini and Kearney, 2013). Finally, the reflex natural frequency was accurately estimated, and the reflex damping was slightly underestimated. However, this did not affect the prediction ability of the estimated models, indicating that the model output is not very sensitive these small differences in the damping.

## 5.3. Experimental Study

We also applied the new method to actual experimental data to estimate the intrinsic and reflex dynamic ankle stiffness during an movement. Results showed that the model structure predicted the output torque to novel perturbation sequences, indicating that the estimated model successfully captured the TV, nonlinear dynamics.

**Figure 10A** shows that the static stiffness changed dramatically during the imposed movement, it increased substantially during the first part of the cycle (from around 35 Nm/rad to 100 Nm/rad) and then sharply decreased (to 20 Nm/rad) in just 200 ms, it maintained a nearly constant value for the remainder of the cycle. **Figure 10C** demonstrates that the ankle static stiffness can take different values for the same ankle angle depending on the immediate history of the movement. This demonstrates a true TV behavior in the joint neuromuscular properties, and not just a staticnonlinear dependency on joint position, as has been previously assumed (Sobhani Tehrani et al., 2013; Jalaleddini et al., 2015).

**Figure 10B** shows there were also large TV changes in the reflex gain, it increased (from around −2 Nm/rad/s to −12 Nm/rad/s) during the first 800 ms of the cycle, then rapidly decreased to almost zero over the next 400 ms, and then remained relatively constant for the remainder of the cycle. Moreover, **Figure 10D** demonstrated that there is a significant historydependent behavior, with the reflex gain showing values with a difference of up to 300% for the same value of joint position. The other components of the reflex pathway did change during the movement.

Furthermore, the reflex gain attained its maximum value at least 100 ms before the intrinsic static stiffness, this is consistent with the idea that the tonic stretch reflex might mediate the changes in muscle activation, leading to an increased intrinsic static stiffness Feldman and Levin (2009). In addition, the history-dependent behavior was observed in both the intrinsic static stiffness and reflex gain; however, as **Figure 10D** demonstrates, this behavior was much more significant for the reflex than the intrinsic component. This might be explained by the fact that reflex dynamic stiffness is generated only by the active muscle response to stretch activation whereas intrinsic dynamic stiffness is generated by both active and passive components.

We conclude that the new algorithm will be a useful tool in the study of dynamic joint stiffness during TV conditions and

#### REFERENCES


that it will help further the understanding of the modulation of this system during function.

## AUTHOR CONTRIBUTIONS

All authors listed, have made substantial, direct and intellectual contribution to the work, and approved it for publication.

#### FUNDING

This work was supported by the Canadian Institutes of Health Research (CIHR), and the Fonds Québécois de la recherche sur la nature et les technologies (FRQNT).

in Engineering in Medicine and Biology Society (EMBC), 2015 37th Annual International Conference of the IEEE (Milan: IEEE), 6740–6743.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Guarín and Kearney. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Optimal Control Based Stiffness Identification of an Ankle-Foot Orthosis Using a Predictive Walking Model

Manish Sreenivasa<sup>1</sup> \* † , Matthew Millard1 †, Martin Felis <sup>1</sup> , Katja Mombaur <sup>1</sup> and Sebastian I. Wolf <sup>2</sup>

<sup>1</sup> Optimization in Robotics and Biomechanics, Interdisciplinary Center for Scientific Computing, Heidelberg University, Heidelberg, Germany, <sup>2</sup> Clinic for Orthopedics and Trauma Surgery, Heidelberg University Hospital, Heidelberg, Germany

Predicting the movements, ground reaction forces and neuromuscular activity during gait can be a valuable asset to the clinical rehabilitation community, both to understand pathology, as well as to plan effective intervention. In this work we use an optimal control method to generate predictive simulations of pathological gait in the sagittal plane. We construct a patient-specific model corresponding to a 7-year old child with gait abnormalities and identify the optimal spring characteristics of an ankle-foot orthosis that minimizes muscle effort. Our simulations include the computation of foot-ground reaction forces, as well as the neuromuscular dynamics using computationally efficient muscle torque generators and excitation-activation equations. The optimal control problem (OCP) is solved with a direct multiple shooting method. The solution of this problem is physically consistent synthetic neural excitation commands, muscle activations and whole body motion. Our simulations produced similar changes to the gait characteristics as those recorded on the patient. The orthosis-equipped model was able to walk faster with more extended knees. Notably, our approach can be easily tuned to simulate weakened muscles, produces physiologically realistic ground reaction forces and smooth muscle activations and torques, and can be implemented on a standard workstation to produce results within a few hours. These results are an important contribution toward bridging the gap between research methods in computational neuromechanics and day-to-day clinical rehabilitation.

Keywords: pathological gait, neuromechanics, movement prediction, model-based optimization, parameter identification

## 1. INTRODUCTION

The clinical treatment of neuromuscular gait abnormality is a complex process that demands significant investment of time and effort from the patient (and caregivers), surgeons and orthotists. Often there may be multiple suitable treatment regimes (surgery, orthotics, rehabilitation exercise, etc.) without a clear indication of an optimal choice. The use of computational methods can assist in these decisions in two ways. First, by estimating internal physiological states that cannot be directly measured to help understand the pathology. Second, by predicting the change in such states under manipulation of virtual patient models to help understand the effects of the possible interventions.

#### Edited by:

Florentin Wörgötter, University of Göttingen, Germany

#### Reviewed by:

Poramate Manoonpong, University of Southern Denmark Odense, Denmark Martin Grimmer, ETH Zürich, Switzerland Oskar Von Stryk, Technische Universität Darmstadt, Germany

#### \*Correspondence:

Manish Sreenivasa manish.sreenivasa @iwr.uni-heidelberg.de

† These authors have contributed equally to this work.

> Received: 25 October 2016 Accepted: 28 March 2017 Published: 13 April 2017

#### Citation:

Sreenivasa M, Millard M, Felis M, Mombaur K and Wolf SI (2017) Optimal Control Based Stiffness Identification of an Ankle-Foot Orthosis Using a Predictive Walking Model. Front. Comput. Neurosci. 11:23. doi: 10.3389/fncom.2017.00023 There is a growing number of studies that apply the former, so called inverse methods, to healthy and pathological movements, e.g., (Nakamura et al., 2005; Damsgaard et al., 2006; Delp et al., 2007; Erdemir et al., 2007; Sreenivasa et al., 2015; Choi et al., 2016). By matching recorded kinematics and ground reaction forces, one may solve for muscle activations under various optimization criteria (Jonkers et al., 2003; Thelen et al., 2003; Erdemir et al., 2007; Groote et al., 2016). Another approach is to use the concept of modularity in neural and muscle recruitment to generate a low dimensional manifold of control signals. Sartori et al. (2013) used this approach to generate EMG signals and joint moments for a lower body neuromuscular model. There are far fewer examples that explore the possibility of predicting the kinematics and dynamics of the body during gait. Here we distinguish between methods that can predict muscle forces given body movements, and those that can predict both muscle forces and body movements. This work focuses on the latter by applying optimal control based methods to predict movements, ground reaction forces and neuromuscular dynamics during walking with and without an orthosis. The goal here is to support an important clinical routine—fitting of an orthosis to a patient with the use of computational methods and patient-specific models.

The ideal combination of model/method would be one that is computationally efficient, includes neuromuscular dynamics, produces realistic ground reaction forces, can be tuned to an individual (healthy or pathological) quickly and accurately, and can predict movements. Each of these requirements is challenging, however, methodological and technological advances have made some of these possible. Anderson and Pandy (2001) famously used 10,000 h on a Cray super-computer to solve for a metabolically efficient gait for a lower-body neuromuscular model. More recently Wang et al. (2012) and Dorn et al. (2015) predicted gait patterns for their models with around 1,000 CPUhours of processing. This level of computational infrastructure and the long time to a solution is not feasible for routine clinical work. In contrast, the works of Schultz and Mombaur (2010), Ren et al. (2007), Felis et al. (2013), Felis and Mombaur (2016), and Srinivasan et al. (2008, 2009) produce results using desktop computers in less than an hour. While the faster solution time is impressive, these works do not include a representation of the muscles, which is necessary to address most clinical questions.

Ackermann and van den Bogert (2010) and Dorn et al. (2015) included muscles and activation dynamics, however, their results were accompanied by ground reaction force peaks that were twice as large as would be expected from healthy human walking. Using an alternative reflexfeedback approach, Geyer and Herr (2010) produced a muscle and reflex-driven simulation of walking that produced ground reaction force profiles that had a comparable form and magnitude to healthy human walking. While these results are impressive, it would be challenging to estimate individualized reflex parameters, especially in a clinical setting.

An alternative to the model-based approaches presented so far is the use of methods from machine learning to adaptively adjust assitive devices to the user. Autonomous learning methods have found application in clinical rehabilitation related to functional electrical stimulation (see e.g., Abbas and Chizeck, 1995; Chang et al., 1997; Ferrante et al., 2004). However, these methods typically require pre-existing datasets and/or a large number of training trials. This makes their extension to the prediction of whole body neuromechanics challenging, as patient data may be sparse or not available at all.

In addition to these computational aspects, a major challenge that must be addressed is the validation of the models and the simulation results. This is a multi-faceted issue that needs to be dealt with at both the technical and clinical fronts. For example, for neuromuscular models a common hurdle is that internal neurological states cannot be measured in vivo, and surface EMG can only roughly approximate muscle function (Farina et al., 2014). In addition, a prospective clinical trial is a major undertaking that needs a close collaboration between research and clinical teams. As an initial step, studies such as the one presented here can at the very least compare their results to those measures that are relatively easy to record (e.g., joint kinematics, ground reaction forces, surface EMG). While this is not a full validation, a model that can match these observations can at least assure the clinician of exhibiting behavior that is physiologically realistic.

In the following we detail a patient-specific model and formulate an optimal control problem (OCP) to identify the optimal individualized stiffness of an ankle foot orthosis that minimizes muscle effort while walking. It is important to note that the identification of the stiffness parameters occurs in advance of the patient walking with the orthosis. We do not identify the stiffness parameters from experimental data, but rather predict what the parameters should be for that patient. In general, an OCP defines a minimization problem where an objective function is minimized while abiding the dynamics describing a physical system (in our case the human body + orthosis dynamics). Such methods have been used successfully for robot and human motion generation in the past (Bobrow et al., 1985; von Stryk and Schlemmer, 1994; Schultz and Mombaur, 2010), and to a limited extent for the design of humanassistive devices (Koch and Mombaur, 2015; Mombaur, 2016). In the current work, we strike a balance between model complexity and computational efficiency by modeling the muscles as lumped torque generators rather than anatomically equivalent linetype actuators. The solutions combine physically consistent neuromuscular dynamics and ground-contact dynamics, and can be achieved in a matter of hours on a standard desktop computer. We implement several OCPs that mimic the patient condition as he walked barefoot as well as with an orthosis. Note that in these OCPs we predict movements, joint torques and ground-reaction forces. For the orthosis OCP, we evaluate two cost functions, one that only minimizes muscle effort, and another that minimizes muscle effort while favoring a higher walking speed. In addition we also present a dynamic fit of the model to the recorded gait kinematics. Our simulation results are compared to experimental recordings from a 7-year old patient with neuromuscular deficits.

## 2. METHODS

## 2.1. Patient Data

Gait data of a 7-year old male (weight 24.7 kg, height 1.25 m) are retrospectively used in this study. The patient presented with multiple bony deformities of neuromuscular origins, which were corrected in a single event multilevel surgery 1.5 years prior to the recording of the gait data. At the time of recordings he presented with a mild crouch, slow walking speed and unstable gait. Recordings were made of the patient walking on level ground with bare feet and with bilateral ankle-foot orthosis. The orthosis stiffness (see Section 2.2.3 for details) was tuned manually by an orthopedic professional overseeing the recordings. Positions of 35 reflective markers attached to the patient's limbs and torso were recorded at 120 Hz during level gait using a 10-camera Vicon system (Vicon, UK). Simultaneous ground reaction forces were recorded at 1080 Hz using Kistler force plates (Kistler GmbH, Germany).

In total 13 barefoot left and right steps and 12 orthosis left and right steps were recorded that contained gait kinematics suitable for further processing. From this set, 3 barefoot left steps and 2 barefoot right steps, as well as 5 orthosis left steps and 4 orthosis right steps, had suitable recorded ground reactions forces. The reduced number of trials with valid ground reaction forces highlight the experimental difficulties associated with getting an under-age patient with neuromuscular deficits to step cleanly on the successive force-plates. The gait recordings were part of a standard clinical routine. Written informed consent was obtained from the parents and the subject. The recordings were conducted according to the guidelines of the Declaration of Helsinki 2013 and approved by the ethics committee of the Medical Faculty Heidelberg of Heidelberg University.

## 2.2. Model Formulation

We model the human body as an articulated multi-body system with 8 segments, each with one rotational Degree of Freedom (DoF) in the sagittal plane. The pelvis is modeled as a floating base with two additional translational DoFs in the X and Z directions (**Figure 1**). Segment lengths were approximated from motion capture data, and segment mass and inertia were calculated based on anatomical regression equations for children as per (Jensen, 1986).

FIGURE 1 | Torque muscles fitted to the patient were used to actuate a sagittal plane rigid-body model. (A–C) Show the normalized active-torque-angle curve, f <sup>A</sup>(θ), and the torque-angular-velocity curve, f <sup>V</sup>(ω), of the torque muscles. (D) Illustrates the degrees of freedom of the rigid-body model along with the modeled ankle-foot orthosis as an adjustable-stiffness torsion spring.

#### 2.2.1. Patient-Specific Muscle Torque Generator

The rotational DoFs at the hips, knees, ankles and the torso were each actuated by a pair of agonist-antagonist Muscle Torque Generators (MTG), which represent the combined torques being generated by muscle forces in that direction (**Figure 1**). The active tension developed by a muscle varies non-linearly with the length and contraction velocity of the muscle, while the passive tension varies non-linearly with its length (Zajac, 1988; Millard et al., 2013) (**Figure 1**). In this study we only model the active components of muscle torque generation. The active torque developed by a MTG varies non-linearly with the angle θ of the muscle and is represented by the normalized active-torqueangle curve **f** <sup>A</sup>(θ) which peaks at a torque of τ M o at an angle of θo. During non-isometric contractions the torque developed by the muscle varies non-linearly with the angular velocity ω of the muscle, which is represented by the normalized torque-angularvelocity curve **f** <sup>V</sup>(ω). Muscle torque τ <sup>M</sup> is computed using these characteristic curves as follows:

$$\mathbf{r}^{\mathcal{M}} = \mathbf{r}\_{\boldsymbol{\alpha}}^{\mathcal{M}}(\boldsymbol{a}\mathbf{f}^{\mathcal{A}}(\boldsymbol{\theta})\mathbf{f}^{\mathcal{V}}(\boldsymbol{\alpha})) \tag{1}$$

where a is the muscle activation. The active-torque-angle and torque-angular-velocity curves are modeled using C<sup>2</sup> continuous Bézier curves (**Figure 1**) fitted to the experimentally derived torque curves of (Anderson et al., 2007). Anderson et al.'s parameterized curves are not used directly because they are not all C<sup>2</sup> continuous, which is required by the OCP solver.

Patient-specific maximum torques in extension for the hip, knee and ankle are estimated under the assumption that during the recorded trials the patient was walking at 90% of his maximum capability (i.e., maximum muscle activations were 0.9). This assumption is motivated by the clinical assessment of this patient's musculature, the pronounced crouch, and slow walking speed observed in the recorded barefoot gait. First, we use inverse dynamics analysis to compute the maximum extension torques generated during the recorded trials. Using a = 0.9 and the θ, ω where this maximum occurred, the corresponding maximum muscle torque in extension is found by solving Equation (1) for τ M o . Maximum flexion torques are then computed based on the extension-flexion torque ratios recorded in the study by Anderson et al. (2007). **Table 1** lists these values for an age-matched and weight-matched healthy child, as well as for the patient considered in this study. Torso strengths are assumed to be the average of the right and left hip strengths. Note that the MTG models developed here do not take into account the active and passive-dynamic coupling effects of muscles that span multiple joints.

#### 2.2.2. Excitation-Activation Dynamics

The physiological activation of muscle is an electrochemical process at the motor unit end plates that converts incoming motor unit action potentials to changes in ionconcentration, and subsequent contraction in muscle fibers. Lumped models provide a simplified representation of this process by relating the overall muscle activation a, to the rate of change of activation a˙ and neural excitation e. Here, we use the formulation by Thelen et al.



(2003):

$$\dot{a} = \begin{cases} (e-a)\left(\frac{e}{\tau\_{\rm A}} + \frac{1-e}{\tau\_{\rm D}}\right) & \text{if } e \ge a\\ \frac{e-a}{\tau\_{\rm D}} & \text{otherwise} \end{cases} \tag{2}$$

where, τ<sup>A</sup> = 0.011, τ<sup>D</sup> = 0.068 denote the activation and deactivation time constants as per (Winters and Stark, 1988).

#### 2.2.3. Parametrized Orthosis Model

The orthosis worn by the patient consisted of custom-built carbon fiber shank and foot segments joined together by an adjustable-stiffness spring-loaded rotational joint at the ankle. The adjustable stiffness joint was constructed using the Neuroswing Joint (Fior and Gentz, Germany) which needs to be tuned to each patient. The foot segment consist of foot-plate fitted to the patient's foot size and inserted into a standard shoe. The masses of the shank and foot segments are estimated to be 0.34 and 0.69 kg, respectively. Note that the foot segment mass referred to here includes the mass of the shoe. In the following, we refer to the gait with the orthosis+shoe combination as orthosis gait. These masses are added to the shank and foot segments of the patient model for the simulations of orthotic gait.

The stiffness of the orthosis is modeled as torques generated at the ankle as a function of the ankle angle. The behavior is divided into 5 stages for the extension-flexion range of motion (**Figure 2**). The parameter θ<sup>0</sup> defines the offset between the neutral pose of the ankle and the orthosis in a torque-free angular position. In a small angle window θ<sup>W</sup> about this neutral pose, a small preload defined by τ<sup>0</sup> acts on the joint. As the shank rotates with respect to the foot, the torques are produced by the joint springs (spring stiffness KD, and, KP). Upon hitting the adjustable hard stops (θDH, and, θPH) the shank may rotate further by flexing the carbon fiber material. This relatively stiffer material results in large torques defined by the parameters, KDH and KPH. The positional parameters listed in **Table 2** were measured by the medical professionals during the clinical process. The stiffness of the orthosis-shoe combination is estimated using inverse

resulting from the stiffness of the orthosis springs and frame are plotted as a function of ankle angle. The shape of this curve changes as a function of the free parameters KD and KP.

#### TABLE 2 | Orthosis parameters.


<sup>a</sup>K<sup>D</sup> and K<sup>P</sup> are free parameters of the optimal control problem (OCP) that are to be determined. Values indicated here are the initial guess provided to the OCP. The other parameter values are fixed in the OCP.

dynamics analysis of recorded orthosis gait (further details in Section 2.4). While the positional parameters and masses can be measured with a high degree of certainty, the stiffness values of the combined orthosis-shoe unit are tougher to measure and is not part of the clinical routine. In the current approach, we place the initial guess for the stiffness values well below the estimate calculated from the torque-angle characteristics.

Note that during the clinical fitting/tuning process, the orthotist would adjust the spring stiffness denoted here by the parameters KD, and, KP. While it is possible to adjust the other orthosis characteristics, these springs are the easiest to access and one can quickly test their effects on gait during a fitting procedure. Consequently, we make these two parameters KD, and, KP, free parameters of the OCP that are to be determined. The other parameters in **Table 2** are fixed, however, future extensions of this approach could include a more extensive parameter set to be identified. Finally, the overall orthosis torqueangle profile is approximated using C<sup>2</sup> continuous Bézier curves that are generated on-the-fly as a function of the changing parameters while running the OCP.

#### 2.3. Gait as an Optimal Control Problem

Gait is formulated as a multi-phase OCP, with each phase defined by the attachment and breaking off of sets of contact constraints between the feet and the ground. Due to left-right asymmetry in the patient's gait we model a consecutive left and right stride with n<sup>p</sup> = 8 phases as follows (see insets in **Figures 4A,B**): Right Flat—Left Toe Off, Right Toe On—Right Heel Off, Right Toe On—Left Heel On, Right Toe On—Left Flat, Left Flat—Right Toe Off, Left Toe On—Left Heel Off, Left Toe On—Right Heel On, Left Toe Off—Right Flat. Here, "Flat" indicates that both heel and toe contacts are active. In addition to the position constraints at foot contacts, contact velocities are also constrained to be zero at the start of phases, to ensure continuity in velocities and ground forces. Forces at the load bearing points of the feet are constrained to ensure strictly positive vertical ground reaction forces during the step. Forces in the anterior-posterior direction are constrained to lie within the limiting friction, assuming a coefficient of friction of 0.8 (Chang and Matz, 2001). Forward dynamics computations for the multi-body system subject to the stepping constraint sets are computed using the method described by Kokkevis (2004), implemented in the open-source dynamics library RBDL<sup>1</sup> by Felis (2017). The OCP then has the general form:

$$\min\_{\underline{\chi}(\cdot), \underline{\mathfrak{u}}(\cdot), \underline{\mathfrak{p}}\underline{\underline{\nu}}} \qquad \sum\_{0}^{n\_{\underline{p}}} \left( \int\_{\upsilon\_{\underline{\jmath}} - 1}^{\upsilon\_{\underline{\jmath}}} \phi\_{\underline{\jmath}}(\underline{\chi}(t), \underline{\underline{\mu}}(t), \underline{\underline{\rho}}) dt \right) \tag{3}$$

$$\begin{aligned} \text{s.t.} \dot{\underline{x}}(t) &= f\_{\dot{\underline{j}}}(t, \underline{x}(t), \underline{\underline{u}}(t), \underline{\underline{p}}) \quad \text{for} \quad t \in [\upsilon\_{\underline{j}} - 1, \upsilon\_{\underline{j}}],\\ &\qquad \dot{\underline{j}} = 1, \ldots, n\_{\underline{p}}, \; \upsilon\_{0} = 0, \upsilon\_{\underline{\eta}\_{\underline{p}}} = T \quad \text{(4)}\\ &\qquad \circ \quad \circ \quad \circ \quad \circ \quad \circ \end{aligned} \tag{4}$$

$$0 = \operatorname{r}\_{eq}(\underline{\mathbf{x}}(0), \dots, \underline{\mathbf{x}}(T), \underline{\underline{\rho}}) \tag{5}$$

$$0 \le \operatorname{r}\_{incq}(\underline{\mathbf{x}}(0), \dots, \underline{\underline{\mathbf{x}}}(T), \underline{\underline{\rho}}) \tag{6}$$

$$0 \le \underset{\delta}{\underline{\delta}}(t, \underline{\underline{x}}(t), \underline{\underline{u}}(t), \underline{\underline{p}}) \quad \overset{\rightharpoonup}{\text{ for } t \in \left[\underline{\nu}\_{j} - 1, \underline{\nu}\_{j}\right]}\tag{7}$$

where, Equation (3) describes a general objective function to be minimized. Equation (4) is a place-holder that denotes the dynamics of the multi-body system. Note that the actual neuromuscular and multi-body dynamics are described by differential algebraic equations (detailed formulation available in Felis et al., 2015; Felis and Mombaur, 2016; Mombaur, 2016). x(t) denotes a vector of state variables (generalized coordinates q, generalized velocities q˙, and muscle activations a). u(t) is a vector of control variables (neural excitations e). p denotes a vector of free model parameters (if any), and, ν is a vector of variable phase switching times with T = tn<sup>p</sup> = overall time for

<sup>1</sup>https://rbdl.bitbucket.io

the motion. Equation (5) denotes coupled and decoupled equality constraints (e.g., switching foot contacts at phase changes), and Equation (6) the inequality constraints (e.g., maintain positive ground reaction force during stepping). Equation (7) denotes all continuous inequality constraints (e.g., bounds of the state variables). Controls u(t) are subject to constraints formulated in Equation (5) to ensure continuity at phase changes. This is done to ensure 2nd order continuity in muscle activations.

To solve the OCP we use a direct multiple-shooting method (Bock and Pitt, 1984) implemented in the software package MUSCOD-II (Leineweber et al., 2003). The direct multiple-shooting approach transforms the infinite dimensional OCP, Equations (4–7), into a finite dimensional non-linear programming problem by first discretizing the continuous controls u(t) on a grid and then solving the resulting boundary value problem using a multiple-shooting method. Note that with this method the system dynamics are also satisfied between the multiple shooting intervals, leading to physically consistent results throughout the simulated motion. The multi-phase problem described above is discretized into 64 shooting nodes. The controls u(t) are modeled as piecewise linear functions between discretization points. The works by Felis et al. (2015); Felis and Mombaur (2016) and Mombaur (2016) provide further detail on the constraint formulation, the solution of the multi-body mechanics, and numerical treatment of the OCP. The models and constraints formulation are available as supplementary software code to this article. In our current study we implement four OCPs:


The LS-Barefoot OCP is used to show that our model is capable of tracking the patient's gait in a dynamically consistent manner. Note that we only apply this fitting-type objective function to the recorded barefoot gait, as in a real-world application the gait with orthosis would not be available in advance. The MAPD-Barefoot OCP is used to test how close the chosen cost function can reproduce recorded barefoot gait of the patient. The MAPD-Orthosis OCP is used to predict the patient gait with an orthosis, and simultaneously identify the orthosis spring stiffness parameters. In initial trials we noticed that the predicted walking speed of the MAPD-Orthosis OCP was slower than that of the patient. To further investigate whether our model could be made to walk as fast as the patient, we implemented the OCP MAPD-WS-Orthosis, that contains an additional objective function term favoring a higher walking speed. Note that the OCPs MAPD-Barefoot, MAPD-Orthosis and MAPD-WS-Orthosis are purely synthetic results and no experimental data is used to compute the solutions.

#### 2.3.1. Dynamic Least-Squares Fit to Recorded Gait

We formulate a fitting-type objective function for the LS-Barefoot OCP that provides a dynamically consistent gait as close as possible to the recorded patient joint kinematics. The objective function is formulated as:

$$\min\_{\underline{\mathbf{u}}(\cdot),\underline{\mathbf{u}}(\cdot)} \quad \sum\_{j=1}^{n\_{\mathcal{P}}} \begin{bmatrix} \sum\_{m=1}^{t\_{M,j}} \{\underline{q}(t\_{jm}) - \underline{q}^{M}(t\_{jm})\}^T \underline{\mathbf{W}}(\underline{q}(t\_{jm})) \\ -\underline{q}^{\underline{M}}(t\_{jm})\} + \delta \int\_{\nu\_{j-1}}^{\nu\_{j}} \underline{\mathbf{u}}(t) \cdot \underline{\mathbf{u}}(t) dt \end{bmatrix} \tag{8}$$

Here, the phase times ν are fixed to those obtained from the recorded gait. Note that the generalized coordinates q <sup>M</sup> are computed using inverse kinematics at discrete measurement points. W is a diagonal scaling matrix that may be used to give preference to a closer fit to a subset of the generalized coordinates. Here, we use an identity matrix which provides an overall good fit to all the coordinates. The second term, δ R νj νj−<sup>1</sup> u(t) · u(t), introduces a small cost that regularizes the control inputs, i.e., it smoothens the control input (neural excitation) and avoids that the solution follows noise in the experimentally recorded data. δ was set to 1e − 4 for our computations. For this regularization term all controls are weighted equally relative to each other.

#### 2.3.2. Gait Prediction with MAPD-Type Objective Functions

We formulate two objective functions for predicting gait: the first minimizes total muscle activations squared per distance walked, and the second contains an additional term that favors a higher walking speed. The first objective function is formulated as:

$$\min\_{\underline{\pi}(\cdot), \underline{\pi}(\cdot), \underline{\pi}, \underline{\eta}} \quad \frac{\sum\_{1}^{n\_{\underline{p}}} \int\_{\upsilon\_{j-1}}^{\upsilon\_{j}} \underline{a}(t) \cdot \underline{a}(t) dt}{r(T)} \tag{9}$$

Note that dividing by the total distance traveled, r(T), provides the impetus for moving forward, as without this term the model has no reason to move. Objective functions similar to the one above are commonly used in literature (Thelen et al., 2003; Damsgaard et al., 2006; Ackermann and van den Bogert, 2010) and are associated with the minimization of muscle effort (Ackermann and van den Bogert, 2010). We introduce additional periodicity constraints on all the state variables and the controls, such that the initial states at the start of the first phase matched the final states at the end of the last phase.

The second objective function includes a term favoring a higher walking speed and is formulated as:

$$\min\_{\underline{r}(\cdot), \underline{\pi}(\cdot), \underline{\pi}, \underline{\eta}} \quad \frac{\sum\_{1}^{n\_p} \int\_{\upsilon\_{j-1}}^{\upsilon\_j} \underline{a}(t) \cdot \underline{a}(t) dt}{r(T)} - \lambda \frac{r(T)}{T} \tag{10}$$

where, λ is a scaling term. The objective function (Equation 9) is used in the OCPs MAPD-Barefoot and MAPD-Orthosis. The objective function (Equation 10) is used in the OCP MAPD-WS-Orthosis. For the OCPs MAPD-Orthosis and MAPD-WS-Orthosis, there are 4 free parameters to be determined during the optimization. These corresponded to the left and right pairs of orthosis spring stiffness parameters (KD,KP). The orthosis dynamics in these OCPs are simulated using the values listed in **Table 2**.

#### 2.4. Evaluation Procedure

We evaluate the model and the predicted results in the following ways:


To separate these differences, we compare the net torque-angle profile of the MAPD-Orthosis results to the corresponding experimental data. If the net ankle torqueangle profiles are similar it is likely that the remaining differences we observe are happening because the patient is no longer walking in a manner that is consistent with our chosen cost function. It is necessary to use the net ankle torque (the sum of the torque contribution of the ankle MTGs and the orthosis) in this comparison because the kinematics and kinetics of the patient's ankle were not recorded separately from the orthosis.

5. We perturb the free orthosis parameters by −5% in the vicinity of the identified optimal values to compute how the cost function value, knee flexion angle (and thus severity of crouch), step lengths and walking speed vary with the stiffness of the orthosis.

## 3. RESULTS

The residual forces from the inverse dynamics analysis for barefoot and orthosis gait are under 3.3 N in the anteriorposterior and vertical directions while the sagittal plane moments are under 0.12 Nm (**Table 3**). The kinematics of the LS-Barefoot



A-P denotes the forces in the anterior-posterior direction, Vert. denotes the forces in the vertical direction, Mom. denotes the moments about the free flier joint.

solution closely matches the patient's barefoot gait kinematics (dashed lines in **Figures 3D–F**), with RMS differences of 0.83◦ at the pelvis, 1.52◦ at the hips, 2.43◦ at the knees, and 2.64◦ at the ankles. The ground reaction forces of the LS-Barefoot solution deviate from the patient's recorded ground reaction forces with RMS differences of 68.24 N in the vertical direction. Note that all RMS differences are computed with respect to the average corresponding recorded gait kinematics and ground reaction forces.

The MAPD-Barefoot gait step lengths and walking speed were within the range recorded on the patient (**Table 4**). The kinematic differences were larger when compared to LS-Barefoot, with RMS differences of 7.57◦ , 12.95◦ , 13.67◦ , 12.22◦ , for the pelvis, hip, knee and ankle angles respectively. To put these kinematic differences in perspective, note that the patient walks with a high degree of variability, exhibiting maximum variances in the barefoot trials of between 5.5◦ and 13.34◦ . The RMS values of the MAPD-Barefoot ground forces are 81.1 N. The MAPD-Barefoot problem took 4 h to solve as a single-thread execution on a 3.6 GHz processor.

The MAPD-Orthosis model walked with a lower cost, less of a crouch, a longer right step, and a higher walking speed than the MAPD-Barefoot trial (**Table 4**). Compared to the MAPD-Barefoot gait the MAPD-orthosis gait extends its right and left knees 9.26◦ and 8.3◦ more during stance, respectively (indicated as filled circles in **Figures 3B,E**). This improvement in knee extension angles matched the trend seen in the patient recordings. The RMS differences for MAPD-Orthosis gait are 8.48◦ at the pelvis, 14.48◦ at the hips, 16.11◦ at the knees, and 8.28◦ at the ankles (**Figures 3A–C**). The RMS differences for ground-reaction forces are 128.04 N for MAPD-Orthosis, which is higher than that for MAPD-Barefoot gait. Even though the orthosis pushed the model to walk faster, the left step length and the walking speed are below the corresponding recorded ranges (**Table 4**). The OCP MAPD-WS-Orthosis with the modified objective function, Equation (10) and a λ = 2, results in a walking speed of 0.77 m/s which is within the recorded range. The MAPD-Orthosis problem took 7 h to solve as a single-thread execution on a 3.6 GHz processor.

Ankle muscle extension torques are substantially reduced for the MAPD-Orthosis gait compared to those for MAPD-Barefoot (**Figures 5A,B**). Despite the faster walking speed for orthosis gait, the corresponding activations and excitations are generally smaller or equivalent to those for MAPD-Barefoot (**Figure 6**).

results from the LS-Barefoot dynamic fit (dashed lines). Note that the LS-Barefoot results have a discontinuity as indicated by asterisks on (D–F). This is due to the difference between the setup of the optimal control problem (starting at left toe off), and the plots (starting at left heel strike), as well as an asymmetry in the patient's gait over one complete left-right stride. We denote 100% along x-axis as the full left and right stride. Insets in panels (A,D) indicate the starting pose of the right and left foot. Filled circles in panels (B,E) indicate the minimum knee angle during stance.

#### TABLE 4 | Comparison of recorded gait characteristics and results from the corresponding optimal control problems.


Overall the objective function cost for the MAPD-Orthosis is smaller than that for MAPD-Barefoot (0.61 and 2.2, respectively). The computed optimal orthosis spring stiffness are K<sup>D</sup> = 45.9 Nm/rad and K<sup>P</sup> = 13.2 Nm/rad for the right ankle orthosis, and K<sup>D</sup> = 62.8 Nm/rad and K<sup>P</sup> = 19.7 Nm/rad for the left ankle orthosis.

The net torque-angle profiles of the MAPD-Orthosis gait have a similar angular offset and slope to the mean torque-angle profiles of the patient (**Figure 7**). Though the peak torques of the MAPD-Orthosis gait are larger than those of the patient, the profiles overlap with the ± 1 standard deviation regions of the patient data (shaded regions). The average slope of the torque-angle profiles from the patient data range from 95.7 to 135.5 Nm/rad, while the slope of the MAPD-Orthosis torqueangle profiles range from 122.3 to 150.1 Nm/rad.

The perturbation analysis reveals a maximum difference of 1.75% in cost function value, min. knee angles, step lengths and walking speeds for −5% changes in the orthosis stiffness parameters (**Table 5**).

## 4. DISCUSSION

We have presented an optimal control approach to generate novel movements with physically consistent dynamics and applied it to the simulation of patient gait. Our simulations result in smooth ground reaction forces (**Figure 4**) as well as muscle torques (**Figure 5**), and can be computed with modest computational resources. The ground reaction forces are continuous and have similar shape and magnitude as the patient observations. This is an improvement from published literature in this field, where large transients as well as deviations upto 150 to 200% of body weight have been reported (Ackermann and van den Bogert, 2010; Dorn et al., 2015). Physiologically realistic ground reaction forces are important, because a discrepancy here propagates through the model resulting in unrealistic joint torques and muscle forces. These characteristics, along with the possibility of tuning the model parameters to reflect weakened muscles, are

important first steps toward applying such methods in clinical settings.

The low residual forces from the inverse dynamics analysis (**Table 3**) indicates that the geometry and mass distribution of the model fit the patient well<sup>2</sup> . For comparison these residual values are 1.0% of the peak ground reaction forces during walking. The LS-Barefoot results reveal that the model is able to follow the recorded patient kinematics with the RMS differences smaller than the stride-to-stride variation in the patient. However, the ground reaction forces are markedly less smooth (dashed lines in **Figure 4B**) when compared to those recorded from the patient. In contrast the model kinematics for MAPD-Barefoot show larger RMS errors than LS-Barefoot, however, the ground reaction forces are smoother. We also note that the duty factor (ratio stance vs. swing time) in our simulated gait is different from that recorded, with shorter double stance durations for MAPD-Orthosis (**Figure 4A**).

Taking these results together, we conclude that the most likely reasons for these differences are the shape of the foot and the enforced sequential nature of the contact phases. Modeling the foot as a flat surface simplifies the resolution of the contact dynamics, however, it overlooks the natural curvature of the foot and the associated influence this can have on the behavior of the rest of the body (Dorn et al., 2012). Foot contact dynamics has been recognized as an issue of significant importance in model based estimation and prediction of gait as the foot forces affect those at the hip, knee and ankle (Dorn et al., 2012; Millard and Kecskeméthy, 2015). The use of a suitable curved foot model would therefore help improve the contact dynamics as well as avoid the strict phases that we have imposed in our current formulation. We expect that a curved foot model would also improve the simulated kinematics of the knee and ankle, which currently show large deviations from recorded behavior.

The orthosis provides additional ankle torque especially during push-off, and the resulting orthosis-equipped model could walk faster, with more extended knees than the barefoot model. The slope of the torque-angle profile of the MAPD-Orthosis is close to that of the patient (**Figure 7**). This indicates that the identified orthosis stiffness values are likely close to those of the patient's orthosis. We recall that the patient's orthosis was manually tuned by the orthopedic professional during the clinical procedure. We remark that while the slopes and angular offsets of the orthosis-equipped model lie within the experimentally recorded variation, the magnitude of the torques were higher in the model. This indicates that either the foot-shape (lever arm during toe-off) or the cost-function need to be updated to better match the patient. Our perturbation analysis reveals a

<sup>2</sup>http://simtk-confluence.stanford.edu:8080/display/OpenSim/Simulation+with+ OpenSim+-+Best+Practices

systematic increase in the cost function value (which is consistent as the perturbation was applied about the optimal solution) and relatively small influence of parameter changes on the gait characteristics (**Table 5**).

Despite the higher walking speed of the orthosis gait, the overall distance-normalized muscle activations based cost is smaller than that for barefoot gait. We observe a strong reduction in the muscle activations for orthosis walking (**Figure 6**). Although this is a desirable effect as it points toward a less fatiguing gait, it is presently unclear whether these changes actually occurred in the patient's real muscular efforts. As we are missing the experimental EMG recordings for this gait, our simulated reduction in muscle activations must be viewed as plausible but unverified. As noted in our Introduction, this is a general open problem with neuromuscular models, which require further experimental efforts as well as technological advances in EMG technology.

## 4.1. Choosing an Optimality Criterion for Gait

Our simulations are driven by an optimization criteria that minimizes the square of muscle activations per distance walked. Higher powers of activations have been suggested to be associated with muscle effort (Ackermann and van den Bogert, 2010), and our results from MAPD-Barefoot show that this formulation provides a reasonable match to the recorded gait characteristics (**Table 4**). For orthosis-equipped gait, we observe that the same formulation (MAPD-Orthosis), resulted in gait that is slower and has smaller steps. With an additional term in MAPD-WS-Orthosis we could drive the simulation toward more desirable characteristics, in this case faster walking. We speculate that there are subtle differences in the patient's walking behavior with orthosis, that are not entirely covered by the MAPD-only formulation.

Note that an alternative explanation for the slow walking speed in MAPD-Orthosis could lie in an underestimation of the maximum isometric torques of the patient's muscles, as well as the missing torques provided by the passive musculotendon components. We explored this avenue by simulating gait of a healthy age-matched, weight-matched child (torque values listed in **Table 4**). The detailed plots are provided in the supplementary section to this article. With healthy muscle strengths, we observed that the model was capable of longer steps and faster walking speed, matching the recorded gait of typically developing children (Schwartz et al., 2008). This leads us to believe that the major reason for the slower gait in MAPD-Orthosis lies in the cost function formulation, and that this deserves further investigation. For example with the use of inverse optimal control methods to identify the particular cost function that best describes experimentally recorded behavior (Mombaur, 2016), and especially the specification of cost functions that are better suited for pathological gait.

From our current work, we show that the specification of muscle strength in our models and the MAPD-type objective function is capable of reproducing, at least in our case study, a range of walking behaviors from healthy to pathological. Overall, it is foreseeable that a generic class of such objective function terms may be made available to the medical specialist, that would correspond to the clinical goals for the patient (e.g., faster walking, less crouch, reduced movement of the center of pressure etc.). The ultimate decision on which of these characteristics are suitable for the patient, would be the responsibility of the orthotist and other medical professionals. Our methods could provide a virtual window into the expected behavior under these conditions without inconveniencing the patient.

## 4.2. Limitations and Perspectives

In addition to the shape of the foot, we believe that another improvement to the model would be to decouple the orthosis and body models. This would allow for a more realistic simulation of the body-orthosis interaction as well take into account the inertial effects of the orthosis independently from the body. Specifically, this decoupled formulation would enable us to calculate a comfort-like cost function term based on the contact forces being generated, and as well simulate the effects of nonaligned rotations between the foot and the orthosis. Together, we believe that these changes will contribute toward more natural looking behaviors in our synthesized gait.

The simulated activations and active muscle forces of our model may be further improved. We estimated the patient muscle strengths based on inverse dynamics analysis and a qualitative clinical assessement of how close the patient was to his maximum strength during the recorded gait. The muscle curves used in our MTG model come from (Anderson et al., 2007), that are based on adult subjects. These curves may look different from children, especially for those with a pathology

FIGURE 6 | Comparison of neural excitations e and muscle activations a for MAPD-Orthosis and MAPD-Barefoot gait. Note that the two simulations resulted in different overall durations and are presented here with respect to % left-right stride. (A–F) Plot the results for the muscles of the left lower limbs, and (G–L) those for the right lower limbs.

that affects the muscle and overall strength. To the best of our knowledge, no such quantative muscle studies exist for children, and it would be of interest to bridge this gap in experimental data in the future. In a general context, the accurate specification of the model to a person is still an open problem. There may be various approaches to solve this, for example by using direct dynamometry information when available, and/or by making the maximum isometric torque as parameters of an OCP. Future iterations of this approach would include passive musculotendon forces in the simulations. To this end we are evaluating methods to estimate passive forces and muscle model coefficients from experimental data. Additionally, modeling the

TABLE 5 | Results from the parameter perturbation analysis.


A −5% perturbation was applied to each of the optimal identified spring stiffness values. Rows indicate the % change in MAPD-Orthosis results for a change in the corresponding orthosis spring parameter. Φ Denotes the cost function value (Equation 9). A negative % change in the min. knee angle indicates a straightening of the knee during stance.

effects of muscles that span multiple joints is an important next step. This may be implemented as a combination of the MTGs used in this work, and some of the major anatomical muscles as line-type models. For the study of pathological gait this may be especially important, as it would then allow the freedom to include the more complicated line-type muscle models based on the specific question/pathology at hand. In this initial work we do not model the feedback dynamics of muscle reflexes like for example those in the work by Geyer and Herr (2010). Including these closed-loop dynamics makes the OCP harder to solve, and we are currently exploring formulations that work well with our framework. While reflexes are typically subdued during normal locomotion (Brooke et al., 1991), they play an important role in making gait robust against perturbation rejection. In addition, neuromuscular pathology can adversely affect the ability to modulate reflexes (Hodapp et al., 2007; Pearson and Gordon, 2013), and any implementation of reflex feedback for pathological gait would necessarily require more detail and study than currently available in the state of the art.

Finally, we have focused so far on movements in the sagittal plane and used a case study to provide an important proof-ofconcept of our methods. Our comparison to experimental data provides a first evaluation of our model and technical platform, that needs to be further validated with a prospective clinical trial and extended to include movements in the transverse plane. We acknowledge that for application in a clinical setting our methods would also need to allow an easy setup and tuning to individual patients. Note that although the setup of the OCPs in this work took a significant amount of time, these efforts do not need to be replicated for each patient. In a future clinical application, we

REFERENCES


envision that a standardized gait simulation may be solved within a few hours with an individualized patient model (which requires relatively little time). This scenario provides a realistic means to apply our methods in a true clinical setting, and would be the ultimate goal of our future efforts related to this work.

#### AUTHOR CONTRIBUTIONS

MS, MM, and SW designed the study. MS, MM, MF, and KM performed the computational work related to this study. All authors contributed to the interpretation of the data and in preparing the manuscript.

## FUNDING

This study was part of the Frontier-Orthosis project supported by the German Excellence Initiative within the third pillar funding of Ruprecht-Karls-Universität Heidelberg. We acknowledge financial support by Deutsche Forschungsgemeinschaft and Ruprecht-Karls-Universität Heidelberg within the funding programme Open Access Publishing.

#### ACKNOWLEDGMENTS

The authors thank Julia Block and Daniel Heitzmann for assistance with processing the clinical data. We also thank the Simulation and Optimization research group of the IWR at Heidelberg University for allowing us to work with the optimal control code MUSCOD-II.

#### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fncom. 2017.00023/full#supplementary-material

Additional results figures are available as Supplementary Material to this article. The patient data, models, and OCP formulations are available as Supplementary Data. Further developments of this work will be maintained as an open-source public repository.<sup>3</sup>


<sup>3</sup>https://github.com/manishsreenivasa/PathWalker


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Sreenivasa, Millard, Felis, Mombaur and Wolf. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Human-Inspired Eigenmovement Concept Provides Coupling-Free Sensorimotor Control in Humanoid Robot

Alexei V. Alexandrov <sup>1</sup> , Vittorio Lippi <sup>2</sup> , Thomas Mergner <sup>2</sup> \*, Alexander A. Frolov 1, 3 , Georg Hettich<sup>2</sup> and Dusan Husek <sup>4</sup>

1 Institute of Higher Nervous Activity and Neurophysiology, Russian Academy of Science, Moscow, Russia, <sup>2</sup> Department of Neurology, University Clinics of Freiburg, Freiburg, Germany, <sup>3</sup> Russian National Research Medical University, Moscow, Russia, <sup>4</sup> Institute of Computer Science, Academy of Science of the Czech Republic, Prague, Czechia

Control of a multi-body system in both robots and humans may face the problem of destabilizing dynamic coupling effects arising between linked body segments. The state of the art solutions in robotics are full state feedback controllers. For human hip-ankle coordination, a more parsimonious and theoretically stable alternative to the robotics solution has been suggested in terms of the Eigenmovement (EM) control. Eigenmovements are kinematic synergies designed to describe the multi DoF system, and its control, with a set of independent, and hence coupling-free, scalar equations. This paper investigates whether the EM alternative shows "real-world robustness" against noisy and inaccurate sensors, mechanical non-linearities such as dead zones, and human-like feedback time delays when controlling hip-ankle movements of a balancing humanoid robot. The EM concept and the EM controller are introduced, the robot's dynamics are identified using a biomechanical approach, and robot tests are performed in a human posture control laboratory. The tests show that the EM controller provides stable control of the robot with proactive ("voluntary") movements and reactive balancing of stance during support surface tilts and translations. Although a preliminary robot-human comparison reveals similarities and differences, we conclude (i) the Eigenmovement concept is a valid candidate when different concepts of human sensorimotor control are considered, and (ii) that human-inspired robot experiments may help to decide in future the choice among the candidates and to improve the design of humanoid robots and robotic rehabilitation devices.

Keywords: human sensorimotor system, neuromechanics, biorobotics, motor control, eigenmovements

## INTRODUCTION

Most human skeletal movements involve several interconnected body segments. Starting from buttressing segments such as the feet when standing, a chain of segments interleaves to the end effector such as the hand in reaching. The joint rotations in such a reaching-while-standing movement occur in a coordinated way, with two aims dictated by physics standing out. A kinematic aim is to maintain the center of mass (COM) of all body segments supported by the ankle joints above the base of support, which is the area under and between the feet, in order to

#### Edited by:

Massimo Sartori, University of Göttingen, Germany

#### Reviewed by:

Yury Ivanenko, Fondazione Santa Lucia (IRCCS), Italy Malte Schilling, Bielefeld University, Germany

\*Correspondence: Thomas Mergner thomas.mergner@uniklinik-freiburg.de

> Received: 31 January 2017 Accepted: 04 April 2017 Published: 25 April 2017

#### Citation:

Alexandrov AV, Lippi V, Mergner T, Frolov AA, Hettich G and Husek D (2017) Human-Inspired Eigenmovement Concept Provides Coupling-Free Sensorimotor Control in Humanoid Robot. Front. Neurorobot. 11:22. doi: 10.3389/fnbot.2017.00022 maintain balance against external forces acting on the body such as gravity. A kinetic aim of movement coordination is to minimize effects of inter-segmental coupling torques. Dysfunction in the matching of timing and torque magnitudes across the chain of segments results in inappropriate compensation for body segment masses and inertia and neural time delays, imposing clinically for the kinematic chain as balance problems (Massion, 1992; Mergner, 2012) and for the kinetic chain as irregular and oscillating movements, a pathological symptom called ataxia that is typically found in cerebellar patients (Bastian, 1997). So far neuroscientists devoted considerable attention to the neural mechanisms underlying human kinematic coordination (Massion, 1992), but paid less attention to the neural mechanisms underlying kinetic coordination, on which this paper focuses.

In the technical domain, where industrial robotic devices are often fixed to the ground, kinematic coordination plays a minor role and the kinetic problem in controlling a chain of serially connected links with coupled dynamics such as a robotic arm can easily be solved. The solution is traditionally done by a full state approach, meaning that feedback and feed forward controls of all joints are computed together in a coordinated way that takes into account a full dynamic model of the arm and solves the inverse dynamics problem. In humanoid robots controlling position of an unstable body posture, the situation is more complex and inter-link force compensation is often performed using servo controllers, one for each joint. If feedback time delays are too large to be fully accounted for by predictive algorithms and if damping of the dynamic coupling effects is insufficient, destabilization of the control may result (Ott et al., 2014, 2016). Also, problems of control stability may arise in humanoid robots with several degrees of freedom (DoF) when the body dynamics are not fully known. Measuring acceleration in each link or a distal link using inertial sensors may help to solve the problem. Also, learning algorithms can be used to produce the needed coordination patterns. Usually reinforcement learning is employed in this context where the desired output is known in terms of performance, but not yet in terms of the needed controller outputs.

In the biological domain, neuroscientists studied for example the electromyographic effects from externally evoked coupling forces in the arm muscles. They observed typical response patterns in muscle activity (Lacquaniti and Soechting, 1986) belonging to the long-latency reflexes, which take into account the current arm configuration (Kurtzer et al., 2009) with a response amplitude scaling that involves the cerebellum (Kurtzer et al., 2013). The underlying neural control mechanisms are still unknown. Theoretically, at least, one could conceive that humans use a neural representation of a full state control. A more parsimonious solution has been suggested in terms of the Eigenmovement (EM) concept (Alexandrov et al., 2001a,b, 2005; Alexandrov and Frolov, 2011). It allows designing the control of the kinematics of the chain in the form of independent SISO (single input, single output) controllers.

Since the EM principle often produces coaction of joint torques in the context of predetermined kinematic synergies, there exists a clear overlap with the important concepts of motor primitives and modular control. Many aspects of motor modularity are discussed by d'Avella et al. (2015) and Flash and Bizzi (2016) including its theoretical and experimental substantiations and robotic applications. According to the modularity concept any movement can be decomposed as a superposition of motor primitives or synergies used as building blocks in a modular control architecture. Each module imposes as a specific pattern of motor activity in terms of kinematic, kinetic or EMG synergy. The specificity of an EM mechanism in this context is that it solves the problem of the dynamic coupling as the basis for controlling each kinematic synergy independently from the others. Although the independent control was demonstrated experimentally so far mostly in relatively simple movements we conceive that the EM concept can be extended to the wide class of multi-joint movements (see Discussion, also for robotic implementations).

Development of the EM concept started from biomechanically describing human hip-knee-ankle coordination during trunk bending as movements along eigenvectors of the motion equation (Alexandrov et al., 2001a,b). After showing that the contribution from the knee joints to this coordination tends to be negligible, the approach was restricted to hip-ankle coordination, and it was shown that the concept is applicable to independently controlled feed-forward and feedback situations (Alexandrov and Frolov, 2011) and that postural reactions to external perturbations can be formalized using a PD (proportional, derivative) control with time delays in the feedback loops (Alexandrov et al., 2005). In their studies, the authors considered the possibility that humans use EM controllers in some neurally implemented form. However, similar as with other concepts of human sensorimotor control, the evidence is indirect and still rather limited as long as analogies are drawn mainly from mathematical calculations or computer simulations, while biological constrains such as neural feedback time delays have experimentally not been considered in face of "real world" challenges such as noisy and inaccurate sensors and nonlinearities from computational and mechanical "dead zones" and friction, backlashes, etc. Such limitations may constrain also the potential use of the EM concept in humanoid robots and robotic rehabilitation devices.

With these reservations in mind, this paper investigates whether the EM concept is able to control a humanoid robot that shows human anthropometrics and is equipped with humaninspired sensors and actuators. The robot used, PostuRob II (**Figure 1**), served already before in experiments that tested a human-derived control concept as robotic implementation (Hettich et al., 2014). Similarly, it is currently used in modified form in another neurorobotics study for the overarching goal to experimentally evaluate the "real-world robustness" of humaninspired control concepts and to obtain back from the robot experiments inspirations for the human sensorimotor research.

The following sections describe first the hip-ankle biomechanics model and the EM controller and its operational capabilities. The subsequent sections describe identification of the robot's specific transfer characteristics by estimating its inertial, gravitational and geometric parameters and the properties of the transformation from joint torque commands

torque sensors); (E), artificial pneumatic "muscles"; (F), ankle joint (joint angle and torque sensors). (See Appendix 1 in Supplementary Material for details) at the controller output to the actual torques that were

experimentally observed at the joints (which in autonomous systems may be achieved by learning). Then, experimental results from testing the robot in a human posture control laboratory are described, including preliminary comparisons with human data, followed by Discussion. Details of the mathematical concepts, the robot, and experimental procedures are given in Appendices.

#### MATERIALS AND METHODS

#### Eigenmovement (EM) Concept and Biomechanical Hip-Ankle Model

Humanoid sagittal movements around hip and ankle joints in the vicinity of vertical body position (**Figure 2**) are described by

$$B\_0 \ddot{q} - G\_0 q = \text{ } \mathfrak{r}^{\text{con}} \tag{1}$$

where **q** is the vector of hip and ankle joint angles, **B**<sup>0</sup> and **G**<sup>0</sup> are the inertial and gravity matrices, and τ con is the vector of joint control torques. The coefficients of **B**<sup>0</sup> and **G**<sup>0</sup> are calculated

via the length-mass parameters of Posturob II, as described in Appendix 1 in supplementary Material.

Each EM is in the linear approach the movement along one eigenvector w<sup>i</sup> that, by definition, satisfies equation

$$\mathcal{B}\_0 \mathbf{w}\_i = \lambda\_i \mathbf{G}\_0 \mathbf{w}\_i, \qquad \text{ (\$i = 1, 2)}\tag{2}$$

where λ<sup>i</sup> is the corresponding eigenvalue (Alexandrov et al., 2001a). The vector ξ of the time courses of the two EMs is obtained by transforming the vector **q** by inversion of equation.

$$q\left(t\right) = \mathcal{W}\xi\left(t\right)\tag{3}$$

where the two columns of matrix **W** are the eigenvectors **w**<sup>i</sup> , (i = 1,2). According to Equations (2) and (3), the dynamic equation (1) takes the following form in terms of EMs.

$$
\Lambda \ddot{\xi} - \xi = \eta^{con} \tag{4}
$$

where 3 is a diagonal matrix with eigenvalues λ<sup>i</sup> , and

$$\boldsymbol{\eta}^{\text{con}} = \boldsymbol{\mathcal{U}} \boldsymbol{\mathfrak{r}}^{\text{con}}, \; \mathbf{U} = \left( \mathbf{G}\_0 \mathbf{W} \right)^{-1} \tag{5}$$

The two columns **u**<sup>i</sup> (i = 1,2) of matrix **U** in Equation (5) are the vectors, whose components define the contributions of ankle and hip joint torques to the EM dynamics.

#### EM PD-Controller

As shown previously (Kuo, 1995; Welch and Ting, 2008; Frolov et al., 2000, 2006), the joint torques τ con , which generates the desired body movement, can be implemented as a PD-controller with time delay 1t in the form

$$\begin{aligned} \mathbf{r}^{\rm con}(t) &= -\mathbf{G}\_0 \mathbf{q}(t - \Delta t) + \mathbf{S} (\mathbf{q}^d(t - \Delta t) - \mathbf{q}(t - \Delta t)) \\ &- \mathbf{V} \dot{\mathbf{q}}(t - \Delta t) \end{aligned} \tag{6}$$

where **q** d and **q** define the time course of the desired and actual changes in joint angles, while **S** and **V** represent "stiffness" and "viscosity" matrices whose elements define the gain coefficients in the feedback loop.

In EMs, Equation (6) takes the form:

$$\begin{aligned} \eta^{con}(t) &= -\xi(t-\Delta t) + \mathcal{S}^{sig}(\xi^d(t-\Delta t) - \xi(t-\Delta t)) \\ &- V^{sig}\dot{\xi}(t-\Delta t) \end{aligned} \tag{7}$$

where

$$\mathcal{S}^{\text{eig}} = (\mathcal{G}\_0 \mathcal{W})^{-1} \mathcal{S} \mathcal{W}, \; V^{\text{eig}} = (\mathcal{G}\_0 \mathcal{W})^{-1} V \mathcal{W} \tag{8}$$

Independent control of each of the two EM means that matrices **S** eig and **V** eig are diagonal, so that the vector Equation (1) in terms of joint angles and torques splits into two scalar equations in terms of EMs (Alexandrov et al., 2001a, 2005; Alexandrov and Frolov, 2011), each equivalent to a PD-control of a single-link inverted pendulum:

$$\begin{aligned} \lambda\_i \dot{\xi}\_i(t) - \dot{\xi}\_i(t) &= -\xi\_i(t - \Delta t) + \mathsf{S}\_i^{\mathrm{eq}}[\xi\_i^d(t - \Delta t) - \xi\_i(t - \Delta t)] \\ &- \mathsf{V}\_i^{\mathrm{eq}} \dot{\xi}\_i(t - \Delta t) \end{aligned} \tag{9}$$

where **S**<sup>i</sup> eig and **V**<sup>i</sup> eig are the diagonal elements of matrices **S** eig and **V** eig , respectively (<sup>λ</sup> <sup>=</sup> J/mgh; <sup>J</sup>, moment of inertia relative to pendulum axis of rotation; m, pendulum mass and h its altitude; g, gravitational acceleration).

The inverse transformation of Equation (8) gives the stiffness and viscosity matrices **S** and **V** in Equation (6) in terms of joint angles:

$$\mathcal{S} = \mathcal{G}\_0 \mathcal{W} \mathcal{S}^{\text{eg}} \mathcal{W}^{-1}, \; V = \mathcal{G}\_0 \mathcal{W} V^{\text{eg}} \mathcal{W}^{-1}. \tag{10}$$

When matrices **S** eig and **V** eig are diagonal, then matrices **S** and **V** are symmetrical, but not diagonal (Alexandrov et al., 2005). Therefore, the PD-control in terms of joint angles needs to take into account not only the kinematics and dynamics of a given joint, but also those of all other joints. The number of feedback control parameters in this kind of control, referred to as "full-state feedback control" (Barin, 1989; Park et al., 2004), is equal to the number of elements in the stiffness and viscosity matrices **S** and **V**. In the EM approach, notably, the number of feedback control parameters is reduced to the number of diagonal coefficients in the matrices **S** eig and **V** eig .

The EM PD-controller is shown schematically in **Figure 3**. The time delays outside the controller represent time delays 1t<sup>1</sup> and 1t<sup>2</sup> between controller commands τ C and actual torque τ applied to the robot segments, which are mainly induced by the generation of the torques in the robot. These "actuation" delays were intentionally equalized in the present study inside the PDcontroller by adding delays of 1t C 1 and 1t C 2 to the respective joints (**Figure 3**) such that the total time delays ∆t<sup>A</sup> and 1t<sup>H</sup> of the transformation for ankle and hip joint torques respectively were equal and amounted to 1t<sup>A</sup> = 1t<sup>1</sup> + 1t C <sup>1</sup> = 1t<sup>H</sup> = 1t<sup>2</sup> + 1t C <sup>2</sup> = 1t.

For the robot experiments (see below), the controller has been implemented as a program in Simulink, which allows controlling the robot in real time. As to the controller inputs, the joint angles q give the desired body position in terms of joint angles with respect to each other and the support surface. This is by itself not sufficient to balance in the general case in which the support surface is not a stable reference. The Posturob platform integrates a human inspired vestibular system, mechanically fixed to the upper body and providing the trunk orientation in space (Mergner et al., 2009). Using the vestibular information, the control can be generalized to the condition of support surface tilt in space. In particular, in experiments with the robot standing on moving platform, the information of leg-in-space angle was calculated with help of the vestibular sensor and used as input signal q<sup>1</sup> for the leg segment control, and the joint angle signal from the hip joint sensor was used for the hip control. With stable platform, angle and vestibular sensor signals were combined for each joint to improve the signal to noise ratio.

#### Theoretical Analysis of Control Stability

The independent control for each EM allows the analysis of whole body control stability by two separate analyses of each EM's stability. The stability of each EM is defined by the roots µ of the secular equation of Equation (9):

$$
\mu^2 \lambda - 1 + \left(\mathbf{S}^{\text{eig}} + 1\right) \mathbf{e}^{-\mu \Delta \text{ t}} + \mu \mathbf{V}^{\text{eig}} \mathbf{e}^{-\mu \Delta \text{ t}} = \mathbf{0}.\tag{11}
$$

When in equation (9) 1t > 0, Equation (11) has an infinite number of complex roots µ = α + iω, where α and ω are the real and imaginary parts of the root and i is the imaginary unit (Alexandrov et al., 2005). The solution of equation (9) is stable if the real part α of all the roots of equation (11) is negative. The maximum value of the real part of all the roots of the secular equation is called Lyapunov index. Thus, the solution of equation (9) is stable if its Lyapunov index α < 0. The Lyapunov index defines the characteristic time ∆tchr = |α| <sup>−</sup><sup>1</sup> of the complex system response to the external perturbation.

The minimization of the Lyapunov index for each EM was used as a criterion for optimizing the PD-controller parameters. The optimum parameters were obtained according to a method based on calculations of the ranges in the space of S eig and V eig in which the Lyapunov index does not exceed given values α (Appendix 2 in Supplementary Material). The main results are shown in **Figure 4**. It shows the minimal Lyapunov index αmin which can be achieved for a given λ and 1t. The values of S eig and V eig which provide αmin were treated as optimal. For each delay 1t there exists some critical value λcrit at which αmin becomes zero. If λ < λcrit, no range space of stability for the given feedback loop delay exists, meaning that the PD-controller does not provide stable control of this dynamic system. Thus, stable PD-control is impossible if feedback loop delay 1t > 1tmax for a given λ or if λ < λcrit for a given 1t.

#### Experimental Setup

The humanoid robot PostuRob II (**Figure 1**) comprises trunk, legs and feet segments interconnected by the hip and ankle joints. Signals from mechatronic vestibular and joint angle sensors are real-time inputs to a PC. The implemented control system controls artificial pneumatic "muscles" (FESTO AG &Co.KG, Esslingen, Germany; Typ MAS20), which generate a desired torque in the hip and ankle joints. The EM control

FIGURE 3 | Scheme of EM control. Shown are the EM controller, the actuation, the plant and the sensors. The PD-controller transforms desired and sensory variables q into EM kinematic variables ξ (boxes W−<sup>1</sup> ) and then into EM dynamic variables η con (boxes Seig <sup>+</sup> 1 and Veig) which are in turn transformed into output joint torques τ <sup>C</sup> (box U) with controller time delays (boxes ∆t <sup>C</sup>). Torque τ <sup>C</sup> becomes effective at the robot segment after an actuation time delay (boxes 1t). Note, that in boxes Seig <sup>+</sup> 1 the unit is added to Seig in order to take into account the first "gravity" term in the right side of Equation (7). Matrices <sup>S</sup> eig , V eig , W and U are defined above.

model was executed as a compiled Simulink model (Real-Time Windows Target, The Math Works Inc., Natick, USA). In the presented experiments, the robot was standing freely on firm support, a 6 DoF motion platform (Mergner et al., 2003), and performed active sinusoidal trunk and leg movements in the sagittal plane with different frequencies and amplitudes, reactive postural responses to external disturbances such as support surface rotation or translation in the sagittal plane. During the experiments, performed in a human posture control laboratory, sensory signals of joint angles and joint angular velocities as well as desired joint angle signals and actual joint torques were recorded with an acquisition rate of 200 Hz (further details in Appendix 1 in Supplementary Material).

In the first series, the robot's characteristics were evaluated in terms of (a) its inertial and gravity matrices **B**<sup>0</sup> and **G**0, (b) the "actuation" time delays 1t<sup>1</sup> and ∆t<sup>2</sup> in the transformation of joint torque commands to effective torques at the robot's joints, (c) the optimal PD-controller parameters, and (d) the dynamic response characteristics of the robot and (e) Experimental transfer functions and dynamic response of the controlled system. For a, preliminary (theoretically optimal) parameters of the PDcontroller were calculated based on the model of the two-linked rigid rods described above, and for b the time delays 1t<sup>1</sup> and 1t<sup>2</sup> for the transformation of torque commands to the effective joint torques were preliminarily estimated to be 100 ms. This allowed for stable PostuRob II movements within the tested conditions and, at a later step (comparing the theoretical with the measured transfer functions), to finally calculate optimal parameters of the PD-controller on the basis of the experimentally obtained matrices **B**<sup>0</sup> and **G**0.

#### Experimental Stimuli

The commanded joint angles in this approach, q d 1 (t) for the ankle joint and q d 2 (t) for the hip joint in Equation (1), had synchronous sinusoidal time courses with seven different frequencies f : 0.05, 0.1, 0.2, 0.4, 0.6, 0.8 and 1.0 Hz. Five pairs of amplitudes A<sup>1</sup> and A<sup>2</sup> of desired signals q d 1 (t) = A1sin(2πft) and q d 2 (t) = A2sin(2πft) were tested for each frequency: (A1, A2) = (0, 10◦ ), (−2.5◦ , 7.5◦ ), (−5 ◦ , 5◦ ), (−4.5◦ , 1.5◦ ), and (3◦ , 0). With these amplitudes, the zero moment point position remained inside the support area defined by the area comprised by the feet.

#### Evaluation of the Robot'S Inertial and Gravity Matrices

The evaluation was performed by integration of Equation (1) with the help of the experimentally recorded joint torques and a given set of elements for matrices **B**<sup>0</sup> and **G**<sup>0</sup> ( compare Kuo, 1995; Alexandrov and Frolov, 2011). The elements of these matrices, which minimize the error between the joint angles obtained by the numerical integration and the experimental joint angles, were taken for estimating the robot's characteristics. Their search was performed by the gradient descent method given in the MATLAB software. Parameters presented in Appendix 1 in Supplementary Material were taken as the initial ones. The search terminated on the step when the tolerance function decreased less than by 10−<sup>4</sup> as compared with the previous step.

#### Evaluation of Time Delays in Transformation from Torque Commands to Effective Joint Torques

The transformation from the joint torque commands at the PDcontroller output to the experimentally obtained joint torques is realized in PostuRob II independently for the hip joint and the ankle joint. Crosstalk between the joint torques is negligible and the transformation is performed in each joint separately with different time delays (boxes 1t1, 1t<sup>2</sup> in **Figure 2**). The matrix **F** τ τ describes the transfer function of the desired torque commands to the effective torques at the robot segments by

$$F^{\tau \tau} = \begin{pmatrix} e^{-i\nu\Delta t\_1} & 0\\ 0 & e^{-i\nu\Delta t\_2} \end{pmatrix} \tag{12}$$

where ω = 2πf, f is the frequency of the sinusoidal robot movement, i is the imaginary unit, and 1t1, 1t<sup>2</sup> are the delays in the transformation of torque commands to the torques in the ankle and hip joints, respectively. The delays ∆t<sup>1</sup> and 1t<sup>2</sup> that provided the best fit with the experimental transfer function were then taken for the subsequent estimations.

#### RESULTS

The experiments with Posturob II were performed on a motion platform in a human posture control laboratory. They comprised in addition to voluntary movement tests also tests of balancing biped stance during external disturbances (details in Appendix 1 in Supplementary Material). We refrained from adjusting the above control parameters to specific experimental conditions even when this was associated with particular technical insufficiencies such as an increased static friction. Proactive lean movement and reactive postural lean responses were tested. Both could be performed either in space coordinates using the artificial vestibular sensor (change in trunk-space angle, TS, or leg-space angle, LS) or in proprioceptive coordinates (change in trunk-leg angle, TL, or leg-foot angle, LF).

### Evaluation of the Robot'S Inertial and Gravity Matrices

The elements of matrices **B**<sup>0</sup> and **G**<sup>0</sup> obtained experimentally amounted to **<sup>B</sup>**<sup>011</sup> <sup>=</sup> 65.01 N m s<sup>2</sup> rad−<sup>1</sup> , **B**<sup>012</sup> = **B**<sup>021</sup> = 10.09 N m s<sup>2</sup> rad−<sup>1</sup> , **<sup>G</sup>**<sup>011</sup> <sup>=</sup> 460.01 N m rad−<sup>1</sup> , **G**<sup>012</sup> = **G**<sup>021</sup> = **G**<sup>022</sup> = 103.02 N m rad−<sup>1</sup> (these values replaced in the following the initial values given in Appendix 1 in supplementary Material). The experimental transfer function **F** qτ <sup>e</sup> was calculated according to Appendix 3 in Supplementary Material for the case that the two components of the signal **X**(t) are the joint angles and the two components of the signal **Y**(t) are the corresponding joint torques recorded during the above 35 described cyclic movements of PostuRob II. The theoretical transfer function **F** qτ <sup>t</sup> was calculated according to equation

$$F\_t^{q\tau} = -\alpha \mathcal{B}\_0 - \mathcal{D}\_0 \tag{13}$$

using the inertial and gravity matrices **B**<sup>0</sup> and **G**<sup>0</sup> obtained experimentally. The root-mean-square error of the mismatch between **F** qτ t and **F** qτ <sup>e</sup> amounted to 9.9%.

#### Evaluation of Delays in Torque Actuators

**Figure 5** shows the experimentally obtained transfer function **F** τ τ e and the theoretical transfer function **F** τ τ t for the transformation of the torque commands to the joint torques. The function **F** τ τ <sup>t</sup> was calculated according to Equation (12) and **F** τ τ e according to Appendix 3 in Supplementary Material. The two components of the signal **X**(t) are the control signals τ C 1 and τ C 2 at the PD-controller output (**Figure 3**) for the ankle and hip joints respectively, and the two components of the signal **Y**(t) are the corresponding experimental recordings of the joint torques. The "actuation delays" that provided the minimum root-mean-square error between the **F** τ τ e and **F** τ τ t amounted to 1t<sup>1</sup> = 0.091 s (ankle) and 1t<sup>2</sup> = 0.053 s (hip). The minimum root-mean-square error amounted to 5.1%. Note that the off-diagonal elements of the experimental transformation matrix **F** τ τ e are small compared to the diagonal elements, indicating very small crosstalk torques between the joints.

#### Readjustments of the Parameters for the EM PD-Controller

The solutions of Equations (2) and (5) for the experimentally obtained matrices **B**<sup>0</sup> and **G**<sup>0</sup> give the following eigenvalues λ<sup>i</sup> , eigenvectors w<sup>i</sup> , and vectors u<sup>i</sup> (i = 1, 2) defined in Appendix 1 in Supplementary Material:

$$\begin{aligned} \lambda\_1 &= 0.15 \text{ s}^2; \ \mathbf{w}\_1 = \begin{pmatrix} -0.89\\ -0.46 \end{pmatrix}; \ \mathbf{w}\_1 = \begin{pmatrix} 455.6\\ 138.9 \end{pmatrix} \\\ \lambda\_2 &= 0.02 \text{ s}^2; \ \mathbf{w}\_2 = \begin{pmatrix} -0.29\\ -0.96 \end{pmatrix}; \ \mathbf{w}\_2 = \begin{pmatrix} 35.6\\ -68.5 \end{pmatrix} \end{aligned} \tag{14}$$

The actuation delays 1t<sup>1</sup> = 0.091 s and 1t<sup>2</sup> = 0.053 s were intentionally equalized by adding delays of τ C <sup>1</sup> = 0.009 s and τ C <sup>2</sup> = 0.047 s to the respective joint inside the PD-controller (**Figure 3**). As a result, the total delays for both ankle and hip joint torques amounted to 1t = 0.1 s. The two markers in **Figure 4** located on the dashed curve for 1t = 0.1 s and the two eigenvalues λ<sup>1</sup> and λ<sup>2</sup> of PostuRob II indicate that the Lyapunov indexes in the experimental movements amounted to αmin<sup>1</sup> = <sup>−</sup>5.6 s−<sup>1</sup> for the first EM and <sup>α</sup>min<sup>2</sup> = −4.1 s−<sup>1</sup> for the second EM.

The following optimal values of stiffness S eig opt and viscosity V eig opt in the PD-controller were calculated according to Appendix 2 in Supplementary Material for the obtained values of λ1, λ2, and 1t = 0.1 s:

$$\begin{aligned} S\_{opt1}^{eq} &= 1.04; & V\_{opt1}^{eq} &= 0.73 \ s; \\ S\_{opt2}^{eq} &= 0.06; & V\_{opt2}^{eq} &= 0.15 \ s. \end{aligned} \tag{15}$$

To obtain the estimate of the transfer function F qq from desired to actual kinematics, cyclic movements of PostuRob II were recorded using the optimal parameters obtained so far. These recordings were used to calculate an experimental transfer

function from desired to actual kinematics (Appendix 3 in Supplementary Material) and were compared to a theoretical transfer function (Appendix 4 in Supplementary Material).

**Figure 6** shows the gain (**Figure 6A**) and phase (**Figure 6B**) characteristics of the transfer function F qq from the desired kinematics to the actual kinematics in terms of joint angles. The experimentally obtained values **F** qq <sup>e</sup> calculated according to Appendix 3 in Supplementary Material are compared with the theoretical ones **F** qq t calculated according to Appendix 4 in Supplementary Material.

The off-diagonal elements F qq <sup>12</sup> and F qq <sup>21</sup> of the transfer function **F** qq are theoretically and experimentally non-zero. However, the off-diagonal elements are small as compared with the diagonal elements F qq <sup>11</sup> and F qq <sup>22</sup> . In general, the experimental data points in **Figure 6** qualitatively correspond to the theoretical results (curves), this despite some data scatter.

#### Proactive Movements of the Robot

Proactive TS and LS movements were performed in addition to the robot experiments also in model simulations. A first overview was obtained with desired sinusoidal TS movements in space coordinates. **Figure 7** shows the "voluntary" signals (desired trunk-space angle, TS!) in comparison with the executed movements signal (TS) for 0.2 and 1.2 Hz sine frequency (A, simulation data; B, robot data). At 0.2 Hz, only a very small coupling effect of TS on LS is visible in the simulations (A1), whereas a small in-phase reaction occurs in the robot (B1). At 1.2 Hz, the evoked LS excursions are increased and shifted toward counter-phase already in the simulation (A2) and more so in the robot (B2).

Noting that the separation between dynamic and static effects and the use of space coordinates Pcomplicate the interpretation of the robots behavior, movement commands with smoothed ramp-like waveform (raised cosine velocity function, see Appendix 1 in Supplementary Material) and proprioceptive coordinates were used in the simulations and robot experiments (**Figure 8**). Stable stance was obtained in both scenarios. In the model simulations a desired trunk-leg angle signal of 4◦ (TL!= 4 ◦ ) leads to a slight TL overshoot and a weak dynamic LF counter excursion (A1). With a desired leg-foot angle (LF!) as command a slight LF overshoot and a clearly larger transient dynamic TL counter responses occurred (A2). In the corresponding robot experiments, the resulting TL movement also shows a dynamic response, mainly attributed to static friction effects (B1). LF showed no considerable dynamic effect, but a static excursion in TL lean direction. With a desired LF command, the resulting LF lean movement showed a dynamic overshoot (B2). The effect on TL consisted of very small dynamic counter-effects and a relatively large static excursion in the direction of LF of approximately 5.5◦ . The large TL response is mainly due to the relatively large weight of the trunk. It should be noticed that the used controller (PD) does not guarantee a null static error in the general case. This is more evident in robot experiments (B2) than in simulations (A2) where the control parameters can be perfectly tuned to the system. Taken together, dynamic coupling in simulations and the robot experiments were not completely abolished, but strongly reduced (e.g., A2), while gravitational

FIGURE 6 | The gain (A) and phase (B) characteristics of the theoretical F qq t (curves) and experimental F qq e (data points) transfer functions of the desired kinematics to the actual kinematics in terms of joint angles q. Experimental points: ◦ - F qq <sup>e</sup>11, <sup>F</sup> qq <sup>e</sup>21,<sup>H</sup> <sup>F</sup> qq <sup>e</sup>12, ♦ <sup>F</sup> qq <sup>e</sup>22.

torque effects where prominent, this mainly when the control operated in proprioceptive coordinates (**Figure 8**) and less so in space coordinates (**Figure 7**).

## Reactive Responses to External Disturbances

Using the pseudo-random ternary sequence (PRTS) stimulus allows to analyze externally evoked LS and TS sway responses over a broad spectrum of frequencies (Hettich et al., 2014; adopted from Peterka, 2002). The method allows analyzing the data in the frequency domain in terms of frequency response functions (FRFs) and coherence functions (see Appendix 1 in Supplementary Material). Examples of the time series of the stimulus and responses in the sagittal plane are given in **Figure 9A** for support surface tilt with vestibular input (control of leg segment operated in coordinates of gravitational space) and in **Figure 9B** for support surface translation (control operated in joint coordinates). Note that the robot successfully maintains balance with relatively small angular leg and trunk excursions in the two experiments shown and in other balancing experiments performed (see also Film in Supplementary Materials). The FRF results for the tilt experiments with pp 2◦ and pp 8◦ are given in **Figure 10**. They show that the robot keeps the orientation of the legs in space and the trunk in space upright. The robot was able to maintain balance also without vestibular input (control of leg segment operated with respect to the feet) when the support surface tilt amplitude was reduced to 3 ◦ or smaller (**Figures 11**, **12**), a performance that qualitatively is similar to that of vestibular loss human subjects (see Discussion).

4 ◦ ; dominant frequency, 0.2 Hz) of TL (desired, TL!) in (A1,B1) and of LF (desired, LF!) in (A2,B2).

## DISCUSSION

Theoretically, the EM concept is a relatively simple and efficient method to cope in a multi-DoF system with the coupling forces between mechanically linked segments. This study tested whether the EM concept is able to cope with coupling forces also in the control of a real-world technical device such as a humanoid robot, in which the control faces non-ideal properties such as noisy, inaccurate and non-linear sensors, friction and backlash, etc. These real world conditions may challenge control robustness in face of human-like feedback time delays. The robot experiments demonstrate that the EM control method copes considerably well with the real-world properties in a humanoid robot with human-like anthropometrics and equipped with human-inspired sensors and actuators. Therefore, we consider the EM control method a valid candidate that should be considered when making inferences on which method humans may use for their sensorimotor control. In the following, we first briefly address general issues of the EM concept before considering our experimental findings and consider alternatives to the EM control method.

Considering the EM method in this study was not meant to reduce the number of degrees of freedom of the system or to solve a redundancy problem, because the number of DOFs is the same when controlling joints or EM spaces. Rather, the benefit of using the EM concept is a control simplification in that multiple EMs can simultaneously be controlled dynamically independently of each other. For example, with a trunk bending forward the robot may simultaneously move the leg segment backwards in order to balance the COM over its feet as base of support (corresponding to a kinematic synergy), but alternatively may maintain the leg segment vertical (coping with forward shift of the COM through corresponding ankle torque). Generally, it should be noted that EMs are not synergies observable in the space of joint angles, but a step in the design of a control system that can produce arbitrary poses and trajectories (within the limit of the robot's dynamic responses). Independent PD-control of separate EMs allows clearly longer time delays in the feedback loop than the independent PD control of separate joints. For example, the Lyapunov index showed that the independent PD-controls of separate EMs in PostuRob II is stable up to a time delay of ∆t = 200 ms, while the limit is ∆t = 150 ms using independent controls of separate joints (see Appendix 5 in Supplementary Material). Conceivably, in a more complex control system such as the human one, additional or other mechanisms may contribute to control stability in face of long time delays.

The definition of the EMs implies a linearization of the system. With the control of upright body posture, a natural choice for the linearization point is the vertical position. Principally, however, the system has previously been shown to work also for a wider range of movements, exploiting successive linearization points as described for arm movements (Frolov et al., 2013). It remains to be shown how complexity increases when the EM method has to deal continuously with large operative spaces.

Despite the fact that the design of Posturob II takes into account the human anthropometry, time delays in the control loop and some human-inspired sensors and actuators it ignores several known constituents of the human posture control system such as load-related proprioceptive sensors (Dietz, 1998), foot deformation (Wright et al., 2012), some minor role of the knee joints (Alexandrov et al., 2001b), and more. Even though ignoring these parts of the human posture control, the EM approach provided stable maintenance of posture and movements in the humanoid robot. Similarly, previous feedback control models, using in the absence of visual information only joint angle proprioceptive and vestibular sensory inputs, were sufficient to quantitatively describe human responses to moderate support surface tilt stimuli in the sagittal plane, as also shown with other control models such as the independent channel (IC) model (SIP biomechanics Peterka, 2002) or the disturbance estimation and compensation (DEC) model (DIP biomechanics; (Hettich et al., 2014); this study includes a direct comparison between data of humans and of Posturob II using the human-derived model). The model of Park et al. (2004); see also (Kuo and Zajac, 1993), one of several currently available models of posture control, used proprioceptive linear full state feedback control to describe human responses to backwards translation and found only moderate improvement when increasing complexity from a 2-segment to a 4-segment model.

The main result of the present study is the experimental demonstration that the EM method copes well with a PDcontrol of a "real-world" mechanical anthropomorphic robot. The feedback loop parameters for the independent control in each EM were calculated from the robot anthropometrics, including human-like feedback time delays. Other characteristics of the robot as a "real-world" system, which typically are not exactly known such as friction, nonlinearities, noise, backlashes, inaccuracies, etc., were ignored. These unaccounted factors led to clear deviations of measured results from model

FIGURE 9 | Time series of stimulus and sway responses of the robot to support surface tilt (A, with vestibular function; leg-in-space tilt, LS, and trunk-in-space tilt, TS, in space coordinates) and support surface translation (B, without vestibular function; leg-in-space tilt, LS, and trunk-in-space tilt, TS, in platform coordinates) using the PRTS stimulus waveform.

predicted results, but influenced relatively little the overall characteristics of the movements and did not contradict the hypothesis that the EM concept can provide in principle stable performance of the robot. This applies to both proactive movements and reactive balancing of stance during unforeseen external disturbances in the sagittal plane (such as horizontal translation of the support surface and, when vestibular information was included into the control, support surface tilt).

Another aspect to be considered in the present approach was that the EM implementation aimed at an optimal stability of the control (see Section Theoretical Analysis of Control Stability). This does not imply that thereby the robot's postural responses automatically become similar to human subjects. In other words,

optimizing the EM control for Posturob II does not mean that the robot's postural responses become human-like because of its human-inspired sensors and actuators alone. The present approach differs from that in a previous study, which also used Posturob II (Hettich et al., 2014). There, human-like postural responses of the robot were obtained by fitting the control parameters of a posture control model, which later controlled the robot, to the human responses. Still, we considered it as interesting to compare in the present context the robot data with human data to visualize differences in the sway response behavior. To this end, we superimposed on the robot's frequency response functions shown in **Figure 12** the results obtained with the same stimulus and set up obtained from a group of vestibular loss human subjects with eyes closed. The reason for this choice was to consider an especially simple control that uses mainly proprioceptive sensors (in humans possibly including force cues; Mergner et al., 2009). Outstanding differences between human and robot data are larger human sway responses in the midfrequency range of the PRTS stimulus, a slightly different phase behavior, and smaller coherence in the low to mid frequency range, possibly indicating higher sensory or motor noise (or more general, effects not fully taken into account by the model). Future studies may use parameter identification methods in order to fit the EM concept to human data.

#### Possible Implications for Robotics

Kinematic synergies are widely used in humanoid robotics, typically with the purpose to simplify movement control. Kinetic synergies, i.e., predefined coordination between joint torques, although considered in numerous human studies (Prattichizzo et al., 2010; Shim et al., 2010), are rather sparsely implemented in explicit form in robots. As to the EM, they are formally kinematic synergies, yet the variables η are kinetic synergies defined by the coefficients of the matrix **U**. Formally, EM are defined to be coupling-free and to cover the whole space of joint motion. They are not producing a simplification of the control in terms of DOFs, yet, being dynamically independent, they simplify the structure of the control problem to the case of the control of several independent SISO systems. This can be advantageous for robotic control both in simplifying the design of the control system and to improve robustness with respect to modifications in control parameters, since modifications to one of the SISO controllers does not affect the others. This can be important in cases when the control parameters need to be dynamically adjusted, e.g., to improve balance performance during different tasks or in different scenarios.

However, the simplification of the control design by using SISO controllers comes at the cost of designing the transformation matrices **U** and **W**. This transformation requires a reliable model of the body dynamics, which may be problematic with certain tasks such as with full body control of a humanoid that involves a large number of DOFs. A further limitation is given by the system linearization required to define the Eigen movements, since this implies the necessity to linearize the system in different points of the control space when trying to cover a wide range of motions and/or when involving several distributed DOFs in complex behaviors such as walking. Overall, considering advantages versus limitations of the EM concept applied to robotics, we conceive that whole body control tasks like balancing upright stance would profit from it, because the limited range of motion makes errors due to linearization negligible. We would expect profits also for applications in which the dynamic requirements of the task and the presence of time delays make the coupling forces highly relevant, such as in the control of fast arm reaching (Frolov et al., 2013). In case that the number of DOFs involved in a task is large so as to make full state solutions non trivial, use of the EM approach with accurate definition of **W** and **U** still may be practicable, this especially if the control task is reduced to a subset of variables by means of integration of task specific constraints.

## CONCLUSIONS

A major conclusion from the present experiments refers to the robot experiments as an experimental tool when studying the human sensorimotor control system. Using a humanoid

## REFERENCES


robot for comparing different bio-inspired control concepts with each other on the same robot will help to define criteria for presumed human-likeness of control algorithms—with potential benefits also for use in humanoid robotics and user acceptance in robotic neurorehabilitation. In doing so, the robot experiments provide a valuable "real world" test that complements model simulations, especially in addressing the problem of control stability in face of human time delays. Finally, an experience from this study is that "learning by doing" in the robot experiments provides inspirations also for the research of the human control system.

## ETHICS STATEMENT

All subjects gave their informed written consent to the study that was approved by the Ethics Committee of the Freiburg University Clinics, in accordance with the Declaration of Helsinki.

## AUTHOR CONTRIBUTIONS

All authors performed the experiments and collected the data. GH, VL, and TM performed the computer simulations and analyzed the robot and human data. All authors contributed to the interpretation of the data and contributed to writing the manuscript. All authors approved the final version of the manuscript for submission.

## FUNDING

The work was supported by the Russian Science Foundation, grant 16-15-00219, for the author AF and by the European Commission FP7 Grant 600698 H2R and 610454 EMBalance for the authors GH, TM, and VL.

## SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fnbot. 2017.00022/full#supplementary-material


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Alexandrov, Lippi, Mergner, Frolov, Hettich and Husek. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Evidence in Support of the Independent Channel Model Describing the Sensorimotor Control of Human Stance Using a Humanoid Robot

Jantsje H. Pasma<sup>1</sup> \*, Lorenz Assländer 2,3, Joost van Kordelaar 1,4, Digna de Kam<sup>5</sup> , Thomas Mergner <sup>2</sup> and Alfred C. Schouten1,4

<sup>1</sup> Department of Biomechanical Engineering, Delft University of Technology, Delft, Netherlands, <sup>2</sup> Department of Neurology, University Clinics Freiburg, Freiburg, Germany, <sup>3</sup> Sensorimotor Performance Lab, University of Konstanz, Konstanz, Germany, <sup>4</sup> Department of Biomechanical Engineering, Institute for Biomedical Technology and Technical Medicine (MIRA), University of Twente, Enschede, Netherlands, <sup>5</sup> Department of Rehabilitation, Donders Centre for Neuroscience, Radboud University Medical Center, Nijmegen, Netherlands

The Independent Channel (IC) model is a commonly used linear balance control model in the frequency domain to analyze human balance control using system identification and parameter estimation. The IC model is a rudimentary and noise-free description of balance behavior in the frequency domain, where a stable model representation is not guaranteed. In this study, we conducted firstly time-domain simulations with added noise, and secondly robot experiments by implementing the IC model in a real-world robot (PostuRob II) to test the validity and stability of the model in the time domain and for real world situations. Balance behavior of seven healthy participants was measured during upright stance by applying pseudorandom continuous support surface rotations. System identification and parameter estimation were used to describe the balance behavior with the IC model in the frequency domain. The IC model with the estimated parameters from human experiments was implemented in Simulink for computer simulations including noise in the time domain and robot experiments using the humanoid robot PostuRob II. Again, system identification and parameter estimation were used to describe the simulated balance behavior. Time series, Frequency Response Functions, and estimated parameters from human experiments, computer simulations, and robot experiments were compared with each other. The computer simulations showed similar balance behavior and estimated control parameters compared to the human experiments, in the time and frequency domain. Also, the IC model was able to control the humanoid robot by keeping it upright, but showed small differences compared to the human experiments in the time and frequency domain, especially at high frequencies. We conclude that the IC model, a descriptive model in the frequency domain, can imitate human balance behavior also in the time domain, both in computer simulations with added noise and real world situations with a humanoid robot. This provides further evidence that the IC model is a valid description of human balance control.

#### Edited by:

Manish Sreenivasa, Universität Heidelberg, Germany

#### Reviewed by:

Robert Edward Kearney, McGill University, Canada Amy R. Wu, École Polytechnique Fédérale de Lausanne, Switzerland

> \*Correspondence: Jantsje H. Pasma j.h.pasma@tudelft.nl

Received: 23 August 2017 Accepted: 26 February 2018 Published: 20 March 2018

#### Citation:

Pasma JH, Assländer L, van Kordelaar J, de Kam D, Mergner T and Schouten AC (2018) Evidence in Support of the Independent Channel Model Describing the Sensorimotor Control of Human Stance Using a Humanoid Robot. Front. Comput. Neurosci. 12:13. doi: 10.3389/fncom.2018.00013

Keywords: balance control model, system identification, parameter estimation, robotics, human balance control

## INTRODUCTION

Human balance control helps us to keep our body in an upright position during daily life activities. In human balance control several systems are involved, like the sensory systems, the nervous system, and the muscles, which interact continuously with each other (Horak, 1997). Visual, proprioceptive (muscle spindles and Golgi tendon organs) and vestibular cues are integrated by the nervous system to obtain body orientation with respect to the visual scene, the support surface and gravitoinertial space, respectively. The nervous system integrates these sensory cues to generate a desired torque signal realized by the muscles. This torque results in a corrective movement to bring the body toward the desired upright position. The body position and velocity thus changes and the new body position and velocity are again sensed by the sensory systems. Thus, balance control can be described as a closed loop control system (Collins and De Luca, 1993; Peterka, 2002; van der Kooij et al., 2005).

To describe and understand the interaction between the underlying systems, human balance control models are useful (Engelhart et al., 2014). The Independent Channel (IC) model is a frequently used linear parametric model describing the interaction between the underlying systems during stance in a closed loop (Peterka, 2002). In this model the human body is modeled as a single inverted pendulum and each sensory system is modeled as a separate feedback channel with a weighting factor, which reflects the contribution of each sensory system during stance. Peterka (2002) quantified the changes in sensory contributions depending on the balancing situation, showing that changes in sensory contributions, referred to as sensory reweighting, plays an important role in human balance control.

The IC model is a simple descriptive model of the balance behavior and is formulated by a transfer function in the frequency domain, which allows easy and fast implementation of parameter estimation (Schoukens et al., 2004). To describe dynamic balance behavior, a non-parametric approach in the frequency domain can be used in combination with a continuous periodic perturbation with specific frequency content for system identification (Johansson and Magnusson, 1991; van der Kooij et al., 2005). Fitting the frequency domain model on the measured balance behavior then provides a limited set of physiologically interpretable parameters describing the underlying systems (Peterka, 2002; Kiemel et al., 2011). Theoretically, however, frequency domain models are a global description over the whole frequency range and may show small imperfections at specific frequencies. They may even include unstable subsystems in the fitting procedure. Furthermore, the IC model is used on averaged data and therefore in approximately noise free situations. As noise is inherent in human balance control, the IC model may not be able to stabilize this noisy system. In addition, the IC model may miss some essential details, as it is a simplified representation of human balance control and the human body is modeled as a linearized inverted pendulum (Peterka, 2003). Thus, the IC model may not always be a valid representation of the human balance behavior in real world situations.

In this study, we evaluated the validity of the IC model (i.e., a frequency domain model) by testing the model in time domain simulations with added noise and in a real world environment with the humanoid robot PostuRob II (Hettich et al., 2014) to show the functionality of a frequency domain model in the time domain and in real world situations. The robot has human-like anthropometrics, pneumatic muscle actuation and noisy and inaccurate sensors. The human IC model was used to control the robot, i.e., to generate a torque command based on the weighted sensory information. Comparing the robot's balance behavior to human balance behavior in similar experimental conditions is an important validation of the model. The real world environment may provide additional insight into human balance control and the robustness of the model.

## MATERIALS AND METHODS

### Independent Channel Model

The IC model (see **Figure 1**) describes the aforementioned process of balance control in the form of a simplified descriptive linear model. The model consists of a single inverted pendulum (body dynamics: BD) controlled by a feedback mechanism with a PD controller (neural controller: NC) and a time delay (TD). The sensory integration mechanism consists of a weighted sum of the sensory contributions, where the weights always sum to unity (Peterka, 2002). The first contribution is the relative orientation of the body to the feet (BF), sensed by the proprioceptive system (Wp). The proprioceptive signal BF refers here to an abstract internal representation of body orientation with respect to the feet instead of the ankle joint angle itself (Peterka, 2002; Mergner, 2010). The second is the body orientation with respect to the space vertical (BS), sensed by the vestibular system (Wves). The third is the visual surround orientation relative to the body (VB), sensed by the visual system (Wvis). The weighted signals are summed with a low-pass filtered positive force feedback (FF; Peterka, 2003), which accounts for a relatively good compensation of body lean at low frequencies. Together they provide an error signal as feedback into the PD controller (see **Figure 1**). In the time delay (TD) all delays in the loop are lumped, including muscle activation, neural delays, and processing time.

### Study Design

To validate the IC model, three steps were performed (**Figure 2A**). First, human balance behavior of healthy participants was obtained in human experiments using support surface rotations in space (SS) and body-in-space sway (BS) measurements (**Figure 2B**). SS rotations evoke body sway through changes of the body angle relative to the feet (BF) sensed by proprioception and the resulting sensory conflict with the body angle in space (BS) sensed by the vestibular system and by vision in a stationary visual surround (**Figure 2B**). The human balance behavior was analyzed in the frequency domain using system identification techniques. Parameter estimation was performed using the IC model (Peterka, 2002) resulting in

FIGURE 1 | Schematic representation of the human balance control (adapted from Peterka, 2003). The human body is represented by an inverted pendulum (body dynamics, BD) controlled by the neuromuscular controller producing a torque (T). The neuromuscular controller consists of the neural controller (NC), represented by a PD controller, a lumped time delay (TD) and the sensory feedback "channels" with their weighting factors [i.e., for proprioception (Wp), vision (Wvis), and vestibular system (Wves)] and force feedback (FF). The external perturbation is a support surface (SS) rotation around the ankle joint axis. It changes the orientation of the feet in space relative to the horizontal (FS) and therefore changes the body angle with respect to the feet (BF, sensed by proprioception). This results in a conflict between the proprioceptive information and the information sensed by the vestibular system [i.e., body angle in space relative to the gravitational vertical (BS)] and vision [i.e., the visual surround orientation relative to the body (VB)] and evokes a change of the body angle in space (BS).

experiments (2) using the same perturbation signal as in the human experiment. BS(t) obtained from the simulations and robot experiments was again used for system identification and parameter estimation and for describing the balance behavior in terms of FRFs and estimated parameters. Finally, balance behavior obtained from computer simulations and robot experiments were compared with balance behavior obtained from the human experiments, using the time series, FRFs, and estimated parameters.

parameters describing the underlying human balance control system.

Secondly, the time domain computer simulations using the IC model with added noise and the estimated parameters from human experiments were performed. Thirdly, the model was implemented in the humanoid robot to test the IC model under real world conditions. To compare the human balance behavior in the time domain, the same perturbations as in the human experiments were used. Again, system identification and parameter estimation were used to describe the balance behavior and to estimate the model parameters representing the balance behavior obtained from the computer simulations and robot experiments. Finally, the balance behavior obtained from the computer simulations and robot experiments were compared with the balance behavior obtained in the human experiments.

#### Human Experiments

#### Participants

Seven healthy young participants (5 males, 2 females, age 26.1 ± 2.1 years, height: 1.79 ± 0.09 m, mass: 77.7 ± 10.8 kg) were included in the study. The participants gave written informed consent prior to participation. The protocol was approved by the medical ethics committee of the Medical Spectrum Twente, Enschede, the Netherlands and was in accordance with the Declaration of Helsinki.

#### Apparatus

A Bilateral Ankle Perturbator (BAP) (Forcelink B.V., Culemborg, the Netherlands) was used to apply support surface (SS) rotations around the ankle joint axis (Schouten et al., 2011). The actual angles of rotation of the SS on the BAP were measured.

The body kinematics of the lower and upper body were measured in anterior-posterior direction using two drawwire potentiometers (Sentech SP2, Celesco, Chatsworth, CA, United States) by connecting them to the participant's trunk and hip. Together with the SS rotation, the body kinematics were measured using a Matlab interface with a sample frequency of 1,000 Hz.

#### Perturbation Signal

A pseudorandom ternary sequence (PRTS) with 80 states and a time increment of 0.25 s was generated, resulting in a signal with a period of 20 s (Davies, 1970; Peterka, 2002). This signal was used as SS angular velocity of both the left and right SS simultaneously. Integration of this signal provided the perturbation signal of the SS rotation with a wide spectral bandwidth where only the odd harmonics contain signal power (Peterka, 2002; **Figure 3**). The even harmonics were not excited by the perturbation and were used to detect nonlinearities in the output (Pintelon and Schoukens, 2001). Each trial consisted of six complete repetitions of the perturbation signal resulting in a trial duration of 2 min. The signal was applied with peak-to-peak amplitudes of 0.5 and 1 degrees.

#### Procedure

During all experiments the participants stood on the BAP wearing socks. The participants were instructed to stand with their arms crossed at chest level and to keep both feet on the support surface. The perturbations were applied at both amplitudes during eyes open and eyes closed conditions, resulting in four trials of 2 min each. Before recording, the participants were given sufficient time to familiarize with the perturbation (∼10 s). The participants wore a safety harness to prevent falling, which did not constrain normal body sway and did not provide support or body orientation information.

#### Preprocessing

Data analysis was performed with Matlab (The Mathworks, Natick, MA, United States). Leg and hip angles were calculated using the potentiometer data and the attachment height of the potentiometers, resulting in the segment angle of the legs relative to the vertical and the joint angle of the trunk relative to the legs. The body-in-space sway (BS), taken as the angular displacement of the whole body Center of Mass without feet (CoM) relative to the vertical, was calculated using the leg and hip angles and body anthropometrics obtained from Winter et al. (1990). The time series of the body sway and the actual SS rotation were used for further analysis.

#### System Identification and Parameter Estimation

The time series of the body sway and SS rotation were segmented into six data blocks of 20 s (i.e., the length of the perturbation signal) and were transformed to the frequency domain. The periodic part of the frequency coefficients was calculated by averaging the frequency coefficients across the six data blocks. The Cross Spectral Density (CSD) of the body sway and the perturbation and the Power Spectral Density (PSD) of the perturbation were calculated.

To test for nonlinearities in the body sway as response to the perturbation, the PSD of the body sway on the odd and even harmonics were calculated. Effect of nonlinearities within each tested condition was quantified by the percentage of total body sway power on the even harmonics, where the perturbation had no power. Thus, a higher percentage indicates excitation of body sway at frequencies which were not excited by the perturbation, and therefore more non-linearities. A low percentage of nonlinearities is a prerequisite for the application of linear system identification techniques.

Next, the Frequency Response Functions (FRFs) representing the sensitivity function of the SS rotation to the body sway were estimated by dividing the CSD of the body sway and the perturbation by the PSD of the perturbation (Equation 1; Peterka, 2002; van der Kooij et al., 2005). Only the excited frequencies (i.e., odd harmonics) were analyzed. In case the amount of nonlinearities is low, the response on the excited frequencies (i.e., odd harmonics) represents the total balance behavior. The function is given by

$$H\_{\rm exp}(f) = \Phi\_{\rm SS,BS}(f) \cdot \left[\Phi\_{\rm SS,SS}(f)\right]^{-1} \tag{1}$$

where 8SS,BS and ΦSS,SS represent the CSD and PSD, respectively. The magnitude and phase represent the relation between the perturbation and body sway per frequency in terms of amplitude ratio and timing, respectively.

The coherence reflects the amount of body sway evoked by the perturbation on the excited frequencies, i.e., the linear response, and decreases with noise and nonlinearities (Pintelon and Schoukens, 2001). The coherence is given by

$$\left|\gamma\_{\text{SS},BS}^2(f) = \left|\Phi\_{\text{SS},BS}(f)\right|^2 [\Phi\_{\text{SS},SS}(f)\,\Phi\_{\text{BS},BS}(f)]^{-1} \tag{2}$$

where 8BS,BS represents the PSD of the body sway. The coherence varies between 0 and 1, with a coherence close to one reflecting a good signal to noise ratio and linear behavior.

The FRFs and coherences were obtained with nonparametric analysis and were averaged across the participants for each condition resulting in four FRFs and coherences (i.e., for the 0.5 and 1 degrees perturbation amplitudes with eyes open and eyes closed), which were used for further validation.

The IC model was fitted on the estimated balance behavior during each condition, represented by the FRFs averaged across participants, using the theoretical transfer function of the IC model, as presented in Equation (3), to obtain parameters describing the balance behavior (Peterka, 2003). To characterize the postural effects evoked by the SS rotation around the ankle axis, the proprioceptive weight (Wp) was estimated from

$$H\_{\rm est}(f, p) = \frac{BS(f)}{SS(f)} = \frac{W\_p \cdot NC \cdot TD \cdot BD}{1 - FF \cdot NC \cdot TD + NC \cdot TD \cdot BD} \tag{3}$$

where NC represents the neural controller, TD the time delay, BD the body dynamics and FF the force feedback. f represents the frequency vector and p the model parameters, namely the mass (m), CoM height (h), moment of inertia (J), proprioceptive weight (Wp), the reflexive stiffness (KP), the reflexive damping (KD), time delay (τD), force feedback time constant (τF) and force feedback gain (KF) (**Figure 1**). As the sum of the weights equals one, in case of eyes closed, Wvis is zero and Wves can be calculated by 1 – Wprop. With the eyes open, the sum of Wvis and Wves equals 1 – Wprop (see Peterka, 2002), where visual and vestibular weight cannot be separated mathematically.

The body mass, CoM height and moment of inertia were used as fixed parameters. The CoM height and moment of inertia were calculated using the method of Winter et al. (1990). The model was fitted on the FRFs (0.05–2.05 Hz) of the averaged human experimental data using a nonlinear least-square fit (Matlab function: lsqnonlin) by minimizing the sum squared error (E), equation (5), in which more more weight was given to the low frequencies and the frequencies with higher coherence (Equation 4).

$$\kappa(f, p) = \sqrt{\frac{\chi\_{\text{SS}, \text{BS}}^2(f)}{1 + f}} \cdot \left| \log \left( \frac{H\_{\text{exp}}(f)}{H\_{\text{est}}(f, p)} \right) \right| \tag{4}$$

$$E = \frac{1}{N} \varepsilon(f, p)^T \varepsilon(f, p) \tag{5}$$

γ 2 SS,BS represents the averaged coherence between the SS rotation and body sway, Hexp the averaged experimental or simulated sensitivity function, Hest the estimated sensitivity function based on the estimated model parameters (p) and N the number of frequencies.

The quality of the model fit was represented by the Variance Accounted For (VAF) (Equation 6) identifying how well the model describes the observed time series averaged across data blocks and participants. The VAF is given in percentage; 100% indicates that the model accounts fully for the experimental data. A lower VAF indicates deviations between the model and the time series averaged across data blocks and participants.

$$VAF = 1 - \frac{\sum\_{t=0.01}^{T} \left| BS\_{\text{exp},t} - BS\_{\text{est},t} \right|^2}{\sum\_{t=0.01}^{T} \left| BS\_{\text{est},t} \right|^2} \* 100\% \tag{6}$$

where BSexp,t represents the body sway measured in the experiment and BSest,<sup>t</sup> represents the body sway obtained from simulations with the estimated model parameters.

The Standard Error of the Mean (SEM) of each parameter represents the sensitivity of the error (ε, Equation 4) to changes in parameters and was calculated using the diagonal of the estimated covariance matrix (P) obtained during the parameter estimation procedure (Ljung, 1999; Equation 7).

$$\stackrel{\frown}{P} = E(\prescript{T}{}{J})^{-1} \tag{7}$$

in which J is the Jacobian (matrix of partial derivatives of each parameter's prediction error ε given in Equation 4) and E the sum squared error. Since the parameters are estimated using the averaged experimental FRFs, the SEM reflects the precision of the estimated parameters and not the amount of variation of the model parameters.

## Computer Simulations

The IC model as described in section Independent Channel Model and **Figure 1** was implemented in Simulink, Matlab (The Mathworks, Natick, MA, United States) with added pink noise to mimic sensory and motor noise (van der Kooij and Peterka, 2011). The human body was modeled as a single inverted pendulum and all parameters (Equation 3, **Figure 1**) were set to the values found in the human experiments. The same perturbation signal (section Perturbation Signal) and analyses (section System Identification and Parameter Estimation) as used in the human experiments were applied resulting in time series, FRFs, and estimated parameters describing the balance behavior simulated by the computer.

## Robot Experiments

#### PostuRob II

To imitate human balance control in a real world situation, the humanoid robot PostuRob II was used (Hettich et al., 2014). The robot was constructed with human-like anthropometric parameters (mass: 51 kg, CoM height above feet: 0.97 m, moment of inertia: 40 kgm<sup>2</sup> ) consisting of trunk, leg, and feet segments interconnected with two actuated ankle and hip joints (hip joints were fixed during this study).

The sensory signals of the vestibular system, joint torque, joint angular position, and velocity were measured using mechatronic sensors. The technical analog for the vestibular system are accelerometers and gyrometers, where the signals are processed to provide body angular velocity and angle with respect to the gravitational vertical, and linear acceleration in the sagittal plane (Mergner et al., 2009). In the current study only the angular orientation with respect to gravity was used in the robot experiments as the IC model only uses this signal.

Torque commands were sent to the robot to actuate artificial pneumatic "muscles" at the ankle joints (Type MAS20, FESTO AG & Co.KG, Esslingen, Germany). An inner torque control loop ensured that the actual torque matches the torque commands. A real time PC with Simulink (Real-Time Windows Target, The Mathworks Inc., Natick, USA) was used as the control module, running the compiled IC model.

#### Apparatus

A custom-built motion platform (Hexapod, Stuart principle; Mergner et al., 2003) was used to apply SS rotations around the ankle axis. The same perturbation signals as in the human experiments were used (see section Perturbation Signal). The body kinematics of the lower and upper body of the robot were measured in anterior-posterior direction using an optical motion capture system with two active markers attached to the robot's hip and shoulder, respectively (Optotrak 3020; Waterloo, Canada). The body kinematics together with the actual angle of rotation were measured using custom made software written in LabView (National Instruments, Austin, USA) with a sample frequency of 100 Hz and were stored for further analysis.

#### Procedure

The same procedure was performed with the PostuRob II as in the human experiments; the experiments consisted of four 2-min long trials with SS rotations and different perturbation amplitudes. Before each trial the parameters estimated from the human experimental data for each condition were implemented in the robot. This step was necessary, as the IC model requires a different set of model parameters for each condition. The neural controller parameters were corrected to account for the difference in mass and CoM height between human participants and the robot. Note that the robot had no visual sensor. Instead, the artificial vestibular sensor was also used for the robot experiments to mimic eyes open conditions, where only the model parameters were adjusted according to the changes identified in the human participants. Data were analyzed according to the procedures described in sections Preprocessing and System Identification and Parameter Estimation.

## RESULTS

#### Time Series

**Figure 4** shows the time series of the averaged body sway for the human experiments, computer simulations, and robot experiments for each condition. Sway responses of computer simulations and robot experiments followed the general pattern of the human sway responses. The VAFs between the human experiments and the computer simulations ("VAFS") were in the range of 94.1–98.7% for all conditions and in the range of 62.9– 79.0% for the robot experiments ("VAFR"; **Table 1**). For both the computer simulations and the robot experiments, the VAF tended to increase with increasing perturbation amplitude and was higher in eyes closed, as compared to eyes open conditions. Computer simulations and robot experiments were robust with respect to the noise and the inaccuracies, and control stability was maintained throughout all conditions.

## Frequency Response Functions

**Figure 5** shows the FRFs with the corresponding coherence for each condition of the human experiments, together with those obtained from the computer simulations and the robot experiments. In general, the pattern of the FRFs and the changes across conditions were similar in humans, computer simulations, and robot experiments. Simulations and the robot experiments showed some differences in the magnitude as

deviation (shaded), the computer simulations (dash-dotted gray lines), and the robot experiments (dotted gray lines).

TABLE 1 | Differences between human experiments and computer simulations or robot experiments per experimental condition given for the time series and estimated parameters.


Data are presented for the four conditions with perturbation amplitude of 0.5 and 1 degrees peak-to-peak and eyes open (EO) versus eyes closed (EC).

compared to humans with the largest difference between robot and human experiments at about 0.7 Hz. The coherence was considerably larger in simulations and the robot as compared to humans, indicating differences in the noise properties. The percentage power of the body sway on the even harmonics varied between 1.8 and 7.4% of total body sway power in the human experiments and therefore did not show strong nonlinearities.

#### Estimated Parameters

The quality of the model fits was represented in the time domain by the VAF. For the human experimental data the VAF varied between 98.0 and 99.1%, for the simulated data between 91.9 and 99.0% and for the robot experiments between 93.4 and 98.3%, indicating only small deviations between the measured (or simulated) data and the fitted model.

**Figure 6** shows the estimated parameters with the corresponding SEM of the human experiments, the computer simulations and the robot experiments. The estimated SEM values indicating the sensitivity of the fitting error to changes in the given parameter were low for most parameters. Exceptional large SEM values stand out in the simulations for the reflexive stiffness (only eyes closed 0.5 degrees) and the force feedback time constants, and in the robot experiments for the time delays, the proprioceptive weight, and the force feedback gain values obtained for the small perturbation amplitudes (0.5 degrees).

The mean relative differences between the parameters is presented in **Table 1**. The differences between the parameters of the computer simulations and the parameters of the human

experiments were in the range of 10.7–16.9%. The differences between the parameters of the robot experiments and the human experiments were in the range of 13.6–27.3%. Again, the differences decrease with increasing perturbation amplitude and with closing the eyes.

correspondingly for computer simulations (dash-dotted gray lines) and robot experiments (dotted gray lines).

Overall, the estimated parameters of the human experiments, the computer simulations, and robot experiments are comparable for all conditions. Clear differences were found in the force feedback time constant and gain, which also show a large SEM. Furthermore, the reflexive stiffness, and reflexive damping showed a larger difference between the human experiments and the robot experiments compared to the other parameters.

## DISCUSSION

In this study we validated the IC model, a commonly used descriptive model in the frequency domain, in the time domain using computer simulations with added noise and in the real world using robot experiments. The results show that both the computer simulations and robot experiments can reproduce human balance behavior, where computer simulations described the human sway responses better compared to the robot. The model simulations showed that the IC model is stable in the time domain with added noise, which adds an important aspect to the descriptive nature of this frequency domain model. Furthermore, the robot, controlled by the IC model, maintained the desired upright position, which showed that the IC model is robust enough to deal with the real-world properties of the robot (i.e., human-like anthropometrics, noisy and inaccurate sensors, and mechanical dead zones).

## Experimental Balance Behavior

The experimental balance behavior was estimated using a linear approach. The low percentage power of the body sway of the healthy participants at even harmonic frequencies shows that no considerable nonlinearities that are effective across frequencies were found within the steady state of one condition. The absence of such nonlinearities allowed the use of a linear approach by only analyzing the excited frequencies (i.e., odd harmonics).

A linear model was fitted for each condition of the experimental data to describe the balance behavior. The high VAF of the model fits indicate that the model explains the data well. The estimated parameters obtained from human experiments are comparable with previous studies and show sensory reweighting, i.e., a change in the use of sensory information (e.g., decrease in proprioceptive weight) with changing perturbation amplitude and sensory condition (i.e., with increasing perturbation amplitude and opening the eyes; Peterka, 2002; Cenciarini and Peterka, 2006; Pasma et al., 2015). Also an increase in reflexive stiffness was found with increasing perturbation amplitude, which is in agreement with previous studies (Peterka, 2002).

## Replication of Human Balance Behavior With Computer Simulations

The results revealed small differences between the time series of the human experiments and those of the computer simulations. Small differences between time series, FRFs, and estimated parameters were expected, as pink noise was implemented in the computer simulations. Furthermore, we observed that the IC model described the human responses in the high frequency range not as accurately as in the low and mid frequency

ranges. This can be explained by the used prediction error function, which gives more weight to the frequencies with a high coherence.

values <sup>&</sup>gt; 5.0 <sup>×</sup> <sup>10</sup>−<sup>4</sup>

rad/Nm.

These results provide evidence that the IC model, a frequency domain model, is able and therefore valid to be used to control a system in the time domain. The stable computer simulations showed that the frequency domain model does not represent an unstable subsystem and is able to tolerate physiologically plausible noise without loss of balance.

Also, the estimated parameters of the human experiments and the computer simulations are comparable. An exception is the force feedback time constant, which mainly affects balance behavior at low frequencies. The large SEM value indicates that the estimate is less reliable. Due to the length of the perturbation signal (i.e., 20 s), the perturbation contained little information in the low frequency range, resulting in the observed low reliability in the estimates. Notably, however, this parameter had only small influences on the time series and FRFs.

## Replication of Human Balance Behavior With Robot Experiments

The main purpose of the study was to show the functionality of the IC model in real world situations using robot experiments. Similar approaches have been used to test other balance control concepts, like the Disturbance Estimation and Compensation concept or the Eigen movement concept (Hettich et al., 2014; Alexandrov et al., 2017). Here, the IC model was able to control the robot in the time domain when adjusting the estimated neural controller parameters (i.e., the reflexive stiffness and damping) to the mass and weight of the robot. The robot's control was stable across conditions and in the presence of manually applied pushes (results not shown). In response to the pushes, the robot showed a compliant behavior (relatively small resistance to the push), which is related to the low loop gain used in the IC model and an important characteristic of healthy human balancing.

The differences between the robot experiments and the human experiments in the time series, the FRFs, and the estimated parameters were larger than those between computer simulations and human experiments. One likely reason is that the mechanical components of the robot introduce additional inaccuracies due to dead zones, friction, etc., which remained unconsidered in the robot's control model. The ability of the IC model to stabilize the robot despite these unconsidered effects suggests a considerable robustness of the control mechanism. This robustness is also a major aspect in human balance control, which lends further support to the evidence that human balance control can be explained by such a simple feedback mechanism as described by the IC model.

A difference between the robot experiments and the human experiments concerned a peak around 0.7 Hz in the magnitude of the FRFs. The peak decreased with increasing perturbation amplitude and increasing torque level. Manual changes in the model parameters and additional experiments (not shown in the results) suggest that this peak might be due to the activation dynamics of the robot's actuators in terms of a resonance peak. As the peak decreased with increasing perturbation amplitude, this suggests a nonlinear behavior of the robot's actuation.

The peak around 0.7 Hz also may explain the higher reflexive stiffness and lower reflexive damping estimates for the robot experiment data as compared to the human experiment data. Furthermore, a clear difference was found for the force feedback gain. As already mentioned above, the force feedback estimates showed a high SEM value and primarily affects the low frequency range, where the differences between the FRFs and between the estimations of the force feedback parameters were largest.

## Decrease in Differences With Increasing Perturbation Amplitude

The results show that, overall, the differences between the human experiments and the computer simulations on the one hand and the human experiments and the robot experiments on the other hand decreased with increasing perturbation amplitude, as shown by the VAFs and the relative mean difference of the parameters shown in **Table 1**. In the computer simulations, these differences can be attributed mainly to the noise injected into the model. The amplitude of the noise was kept constant across the conditions. This means that with a higher perturbation amplitude the noise had less influence on the time series (resulting in a better signal to noise ratio) and therefore had less influence on the FRFs and the estimated parameters. This may explain why the differences between the human experiments and the computer simulations became less with increasing perturbation amplitude and why the SEM of the estimated parameters became smaller.

The argument could also hold for the robot experiments if one assumes that with increasing perturbation amplitude, the effects of the sensory and motor noise became relatively smaller, and similarly also the effects of the activation dynamics and mechanical inaccuracies. The reduced difference with increasing perturbation amplitude and with closing the eyes suggests that the robot controlled by the IC model is able to reproduce human balance behavior.

#### Limitations

The IC model is a simplification of the human balance control, in which the human body is modeled as an inverted pendulum pivoting around the ankle joint axis and the equations of motion are linearized. The model can be used to describe balance behavior in the frequency domain at a specific operating point as long as the balance conditions are not changed and the deviations from this point are small. Therefore, the model can only be used during steady state conditions, e.g., within one amplitude, which might also be possible with other models.

In case of larger perturbations, which result in larger deviations and also in rotation around the hip joints in addition to the rotation around the ankle joints, the IC model would miss essential details as suggested by studies which used balance control models that incorporate also the hip joints, modeling the human body as a double inverted pendulum (Qu and Nussbaum, 2012; Boonstra et al., 2013; Hettich et al., 2014; Engelhart et al., 2015; Hwang et al., 2016). These models are able to identify both the control of the upper and lower body separately and the intersegmental coupling.

The somewhat lower coherence of the human experimental data likely originates from noise and variability present in the measured time series given the low amount of nonlinearities in the system. The coherence is plausible since the sway amplitude evoked by the small perturbation was comparable to the sway amplitude not evoked by the perturbation (i.e., spontaneous sway in quiet stance). The coherence values are also comparable to other studies (Pasma et al., 2012, 2015; Boonstra et al., 2013). Despite the somewhat lower coherence in the human experimental data, the sensitivity function described the linear balance behavior in the humans rather well and can be explained well by the IC model, as shown by a high variance accounted for.

Intrinsic dynamics of the passive tissue and tendon structures were neglected in the formulation of the IC model. This simplification was implemented based on earlier studies suggesting that the intrinsic dynamics contribute only about 10% to the overall torque generated by the active muscle contractions (Peterka, 2002; Maurer et al., 2005; Cenciarini and Peterka, 2006; Assländer et al., 2015; Vlutters et al., 2015; Wiesmeier et al., 2015). Furthermore, previous studies showed that it is difficult to experimentally determine the intrinsic dynamics during balance control (Peterka, 2002; Pasma et al., 2015; Engelhart et al., 2016). As our model was able to explain the obtained sensitivity functions well without the intrinsic dynamics, we decided to dismiss them here.

## CONCLUSIONS

This study showed that the IC model, a descriptive linear model in the frequency domain, is able to imitate human balance behavior in both the time and frequency domain, this both in computer simulations and robot experiments. Therefore, the IC model represents a good descriptor of human balance control. The capability to tolerate noise and keep the robot in an upright position, while being externally perturbed, indicates that the IC model is robust in the time domain and in a real world situation.

The IC model may help in the future to obtain further insights into human balance control and to develop better and more human-like balance control mechanisms for robotic assistive devices such as exoskeletons. Furthermore, the robot implementation is useful for educational purposes, as it opens the possibility to experience the functionality of the IC model in a direct interaction with the human-like behaving robot. It remains to be shown to what extent the IC model can help to detect and classify changes underlying impaired balance control.

#### AUTHOR CONTRIBUTIONS

DdK and JvK: performed the human experiment and collected the data; JP and LA: performed the robot experiments; JP: performed the computer simulations and analyzed the data. All

#### REFERENCES


authors contributed to the interpretation of the data. JP and LA: made a draft of the manuscript; DdK, JvK, TM, and AS: critically revised the manuscript. All authors approved the final version of the manuscript for submission.

#### FUNDING

This research is supported by the European Union FP7-ICT project 610454 EMBalance and by the Dutch Technology Foundation STW (NeuroSIPE #10737 BalRoom), which is part of the Netherlands Organization for Scientific Research (NWO), and which is partly funded by the Ministry of Economic Affairs.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The handling editor is currently editing a Research Topic with one of the authors AS, and confirms the absence of any other collaboration.

Copyright © 2018 Pasma, Assländer, van Kordelaar, de Kam, Mergner and Schouten. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# **A Functional Subnetwork Approach to Designing Synthetic Nervous Systems That Control Legged Robot Locomotion**

*Nicholas S. Szczecinski <sup>1</sup> \*, Alexander J. Hunt <sup>2</sup> and Roger D. Quinn<sup>1</sup>*

*<sup>1</sup>Biologically Inspired Robotics Laboratory, Department of Mechanical and Aerospace Engineering, Case Western Reserve University, Cleveland, OH, United States, <sup>2</sup>Department of Mechanical and Materials Engineering, Portland State University, Portland, OR, United States*

A dynamical model of an animal's nervous system, or synthetic nervous system (SNS), is a potentially transformational control method. Due to increasingly detailed data on the connectivity and dynamics of both mammalian and insect nervous systems, controlling a legged robot with an SNS is largely a problem of parameter tuning. Our approach to this problem is to design functional subnetworks that perform specific operations, and then assemble them into larger models of the nervous system. In this paper, we present networks that perform addition, subtraction, multiplication, division, differentiation, and integration of incoming signals. Parameters are set within each subnetwork to produce the desired output by utilizing the operating range of neural activity, *R*, the gain of the operation, *k*, and bounds based on biological values. The assembly of large networks from functional subnetworks underpins our recent results with MantisBot.

#### *Edited by:*

*Manish Sreenivasa, Heidelberg University, Germany*

#### *Reviewed by:*

*Shinya Aoi, Kyoto University, Japan Sander Bohte, Centrum Wiskunde & Informatica, Netherlands Elisa Donati, Sant'Anna School of Advanced Studies, Italy*

#### *\*Correspondence:*

*Nicholas S. Szczecinski nicholas.szczecinski@case.edu*

*Received: 30 December 2016 Accepted: 17 July 2017 Published: 09 August 2017*

#### *Citation:*

*Szczecinski NS, Hunt AJ and Quinn RD (2017) A Functional Subnetwork Approach to Designing Synthetic Nervous Systems That Control Legged Robot Locomotion. Front. Neurorobot. 11:37. doi: 10.3389/fnbot.2017.00037* **Keywords: synthetic nervous system, design tools, functional subnetworks, leaky integrator, arithmetic, differentiator, memory**

## **1. INTRODUCTION**

The development of robotic control that can closely match the dexterity and adaptability found in the animal kingdom has so far remained elusive. This is because the control of locomotion is a complex process controlled by dynamic systems which are not fully understood. However, recent advances in neural imaging and recording has lead to an increase in the abundance and detail of our knowledge of how an animal's nervous system controls its body within the context of its environment (for a recent review, see Buschmann et al. (2015)).

These advances have lead to an explosion of bio-inspired robotic systems in recent years (for a review, see Ijspeert (2014)). These models can be broadly categorized into a range of template and anchor models. In a template model, biological principles are abstracted, such using as a springloaded inverted pendulum (SLIP) model to investigate bipedal locomotion (Blickhan, 1989) or using Whegs to investigate insect locomotion (Allen et al., 2003; Schroer et al., 2004). These models seek to explain how specific characteristics of animal locomotion lead to desired behaviors, or they exploit certain principles of animal locomotion for more agile robotic systems. Anchor models, in contrast, seek to directly mimic particular mechanical or control mechanisms from animals, in order to understand how they function. Robots such as Pleurobot (Karakasiliotis et al., 2016), Puppy (Hunt et al., 2017), MantisBot (Szczecinski and Quinn, 2017; Szczecinski et al., 2017a), and others are relevant anchor models, because they seek to use highly articulated robots with central pattern generator controllers to understand how specific animals are capable of providing adaptable locomotion with their unique morphology and physical constraints.

The template versus anchor model distinction is not limited to physical models; it can also be applied to control systems. The majority of robotic controllers so far have been template models, either mathematical abstractions of neural systems, or black box artificial neural networks. This is because effective tools for setting parameters in more realistic, dynamic neural models to produce reliable behavior in a robotic system do not yet exist. In spite of growing knowledge about the neural connectivity that underlies locomotion control, detailed data for tuning these systems (neural time constants, ion channel conductivities, synaptic conductivities, etc.) remain largely unavailable, requiring the modeler or engineer to tune these parameter values. However, this is an inherently difficult task because there are many parameters to be tuned in a model, and likely many different parameter combinations that lead to indistinguishable performance (Prinz et al., 2004; Marder and Taylor, 2011). Thus, the emphasis in choosing parameter values should not be on selecting the singular "correct" values, but rather sufficiently "effective" values. In this work, we tune parameter values in functional subnetworks for addition, subtraction, multiplication, division, differentiation, and integration of incoming signals and use analytical techniques to identify *constraints* on the parameter values that must be met for the intended calculations to occur. Larger networks can then be assembled from these subnetworks with no additional tuning (Szczecinski and Quinn, 2017; Szczecinski et al., 2017a). In this manuscript, "tuning" refers to selecting the static parameter values for a network; "learning" refers to changing the parameter values while the model performs a task, either in simulation or in a robot, based on its performance; and "adapting" refers to a model qualitatively changing its behavior (e.g., walking more slowly), either with or without "learning."

Neuromechanical models may be tuned in a supervised or unsupervised way. A supervised tuning method adjusts the parameters of the model until the model replicates animal data. This includes tuning the model by hand (Daun-Gruhn and Tóth, 2010; Szczecinski et al., 2014; Markin et al., 2016), which is a time-consuming and imprecise process. Such imprecision may be acceptable in simulation studies, but provides many difficulties for robots that must interact with real environments. Techniques do exist for tuning controllers based on animal locomotion data (Schilling et al., 2013; Hunt et al., 2015b, 2017; Karakasiliotis et al., 2016). However, collecting kinematic and dynamic data from animals is time-consuming and expensive, and once collected, must be further processed to scale the dynamics of the animal to the robot (Karakasiliotis et al., 2016; Hunt et al., 2017). In addition, using cross-individual average values for tuning dynamical neural models may fail in many cases, because the average may not represent any one individual (Golowasch et al., 2002; Marder and Taylor, 2011). Large amounts of animal data may be used to tune a control network of abstracted artificial neural networks (Schilling et al., 2013). Methods like back-propagation can be used to adjust synaptic weights in the network until it captures the animal data arbitrarily closely, if it has enough connections (Trappenberg, 2009). However, because the control network

is abstracted, so are the biological insights gained from the model.

Unsupervised tuning methods, in contrast, tune the model based on how well the model accomplishes a task, such as navigating toward a goal, without comparison to animal data. These methods frequently use genetic algorithms (GAs) (Beer and Gallagher, 1992; Haferlach et al., 2007; Agmon and Beer, 2013; Izquierdo and Beer, 2013) or reservoir computing (RC) (Dasgupta et al., 2015) to test many different networks and parameter values, based on a simulated agent's performance. GAs can be effective at finding networks that perform specific operations, such as oscillating (Beer and Gallagher, 1992), navigating (Haferlach et al., 2007), or switching between foraging tasks (Agmon and Beer, 2013). However, this approach has some drawbacks. Specifically, the evolution process is slow, requiring the simulation of hundreds or thousands of parameter combinations (Agmon and Beer, 2013), which may take days without great computing power. The speed and likelihood of success can be increased by embedding functional subnetworks in the network (Pasemann et al., 2001; Haferlach et al., 2007), which may be identified by brute-force (Prinz et al., 2003), dynamical systems analysis (Hunt et al., 2017), or constraints on network connectivity and parameter values (Haferlach et al., 2007). In this paper, we analytically derive parameter constraints to eliminate the need for GAs altogether, and guarantee network performance.

RC methods simulate large "reservoirs" of randomly connected dynamical neuron models, and then use optimization methods to map reservoir activity to learned useful values. While this method can produce capable robotic controllers (Dasgupta et al., 2015), the final system is likely more complicated than is ultimately necessary, increasing its computational cost to implement. In addition, the final system is a black box, which does not provide any insights about nervous system function. The methods in this paper enable the direct assembly and tuning of dynamical networks without the need of large reservoirs of neurons.

This work analytically derives constraints that govern the behavior of synthetic nervous systems (SNSs) built from dynamical neural networks. These constraints were derived as a result of our previous network design work (Szczecinski et al., 2017b) and have enabled the rapid assembly and testing of our recent robot control networks (Szczecinski and Quinn, 2017; Szczecinski et al., 2017a). An SNS designer can apply these constraints to find parameter values needed for a functional network. Section 2 presents the neural and synaptic models and explains how the neural system encodes mechanical inputs and outputs. Section 3 derives two basic synapse types, "signal transmission" and "signal modulation," and uses them to derive constraints on synaptic parameters in networks performing addition, subtraction, multiplication, and division of two incoming signals. Section 4 derives constraints on neural and synaptic parameters in networks that differentiate and integrate incoming signals as a function of time. Results showing that the networks perform as intended are provided throughout the manuscript, and **Tables 1** and **2** summarize the design constraints. Finally, Sec. 6 explores how these techniques may be used to tune robot controllers and neuromechanical models of animals, and how they may be improved in the future.

#### **2. METHODS: MODELS AND APPROACH**

We model neurons as non-spiking Hodgkin–Huxley compartments (Cofer et al., 2010), the same basic model as used in continuous-time recurrent neural networks (Haferlach et al., 2007; Agmon and Beer, 2013). The leaky integrator dynamics capture the most basic behavior of neurons and allow more complex behaviors to be added with additional ion channels, if desired. This work is not concerned with the specifics on how action potentials are generated and have left out Hodgkin–Huxley sodium and potassium currents. The membrane voltage, *V*, may be seen as a proxy for the spiking frequency of a spiking neuron. *V* varies according to the differential equation

$$C\_m \frac{dV}{dt} = I\_{lank} + I\_{syn} + I\_{app} \tag{1}$$

where

$$I\_{leak} = G\_m \cdot (E\_r - V),\tag{2}$$

$$I\_{\rm syn} = \sum\_{i=1}^{n} \mathcal{G}\_{\rm s,i} \cdot (E\_{\rm s,i} - V), \tag{3}$$

and *Iapp* is an optional external stimulus. Equations (2) and (3) define the leak and synaptic currents, respectively. Both follow the same basic form of a conductance *G* multiplied by the difference between the current membrane voltage, *V*, and a constant reference voltage (i.e., reversal potential), *E*. *E<sup>r</sup>* is the resting potential of the neuron, and *C<sup>m</sup>* and *G<sup>m</sup>* are the capacitance and conductance of the cell membrane, respectively. Unless otherwise noted, all units in this paper are scaled to nA for current, mV for potentials, nF for capacitances, and *µ*S for conductances.

Neurons communicate via synapses. The conductance, *Gs,i* in equation (3), is a threshold linear function of the *i*th incoming (i.e., presynaptic) neuron's voltage. Synapses communicate via piecewise-linear functions described as

$$\mathcal{G}\_{s,i} = \begin{cases} 0, & \text{if } V\_{pre} < E\_{lo}, \\ \mathcal{g}\_{s,i} \cdot \frac{V\_{pre} - E\_{lo}}{E\_{hi} - E\_{lo}}, & \text{if } E\_{lo} < V\_{pre} < E\_{hi}, \\ \mathcal{g}\_{s,i}, & \text{if } V\_{pre} > E\_{hi}. \end{cases} \tag{4}$$

The parameters *gs,i*, *Elo*, and *Ehi* are constants representing the synapse's maximum conductance, its lower threshold, and its upper threshold, respectively. The relationship between the presynaptic neuron voltage, synaptic conductance, and postsynaptic neuron voltage is illustrated in **Figure 1A**.

We prefer this piecewise-linear representation better than a sigmoidal function for several reasons. First, the thresholds ensure that for low activations, synapses conduct exactly 0 current. This could represent a reduced model of a spiking neuron, which transmits no information while it is not spiking. Second, equation (4) contains no transcendental terms, facilitating analytical manipulation of the equations. A discontinuous system does complicate traditional gradient-based optimization methods, but this structure can be exploited to make these methods unnecessary. In the following sections, we show how networks of three or four neurons with synapses between them can be constructed and analytically

between mechanical and neural values. **(A)** Graphical representation of how synapses couple neural dynamics. Note that *R* is marked on the plot. **(B)** Enhanced version of the motor control network from Szczecinski et al. (2017b), showing how *R* relates mechanical and neural values. Mechanical values are drawn in red, and neural values are drawn in blue. Gray shaded boxes map from mechanical to neural values (i. and ii.), or from neural to mechanical values (iii.).

tuned to perform mathematical operations on the input signals, such as addition or differentiating with respect to time.

Instead of analyzing *V* when designing these networks, we shift the neural activity to simplify analysis. For each neuron, we substitute *U* = *V − Er*, the activation level above the resting voltage. A typical value is *E<sup>r</sup>* = *−*60 mV, but using *U* for analysis rather than *V* lets us apply the same analysis no matter what *E<sup>r</sup>* is. We also set *G<sup>m</sup>* = 1 *µ*S, which is a typical value (Daun-Gruhn et al., 2009; Daun-Gruhn, 2010).

For the synapses, we set *Elo* = *E<sup>r</sup>* of the presynaptic neuron, and introduce a new parameter *R* = *Ehi − Elo*. Thus, a synapse's conductivity rises as the presynaptic neuron's voltage rises above its resting potential, and exhibits an "operating range" of *R* mV. The constraints we apply ensure that *Upre ∈*[0, *R*], meaning that the synapse is always active, but never saturates. Thus, we can replace *G<sup>s</sup>* with the second line of equation (4). Applying the substitutions described so far,

$$\mathcal{G}\_s = \mathcal{g}\_s \cdot \frac{V\_{pre} - E\_{lo}}{E\_{hi} - E\_{lo}} = \mathcal{g}\_s \cdot \frac{U\_{pre}}{R} = \frac{\mathcal{g}\_s}{R} \cdot U\_{pre}.\tag{5}$$

For each synapse, we also introduce the parameter ∆*Es,i* = *Es,i − Er,post*, where *Er,post* is the resting potential of the postsynaptic, or receiving neuron.

Making all of these substitutions in equations (1)–(3) gives the response

$$C\_m \frac{dU}{dt} = -U + \sum\_{i=1}^{n} \frac{\mathcal{g}\_{s,i}}{R} \cdot U\_{pre,i} \cdot (\Delta E\_{s,i} - U) + I\_{app}.\tag{6}$$

When *U* = *R*, the neuron is fully active, and when *U* = 0, the neuron is inactive. We can use this knowledge to categorize synapses as excitatory or inhibitory, depending on the sign of ∆*Es,i*. If ∆*Es,i ≥ R*, then the *i*th synapse will always transmit positive current, no matter the instantaneous value of *U*. Thus, this synapse will cause *U* to increase and is, therefore, excitatory. Similarly, if ∆*Es,i ≤* 0, then the *i*th synapse will always transmit negative current, no matter the instantaneous value of *U*. Thus, this synapse will cause *U* to decrease and is, therefore, inhibitory.

#### **2.1. Mapping between Neural and Mechanical Values**

The nervous system encodes physical quantities as neural activity. In insects, the firing rate of sensory neurons encode the stretch of chordotonal organs (Field and Matheson, 1998) and the strain of campaniform sensilla (Zill et al., 2004), among other physical quantities. Typical robot controllers perform operations on these signals to provide meaningful information for control actions. These operations may include the subtraction of measured and reference values, differentiation or integration of error values, or gain adjustments. Neural systems perform these same operations, but in a transformed space. The exact transformation that nervous systems use is not known, but for reliable behavior, it is necessary that sensory information is mapped to neural activity in apredictable way. Thus, we map any sensory input, *θ*, to an applied current

$$I\_{\rm app} = R \cdot \frac{\theta - \theta\_{\rm min}}{\theta\_{\rm max} - \theta\_{\rm min}},\tag{7}$$

where *R* is the "operating range" specified in the previous section. **Figure 1B** illustrates such a transformation within a diagram of a neural feedback loop that controls the position of a motor. The purpose of this paper is not to analyze how this particular network functions; for a detailed analysis of this network and its function, see Szczecinski et al. (2017b). Instead, the purpose is to show how *R* and other functional values (time constants, gains, etc.) may be used to constrain neural and synaptic parameter values.

**Figure 1B** (i,ii) graphically illustrate the mapping in equation (7), and **Figure 1B** (iii) graphically illustrates the inverse relationship (i.e., neural value to mechanical value). If a sensory neuron has only this applied current and leak current, equation (6) shows that

$$\mathcal{C}\_{m}\frac{dU}{dt} + U = \mathcal{R} \cdot \frac{\theta - \theta\_{\rm min}}{\theta\_{\rm max} - \theta\_{\rm min}}.\tag{8}$$

This means that the sensory neuron acts as a low-pass filter with time constant *τ* = *Cm*. It is trivial to show that, when the neuron is at equilibrium (i.e., *dU*/*dt* = 0),

$$U^\* = \mathcal{R} \cdot \frac{\theta - \theta\_{\rm min}}{\theta\_{\rm max} - \theta\_{\rm min}},\tag{9}$$

where the superscript "*∗*" specifies the equilibrium value. (Throughout this manuscript, the equilibrium activation of neuron *U* will be referred to as *U ∗* , and the neuron itself will be referred to as *U*.) Equation (9) means that the neuron's activation above its rest potential encodes the sensory signal. In addition to perceiving sensory information, commands must be issued in the same transformation. Thus, we map the commanded sensory quantity, *θ*comm, to the commanded neural activation, *U*comm, with the inverse function of equation (9),

$$
\theta\_{\rm comm} = \theta\_{\rm min} + \frac{U\_{\rm comm}}{R} \cdot (\theta\_{\rm max} - \theta\_{\rm min}).\tag{10}
$$

In this way, the nervous system may specify an intended motion, such as the rotation of a joint, encoded in neural activity. In our synthetic nervous systems, *R* specifies how mechanical quantities and neural activation are related. Thus, the tuning of every functional subnetwork described in this work relies on *R*, which the designer specifies before tuning the rest of the network. Two other parameters are critical for tuning these subnetworks: the amplification of synaptic transmission, *ksyn* (discussed in Sec. 3.1), and the synaptic reversal potential, ∆*Es*. From these values, biological parameters such as synaptic conductance and neural tonic drive can be directly calculated. This makes network design intuitive, enabling the designer to select biological parameter values based on functional ones.

#### **3. METHODS: ARITHMETIC SUBNETWORKS**

This section describes how to use typical engineering quantities to design neural and synaptic pathways. We can understand how these pathways work by manipulating their equilibria, something that naive optimization does not leverage. The steady-state activation *U ∗* is calculated by solving for *U* when *dU*/*dt* = 0

$$0 = -U^\* + \sum\_{i=1}^n \frac{\mathcal{g}\_{s,i}}{R} \cdot U\_{\text{pre},i} \cdot \left(\Delta E\_{s,i} - U^\*\right) + I\_{app}.\tag{11}$$

Moving all *U ∗* terms to the left hand side

$$U^\bullet \cdot \left(1 + \sum\_{i=1}^n \frac{\mathcal{g}\_{s,i}}{R} \cdot U\_{pre,i}\right) = \sum\_{i=1}^n \frac{\mathcal{g}\_{s,i}}{R} \cdot U\_{pre,i} \cdot \Delta E\_{s,i} + I\_{app} \tag{12}$$

Solving for *U ∗* ,

$$U^\* = \frac{\sum\_{i=1}^n \frac{\xi\_{s,i}}{R} \cdot U\_{pre,i} \cdot \Delta E\_{s,i} + I\_{app}}{1 + \sum\_{i=1}^n \frac{\xi\_{s,i}}{R} \cdot U\_{pre,i}}.\tag{13}$$

This solution is the basis for the remainder of Sec. 3.

#### **3.1. Signal Transmission Pathways**

The goal of a signal transmission pathway is to cause the postsynaptic neuron's voltage to be some ratio of the presynaptic neuron's voltage. We call this ratio *ksyn*. The *Upre,i* terms in the denominator of the right hand side of equation (13) mean that *ksyn* changes as *Upre* changes, so we approximate *ksyn* as *Upost*/*Upre* when the presynaptic neuron is fully activated (i.e., *Upre* = *R*). The steady-state response of a neuron with a single synaptic input and no applied current can be written based on equation (13), as below:

$$U\_{post}^{\*} = \frac{\frac{\mathcal{G}\_{\mathfrak{t}}}{R} \cdot U\_{pre} \cdot \Delta E\_{s}}{1 + \frac{\mathcal{G}\_{\mathfrak{t}}}{R} \cdot U\_{pre}}.\tag{14}$$

To find *ksyn* for this synapse, we first divide both sides of equation (14) by *Upre*,

$$\frac{U\_{\rm post}^{\*}}{U\_{\rm pre}} = \frac{\frac{\mathcal{E}\_{\rm \epsilon}}{\mathcal{R}} \cdot U\_{\rm pre} \cdot \Delta E\_{\rm s}}{U\_{\rm pre} \cdot \left(1 + \frac{\mathcal{E}\_{\rm \epsilon}}{\mathcal{R}} \cdot U\_{\rm pre}\right)}.\tag{15}$$

Next, we want to find *ksyn*, which can be calculated for any value of *Upre*. To simplify analysis and improve the clarity of this derivation, we set find *ksyn* when *Upre* = *R*. Then, we show how to set parameter values to keep *ksyn* nearly constant, even as *Upre* changes. Making this substitution,

$$\frac{U\_{\rm post}^{\star}}{R} = k\_{\rm sym} = \frac{\frac{\mathcal{E}\_{\rm \parallel}}{R} \cdot R \cdot \Delta E\_{\rm s}}{R \cdot \left(1 + \frac{\mathcal{E}\_{\rm \parallel}}{R} \cdot R\right)}.\tag{16}$$

Finally, reducing *R*/*R* terms reveals

$$k\_{\rm sym} = \frac{\mathbf{g}\_{\rm s} \cdot \Delta E\_{\rm s}}{R \cdot (1 + \mathbf{g}\_{\rm s})}.\tag{17}$$

Rearranging to solve for *gs*,

$$\mathbf{g}\_{\mathbf{s}} = \frac{k\_{\rm syn} \cdot \mathbf{R}}{\Delta E\_{\mathbf{s}} - k\_{\rm syn} \cdot \mathbf{R}}.\tag{18}$$

Because *g<sup>s</sup>* must be positive, and the numerator of equation (18) is always positive, equation (18) is also subject to the constraint

$$
\Delta E\_{\sf s} > k\_{\sf spn} \cdot \mathbb{R}.\tag{19}
$$

Equation (18) will be used to tune addition, subtraction, multiplication, and division networks (Secs. 3.3 through 3.6).

#### **3.2. Signal Modulation Pathways**

We may also use synapses to modulate a neuron's sensitivity to other inputs. Based on equation (13), the steady-state response of a neuron with only an applied current *Iapp* is simply

$$U\_{\text{post}}^{\*} = I\_{\text{app}},\tag{20}$$

if we set *G<sup>m</sup>* = 1. For example, this is the case for a sensory neuron that receives applied current proportional to a sensor's state, such as a joint angle (**Figure 1B**), muscle stretch, or touch sensor. However, the nervous system may need to actively increase or reduce the sensitivity of the sensory neuron depending on context. Hyperpolarizing or depolarizing the neuron, however, would cause sensory information to be truncated (i.e., *Vpre < Elo*). We can change the sensitivity of this neuron without losing sensory information by adding a synaptic input to the response from equation (20):

$$U\_{\rm post}^{\*} = \frac{\frac{\xi\_{\rm \epsilon}}{\overline{\mathcal{R}}} \cdot U\_{\rm pre\epsilon} \cdot \Delta E\_{\rm s} + I\_{\rm app}}{1 + \frac{\xi\_{\rm \epsilon}}{\overline{\mathcal{R}}} \cdot U\_{\rm pre\epsilon}}. \tag{21}$$

To quantify how *Upre* modulates *U ∗ post* for a given *Iapp*, we introduce the parameter*csyn*, which quantifies this degree of modulation. We define *csyn* as *U ∗ post/Upre*, the same as *ksyn*, but with the understanding that *Upre* will decrease *U ∗ post* in this case. Dividing both sides of equation (21) by *Upre* and using the definition of *csyn*,

$$\frac{U\_{\rm post}^\*}{U\_{\rm pre}} = c\_{\rm syn} = \frac{\frac{\xi\_t}{R} \cdot U\_{\rm pre} \cdot \Delta E\_s + I\_{\rm app}}{U\_{\rm pre} \cdot \left(1 + \frac{\xi\_t}{R} \cdot U\_{\rm pre}\right)}. \tag{22}$$

As in the previous section, we will solve for *csyn* when *Upre* = *R* to simplify analysis. Making this substitution and reducing *R*/*R* terms,

$$
\omega\_{\text{sym}} \cdot R = \frac{\mathbf{g}\_{\text{s}} \cdot \Delta E\_{\text{s}} + R}{1 + \mathbf{g}\_{\text{s}}}.\tag{23}
$$

Multiplying both sides by the denominator of the right hand side and expanding,

$$
\boldsymbol{\omega}\_{\rm sym} \cdot \boldsymbol{\mathcal{R}} + \boldsymbol{\mathcal{c}}\_{\rm sym} \cdot \boldsymbol{\mathcal{R}} \cdot \boldsymbol{\mathcal{g}}\_{\rm s} = \boldsymbol{\mathcal{g}}\_{\rm s} \cdot \boldsymbol{\Delta} \boldsymbol{E}\_{\rm s} + \boldsymbol{\mathcal{R}}.\tag{24}
$$

Collecting *g<sup>s</sup>* terms on the left hand side,

$$
\mathfrak{c}\_{\rm sym} \cdot \mathbb{R} \cdot \mathfrak{g}\_{\rm s} - \mathfrak{g}\_{\rm s} \cdot \Delta E\_{\rm s} = \mathbb{R} - \mathfrak{c}\_{\rm sym} \cdot \mathbb{R}.\tag{25}
$$

Solving equation (25) for *gs*,

$$\mathbf{g}\_s = \frac{\boldsymbol{\varepsilon}\_{\rm sym} \cdot \boldsymbol{R} - \boldsymbol{R}}{\Delta E\_s - \boldsymbol{\varepsilon}\_{\rm sym} \cdot \boldsymbol{R}}.\tag{26}$$

Just as in Sec. 3.1, *g<sup>s</sup> >* 0 depends only on *R*, which the designer specifies beforehand, ∆*Es*, which is limited by biological constraints, and *csyn*, which the designer picks based on network function. ∆*E<sup>s</sup>* should be negative, and as close to 0 as possible to minimize hyperpolarization of the postsynaptic neuron. Equation (26) will be used to tune division and multiplication networks (Secs. 3.5 and 3.6).

#### **3.3. Addition**

A subnetwork that approximates linear addition of the form *U ∗ post* = *ksyn ·* (*Upre,*<sup>1</sup> + *Upre,*2) may underlie positive feedback mechanisms, which increase motor neuron activation proportional to sensory inputs such as force sensing organs (Zill et al., 2004), or used to sum sensory signals from different body segments (Mittelstaedt, 1957). We construct such a network by using two Signal Transmission pathways as presented in Sec. 3.1.

Let us rewrite equation (13) here, for clarity:

$$U^\* = \frac{\sum\_{i=1}^n \frac{\xi\_{s,i}}{R} \cdot U\_{pre,i} \cdot \Delta E\_{s,i} + I\_{app}}{1 + \sum\_{i=1}^n \frac{\xi\_{s,i}}{R} \cdot U\_{pre,i}}.\tag{27}$$

This equation shows *Upre,i* in both the numerator and denominator. To capture addition, we wish to minimize the impact of *Upre,i* on the denominator. This is accomplished by minimizing *gs*. However, if *g<sup>s</sup>* = 0, then the network will not function at all. Therefore, we instead maximize ∆*Es*, which yields a small *g<sup>s</sup>* (equation (18)). Mathematically, there is no limit on ∆*Es*, but synaptic potentials are limited in biological systems. In our work, we choose the reversal potential of calcium (*E<sup>s</sup>* = 134 mV), which yields ∆*E<sup>s</sup>* = *E<sup>s</sup> − E<sup>r</sup>* = 134 *−* (*−*60) = 194 mV, and specify *R* = 20 mV. To design a pathway where *ksyn* = 1, for example, we

**TABLE 1** | This table assumed that the designer has already selected a value of *R* for the subnetwork.


*In this table, "minimize" refers to making a value as negative as possible and "maximize" refers to making a value as positive as possible.*

plug these values into equation (18), which gives *g<sup>s</sup>* = 115 nS. The contour plots in **Figure 2A** show that the network matches the ideal behavior very closely over the operating range *Usum ∈*[0, *R*]. These design constraints are summarized in **Table 1**.

### **3.4. Subtraction**

A subnetwork that approximates linear subtraction of the form *U ∗ post* = *ksyn ·* (*Upre,*<sup>1</sup> *− Upre,*2) may underlie negative feedback mechanisms, which are important for controlling many parameters in locomotion (Pearson, 1993; Peterka, 2003; Buschmann et al., 2015). Just as in the previous section, equation (18) is used to find *g<sup>s</sup>* for each pathway.

Designing a subtraction network requires that we pay attention to how the two synapses affect one another. Since the reversal potentials of hyperpolarizing ion channels are not much more negative than typical resting potentials, larger *gs,*<sup>2</sup> values are required to transmit information than for depolarizing ion channels. This makes it harder to minimize *g<sup>s</sup>* like we did in the previous section. Equation (13) enables us to constrain *gs*,2 such that when *Upre*,1 = *R* and *Upre*,2 = *R*, *U ∗ post* = 0. Starting with the neuron response in equation (13) for two synaptic currents and no applied current,

$$\mathbf{U}\_{\rm post}^{\*} = \frac{\mathbf{g}\_{\rm s,1}/\mathbf{R} \cdot \mathbf{U}\_{\rm pre,1} \cdot \Delta E\_{\rm s,1} + \mathbf{g}\_{\rm s,2}/\mathbf{R} \cdot \mathbf{U}\_{\rm pre,2} \cdot \Delta E\_{\rm s,2}}{1 + \mathbf{g}\_{\rm s,1}/\mathbf{R} \cdot \mathbf{U}\_{\rm pre,1} + \mathbf{g}\_{\rm s,2}/\mathbf{R} \cdot \mathbf{U}\_{\rm pre,2}}.\tag{28}$$

Substituting in *Upre*,1 = *R*, *Upre*,2 = *R*, and *U ∗ post* = 0,

$$\mathbf{0} = \frac{\mathbf{g}\_{\mathbf{s},1}/\mathbf{R}\cdot\mathbf{R}\cdot\Delta E\_{\mathbf{s},1} + \mathbf{g}\_{\mathbf{s},2}/\mathbf{R}\cdot\mathbf{R}\cdot\Delta E\_{\mathbf{s},2}}{1 + \mathbf{g}\_{\mathbf{s},1}/\mathbf{R}\cdot\mathbf{R} + \mathbf{g}\_{\mathbf{s},2}/\mathbf{R}\cdot\mathbf{R}}\tag{29}$$

$$\mathbf{0} = \frac{\mathbf{g}\_{\mathbf{s},1} \cdot \Delta E\_{\mathbf{s},1} + \mathbf{g}\_{\mathbf{s},2} \cdot \Delta E\_{\mathbf{s},2}}{1 + \mathbf{g}\_{\mathbf{s},1} + \mathbf{g}\_{\mathbf{s},2}} \tag{30}$$

$$\mathbf{0} = \mathbf{g}\_{\mathbf{s},1} \cdot \Delta E\_{\mathbf{s},1} + \mathbf{g}\_{\mathbf{s},2} \cdot \Delta E\_{\mathbf{s},2} \tag{31}$$

$$\mathbf{g}\_{\mathbf{s},2} = \frac{\Delta E\_{\mathbf{s},1}}{\Delta E\_{\mathbf{s},2}} \cdot -\mathbf{g}\_{\mathbf{s},1} \,. \tag{32}$$

Substituting equation (18) for *gs*,1,

$$\mathbf{g}\_{\mathbf{s},2} = \frac{\Delta E\_{\mathbf{s},1}}{\Delta E\_{\mathbf{s},2}} \cdot \frac{-k\_{\rm syn} \cdot \mathbf{R}}{\Delta E\_{\mathbf{s},1} - k\_{\rm syn} \cdot \mathbf{R}}.\tag{33}$$

To be physically realizable, *gs*,2 *>* 0. Because *gs*,1 *>* 0 and ∆*Es*,1 *>* 0, *gs*,2 *>* 0 if and only if ∆*Es*,2 *<* 0. Thus, it is critical that ∆*Es*,2 *<* 0.

Just as for the addition network, we minimize *gs*,1 by maximizing ∆*Es*,1. If *R* = 20 mV and *ksyn* = 1, then *gs*,1 = 115 nS and ∆*Es*,1 = 194 mV. To tune *gs*,2, we first select ∆*Es*,2 = *−*40 mV, then we solve equation (33) to find *gs*,2 = 558 nS. These design constraints are summarized in **Table 1**, and **Figure 2B** graphically shows the accuracy of the subtraction network.

#### **3.5. Division**

A subnetwork that approximates division of the form

$$U\_{\text{post}}^{\*} = \frac{U\_{\text{pre},1}}{1 + \frac{1 - \varepsilon\_{\text{pre}}}{\varepsilon\_{\text{pre}} \cdot \mathbb{R}} \cdot U\_{\text{pre},2}} \tag{34}$$

**FIGURE 2** | Data demonstrating the function of arithmetic networks. Each contour plot represents cross sections of the response surface, as depicted at the top. The network diagram, relevant parameters, and data are shown for addition **(A)**, subtraction **(B)**, division **(C)**, and multiplication **(D)**. Triangular synaptic terminations stand for excitatory inputs, and filled round terminations stand for inhibitory inputs. For each operation, the contour on the right is the ideal output, and the contour on the left is the actual operation for the parameter values listed. Free parameters from **Table 1** are highlighted in gray.

replicates the function of GABA synapses that regulate activity in the brain. A key reason for this behavior is that the reversal potential of GABA-ergic synapses is about equal to the resting potential of the postsynaptic neuron (Trappenberg, 2009). Equation (26) is used to find *g<sup>s</sup>* for the division pathway.

The synapse from *Upre*,1 to *Upost* is tuned as an excitatory Signal Transmission pathway with *k* = 1, as in Sec. 3.1. In our work, *R* = 20 mV, ∆*Es*,1 = 194 mV, and equation (18) tells us that *gs*,1 = 115 nS. Such a small *g<sup>s</sup>* ensures that the signal from *Upre*,1 to *Upost* is transmitted without greatly affecting the sensitivity of *Upost* to inputs. That is, the effect of *Upre*,1 on the denominator of *U ∗ post* is very nearly 0.

The synapse from *Upre*,2 to *Upost* is tuned as a Signal Modulation pathway, as analyzed in Sec. 3.2. Setting ∆*Es*,2 = 0 will eliminate *Upre*,2's influence on the numerator of *U ∗ post*. Substituting this case into equation (26) and reducing,

$$\mathbf{g}\_{5,2} = \frac{1 - \mathfrak{c}\_{\text{sym}}}{\mathfrak{c}\_{\text{sym}}},\tag{35}$$

where *U ∗ post* = *csyn · R* when *Upre*,1 = *Upre*,2 = *R*, their maximal value. Equation (35) also reveals that since *gs*,2 *>* 0, 0 *< csyn <* 1.

The steady-state response of the network is the result of these two synaptic inputs, as written in equation (28). Substituting equation (35), and specifying that *ksyn*,1 = 1, *U ∗ post* simplifies to

$$\begin{array}{l} U\_{\text{post}}^{\*} = \\ \frac{\mathcal{G}\_{\text{s},1}/\mathcal{R} - \Delta E\_{\text{s},1} \cdot \stackrel{\bullet}{U}\_{\text{pre},1}^{1} + \mathcal{g}\_{\text{s},2}/\mathcal{R} \cdot \Delta E\_{\text{s},2} \cdot \stackrel{\bullet}{U}\_{\text{pre},2}^{0}}{1 + \mathcal{g}\_{\text{s},1} \cdot ^{0}U\_{\text{pre},1}/\mathcal{R} + \frac{1-\varepsilon\_{\text{pre}}}{\varepsilon\_{\text{pre}}} \cdot U\_{\text{pre},2}/\mathcal{R}} \approx \frac{U\_{\text{pre},1}}{1 + \frac{1-\varepsilon\_{\text{pre}}}{\varepsilon\_{\text{pre}} \cdot \mathcal{R}} \cdot U\_{\text{pre},2}} \end{array} \tag{36}$$

In our network, we wished *U ∗ post* = 1 when *Upre*,2 = *R*, so we set *csyn* = 1/*R* = 0.05, which makes *gs*,2 = 19 *µ*S. When *csyn* is close to 0, *Upre*,2 can strongly reduce *Upost*'s sensitivity to inputs. When *csyn* is close to 1, *Upre*,2 can only weakly reduce *Upost*'s sensitivity to inputs. **Figure 2C** shows that this network performs the intended division of the signals. **Table 1** summarizes these design constraints.

#### **3.6. Multiplication**

A subnetwork that approximates multiplication of the form *U ∗ post* = *Upre,*<sup>1</sup> *· Upre,*2*/R* can be used to control the gain of a sensory feedback loop, a frequently observed characteristic of neural systems that control locomotion (Cruse, 1981; Gabriel and Büschges, 2007) and posture (Peterka and Loughlin, 2004).

A multiplication network can be assembled by replacing the Modulatory Pathway in the division network with two identical Modulatory Pathways in series, connected into a disinhibitory network (see **Figure 2D**). This works because the product of two numbers, *a·b* = *a*/(1/*b*). However, tuning the Modulatory Pathway for the multiplication network differs from tuning the division network. This is because the right-side pathway of the network in **Figure 2D** must make *U ∗ post* = 0, no matter how active *Upre*,1 becomes (because *a·*0 = 0, no matter the value of *a*). Thus, according to equation (22), *csyn* = 0, unlike the division network, for which 0 *< csyn <* 1. Solving equation (26) when *csyn* = 0 reveals that

$$g\_{\mathfrak{s},2} = -\mathbb{R}/\Delta E\_{\mathfrak{s},2}.\tag{37}$$

To solve for *gs*,2, we must first select ∆*Es*,2. If ∆*Es*,2 = 0 like for the division network, then equation (37) divides by 0. If ∆*Es*,2 *>* 0, then *gs*,2 *<* 0, which is physically not realizable. Therefore, we must choose a value ∆*Es*,2 *<* 0. The more negative ∆*Es*,2 is, the more small-amplitude signals are clipped; however, the less negative it is, the larger *gs*,2 must be. Therefore, *gs*,2 is the limiting factor to maintain biological realism. We have chosen *gs*,2 = 20 *µ*S and *R* = 20 mV, making ∆*Es*,2 = *−*1.

Now that we have designed one of the Modulatory synapses, we can calculate the response of the complete multiplication network seen in **Figure 2D**, which includes two identical Modulatory Pathways in series. When *Upre*,2 is inactive, then it does not inhibit *Uinter*, which is tonically active. In this case, *Uinter*'s activity completely desensitizes *Upost* to inputs. When *Upre*,2 is active, then it inhibits *Uinter*. In this case, *Uinter* is hyperpolarized, and cannot desensitize *Upost* to inputs. To show that this is the case, let us find the full response of the system. We first calculate *U ∗ inter*, which has one Modulatory Pathway input and a tonic applied current *Iapp* = *R*. Its response is the same as in equation (21), with the constraint from equation (37), which causes terms to cancel:

$$U\_{inter}^{\*} = \frac{\frac{\mathcal{G}\_{\iota,2}}{R} \cdot U\_{pre,2} \cdot -\frac{R}{\mathcal{G}\_{\iota,2}} + R}{1 - \frac{U\_{pre,2}}{\Delta E\_{\iota,2}}} = \frac{R - U\_{pre,2}}{1 - \frac{U\_{pre,2}}{\Delta E\_{\iota,2}}}.\tag{38}$$

*Upost* has two presynaptic neurons, *Upre*,1 and *Uinter*. The synapse from *Upre*,1 is a Signal Transmission synapse, and the synapse from *Uinter* is a Signal Modulation synapse. Its response is found via equation (13),

$$U\_{post}^{\*} = \frac{\frac{\mathcal{E}\_{\text{t},3}}{\mathcal{R}} \cdot U\_{inter} \cdot \Delta E\_{\text{s},3} + \frac{\mathcal{E}\_{\text{t},1}}{\mathcal{R}} \cdot U\_{pre,1} \cdot \Delta E\_{\text{s},1}}{1 + \frac{\mathcal{E}\_{\text{t},3}}{\mathcal{R}} \cdot U\_{inter} + \frac{\mathcal{E}\_{\text{t},1}}{\mathcal{R}} \cdot U\_{pre,1}}.\tag{39}$$

We showed in Sec. 3.3 that equation (18) can be used to design a synapse that transmits the presynaptic neuron's activity to the postsynaptic neuron, while minimizing its impact on the denominator of the postsynaptic neuron's steady-state response, *U ∗ post*. This enables us to approximate *Upre*,1's effect on *U ∗ post* as an applied current *Iapp ≈ Upre*,1. Making this substitution in equation (39),

$$U\_{post}^{\*} \approx \frac{\frac{\mathcal{E}\_{t,3}}{R} \cdot U\_{inter} \cdot \Delta E\_{s,3} + U\_{pre,1}}{1 + \frac{\mathcal{E}\_{t,3}}{R} \cdot U\_{inter}}.\tag{40}$$

Because we previously specified that the Modulatory Pathways are identical, we can apply the constraint from equation (37),

$$U\_{\text{post}}^{\*} = \frac{U\_{\text{pr},1} - U\_{\text{inter}}}{1 - \frac{U\_{\text{inter}}}{\Delta E\_{\text{i,3}}}}.\tag{41}$$

We can now substitute equation (38) for *Uinter*,

$$U\_{post}^{\*} = \frac{U\_{pre,1} - \frac{R - U\_{pre,2}}{1 - \frac{U\_{pre,2}}{\Delta E\_{i,2}}}}{1 - \frac{1}{\Delta E\_{i,3}} \cdot \frac{R - U\_{pre,2}}{1 - \frac{U\_{pre,2}}{\Delta E\_{i,2}}}} . \tag{42}$$

This expression can be simplified. First, as noted previously, synapses 2 and 3 are identical, so ∆*Es*,2 = ∆*Es*,3 = ∆*Es*. Second, we can multiply the first term in both the numerator and denominator by the factor (1 *− Upre*,2/∆*Es*), which enables us to combine terms. Performing these simplifications,

$$U\_{\rm pot}^{\*} = \frac{U\_{\rm pre,1} - U\_{\rm pre,1} \cdot U\_{\rm pre,2}/\Delta E\_{\rm s} - R + U\_{\rm pre,2}}{1 - U\_{\rm pre,2}/\Delta E\_{\rm s} - R/\Delta E\_{\rm s} + U\_{\rm pre,2}/\Delta E\_{\rm s}},\tag{43}$$

$$\mathcal{U}\_{\rm pot}^{\*} = \frac{-U\_{\rm pre,1} \cdot U\_{\rm pre,2}/\Delta E\_{\rm s} + U\_{\rm pre,1} + U\_{\rm pre,2} - R}{1 - R/\Delta E\_{\rm s}}.\tag{44}$$

Equation (44) contains a lot of information about how the multiplication network functions. First, *Upost*'s response indeed contains a term that multiplies *Upre*,1 and *Upre*,2. When ∆*E<sup>s</sup>* = *−*1, then *U ∗ post* scales with *Upre*,1*·Upre*,2 in a 1:1 fashion. Second, the numerator will be *≤* 0 if either *Upre*,1 = 0 or *Upre*,2 = 0, *U ∗ post ≤* 0. This is because *Upre*,1 and *Upre*,2 must each be less than or equal to *R*. If either input is greater than *R*, then their synaptic inputs to *Upost* will saturate (see equation (4)), preventing this condition from being violated. Third, the denominator does not depend on the input values. Technically, because of the approximation made in equation (40), the denominator does change slightly with *Upre*,1. However, with our chosen values of *R* (20), ∆*E<sup>s</sup>* (*−*1), and *gs*,1 (0.115), this change is less than 1%, justifying this approximation. **Figure 2D** demonstrates that this network multiplies the two inputs.

**Table 1** summarizes the function, component pathways, constraint equations, and free parameters of each network from this section. This analysis enables direct construction and parameter selection for functional subnetworks that can be assembled into more complex networks capable of performing real-time robotic control (e.g., Szczecinski and Quinn (2017) and Szczecinski et al. (2017a)). Additionally, one of the key advantages to using dynamic neural systems for motor control is the handling of time varying signals. The next section examines how the dynamics of these neurons can be exploited to perform calculus on signals.

#### **4. METHODS: DYNAMIC NETWORKS**

The differential equation for a single neuron's response (equation (1)) can be solved analytically. Solving an equation *dx*/*dt* = *f*(*x*) is simplified if the equilibrium state is *x <sup>∗</sup>*= 0, so as in Sec. 3, the substitution *U* = *V − E<sup>r</sup>* is made. Additionally, the membrane conductance *G<sup>m</sup>* and capacitance *C<sup>m</sup>* can be combined into a new parameter *τ* = *Cm*/*Gm*, which is a more intuitive parameter when discussing dynamic networks. This section uses analysis from the previous section, plus additional analysis, to derive design constraints for networks that differentiate or integrate input signals over time.

#### **4.1. Differentiation**

One dynamic response neural systems are known to utilize is differentiation of signals. Early examination of neural networks led to the discovery of the Reichardt detector network (Reichardt, 1961), an autocorrelation network with delays that approximates the differential of an incoming signal. Other examples include human balance, which relies on feedback proportional to the position, velocity, and acceleration of the center of mass (Peterka, 2003; Safavynia and Ting, 2012). Also, positive velocity feedback plays an important role in insect muscle control (Cruse, 1981).

We have developed differentiation networks based on the Reichardt detector network, shown in **Figure 3A**. We can understand its function by examining a neuron's response to a ramp input, *Iapp* = *A·t*, where *A* is an arbitrary slope of the ramp. The response of the network should be a step with a magnitude proportional to *A*, as shown in **Figure 3B**. Inserting this applied current into equation (6), a single neuron's response is

$$C\_m \cdot \frac{dU}{dt} = -U + A \cdot t \tag{45}$$

$$\mathbf{C}\_{m} \cdot \frac{dU}{dt} + \mathbf{U} = \mathbf{A} \cdot \mathbf{t}.\tag{46}$$

The response of the neuron, *U*(*t*), is the sum of the particular and homogeneous solutions to equation (46), *Up*(*t*) and *Uh*(*t*), respectively. Simulating the dynamics of equation (46) suggests **TABLE 2** | This table assumed that the designer has already selected a value of *R* for the subnetwork.


*In this table, "minimize" refers to making a value as negative as possible and "maximize" refers to making a value as positive as possible.*

that the particular solution is a ramp of slope *A*, which lags behind the input with a time constant *Cm*. To confirm this, we can substitute a candidate solution and its derivative into equation (46), and check for equality. The result is the particular (i.e., steady-state) response,

$$U\_p(t) = A \cdot (t - C\_m) \tag{47}$$

This means that if the same *Iapp* were injected into neurons with different *C<sup>m</sup>* values, and then their outputs were subtracted from one another with a network from Sec. 3.4, the network would perform a finite-difference approximation of the derivative of *Iapp*, once the transient response decays (illustrated in **Figures 3A,B**).

Calculating the homogeneous solution, *Uh*(*t*), informs us how quickly the transient response decays. The homogeneous solution to first-order linear equation like equation (46) is well-known, *Uh*(*t*) = *b·*exp(*− t*/*Cm*). The constant *b* is found by plugging the initial condition into the full response, *U*(*t*) = *Up*(*t*) + *Uh*(*t*),

$$b = A \cdot \mathcal{C}\_m.\tag{48}$$

To tune this network, the response of *Upost* is written as the difference between neuron *Upre*,1 with *Cm*,1 and neuron *Upre*,2 with *Cm*,2 *> Cm*,1,

$$\begin{split} U\_{\text{pot}}(t) &= U\_{\text{prc},1}(t) - U\_{\text{prc},2}(t) \\ &= A \cdot t - A \cdot \mathbb{C}\_{m,1} \cdot \left( 1 - \exp(-t/\mathbb{C}\_{m,1}) \right) \\ &\quad - \left( A \cdot t - A \cdot \mathbb{C}\_{m,2} \cdot \left( 1 - \exp(-t/\mathbb{C}\_{m,2}) \right) \right) . \end{split} \tag{49}$$

Canceling the terms that are linear in *t* and expanding,

$$\begin{split} U\_{\text{post}}(t) &= A \cdot \left( \mathbf{C}\_{m,2} - \mathbf{C}\_{m,1} \right) \\ &+ A \cdot \left( \mathbf{C}\_{m,1} \cdot \exp(-t/\mathbf{C}\_{m,1}) - \mathbf{C}\_{m,2} \cdot \exp(-t/\mathbf{C}\_{m,2}) \right). \end{split} \tag{50}$$

Properly tuning a differentiator network requires tuning *Cm*,1 and *Cm*,2 to obtain the intended gain of the network, *kd*, and an appropriately high cutoff frequency, *ωc*. Equation (50) reveals how these may be tuned. First, the steady-state response of this network to a ramp input defines *k<sup>d</sup>* = (*Cm*,2 *− Cm*,1). Second, the cutoff frequency *ω<sup>c</sup>* = 1/*τ <sup>d</sup>* quantifies the frequency of incoming signals (i.e., *Iapp* = *A·*sin(*ω·t*)) above which the network's response has less than half the energy of a lower-frequency signal. This is especially useful because although differential calculations amplify high-frequency noise, this network filters out noise with a frequency *ω > ωc*. Because *Cm*,2 *> Cm*,1, the time constant *τ <sup>d</sup>* = *Cm*,2.

**Figure 3C** shows contours of *k<sup>d</sup>* and *τ <sup>d</sup>* as *Cm*,1 and *Cm*,2 change. The plots show that increasing *Cm*,2 relative to *Cm*,1 increases *kd*, which may be valuable for amplifying signals. However, this also increases *τ <sup>d</sup>*, making *ω<sup>c</sup>* impractically low, which will cause the network's output to lag behind the input substantially. The contour for *k<sup>d</sup>* = 1 is drawn on the contour of *τ <sup>d</sup>*, showing that the smallest *τ <sup>d</sup>* achievable for this gain value is 1,000 ms, which would filter out all incoming signals for which *ω > ω<sup>c</sup>* = 1/(1 *s*) = 1 rad/s (0.159 Hz).

We can gain further insight into tuning *τ <sup>d</sup>* using our FeedbackDesign tool (Szczecinski et al., 2017b). **Figure 3D** shows Bode plots for this network's response, given two different values for *Cm*,2. When *Cm*,2 = 1,000 nF, like in **Figure 3B**, the network functions properly for inputs with *ω <* 1 rad/s, as predicted in the previous paragraph. Lowering *Cm*,2 to 50 nF increases *ω<sup>c</sup>* to 20 rad/s (3.18 Hz). Lowering *Cm*,2 also lowers the magnitude response as a function of *ω*, that is, it decreases *kd*. To regain this lost gain, we may increase *ksyn* in the subtraction network. **Figure 4** shows simulation data that explores this tradeoff. **Table 2** lists how to use *τ <sup>d</sup>* and *k<sup>d</sup>* to tune the entire differentiation network.

## **4.2. Integration**

Our neuron model is a leaky integrator, which means that the membrane voltage will integrate an applied current, but "leak" current to return to its resting potential. As a result, data cannot be stored in individual neurons, because neurons only have one stable equilibrium point. A network that is constructed to have a marginally stable equilibrium curve (or subspace) will not leak. A network will have this property if the determinant of the Jacobian matrix is 0, or in other words, if it is not full rank (Khalil, 2002). Instead of leaking, it will maintain its activation when no external currents are applied; when currents are applied, the state of the system will change continuously. This is analogous to the position of a box on a table with friction; it will remain wherever it is placed indefinitely, unless an external force is applied. In this section, we expand on previous work (Szczecinski et al., 2017a) to show how to construct a network that is marginally stable by applying constraints to reduce the rank of its Jacobian matrix; demonstrate that such a network can be used to integrate signals over time; and relate the integration rate, *ki*, to the parameter values of the network, such that *U*˙ <sup>1</sup> = *k<sup>i</sup> · Iapp*.

Marginally stable networks are hypothesized to play an important role in navigation (Haferlach et al., 2007) and the regulation of muscle forces in posture (Lévy and Cruse, 2008). Some memory models use carefully tuned self-excitation to cancel the leak current with excitatory synaptic current (Seung et al., 2000). In a similar vein, our network uses self-disinhibition (**Figure 5A**) to produce a line-attractor network in which a continuum of marginally stable equilibrium states exist. Simulation data in **Figure 5B** shows that stimulating *U*<sup>1</sup> with an applied constant current *u* causes *U*<sup>1</sup> to increase at an apparently constant rate, and when *u* is removed, neither *U*<sup>1</sup> nor *U*<sup>2</sup> leak to their rest potentials.

**FIGURE 5** | **(A)** A disinhibitory network can exploit neural dynamics to compute the integral of an incoming signal. **(B)** When given an applied current in the form of a step, the network response is a ramp whose slope is proportional to the amplitude of the step. **(C)** A plot of this data in the (*U*1, *U*2) phase space shows that when stimulated by applied current *u*, the system state, *x*(*t*) = [*U*1(*t*), *U*2(*t*)]*<sup>T</sup>* (blue), moves in the *X*<sup>1</sup> direction (green) while maintaining a constant distance from the equilibrium subspace (dashed violet) in the *X*<sup>2</sup> direction (red). This difference in behavior in each direction is because the eigenvalue associated with eigenvector *X*1, *λ*<sup>1</sup> = 0, and the eigenvalue associated with eigenvector *X*2, *λ*<sup>2</sup> *<* 0. *X*<sup>1</sup> and *X*<sup>2</sup> are drawn in multiple places because they depend on *x*(*t*), as shown in Appendix. **(D)** The mean rate of integration, *ki*,mean (left), and the range of the rate of integration, *ki*,range (right), depend on the synaptic conductance of mutual inhibition, *gs*, and the membrane capacitance of the neurons, *Cm*. Note that the *x*-axis of these plots are 1/*Cm*, to better space the contour lines.

This is the behavior of an integrator, as described in the previous paragraph.

Let us write the response of the integrator network as shown in **Figure 5A** to find its equilibrium states. Each neuron has leak current, synaptic current, and a constant applied current. Let all parameter values be symmetrical between the two neurons. We make the same substitutions as before; *U* = *V − Er*, *E<sup>r</sup>* = *Elo*,

$$
\Delta E\_{\sf s} = E\_{\sf s} - E\_{\sf r} \text{, and } R = E\_{hi} - E\_{lo}. \text{ If } I\_{app} = R,
$$

$$\mathbf{C}\_{m} \cdot \frac{dU\_{1}}{dt} = -U\_{1} + \mathbf{g}\_{s} \cdot \frac{U\_{2}}{R} \cdot (\Delta E\_{s} - U\_{1}) + R \tag{51}$$

$$\mathbf{C}\_{m} \cdot \frac{d\mathbf{U}\_{2}}{dt} = -\mathbf{U}\_{2} + \mathbf{g}\_{s} \cdot \frac{\mathbf{U}\_{1}}{R} \cdot \left(\Delta E\_{s} - \mathbf{U}\_{2}\right) + R.\tag{52}$$

Moving dynamical terms to the left hand side, and applied current to the right hand side,

$$\frac{dU\_1}{dt} + \frac{1}{C\_m} \left( U\_1 - \mathbf{g}\_s \cdot \frac{U\_2}{R} \cdot (\Delta E\_s - U\_1) \right) = \frac{R}{C\_m} \tag{53}$$

$$\frac{dU\_2}{dt} + \frac{1}{C\_m} \left( U\_2 - \mathbf{g}\_s \cdot \frac{U\_1}{R} \cdot (\Delta E\_s - U\_2) \right) = \frac{R}{C\_m} \tag{54}$$

Solving equation (53) when *dU*1/*dt* = 0 reveals the equilibrium curve

$$U\_2 = \frac{R \cdot (U\_1 - R)}{g\_s \cdot (\Delta E\_s - U\_1)}.\tag{55}$$

Solving equation (54) when *dU*2/*dt* = 0 reveals the equilibrium curve

$$U\_1 = \frac{\mathcal{R} \cdot (U\_2 - \mathcal{R})}{\mathcal{g} \cdot (\Delta E\_s - U\_2)},\tag{56}$$

which can be algebraically rearranged to be the same as equation (55) as long as *g<sup>s</sup>* and ∆*E<sup>s</sup>* are constrained such that

$$
\mathbf{g}\_s \cdot \Delta E\_s = -\mathbf{R}.\tag{57}
$$

Multiplying both sides of equation (56) by the denominator of the right hand side, and expanding,

$$\mathbf{g}\_s \cdot \Delta E\_s \cdot U\_1 - \mathbf{g} \cdot U\_1 \cdot U\_2 = \mathbf{R} \cdot U\_2 - \mathbf{R}^2.\tag{58}$$

Collecting multiples of *U*<sup>2</sup> and applying equation (57),

$$U\_2 = \frac{R \cdot (U\_1 - R)}{\mathbf{g}\_s \cdot (\Delta E\_s - U\_1)}.\tag{59}$$

Thus, equations (55) and (56) are the same equilibrium curve if *g<sup>s</sup>* and ∆*E<sup>s</sup>* satisfy equation (57). This curve, drawn on the phasespace diagram in **Figure 5C**, describes every equilibrium state that this network can have. In other words, a [*U*1, *U*2] pair is an equilibrium state of the system if and only if it satisfies equation (55). In the coming paragraph, we will use eigenvalue analysis to show that this network always functions as an integrator, as long as equation (57) is satisfied.

To find the system's eigenvalues, let us write equations (53) and (54) together in matrix form,

$$
\cdot \begin{bmatrix} \dot{U}\_1 \\ \dot{U}\_2 \end{bmatrix} + \frac{1}{C\_m} \cdot \begin{bmatrix} 1 + U\_2 \cdot \frac{\underline{g}}{R} & \frac{-\underline{g}}{R} \cdot (\Delta E\_t - U\_1) \\\\ \frac{-\underline{g}}{R} \cdot (\Delta E\_t - U\_2) & 1 + U\_1 \cdot \frac{\underline{g}}{R} \end{bmatrix} \cdot \begin{bmatrix} U\_1 \\ U\_2 \end{bmatrix} = \frac{1}{C\_m} \cdot \begin{bmatrix} R \\ R \end{bmatrix} \tag{60}
$$

in which the square matrix is *J*, the system Jacobian. Because *J* contains *U*<sup>1</sup> and *U*<sup>2</sup> terms, it is not constant, but still describes the stability of the system, given specific values of *U*<sup>1</sup> and *U*2. To construct a marginally stable equilibrium subspace for the network, we must show that *J* has insufficient rank (i.e., the rows are identical) when *U*<sup>1</sup> and *U*<sup>2</sup> are at equilibrium (i.e., equation (55) is satisfied). However, the rows are identical, no matter the values of *U*<sup>1</sup> and *U*2, if we apply the constraint from equations (57) to (60),

$$
\begin{bmatrix} \dot{U}\_1 \\ \dot{U}\_2 \end{bmatrix} + \frac{1}{C\_m} \cdot \begin{bmatrix} 1 + \frac{\mathcal{G}\_t}{R} \cdot U\_2 & 1 + \frac{\mathcal{G}\_t}{R} \cdot U\_1 \\ 1 + \frac{\mathcal{G}\_t}{R} \cdot U\_2 & 1 + \frac{\mathcal{G}\_t}{R} \cdot U\_1 \end{bmatrix} \cdot \begin{bmatrix} U\_1 \\ U\_2 \end{bmatrix} = \frac{1}{C\_m} \cdot \begin{bmatrix} R \\ R \end{bmatrix} . \tag{61}
$$

Thus, the system will always have one null direction, and we do not need to calculate *J* for specific equilibrium conditions to determine the system's stability. To make notation more compact, let us define

$$a = 1 + \lg\_s / \mathcal{R} \cdot U\_1 \tag{62}$$

$$b = 1 + \mathfrak{g}\_{\mathfrak{s}}/\mathbb{R} \cdot U\_2. \tag{63}$$

These expressions let us write equation (61) as simply

$$
\begin{bmatrix} \dot{U}\_1 \\ \dot{U}\_2 \end{bmatrix} + \begin{bmatrix} b/\text{C}\_m & a/\text{C}\_m \\ b/\text{C}\_m & a/\text{C}\_m \end{bmatrix} \cdot \begin{bmatrix} U\_1 \\ U\_2 \end{bmatrix} = \begin{bmatrix} R/C\_m \\ R/C\_m \end{bmatrix} . \tag{64}
$$

Plotting the simulation data of the network's forced response from **Figure 5B** on a phase-space diagram (**Figure 5C**) suggests that *u* causes *U*<sup>1</sup> and *U*<sup>2</sup> to change in such a way that the state of the system (*⃗x*(*t*), blue) moves*tangent to* the equilibrium curve (dashed violet), with some constant distance *away* from it. These curves do not overlap because the forced response is not the same as the equilibrium condition while the external current *u* is applied. Motion in the *X*<sup>2</sup> direction is resisted by the neural dynamics, much how a spring resists the translation of an object with an applied force.

Nonetheless, these direction-dependent responses suggest that the state can be generalized into two decoupled degrees of freedom in the phase-space: unresisted, marginally stable motion parallel to the equilibrium curve (*X*1, green in **Figure 5C**); and resisted, stable motion away from the equilibrium curve (*X*2, red). The natural coordinates, *⃗x* = [*U*1*, U*2] *T* , are transformed into generalized coordinates, *⃗q* = [*q*1*, q*2] *T* , by a matrix *X* comprised of the eigenvectors of *J*. This same transformation matrix is used to transform *J* into the generalized coordinate system, yielding *Jq*. *J<sup>q</sup>* is diagonal, decoupling the dynamics of the generalized coordinates and enabling us to quantify how quickly *⃗x* moves parallel to the equilibrium curve.

Appendix shows the calculation of *X*, with *q*<sup>1</sup> representing the marginally stable mode and *q*<sup>2</sup> representing the stable mode. Using *X*, we can transform the system into generalized coordinates. First, we write the dynamics from equation (64) in a compact format.

$$
\dot{\vec{x}} + J\vec{x} = \vec{F},\tag{65}
$$

where *J* is the square matrix in equation (64) and

$$\vec{F} = \begin{bmatrix} \mathbf{R}/\mathbf{C}\_m + \boldsymbol{\mu}/\mathbf{C}\_m \\ \mathbf{R}/\mathbf{C}\_m \end{bmatrix} . \tag{66}$$

The generalized coordinates, *⃗q*, are defined as

$$
\vec{x} = X\vec{q}.\tag{67}
$$

To transform equation (65) into generalized coordinates, premultiply both sides of equation (65) by *X −*1 ,

$$
\dot{\vec{q}} + J\_q \vec{q} = \vec{Q}, \tag{68}
$$

where *J<sup>q</sup>* = *X −*1 *JX* and *Q⃗* = *X <sup>−</sup>*<sup>1</sup>*⃗F*. The top and bottom rows of equation (68) are decoupled because *J<sup>q</sup>* is a diagonal matrix. Furthermore, *J i,i <sup>q</sup>* = *λi*, meaning that *J* 1*,*1 *<sup>q</sup>* = 0, so the system simplifies even further.

To find the particular solution of this system, we can guess the form of *qp*,1 and *qp*,2, and substitute those in to equation (68). We observe that *q*˙ <sup>1</sup>(*t*) = *B · u* in steady state, where *B* is a constant that relates *q*˙ <sup>1</sup>(*t*) and *u*. *q*1(*t*) would be the integral of *q*˙ <sup>1</sup>(*t*), but because the top row of*J<sup>q</sup>* is zeros, it will not appear in the particular solution, and thus need not be explicitly included. We also observe that *q*˙ <sup>2</sup>(*t*) = 0 in steady state, so *q*2(*t*) = *D*, a constant. We can calculate *Q⃗* = *X −*1 *F* using *X −*1 , which is calculated in Appendix (equation (A9)). Solving for the particular solution of this system, ˙*⃗qp* (*t*),

$$\dot{\vec{q}}\_{\mathcal{P}}(t) + f\_{q}\vec{q}\_{\mathcal{P}} = \begin{bmatrix} B \cdot u \\ \mathbf{0} \end{bmatrix} + \begin{bmatrix} \mathbf{0} & \mathbf{0} \\ \mathbf{0} & \frac{a+b}{C\_{m}} \end{bmatrix} \cdot \begin{bmatrix} q\_{p,1} \\ D \end{bmatrix}$$

$$= \begin{bmatrix} \frac{a \cdot d}{C\_{m} \cdot (a+b)} \cdot u \\ \frac{R\sqrt{2}}{C\_{m}} + \frac{b\sqrt{2}}{C\_{m} \cdot (a+b)} \cdot u \end{bmatrix} \tag{69}$$

$$\mathcal{B} = \frac{a \cdot d}{\mathcal{C}\_m \cdot (a+b)},\tag{70}$$

where *d* is defined in equation (A5). *B* describes how quickly *qp*,1 varies with *u*, but we want to know how quickly *U*<sup>1</sup> varies with *u*. Therefore, we use equation (67) to transform*⃗q*˙ *<sup>p</sup>* = [*B · u,* 0] *T* into natural coordinates to find ˙*⃗x*,

$$
\dot{\vec{x\_p}} = X \dot{\vec{q\_p}}\tag{71}
$$

$$
\begin{bmatrix}
\dot{U}\_{1,p}(t) \\
\dot{U}\_{2,p}(t)
\end{bmatrix} = \begin{bmatrix}
1/d & 1/\sqrt{2} \\
\end{bmatrix} \cdot \begin{bmatrix}
\frac{a \cdot d}{\mathcal{C}\_m \cdot (a+b)} \cdot u \\
0
\end{bmatrix} \tag{72}
$$

$$\dot{U}\_{1,\emptyset}(t) = \frac{a}{C\_m \cdot (a+b)} \cdot u \tag{73}$$

$$k\_l = \frac{a}{C\_m \cdot (a+b)}.\tag{74}$$

Recall that *a* and *b* are functions of *U*<sup>1</sup> and *U*2, respectively. This means that *ki*, the integral gain of the network, is not a constant. To place bounds on *ki*, let us substitute equations (62) and (63) into equation (74),

$$k\_i = \frac{1 + \mathbf{g}\_s/\mathbf{R} \cdot \mathbf{U}\_1}{\mathbf{C}\_m \cdot (2 + \mathbf{g}\_s/\mathbf{R} \cdot (U\_1 + U\_2))}.\tag{75}$$

We can now plug in different values of *U*<sup>1</sup> and *U*<sup>2</sup> to see how *k<sup>i</sup>* varies. Using equations (55) and (56), we find that the most extreme cases are when [*U*1, *U*2] = [0, *R*] and [*U*1, *U*2] = [*R*, 0]. We can plug these cases into equation (75) to find the minimum and maximum values for *ki*,

$$k\_{i, \text{min}} = \frac{1 + \text{g}\_{\text{s}}/\text{R} \cdot \text{0}}{\text{C}\_{\text{m}} \cdot (2 + \text{g}\_{\text{s}}/\text{R} \cdot (\text{0} + \text{R}))} = \frac{1}{\text{C}\_{\text{m}} \cdot (2 + \text{g}\_{\text{s}})} \tag{76}$$

and

$$k\_{i, \max} = \frac{1 + \mathcal{g}\_s / \mathcal{R} \cdot \mathcal{R}}{\mathcal{C}\_{\mathfrak{m}} \cdot (2 + \mathcal{g}\_s / \mathcal{R} \cdot (\mathcal{R} + 0))} = \frac{1 + \mathcal{g}\_s}{\mathcal{C}\_{\mathfrak{m}} \cdot (2 + \mathcal{g}\_s)}.\tag{77}$$

The difference between *ki*,min and *ki*,max:

$$k\_{i, \text{range}} = \frac{1 + \text{g}\_s}{\text{C}\_m \cdot (2 + \text{g}\_s)} - \frac{1}{\text{C}\_m \cdot (2 + \text{g}\_s)} = \frac{\text{g}\_s}{\text{C}\_m \cdot (2 + \text{g}\_s)}.\tag{78}$$

To find the mean rate of integration, we can calculate *ki*,mean = (*ki*,min + *ki*,max)/2,

$$k\_{i, \text{mean}} = \frac{1}{2 \cdot C\_m}.\tag{79}$$

This is the same value of *k<sup>i</sup>* obtained from computing *k<sup>i</sup>* when *U*<sup>1</sup> = *U*2. This simple expression is a useful relationship for tuning the integrator network. One may select *C<sup>m</sup>* to obtain the intended mean integration rate, and then minimize the variation of the integration rate by minimizing *gs*, as long as equation (57) is satisfied.

**Figure 5D** graphically demonstrates how *ki*,mean and *ki*,range determine *C<sup>m</sup>* and *gs*. Just as in equation (79), *ki*,mean is a function only of *Cm*. Therefore, the contour only shows vertical lines. The value of *ki*,range is minimized by decreasing either *g<sup>s</sup>* or *C −*1 *<sup>m</sup>* (i.e., increasing *Cm*). **Figure 6** shows simulation data of the integrator's response to a step input with eight different parameter value combinations. In every case, the change in *U*<sup>1</sup> is bounded by the values of *ki*,mean and *ki*,range. As shown in **Figure 5D**, increasing *C<sup>m</sup>* decreases the integration rate, and increasing *g<sup>s</sup>* increases the variation in the integration rate.

**Table 2** summarizes the design approach for this integrator network. The mean and range of the integration rate are free parameters that are determined by the intended network performance. Using these values and the constraint in equation (57), the neurons' *C<sup>m</sup>* value and the synapses' *g<sup>s</sup>* and ∆*E<sup>s</sup>* values can be fully specified.

#### **5. APPLICATION TO A ROBOT CONTROLLER**

We have used the methods in this paper to tune (i.e., select parameter values for) several different networks that control robotic stepping (Szczecinski and Quinn, 2017; Szczecinski et al., 2017a) and visual tracking (Szczecinski et al., 2017a). Once a network layout is determined, whether hypothetical or based on neurobiological findings, individual subnetworks can be identified and tuned to work together. **Figure 7** shows a simplified joint-control network in which different functional pathways are color-coded. This illustrates how these functional subnetworks enable the direct assembly of control networks based on neurobiology. The neurobiological inspiration for these networks and the results of robotic experiments are presented in Szczecinski and Quinn (2017) and Szczecinski et al. (2017a), and so are omitted here.

The joint network in **Figure 7** uses three simple descending commands (body heading, stride length, and reference leg load) to control the walking motion of one joint of a leg. The descending commands modulate the output of a central pattern generator (CPG) to control the speed of the motion, and sensory feedback is used to adjust both the timing and amplitude of motor output. Addition pathways are drawn in red. These include the

mapping between body heading and stride length (i.e., descending commands, drawn in gray) to the PEP and AEP (Szczecinski and Quinn, 2017). The PEP can also be modulated by force feedback, which compares the load on the leg to a reference value (Szczecinski and Quinn, 2017, in review). This requires a subtraction network, drawn in orange, to compute if there is too much or too little load on the leg. The difference is used to adjust the PEP Memory network, which is an integration network, drawn in blue. This network adjusts the PEP over time, and remembers the motor command that produces the intended force.

The output of the CPG, drawn in purple, excites the motor neurons. Tuning CPG dynamics is discussed in our previous work (Szczecinski et al., 2017b). The PEP and AEP neurons adjust the motor output via multiplication pathways, drawn in green, which scale CPG output to the motor neurons based on the intended range of motion. Motor neuron activity controls the motor velocity, and the *θ* neuron receives position feedback from the motor via the mappings in **Figure 1B**. The motor velocity, computed by the cyan differentiation pathway, reinforces ongoing CPG behavior through the ˙*θ* neuron (Szczecinski et al., 2017b). A division pathway (not shown) can be used to normalize the velocity feedback to the joint's commanded range of motion, simplifying the control of stepping frequency. The ˙*θ* neuron also receives some input from the Load neuron, ensuring that stance phase is stable (Szczecinski and Quinn, 2017, accepted). By using the functional subnetworks and the design constraints presented in this paper, we can rapidly and directly assemble models of neural systems that perform as intended without hand-tuning or optimization methods.

How are the "Free Parameters" in **Tables 1** and **2** chosen? The free parameters fall into two classes: reversal potentials (i.e., ∆*Es*) and dynamical constants (e.g., *k*, *τ* , etc.). The reversal potentials are informed by biology. In this paper, we kept *−*40 *<* ∆*E<sup>s</sup> <* 194 mV (i.e., *−*100 *< E<sup>s</sup> <* 134 mV). The modeler could use reversal potentials from specific synapses if that data were available. The dynamical constants are informed by the function of the robot. For example, the *ksyn* of the subtraction network in **Figure 1B** controls the stiffness of the controller, and may destabilize the system if not tuned to match the mechanical properties of the robot (Szczecinski et al., 2017b).

As another example, *τ<sup>d</sup>* and *k<sup>d</sup>* of the differentiator network in **Figure 7** determines the robustness of CPG rhythms, and how well it entrains to sensory feedback (Szczecinski et al., 2017b). A slow, adaptively-walking robot may want a high *k<sup>d</sup>* to regularize CPG oscillations, whereas a fast running robot may want a low *k<sup>d</sup>* to be less sensitive to sensory feedback. Picking specific values for these free parameters ultimately depends on the intended behavior of the robot. The constraints in this paper enable the designer to think in terms of more traditional robotics quantities, and use these to set neural and synaptic parameter values, which may be less intuitive.

## **6. DISCUSSION**

In this paper, we presented analytical methods for setting parameters in dynamical neural networks that can add, subtract, multiply, divide, differentiate, and integrate incoming signals. Such operations are at the core of control, and these techniques enable control networks to be assembled rapidly and tuned directly. This work primarily identifies constraint equations, not unique values, that govern how parameters should be tuned. Thus, many different networks may perform the same function with different parameter values, as observed in real neural circuits (Prinz et al., 2004). Since these results are analytical, not based on machine learning or optimization, there is no concern about these networks over- or under-fitting training data, and their behavior is provable. These techniques build on our previous analysis of synthetic nervous systems (Szczecinski et al., 2017b) and have been validated through several studies with our robot, MantisBot (Szczecinski and Quinn, 2017; Szczecinski et al., 2017a).

All of the results from this paper make it easier to tune neuromechanical models of animals, as well. Many such models have been created to study the principles underlying insect (Daun-Gruhn and Tóth, 2010; Szczecinski et al., 2014) and mammalian (Hunt et al., 2015a; Markin et al., 2016) locomotion alike. Oftentimes, parameters of these models are tuned by hand to obtain the intended motion, which is a painstaking, slow, and imprecise process. The analysis in this paper can make neuromechanical models come together more quickly, and have more predictable behavior, leading to more thorough scientific investigations. More precise tuning methods enable more thorough validation or invalidation of hypotheses. Faster tuning methods enable more rapid validation or invalidation of hypotheses. For example, these methods could be used to improve the coordination our previous cockroach model (Szczecinski et al., 2014). In the model, curve walking of varying radii was achieved by modulating muscle activations with broad descending commands. However, the coordination, reliability, and repeatability of such motion could be improved with the methods of this paper, enabling us to improve or reject the model.

## **6.1. Simplifications**

Some of the calculations in this paper are based on approximations, which lead to inaccuracies in the calculations of the subnetworks. One example is that the subtraction network does not produce linear output. This non-linearity occurs because the reversal potentials of synapses are rarely much lower than the resting potentials of neurons, requiring large values of *gs*,2 to build a subtractor where *ksyn* = 1. A large *gs*,2 value increases *Upre*,2's effect on the denominator of *U ∗ post*'s response, causing the synaptic input to reduce *Upost*'s sensitivity to inputs. This is particularly noticeable in the differentiator's response (**Figure 4**), especially as *ksyn* increases.

Another example of a simplification we made is that our calculation of *k<sup>i</sup>* only used the particular solution of the system. This means that a transient response also exists, which we did not compute. In addition, *k<sup>i</sup>* is a function of *U*<sup>1</sup> and *U*2. This means that *k<sup>i</sup>* is not a constant for this network. However, the impact of *U*<sup>1</sup> and *U*<sup>2</sup> on *k<sup>i</sup>* can be minimized by minimizing *g<sup>s</sup>* and maximizing *R*, as we showed in Sec. 4.2.

However, the developed networks are not intended to act as perfect analogs to their mathematical counterparts. These networks are intended to act as representations of real neural circuits, which likely do not act as perfect adders, multipliers, differentiators, etc. Dynamic and transient effects are a real part of biological control systems, and effective neural controllers have developed around these idiosyncrasies and have likely evolved to even exploit many of these aspects. In spite of these issues, the methods in this paper are valuable. Our recent robotics work (Szczecinski and Quinn, 2017; Szczecinski et al., 2017a), as well as related work in progress, is proof of the effectiveness of this approach.

## **6.2. Why Put Neurons in the Way?**

The methods in this paper enable the direct construction of networks that perform arithmetic and dynamic calculations. Why bother building neural networks just to recreate mathematical operators? We believe there are several reasons to take this approach. From a neurobiology perspective, the constraints that we have identified may help explain why certain structures are common in the nervous system (David Friel, personal correspondence). For instance, mutually inhibitory parallel pathways are common in the thoracic control of insect locomotion (Büschges and Wolf, 1995), which may function as subtraction networks in negative feedback loops. As another example, networks in the retina of the rabbit are selectively sensitive to motion in one direction or the other (Barlow and Levick, 1965). Such a network could be constructed by using adjacent cells in the retina as inputs to differentiator networks. This would be consistent with both the function of direction-sensitivity, as well as the laterally inhibitive structure. Even though such consistency does not guarantee that the animal's nervous system functions precisely this way, the design methods in this paper may aid in understanding the function of neural networks found in animals.

Additionally, the constraints that we identified may be used to constrain parameter values in large brain models. Rather than using global search techniques to understand the dynamics of a large pool of neurons, we believe it may be faster to begin with a number of functional subnetworks, and then use local search techniques to tune the connections between them. In this way, the designer is certain that parts of the network perform specific, useful computations, rather than naively optimizing a large network (Haferlach et al., 2007;Agmon and Beer, 2013;Izquierdo and Beer,

#### **REFERENCES**


2013). The end result is something like a genetic program, but in a neuroscience context.

#### **AUTHOR CONTRIBUTIONS**

NS led research on functional subnetworks and led the preparation of the manuscript. AH aided in the preparation of the manuscript. RQ provided critical oversight of the research and aided in the preparation of the manuscript.

## **FUNDING**

This work was supported by NASA Space Technology Research Fellowship NNX12AN24H, as well as a GAANN Fellowship.


to turning of the cockroach *Blaberus discoidalis*. *Biol. Cybern.* 108, 1–21. doi:10.1007/s00422-013-0573-3


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2017 Szczecinski, Hunt and Quinn. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## **APPENDIX**

## **A. Derivation of Integrator Eigenvalues and Eigenvectors**

We find the eigenvalues *λ*<sup>1</sup> and *λ*<sup>2</sup> and the associated eigenvectors *X*<sup>1</sup> and *X*<sup>2</sup> of the Jacobian matrix by the eigenvalue problem,

$$\det(I - \lambda\_i \cdot I) = 0 \tag{A1}$$

$$J \cdot X\_i = \lambda\_i \cdot X\_i,\tag{A2}$$

where *i* is the index of the eigenvalue (1 or 2), *I ∈* R 2*×*2 is an identity matrix, and *J* is the square matrix from equation (64). Solving for *λ*,

$$
\lambda\_1 = 0, \ \lambda\_2 = \frac{a+b}{C\_m} > 0. \tag{A3}
$$

Because *J* is on the same side of the equation as ˙*⃗x* (see equation (65)), *λ*<sup>2</sup> *>* 0 indicates a stable system (e.g., as the stiffness matrix of a physical system). *λ*<sup>2</sup> *>* 0*∀⃗x*, because *a >* 0 and *b >* 0. The definition of *a* in equation (62) shows that *a >* 0 because *g<sup>s</sup> >* 0 (it is a physical quantity) and *U*1/*R∈*[0, 1]. The same reasoning applies to *b*.

We use the eigenvalues to find their associated eigenvectors,

$$X\_1 = \begin{bmatrix} 1 \\ -b/a \end{bmatrix}.\tag{A4}$$

Normalizing *X*<sup>1</sup> to 1,

$$X\_1 = \begin{bmatrix} \frac{1}{\sqrt{1^2 + (-b/a)^2}}\\ \frac{-b/a}{\sqrt{1^2 + (-b/a)^2}} \end{bmatrix} = \begin{bmatrix} 1/d\\ -b/(ad) \end{bmatrix}, \quad d = \sqrt{1^2 + (-b/a)^2}. \tag{A5}$$

Next, we calculate

$$X\_2 = \begin{bmatrix} 1 \\ 1 \end{bmatrix} . \tag{A6}$$

Normalizing *X*<sup>2</sup> to 1,

$$X\_2 = \begin{bmatrix} 1/\sqrt{2} \\ 1/\sqrt{2} \end{bmatrix}.\tag{A7}$$

We now know the transformation matrix between the natural coordinates, *⃗x* = [*U*1*, U*2] *T* , and the generalized coordinates, *⃗q* = [*q*1*, q*2] *T* :

$$\vec{X} = X \cdot \vec{q}, \qquad X = [X\_1, X\_2] = \begin{bmatrix} 1/d & 1/\sqrt{2} \\ -b/(ad) & 1/\sqrt{2} \end{bmatrix}. \tag{A8}$$

We will also make use of *X <sup>−</sup>*<sup>1</sup> when transforming between natural and generalized coordinates. We can analytically invert *X* from equation (A8),

$$X^{-1} = \frac{a \cdot d \cdot \sqrt{2}}{a+b} \cdot \begin{bmatrix} 1/\sqrt{2} & -1/\sqrt{2} \\ b/(ad) & 1/d \end{bmatrix} = \begin{bmatrix} \frac{a \cdot d}{a+b} & \frac{-a \cdot d}{a+b} \\ \frac{b \cdot \sqrt{2}}{a+b} & \frac{a \cdot \sqrt{2}}{a+b} \end{bmatrix} \tag{A9}$$

# Human-Derived Disturbance Estimation and Compensation (DEC) Method Lends Itself to a Modular Sensorimotor Control in a Humanoid Robot

#### Vittorio Lippi\* and Thomas Mergner

*Neurology, University Clinics of Freiburg, Freiburg, Germany*

The high complexity of the human posture and movement control system represents challenges for diagnosis, therapy, and rehabilitation of neurological patients. We envisage that engineering-inspired, model-based approaches will help to deal with the high complexity of the human posture control system. Since the methods of system identification and parameter estimation are limited to systems with only a few DoF, our laboratory proposes a heuristic approach that step-by-step increases complexity when creating a hypothetical human-derived control systems in humanoid robots. This system is then compared with the human control in the same test bed, a posture control laboratory. The human-derived control builds upon the identified disturbance estimation and compensation (DEC) mechanism, whose main principle is to support execution of commanded poses or movements by compensating for external or selfproduced disturbances such as gravity effects. In previous robotic implementation, up to 3 interconnected DEC control modules were used in modular control architectures separately for the sagittal plane or the frontal body plane and successfully passed balancing and movement tests. In this study we hypothesized that conflict-free movement coordination between the robot's sagittal and frontal body planes emerges simply from the physical embodiment, not necessarily requiring a full body control. Experiments were performed in the 14 DoF robot Lucy Posturob (i) demonstrating that the mechanical coupling from the robot's body suffices to coordinate the controls in the two planes when the robot produces movements and balancing responses in the intermediate plane, (ii) providing quantitative characterization of the interaction dynamics between body planes including frequency response functions (FRFs), as they are used in human postural control analysis, and (iii) witnessing postural and control stability when all DoFs are challenged together with the emergence of inter-segmental coordination in squatting movements. These findings represent an important step toward controlling in the robot in future more complex sensorimotor functions such as walking.

Keywords: sensory-motor system, humans, neuromechanical modeling, modular control architecture, humanoid robot experiments

#### Edited by:

*Manish Sreenivasa, Heidelberg University, Germany*

#### Reviewed by:

*Egidio Falotico, Sant'Anna School of Advanced Studies, Italy Malte Schilling, Bielefeld University, Germany*

#### \*Correspondence:

*Vittorio Lippi vittorio.lippi@uniklinik-freiburg.de*

Received: *31 January 2017* Accepted: *24 August 2017* Published: *08 September 2017*

#### Citation:

*Lippi V and Mergner T (2017) Human-Derived Disturbance Estimation and Compensation (DEC) Method Lends Itself to a Modular Sensorimotor Control in a Humanoid Robot. Front. Neurorobot. 11:49. doi: 10.3389/fnbot.2017.00049*

## INTRODUCTION

## Human Posture Control Modeling and Neurorobotics

Human postural control attracts considerable interest in healthcare research worldwide for reasons such as "fall of the elderly" and neurological impairments such as ataxia in cerebellar patients or deficient movement control in Parkinson's disease. Developing model-based diagnostics as well as therapeutic and rehabilitative interventions is an important aim of this research. Engineering-inspired approaches to model the human sensorimotor control and its failures have a long tradition (e.g., Nashner, 1972; Hajos and Kirchner, 1984; Johansson et al., 1988; Kuo, 1995; Fitzpatrick et al., 1996). These approaches often address reactive postural responses to well-controlled external stimuli, which lend themselves to system identification, whereas the exact input for voluntary movements is generally unknown. Two of the following recent modeling approaches were especially influential, yet their clinical application can often prove to be problematic.

One modeling approach used system state estimation based on multi-sensory integration under noise optimization principles. This approach builds on biological textbook knowledge on human sensors and anthropometrics and, based on this knowledge and engineering principles, analyzes the human posture control using noise optimizing principles (van der Kooij et al., 2001; Kuo, 2005). For example, it predicts that, whenever possible, posture control preferentially uses proprioceptive rather than vestibular information, as the vestibular information is the one containing more noise. These models allow general predictions on human preferences for certain sensory environments and may define which sensory deficit in patients tends to increase the danger of falling. However, poor correspondence of these models with human neurophysiology and anatomy restricts its clinical usefulness for diagnosing more specific posture control problems of an individual patient.

The other modeling approach used time series or frequency domain data gathered from posture control experiments in humans to establish the simplest model compatible with known human physiology and anatomy that would allow reproduction of the data with identified model parameters. Best known is the independent channel (IC) model of Peterka (2002) for single inverted pendulum (SIP) scenarios. It identifies sensory weights depending on the proper selection and interpretation of results using different stimuli and test conditions. From extensions of this model to more complex scenarios it was concluded that available engineering methods are in principle capable of arriving at multi-segmental control models. However, the increasing model complexity and the proliferation of parameters tend to reduce the chance of unequivocally identifying the control parameters in health and disease (Mergner and Peterka, 2017). The problem is aggravated by non-linearities of the human control system (described as detection thresholds in Maurer et al., 2006).

As such, there currently exists a dilemma as to which methodology can be used to establish model-based diagnostics and therapeutic or rehabilitative interventions for patients with impaired postural control. The goal of the present study is to contribute to a heuristic solution. By this we mean a practicable solution that proceeds from an established model of the human control in the SIP scenario and uses plausible arguments and steps for its extension to more degrees of freedom (DoFs). To evaluate the appropriateness of these steps, we test their effects in special-purpose humanoid robots with human-inspired anthropometrics, sensors, and actuators. In this way a "real world" challenge is imposed which accounts for noisy and inaccurate sensors and non-ideal actuation and mechanics. For as much correspondence as possible to the human situation, these tests are performed in the same testbed that is also used for the human subjects, i.e., in a human posture control laboratory. As described below, first steps in this heuristic approach have successfully been performed. Point of departure was the "disturbance estimation and compensation" (DEC) model for the SIP scenario, which shares basic similarities with the IC model (Mergner et al., 2003; Maurer et al., 2006; Mergner, 2010). While several extension steps have been described previously (see section Previous and Current Steps in the Heuristic Approach), here we report on an extension to a 14 DoF robot and examine whether a modular architecture consisting of a net of DEC controls in the robot's sagittal plane cooperates in a conflict-free way with a corresponding but independent net of DEC controls in the frontal plane during both disturbance compensation and commanded ("voluntary") movements.

The following subsections aim to combine state of the art human sensorimotor issues with recent robotics issues dealing with posture control. Here and in later sections, we include brief descriptions on the current state of our bottomup implementation of a human-like postural control in robots. First, we briefly review the biological basis of the DEC model (section Main Features of the Human DEC Model), then consider related issues in neuroscience and robotics (section Modular Control Issues in Neuroscience and Robotics), and finally present a list of the previously performed steps in our neurorobotics approach and the new steps taken in this study (section Previous and Current Steps in the Heuristic Approach).

#### Main Features of the Human DEC Model

Both the DEC model and the clearly simpler IC model can describe results from the same protocols for human balance control experiments. The IC model is a linear model that analytically describes human sway responses to support surface tilt in the frequency domain based on a control by proprioceptive, vestibular, and visual feedback channels (Peterka, 2002). The model allows for identification of important features of the human postural control system, the most important ones being time delays in the order of 100–200 ms associated with low loop gain and, as a consequence, soft mechanical compliance, and low energy consumption. It also identifies sensory reweighting, meaning that humans adjust their use of sensory information to changes in perturbation amplitude and modality. While the IC model describes this feature by using different sets of control parameters, the DEC model (**Figure 1**) is able

FIGURE 1 | Schema of DEC control of the ankle joint in the single inverted pendulum, SIP, scenario with support surface tilt forward (this model is the basis for the modular control architexture in multi-DoF systems). Four external disturbances (external force *Fext*, gravitational force *Fg*, foot-space rotation α*FS* and foot-space linear acceleration *<sup>x</sup>*¨*FS*) give rise to the joint torques (*Text*, *<sup>T</sup>grav*, *<sup>T</sup>prop*, and *<sup>T</sup>in*(=*inertial*) , respectively). Sensors (VEST, vestibular; PROP, proprioceptive; TORQUE) are actually networks that combine signals from a variety of transducers (e.g., 3 VEST sensors result from vestibular canal-otolith intractions; Mergner et al., 2009) to yield information on the following physical quantities: *x*¨*hx,* head acceleration in x (sagittal forward) direction, and α*hs* and α˙ *hs*, head-in-space angle and angular velocity (dashes indicate proprioceptive coordinate transformation to lower body segments, see Figure 2; here they point out that in the SIP scenario these vestibular signals refer to the whole body above the anke joints, includinging the legs, i.e., α*ls* and α˙*ls*); α*lf* and α˙*lf* represent leg-to-foot angle and angular velocity, respectively. Colored boxes derive disturbance estimates (indicated by hats) from the reported physical quantities. Variable foot-in-space angular velocity (α˙*fs*) is time-integrated leading to the estimate αˆ*fs* of foot-in-space position; by means of this signal the proprioceptive signal α*lf* is being "upgraded" (transformed) into space coordinates (α*ls*′ ). C, neural controler; 1t, lumped neural time delay; α*ls!*, desired leg-space angle. Box 1/*mgh* transforms torque into an equivalent of an angle. Passive stiffness and its modulation (in humans achieved by muscle co-contration) is omited here for simplicity (compare Ott et al., 2016 as to its role for control stability). In current versions of the DEC concept, the input may be varied in three ways depending on the task. The task shown in this figure is to reach and maintain a given orientation of the supported body segment, here the leg segment (in fixed alignment with upper body), in space (α*ls!*). In situations where the body COM changes with the body configuration or load distribution, the task refers to the body COM in space (*bs*; α*bs!*) and requires that the current COM location is taken into account in the control. This has been experimentally tested and modeled for human responses in ankle and hip joints to support surface tilts in the sagittal plane (Hettich et al., 2014). As an intuitive third possibility, one may want to reach and maintain a given joint angle, which in the scenario of this figure would mean to command the leg-foot angle α*lf !* as input and to neglect estimate αˆ*fs*.

to predictively describe the data from various experimental conditions with one set of control parameters. The DEC model contains synthetic and holistic features-synthetic in the sense that it uses disturbance estimations inspired from studies on multisensory fusions in human self-motion perception (Mergner et al., 1997; Mergner and Rosemeier, 1998), holistic in the sense that it also integrates the control of movements in a single structure.

The DEC model extends upon the engineering concept of the servo control by negative feedback (see Wiener, 1948, for the early developments of the concept). In neurology, Merton (1953) used this concept to explain the role of the muscle stretch reflex for the control of posture and movements. He posits that a PD-controller adjusts the force of the muscles so as to produce the desired pose or movement. This would be achieved through negative feedback from proprioceptive sensors, i.e., by feeding the controller with the difference between the desired and the sensed joint angle. However, later researchers considered this concept to be problematic. One reason was that the biological time delay in the feedback loop does not allow for stable performance when large external disturbances such as gravity require high loop gains (see McIntyre and Bizzi, 1993). The DEC model overcomes this problem by estimating the external disturbances and commanding the servo to produce the extra force required for their compensation. The underlying principle is known in control theory under different names, e.g., "feed forward disturbance correction" (Roffel and Betlem, 2006); similar (Luecke and McGuire, 1968; Zhong et al., 2012); in German consistently "Störgrößenaufschaltung;" Bleisteiner et al., 1961). The DEC mechanisms operate context- and intent-dependently (Mergner, 2010). In the terminology of sensorimotor physiology, we subsume them in contradistinction to "short latency reflexes" under "long latency reflexes," which are known to be modifiable by higher brain centers such as the cerebral cortex (see Pruszynski and Scott, 2012). Disturbance compensation in the DEC is thought to arise reactively with unforeseen external disturbances

FIGURE 2 | (A) Schematic representation of the interconnections between the DEC control modules, in terms of down-channeling and up-channeling of sensory information in a given body plane (here sagittal plane; compare Figure 4 for analogies in the frontal plane). Shown by solid downward arrows is the distribution of the vestibular signal "head angle with respect to the gravitational vertical" (α*hs*). Analog distributions hold for the angular velocity signal α˙ *hs* and the head linear acceleration signal in x (sagittal forward) direction, *x*¨*Hx*. The gray upward arrows show the up-channeling of processed signals, representing the causal physical interactions in a stack of superimposed body segments. For example, up channeling of αˆ*fs* from the ankle module to the knee module and fusing it with the proprioceptive α*lf* signal may be used in the knee module as a predictor of α*hs* (a mechanism that bypases the sensory feedback chain and increases control stability). This up-channeling is especially effective if the support surface is stationary, whereby αˆ*fs* becomes subthreshold (through a velocity threshold) and with it the effect of the particularly noisy vestibular α˙ *hs* signal; see Mergner et al., 2009). (B) Picture of Lucy Posturob (from right and behind) balancing on the motion platform in the human posture control laboratory.

as well as in the form of predicted-sensory estimates, issued by higher brain centers with foreseen external or self-produced disturbances. Such a "proactive" compensation is assumed to be advantageous compared to a "reactive" one due to short central time delays and absence of sensory noise, leading to more stable control (see Maurer et al., 2006). Noticeably, several reactive and proactive compensatory actions may arise simultaneously, even during voluntary movements (i.e., the "superposition law" applies despite the non-linarities in the disturbance estimates).

As previously explained in more detail (Maurer et al., 2006), humans use several sensory inputs (proprioceptive, vestibular, visual, haptic contact, torque, and pressure) for posture control. They combine this information in various ways to estimate physical variables such as joint angle and angular velocity from muscle spindle, skin receptor, tendon organ, and torque inputs, as well as head linear and angular acceleration from vestibular otolith and semicircular canal inputs, for example. From the estimated physical variables, the DEC model then derives estimates of four classes of disturbances that may influence the torque acting on the skeletal joints (colored boxes in **Figure 1**): (1) Rotation and (2) translation of the support (be this a supporting body segment or an external support), (3) gravity and other field forces, and (4) contact forces ("external torque"). As depicted in **Figure 1** (which refers to the SIP scenario), in the case of balancing, the input of the model is the desired angle of the leg segment (here for SIP) in space αls!, a signal that is thought to stem from higher brain centers including the cerebral cortex. The cortex builds a coherent and continuously updated sensorimotor model of the body as a whole from many input sources. This is often referred to as "body schema:" a conception from neurology that has found its way into humanoid robotics (Morasso, 2013). In the DEC model, noticeably, both the cortical movement commands from the body schema and the conscious perception of the produced movement almost exclusively reflect kinematics. Signals related to stereotypically occurring forces from gravity, link inertia, inter-link coupling forces, etc. are hidden, so to speak, because they are compensated by the DEC mechanism (details in Mergner, 2010).

In this form, the DEC concept qualified for a modular control architecture in scenarios with >1 DoF (see Section Previous and Current Steps in the Heuristic Approach). Modular control as a concept has attracted considerable interest earlier in neuroscience and robotics. The following section points out differences to the present concept.

## Modular Control Issues in Neuroscience and Robotics

In the DEC concept, a modular control by two DEC modules has been used for the modeling of human ankle joint and hip joint responses to support surface tilt and for mimicking the responses in a robot (Hettich et al., 2014). Further work supported this idea of a modular control, for example when we added a third DEC control in a robot for proactive squatting in the knee joints during reactive balancing (Ott et al., 2016 and https://www.youtube. com/watch?v=3ALCTMW3Ei4).

Generally, modularity is often applied at the level of kinematics, as is the case in neuroscience with "movement synergies," where each control module commands a subset of the controlled DoFs. Specifically, a kinematic synergy is often defined as a function mapping a scalar value to several DoFs. Such synergies are often used to reduce the dimensionality of the control problem with the result that the number of controlled synergies is smaller than the total number of DoFs. The principle has not only inspired research of human walking (e.g., Ivanenko et al., 2003), but also various fields of humanoid control (Hauser et al., 2007). Noticeably, however, the number of required synergies in complex movements may reach the number of DoFs. This may happen for example when movement control is organized in terms of eigenmovements (Alexandrov et al., 2017) where the aim is to free the control from coupling forces stemming from other movements. Certain synergies studied in humans, such as torque synergies, are so far largely neglected in robotics. Another related concept is that of motor primitives, originally meaning basic kinematic, dynamic, or muscular building blocks of movements arising at neuronal levels (Flash and Hochner, 2005). Generally, motor primitives can be combined in several ways, i.e., to act simultaneously by summing them or to obtain superposition of effects, as is the case with synergies, or serial effects when using them in a sequence, as with predefined trajectories or velocity profiles. The concept of motor primitives has found extensive application in robotics for learning of motor tasks (Schaal et al., 2003). Some implementations of modularity are not only advantageous in that they reduce complexity in control design, but also in increasing control robustness (in case one module fails, remaining modules can take over).

Compared to these modularity concepts, the architecture of combining DEC modules is clearly distinct. Here, each DoF of the human skeletal system is controlled by one DEC module. This even applies to situations where two or more modules do the same job and conceptually may be viewed as one module (e.g., during sagittal body sway about the two ankle joints with aligned axes and the body weight equally distributed on both feet). Each module determines the torque to be applied to the controlled DoF. The desired trajectory is specified as an input to each module. All modules have essentially the same structure and have no internal model of the whole system. Yet, the modules operate not completely independently of each other, because they exchange sensory information through coordinate transformation across the joints that interconnect the body segments (**Figure 2A**). Compared to the above examples of modular control, the DEC model can be defined as a low level control system that takes care of the fundamental task of posture control for undisturbed motor execution and acts at the level of joint kinematics. Coordination between different joints may emerge from the interaction between the modules and the body mechanics under task, even if no kinematic synergy is explicitly specified. For example, hip-ankle coordination in the experiments of Hettich et al. (2014) emerged from the tasks for the ankle and hip controllers to maintain the body COM over the feet as base of support and the trunk upright, respectively.

Further aspects will be considered in section Discussion.

## Previous and Current Steps in the Heuristic Approach

In previous works, the DEC model was subjected to "real world" tests, separately for the sagittal and the frontal plane. The tests were performed with humanoid robots equipped with humaninspired sensors, actuation, and anthropomorphic properties in the human posture control laboratory. Robot responses to support surface tilt in the sagittal plane were successfully compared to human responses once in a SIP scenario (Mergner et al., 2009) and later in a DIP scenario (Hettich et al., 2014). The latter study also showed that it sufficed to use interconnected DEC control modules for the ankle and the hip joints in order to simulate the human responses in the robot (for simplification, the joints on both sides were mechanically coupled). Interestingly, the ankle-hip coordination emerged from the interconnection of the two modules (see section Modular Control Issues in Neuroscience and Robotics). Subsequent experiments focused on a transfer of DEC to the frontal plane (14 DoF robot; Lippi et al., 2016) and the use of the knee joints to test the control of squatting movements in the sagittal plane. To this end, realization of the control in Simulink (The MathWorks Inc., Natick, USA) and using force controlled actuation allowed us to transfer the DEC control to the DLR robot TORO for comparison with a fully model-based control (Ott et al., 2016 and https://www.youtube. com/watch?v=3ALCTMW3Ei4).

This study investigates whether the networks of DEC modules previously controlling the robots' sagittal plane and frontal plane separately would adequately cooperate when combined during movements in intermediate planes. Specifically, we asked whether the physical linkages given by the robot's body suffice to guarantee adequate cooperation between the sensorimotor controls, or whether a supervising full body model or some other form of software linkage between the two networks would be required. The experiments are performed in a robot with 14 DoFs (Lucy Posturob, **Figure 2B**) using one and the same set of control parameters for non-trivial scenarios that draw on the scalability of the DEC modular control. The tests included both voluntary movements and balancing responses to passive body motions in an intermediate plane. Postural and control stability across all DoF of the robot were tested when the robot performed squatting movements in the knee joints, evoking the emergence of inter-segmental coordination in the ankle and hip joints. Further experiments aimed to provide a quantitative characterization of the robot's balancing responses in terms of FRFs (as they are used in human postural control analyses) and tested postural stability when increasing frequency of voluntary movements simultaneously in the sagittal and frontal body planes up to the performance limits.

## METHODS—ROBOTIC PLATFORM AND EXPERIMENTAL PROCEDURES

#### Humanoid Robot Lucy Posturob

The Posturob III robot, Lucy Posturob (details in **Figure 3**), was constructed as a humanoid of 1.5 m body height and ∼20 kg body weight with anthropometric parameters inspired by Winter (1990). Its body consists of the upper body (HAT, for head, arms, and trunk), pelvis, the two thighs, shanks (lower leg), and foot segments, all made of aluminum and interconnected by hinge joints. The total of 14 DoF (**Figures 3A,B**) comprise 2 DoF per ankle joint, 1 per knee, 3 per hip, and 2 DoF for the lower vertebral column ("lumped" DoFs across the vertebrae of the back bone).

Using an analog acquisition board, a PC read from mechatronic sensors the signals joint torque, joint angular position and velocity, and the anterior-posterior and mediolateral pressure distribution under each foot (not used in the present experiments). The same computer also read via USB the signals from a custom-made human-inspired artificial vestibular system (Mergner et al., 2009). The DEC control model was implemented and executed as a compiled Simulink model (Real-Time Windows Target, The MathWorks Inc., Natick, USA). The control system worked at 200 Hz. The delay of the system as a whole is estimated to be 20 ms (which corresponds approximately to the value to which we adjusted all sensory signals for correct time-matched interactions in the disturbance estimations). This delay is, admittedly, shorter that the lumped delays identified in humans (see section Main Features of the Human DEC Model). However, additional features have to be added to the DEC control modules in the Lucy robot before we can aim to repeat some of our experiments with more human-inspired time delays (as in Hettich et al., 2014).

The vestibular sensor was fixed to the robot's HAT segment. A bio-inspired algorithm (Mergner et al., 2009) fuses the inputs from 3 accelerometers and 3 gyrometers to estimate (i) HAT angular velocity in space, (ii) HAT angle with respect to the gravitational vertical, and (iii) linear acceleration of the upper HAT end representing head position (compare **Figure 3A**). Joint actuation was achieved by DC electric motors (part a in **Figure 3C**) that rigidly interconnected the two links by a screw/spindle system (part b). The spindle drive transformed the rotational movement of the motors into a lever movement, as measured by a linear potentiometer, part c, for producing joint angle. An inner torque control loop was implemented on the onboard robot electronics, receiving the torque command signal from the higher-level bio-inspired DEC algorithms on the PC. The force sensor, part d in **Figure 3C**, measured the tangential force acting on the joint in order to compute the joint torque signal used (i) in the DEC control loop and (ii) in the on-board control of the torque.

FIGURE 3 | Details of the humanoid robot Lucy. (A) Scheme of the 14 degrees of freedom. (B) Picture of Lucy Posturob (arms represented by weights; vestibular system is located in head; no visual system is currently provided by the eyes; electronics for actuation is contained in the pelvis). (C) Details of foot with actuation (a, DC motor; b, spindle; c, enconder; d, load cell as torque sensor). Actuation is the same in all DoFs.

## Disturbance Estimation in Multiple DoF System

The four estimated disturbances in a DEC control module are not directly available as sensory inputs and are hence reconstructed through inter-sensory interactions. The sensory inputs used in these and our previous robot experiments with the DEC model are from the vestibular system, joint proprioception and joint torque (compare **Figure 1**). Currently unconsidered are visual self-motion cues for balancing (see Assländer et al., 2015), and foot pressure cues.

In the multi-segment human body, the sensor fusions for the disturbance estimates use signals from both local sensors (proprioceptive joint angle and angular velocity sensors and torque sensors) and remote (e.g., vestibular) sensors. The vestibular signals in Lucy are conveyed to the control modules through down-channeling from their source in the head, as schematically shown in **Figure 2**. Additional information is required for the disturbance estimations concerning mass and inertia distribution of the supported body segments with respect to the supporting joint, as described for the SIP scenario in Mergner (2010). To further account for momentary changes of these parameters in the multi-segment system, Lippi et al. (2013) provided a general description of the down-channeling of the processed sensory information. In particular, for the two disturbance estimators of support rotation and gravitational torque, αˆfs and Tˆ grav (see **Figure 1**), the COM location of all segments above a given supporting joint is calculated step-wise downwards. This allows for treating the COMs of all supported segments in a given joint as if they were the COM of a singlesegment body. For example, with sagittal support surface tilts the hip joints compensate the gravity impact from any upper body lean, while the ankle joints are compensating the gravity effect due to the lean of the whole body COM. This example also shows that, while the action of a control module is local, the estimated disturbances represent global effects acting on the body. For the support surface translation estimates Tˆ in, the simplification by down-channeling concerns the joint torque produced by the combined inertial forces exerted by all upper body segments, while for the contact force estimates Tˆ ext it is the torque produced by combined forces having impact on the supported segments. The sensor fusions used to reconstruct these global variables are distributed among modules and are based on signal exchanges between modules.

The present study explains the generalized concepts of disturbance estimations with down and up channeling between modules (**Figure 2**) in more detail. It is worth noting that an advantage of using up-channeling is that the information of the physical variables conveyed upward are already processed at a lower level. This is especially relevant for the estimate of support surface tilt in the ankle joint. Running the input velocity signal through a threshold reduces noise of this estimate (see below). The up-channeling of this signal is used in upper segments, instead of the local input signals in these segments, which carry more noise. This up-channeled foot(support)-in-space signal contributes to the computation of the variables controlled by the servo loops of the higher modules. Module inputs and outputs are shown in **Table 1**. Included are also the inputs coming directly from the sensory system (e.g., joint angle). These sensor fusions are described here in more detail than before, using a generalized notation that allows for an arbitrary number of DoFs.

#### Support Surface Tilt

In the SIP scenario of **Figure 1**, an estimate of support surface tilt is obtained by combining a vestibular derived leg-in-space signal and a local ankle proprioceptive leg-on-foot signal. Specifically, the signal used to compensate the support surface rotation is derived from the rotation speed of the foot in space, when the foot is in firm contact with the support surface ˙α 1 fs.

$$\widehat{\alpha^1\_{fs}} = \int\_0^t \rho(\widehat{\alpha^1\_{fs}}) - k \widehat{\alpha^1\_{fs}} \, d\mathfrak{r} \tag{1}$$

In the generalized notation used, the pedix fs stands for foothold in space, i.e., the orientation in space of the link under the controlled joint, which represents an extension of the variable foot in space. Similarly, in the following the variables ls for link in space and lf for link to foothold denote the generalization of the variables leg in space and leg to foot as used in the SIP and the DIP (double inverted pendulum) case before (Mergner, 2010; Hettich et al., 2013; Lippi et al., 2013). This notation now allows for a description of the sensor fusion process with a generic number of modules, where the index n represents the position of the controlled joint (n = 1 is the ankle, n = 2 the knee, etc.). For specific cases, a notation referring to the names of body segments can be used for simplicity, as shown in **Figure 2A** where ts for example means thigh in space. All the signals in (1) are estimates, with the hat denoting here the up-channeled estimate, ρ(·) is the threshold function and t the current time. The integration is leaky (modulated by the term k).

Given the threshold 2 >0, the function ρ(·) is defined as

$$\rho \,(\alpha) = \begin{cases} \alpha + \theta & \alpha \le -\theta \\ 0, & -\theta < \alpha < \theta \\ \alpha - \theta & \alpha \ge \theta \end{cases} \tag{2}$$

The value of 2 is a parameter of the control module. The presence of the threshold function ρ (·) introduces a gain nonlinearity in the balance behavior (i.e., larger support surface tilts are more compensated than smaller ones), as previously observed in human experiments (e.g., Hettich et al., 2014).

The estimate of support surface tilt from Equation (1) is used to reconstruct the orientation in space of the link above the controlled joint αˆ<sup>n</sup> ls inside each module. The signal is upchanneled and fused with the proprioceptive input throughout all the modules of a given body plane. The resulting value in the nth module is

$$
\widehat{\alpha^n\_{ls}} = \widehat{\alpha^1\_{fs}} + \sum\_{k=1}^n \alpha^k\_{l\bar{f}} \tag{3}
$$

This is different with the other three sensory disturbance estimates (below), in that these are based on the down-channeled position. This scheme reproduces human-like responses in robot experiments and model simulations. However, the interactions TABLE 1 | Signals and parameters used in the sensor fusion process.


between up-channeled and down-channeled signals in the general case are still under research.

#### Field Forces Such As Gravity

The gravity torque T<sup>g</sup> in the general case is calculated by

$$T\_{\mathcal{g}} = m\_{\upmu\mathfrak{g}}^n \text{gCOM}\_{\mathfrak{x}}^n \tag{4}$$

where COM<sup>n</sup> x is the horizontal component of the position of the center of mass **COM<sup>n</sup>** of all the segments above the controlled joint. The estimated **COM<sup>n</sup>** is computed performing the weighted sum

$$\overline{\text{CO}}\overline{\text{M}}^{\text{n}} = \frac{\left(\overline{\text{CO}}\overline{\text{M}}^{\text{n}+1} + L^{\text{n}} \begin{bmatrix} \cos(\widetilde{\alpha\_{ls}^{n}})\\ \sin(\widetilde{\alpha\_{ls}^{n}}) \end{bmatrix} \right) m\_{\text{up}}^{n+1} + h^{\text{n}} \cos(\widetilde{\alpha\_{ls}^{n}}) m^{\text{n}}}{m\_{\text{up}}^{n}} \tag{5}$$

where L n is the length of the link controlled by the joint and h n is the distance of the COM of the nth link from the nth joint, m<sup>n</sup> is the mass of the nth link, and m<sup>n</sup> upthe total mass of all the links above the nth joint. The inverted hat denotes estimators based on down-channeled signals. The modules can be set to use COM<sup>n</sup> x or to apply a small angle approximation as done in previous experiments (Mergner, 2010; Hettich et al., 2014). The estimate then becomes

$$
\widetilde{T\_{\xi}} = m\_{\nu\rho}^n g \widetilde{h}^n \widetilde{\alpha\_{bs}} \tag{6}
$$

where the expression <sup>α</sup>|<sup>n</sup> bs <sup>=</sup> atan2(COM n y ,COM n x ) represents the angular sway in space of the center of the body mass above the nth joint, while ˜h n is the average height of the COM of all the segments above the controlled joint.

#### Support Surface Linear Acceleration

In the presence of support surface acceleration described in the reference system of the support, an inertial force on the center of mass of the body arises. The external acceleration is computed for each joint. The part of the vestibular head acceleration signal not explained by trunk rotation at the hip or at any joint below is taken to stem from support surface acceleration. This is expressed as

$$
\alpha\_{\text{EXTERAL}} = \alpha\_{\text{VETIBULAR}} - \alpha\_{\text{SELF}} \tag{7}
$$

where the acceleration produced by the joint movements is:

$$\widetilde{\mathbf{a}\_{SELF}} = \widetilde{\mathbf{a}\_{SELF}} + L\_n \frac{d^2}{dt^2} \begin{bmatrix} \sin(\widetilde{\alpha\_{ls}}^n) \\ \cos(\widetilde{\alpha\_{ls}}^n) \end{bmatrix} \tag{8}$$

The disturbance inertial torque then results from

$$
\widetilde{T}\_{in} = \breve{\mathbf{a}}\_{\text{EXTERNAL}} \breve{\mathbf{COM}}^{\mathbf{n}} m\_{\upmu}^{n} \tag{9}
$$

For simplicity, here only the horizontal translation is considered

$$T\_{in} = \ddot{\varkappa}\_{\text{EXTERNAL}} \text{COM}\_{\text{y}}^{\mathbf{n}} m\_{\text{up}}^{n} \tag{10}$$

Frontiers in Neurorobotics | www.frontiersin.org

#### Contact Force Disturbance (Such as a Push or Pull)

Humans may sense amount and location of a force exerting impact on the body directly (locally), and in addition may sense the impact that the force has on their balancing in terms of the evoked center of pressure (COP) shifts under the feet, which is proportional to ankle torque. Recall the SIP case scenario of **Figure 1**, where the external torque Text affects the sensory measure of the actively produced torque (Ta). It contributes together with the gravitational torque, inertial torque, passive torque (from muscle and connective tissues) and total torque (T<sup>g</sup> , Tin, Tp, and TA, respectively) to Text, which thus is given by

$$T\_{\text{ext}} = T\_A - T\_\text{g} - T\_{in} - T\_\text{p} - T\_a \tag{11}$$

The term <sup>T</sup><sup>A</sup> is computed using the down-channeled signal <sup>α</sup>|<sup>n</sup> bs and, J n up, the moment of inertia of the supported body segments as

$$T\_A = \frac{d}{dt} \left( \frac{d\widetilde{\alpha^n\_{bs}}}{dt} J^n\_{up} \right) \tag{12}$$

This expression takes in account that also J n up may change with respect to time due to the movements of other DoF. Similarly to m<sup>n</sup> up, also J n up is computed down-channeling information through the modules. For simplicity the down-channeled value is the moment of inertia computed as

$$J\_{up}^{\tilde{n}} = J\_{up}^{\tilde{n+1}} + m\_{up}^n \left\| \overline{\text{COM}}^{\mathbf{n}+1} - \overline{\text{COM}}^{\mathbf{n}} \right\|^2 + l^n$$

$$+ \left. m^n \left\| \text{COM}\_{\text{link}}^{\mathbf{n}} - \overline{\text{COM}}^{\mathbf{n}} \right\|^2 \right\|^2 \tag{13}$$

Where J n and **COM<sup>n</sup> link** are, respectively, the moment of inertia and the position of the COM of the nth link. The moment of inertia used in Equation (12) is

$$J^{n}\_{\upmu\text{p}} = J^{n\text{\textquotedblleft}}\_{\upmu\text{\textquotedblright}} + m^{n}\_{\upmu\text{\textquotedblleft}} \left\| \overrightarrow{\text{COM}}^{\text{n}} \right\|^{2} \tag{14}$$

Compensating the estimated Text may imply that, in a system with neural time delays, positive feedback in T<sup>a</sup> requires a limitation of the compensation (gain < 1, low-pass filtering), similarly to the case of the translation estimator. Model simulations suggest that humans may deal with these flaws by transiently increasing passive stiffness, which has zero time delay, through co-contraction of antagonistic muscles pairs.

#### **Servo Loop and Compensation of External Disturbances**

The disturbance compensation is implemented by summing the disturbance estimates with the input of the controller (PD) with negative sign for compensation (compare **Figure 1**). In the case of the support surface tilt this can be seen as a coordinate transformation of the controlled variable from joint coordinates to space coordinates. The other disturbances are normalized by m<sup>n</sup> upg ˜h n , which represents an angle equivalent to the torque, i.e., the angle that would produce the torque evoked by gravity during body (or segment) lean in linear approximation. This makes the input of the PD controller homogeneous. The torque commanded by the servo controller in the nth module is defined by

$$\begin{aligned} T\_a &= K\_\mathcal{p} \left[ \varepsilon - \left( T\_\mathcal{g} + T\_{in} + T\_{\text{ext}} \right) / m\_{\text{up}}^n \mathbf{g} \tilde{h}^n \right] \\ &+ K\_d \left[ \dot{\varepsilon} - \left( \dot{T}\_\mathcal{g} + \dot{T}\_{in} + \dot{T}\_{\text{ext}} \right) / m\_{\text{up}}^n \mathbf{g} \tilde{h}^n \right] \end{aligned} \tag{15}$$

where K<sup>p</sup> and K<sup>d</sup> are the proportional and the derivative gain, respectively, and ε is the error of the controlled variable as computed using the up-channeled information

$$\boldsymbol{\varepsilon} = \begin{cases} \alpha\_{ls}^{n}! - \widehat{\alpha}\_{ls}^{n} & \text{if the controlled variable is } \alpha\_{ls}^{n} \\ \alpha\_{bs}^{n}! - \alpha\_{bs}^{n} & \text{if the controlled variable is } \alpha\_{bs}^{n} \\ \alpha\_{lf}^{n}! - \alpha\_{lf}^{n} & \text{if the controlled variable is } \alpha\_{lf}^{n} \end{cases} \tag{16}$$

where <sup>α</sup>f<sup>n</sup> bs is an estimate of the COM sway of all the links above the controlled joint, as computed using the up-channeled variable αbn ls.

The effect of each disturbance input and of the error signal of the servo loop can be adjusted by gains (specified as control module parameters). In contrast, the relation between proportional (Kp) and derivative (Kd) gain is fixed for all disturbances. Gravity compensation with lasting body lean represents a special case. Modeling of human responses to slow support surface tilts yielded better results when using a PID controller instead of a PD controller (Mergner et al., 2003; Peterka, 2003) For force feedback as an alternative for the I, see Peterka (2009). The solution used here is a gain elevation in the gravity estimator for low frequencies (Schweigart and Mergner, 2008). As an alternative, Ott et al. (2016) used a PID controller for the servo in robot experiments and in addition a specific PD controller for each disturbance estimator.

The DEC implementation used in the robot experiments of Ott et al. (2016) included feedback from passive joint stiffness and damping. In humans, passive stiffness and damping amount to ∼10% of the active stiffness and damping, stemming mainly from connective tissue properties of muscles and tendons. Having impact with virtual zero latency, they improve control stability in face of the considerable time delays of the reflexive loops (lumped delay ≥100 ms for the ankle joint). Their implementation in the robot experiments of Ott et al. (2016) helped in stabilizing the control, as was the case in our previous experiments with Posturob II (Hettich et al., 2014). Implementation of passive stiffness and damping in the Lucy robot is still pending. Further, the implementation of predicted-sensory estimates with proactive movements, whose effectiveness has been shown before in Posturob I (Mergner, 2010), awaits implementation in Lucy.

## DEC Control in the Frontal Plane and Combined in the Sagittal and Frontal Planes

Conceptually, one DEC control module can be used to control both legs during stance control in the sagittal plane, as realized in our robot experiments in Posturob I and II (Mergner et al., 2009; Hettich et al., 2014). This applies when the two legs and feet are aligned in parallel such that the rotations in the two ankle joints occur approximately around a common axis (for special conditions in which humans weight the proprioceptive input into the controls of the two legs independently of each other, see Pasma et al., 2012). Analogous simplifications hold for the hip joints when extending the control to upper body rotations, and to the knee joints when considering vertical body movements. A mechanically special situation ("four bar linkage") with only one DoF is given with biped balancing in the frontal plane. In this study, stance width with approximately parallel legs is exclusively considered (**Figure 3A**). In particular, the torque produced in the frontal plane around the ankle joints is relatively small, but the legs can overall produce a large torque on the support surface and hence on the body COM for body stabilization and movements.

Inspired by a model interpretation of human balancing in the frontal plane (Goodworth and Peterka, 2010), Lippi et al. (2016) suggested for the Lucy robot a preliminary generalization of the DEC control to the frontal plane, formalizing the body kinematics as a double inverted pendulum. The four bar linkage system with one degree of freedom for the lower body control applies if the knee movements are negligible (true with moderate disturbances) and when the feet are continuously kept in contact with the support surface. Then, a simplified model in terms of the SIP scenario applies to lower body sways about a virtual ankle joint between the two actual feet and can be connected by a virtual link to the pelvis joint (**Figure 4A**). Body balancing is then achieved in terms of controlling the position of the COM of the whole body in the frontal plane by applying the appropriate torque to the support surface and maintaining the vertical orientation of the HAT segment above the pelvis. The desired torque for the virtual joint is distributed in the robot on the four actuated joints (ankles and hips). Using the control equations of the previous section, two DEC modules suffice for the control in the frontal plane, for simplification, one for the lower body and one for the upper body.

The frontal and sagittal planes are mechanically coupled in the physical robot, as are the intermediate planes between them. Theoretically, the interaction between the planes could be considered problematic in terms of coupling forces if the two planes are controlled independently of each other. Our hypothesis in this study for the control system of Lucy is that, in face of the low loop gain for the disturbance compensations used in the control, the coupling between the dynamics in the two planes is not critical and should allow us to control the two planes independently of each other. Independent control in the frontal and sagittal planes based on kinematic synergies has been successfully applied for posture control in a small position-controlled humanoid in earlier studies (Hauser et al., 2007).

In the experiments reported below, Lucy was freely standing while actively controlling its balance in the sagittal and frontal planes using the two sets of DEC control modules: one set for the sagittal plane and the other set for frontal plane, while the controls of the horizontal-plane in the hip joints were "passive" through local joint angle proportional and derivative (PD) "proprioceptive" feedback (indicated in red in **Figure 4B**).

#### Experimental Procedures and Testbed

Performance of the DEC system was investigated experimentally when it controlled the sensorimotor behavior of the humanoid robot during balancing of upright stance while compensating external disturbances and self-produced disturbances arising from voluntary movements. Voluntary movements were produced by defining a reference trajectory for the specific control variables (see below). During the experiments (**Figure 2B**), the robot was standing on a 6 DoF motion platform (Stewart platform) in a human posture control laboratory (Mergner et al., 2003). Motions of the robot were recorded by capturing motion signals from the robot's internal sensors (same as used for its control) for comparison between desired and actual poses or movements. The responses to external disturbances were recorded using an external opto-electronic device (Optotrak 3020; Northern Digital Inc.; Waterloo, Canada).

List of experiments performed:

(1) Test: Balancing of COM during upright stance on periodically tilting support surface (SS)—this tested steady state balancing performance. Disturbance: Sinusoidal SS tilts of peak-to-peak (pp) 2◦ at 0.1 and 0.2 Hz were applied in three trials: (a) sagittal plane, (b) frontal plane, and (c) 45◦ with respect to the sagittal and frontal planes.

Aim: Demonstration of stable balancing in all three planes, with cooperative effects for c from the simultaneous balancing in the sagittal and frontal planes.

(2) Test: Balancing of COM during upright stance as in (1), but with transient tilts of the SS. This experiment allowed us to distinguish between static and dynamic balancing performance.

Disturbance: SS tilts with raised cosine velocity profile (dominant frequency, 0.2 Hz) of peak-to-peak 4◦ were applied again in three trials: (a) in the sagittal plan, (b) in the frontal plane, and (c) 45◦ with respect to frontal and sagittal planes.

Aim: Evaluation of static and dynamic balancing performances in the sagittal and frontal planes and their cooperative effects occurring in the intermediate plane.

(3) Test: Balancing of COM during upright stance while pseudorandomly tilting the SS using the PRTS stimulus<sup>1</sup> . This stimulus allowed us to describe the balancing behavior in terms of FRFs.

Disturbance: Support surface tilts of peak-to-peak (pp) 1◦ , 2◦ , 4 ◦ , and 8◦ .

Disturbance waveform: PRTS<sup>1</sup> (0.016–2.2 Hz), applied again in the sagittal, the frontal, and the 45◦ intermediate planes.

Aim: Demonstration of stable balancing in the sagittal and frontal planes across a broad spectrum of tilt frequencies and the cooperative effects occurring between the balancing

Analysis: Spectral analysis of the angular excursions of the body COM in space.

<sup>1</sup>PRTS stands for pseudo-random ternary sequence stimulus (see Peterka, 2002). It allows evaluation of gain of the disturbance-evoked body excursion and its phase and coherence (e.g., with support surface tilt) over a defined frequency range. Data processing comprises a spectral analysis of the stimulus vs. the body (e.g., COM) angular excursions in space using a discrete Fourier transform. The pseudorandom sequence of the tilts makes them unpredictable for humans. See Hettich et al. (2014) for details of the stimulus used.

in the two planes. Varying stimulus amplitude would allow us to test for the human-like non-linearity of responses (expected from the velocity threshold in the ankle module, see Equations 1 and 2).

(4) Test: Voluntary rapid full body movements in the intermediate sagittal-frontal plane.

Movement command: Starting from a leaning body COM position in an intermediate plane with 45◦ orientation with respect to sagittal and frontal planes, a fast voluntary movement was commanded to reorient the COM into the vertical position above the feet, using step function references for all the commanded DoFs. The movement was repeated six times to observe the variability of the response. A similar movement was also performed in the frontal plane and in the sagittal plane separately. This allowed us to observe how the coupling effects are affecting the dynamic response in the two planes.

Aim: Demonstrating proactive movements and testing the robustness of the system in face of strong self-produced disturbances including coupling effects between different joints and between the two controlled planes.

(5) Test: Squatting movements (knee bending).

Movement and task commands: Raised sinusoids with 4◦ amplitude at 0.17 Hz were used for commanding kneebending in repetitive cycles from and back to straight. With the instructed tasks of COM balancing in the ankle and hip joints, the commanded knee-bending was associated with reactive compensatory movements of the whole body in the ankle joints and of the upper body in the hip joints. The task of the pelvis-HAT joint was to maintain a vertical HAT orientation in space.

Aim: This test challenged the robot's postural stabilization in that all DoFs in the sagittal and frontal planes were interacting. Demonstrating postural stability with the "emerging" movements of the whole body in the ankle joints and of the upper body in the hip joints were secondary aims of the test. For corresponding experiments restricted to the sagittal plane in a 3 DoF DEC implementation in the TORO robot of DLR, see: https://www.youtube.com/watch? v=3ALCTMW3Ei4).

(6) Test: Voluntary full body movement at increasing speed. Movement and task commands: Voluntary body sway movements were commanded simultaneously in the sagittal and frontal planes (the result was a combined movement in some intermediate plane). The common reference signal followed a sinusoidal function with linearly increasing frequency (i.e., chirp signal). The amplitude of the reference signal was set to 1◦ in the sagittal plane and to 2◦ in the frontal plane. The frequency range was ∼0.2–0.7 Hz.

Aim: This experiment tests the frequency limits of the active body sway in the sagittal and frontal planes of the ankle joints. While the previous tests were performed within the margins of the system's stability (to characterize the normal behavior of the system), this test pushes the robot beyond these margins to performance limits.

## EXPERIMENTAL RESULTS

In the robot experiments, we investigated how the DEC control mechanisms in the sagittal and frontal body planes interact through mechanical coupling of the robot's body in producing reactive responses to external disturbances applied in the 45◦ intermediate plane (sections Responses to Sinusoidal Support Surface Tilts; Responses to Raised Cosine Support Surface Tilts; Responses to PRTS Support Surface Tilts). A further experiment investigated this interaction for rapid voluntary full body lean movements in the intermediate plane, which were generated by commanding combined action in the sagittal and frontal planes (section Commanded Fast Full-Body Movements in the Intermediate Plane). A fifth experiment (section Voluntary Squatting Movements) tested control stability across all DoFs of Lucy during commanded squatting.

## Responses to Sinusoidal Support Surface Tilts

Lucy's steady state postural responses in terms of body COM sway evoked by the sinusoidal support surface tilts are shown in **Figure 5**. The responses to the ±2 ◦ tilts in the sagittal and frontal plane at 0.2 Hz (**Figures 5A**1**,B**1) and 0.1 Hz (**Figure 5A**2**,B**2) are compared with corresponding responses in the intermediate 45◦ plane (**Figures 5C**1**,C**2). Note that compensation is similarly stable in all three stimulus planes, with some residual COM lean resulting in the direction of the tilt. This under-compensation is "human-like," stemming mainly from imperfect support surface tilt estimations in the DEC modules with gain < 1 and velocity threshold (see notches around maxima and minima).

## Responses to Raised Cosine Support Surface Tilts

**Figure 6** shows Lucy's COM transient and static sway responses in the sagittal, frontal, and intermediate plane to support surface tilt stimuli with raised cosine velocity profile and amplitude of 4◦ with respect to the horizontal. Dynamic and static response components reflect under-compensation of the tilt stimulus similarly as observed in **Figure 6**. The responses in the intermediate plane approximately reflect the sum of the responses in the sagittal and frontal planes. The finding of larger dynamic responses in the sagittal as compared to the frontal plane mainly reflects a stronger effect from body inertia. We can further note an absence of static error when the support surface has moved back to the horizontal position, owing to properties of the support surface tilt estimator with velocity threshold and leaky integrator (see Equation 1).

## Responses to PRTS Support Surface Tilts

Testing the robot with the PRTS stimulus allowed us to more comprehensively characterize the frequency and amplitude behavior of the system. Being composed of several velocity step

functions, this stimulus has a power spectrum with significant components over the whole range of the frequencies that are of interest here to describe the system dynamics [compare section Experimental Procedures and Testbed (3)]. The stimulus was applied with different amplitudes (pp 1, 2, 4, and 8◦ ) for the support surface tilts in the robot's sagittal, frontal, and intermediate body plane. Lucy's PRTS responses in terms of time series data are shown in **Figure 7**. They again show undercompensation as in the previous experiments with the sine and raised cosine stimuli. Here, the under-compensation exhibits a non-linearity upon increase in stimulus amplitude. The nonlinearity stands out better in **Figure 8** where the responses are expressed in the upper panels of the corresponding FRFs as sway (error) gain (zero with full compensation and unity if body motion equals platform motion). Note that gain in **Figures 8A,B** decreases with increasing stimulus amplitude, being lowest with the pp 8◦ stimulus—again as can be expected from the velocity threshold contained in the support surface tilt estimation. The basic feature in terms of gain, phase, and coherence resemble each other across the tests in the three body plane tested (**Figures 8A–C**). Interestingly, we observe a smaller amplitude non-linearity for low frequencies in the intermediated plane (**Figure 8C**; also compare across in **Figures 7B1–B3**). Again, we can observe that the coupling between the two planes does no harm to the stability of the system.

### Commanded Fast Full-Body Movements in the Intermediate Plane

This test challenged the robustness of the modular control in face of rapid self-produced movements in the sagittal-frontal 45◦ intermediate plane. The movement is associated with strong disturbances acting mainly through coupling forces arising between most of the robot's controlled DoFs. The test may be critical in a system with distributed modular controls that have time delays. **Figure 9A** shows the path of the COM sway in the two planes from the starting position at the right upper corner back to primary position (coordinates x = 0 cm, y = 0 cm). The complex path of the return reflects differences in the system's control and mechanical compliance in the two planes. The **Figure 9B** reports the temporal relaxation in the frontal plane (red) and the sagittal plane (green) following the step. The panel shows decaying oscillations of the relaxation. In the frontal plane, the rise time amounted to 0.584 s, the settling time to 5.115 s, and the overshoot to 33.2% of the step with a peak time of 1.620 s. The corresponding values for the sagittal plane were 0.621 s rise time, 5.141 s for the settling time, and an overshoot of 60.3%. This experiment shows differences in the dynamics in the two body planes, with larger oscillations in the sagittal plane due to higher compliance in this plane. Overall, it demonstrates stable performance when the control and mechanics in the robot's sagittal and frontal planes are dynamically interacting in rapid movements.

In the additional experiments shown in panels **Figures 9C,D**, the movement was restricted to the sagittal and frontal plane, respectively. They demonstrate that a rapid movement in one plane has only a small effect on the COM in the other plane. The cross talk from the sagittal to the frontal plane is again clearly larger than vice versa. Note that the responses in **Figures 9C,D** are similar to those shown in **Figure 9B** with respect to the actively moved component. This suggests that the controllers

efficiently compensate the disturbance produced by the coupling between the planes. Note also that in all three experiments (**Figures 9A–C**) the control was active in all the DoFs of the sagittal and frontal planes in order to keep the body upright.

## Voluntary Squatting Movements

In this experiment, Lucy performed repetitive squatting knee-bending movements with reactive balancing movements occurring mainly in the ankle and hip joints [see section Experimental Procedures and Testbed (5)]. This experiment challenged Lucy's movement and balancing performance in a situation where all DoFs in the sagittal and frontal planes were interacting. The robot successfully performed this test (see film sections in **Figure 10** and film in Supplementary Materials). Lucy executes the voluntary and reactive movements in the sagittal plane, while the frontal plane controls prevent falling by keeping the body upright. No kinematic synergy is imposed explicitly and the resulting joint configurations are produced by the interactions between the modules.

## Voluntary Full Body Movement at Increasing Frequency

In this experiment, Lucy performed voluntary body sway movements in the frontal plane and in the sagittal plane simultaneously. The reference trajectory was a sinusoid with increasing frequency (chirp signal) with the amplitude of 1◦ in the frontal plane and 2◦ in the sagittal plane. In contrast to the previous experiments, which were aimed to characterize the behavior produced by the DEC control within the margins of stability of the system (frequency range similar to that used in human posture control experiments), this trial was performed to the limit of failure (until the robot's feet lost contact with the support surface). The resulting movements are shown in **Figure 11**. In the low frequency range up to 0.4 Hz, tracking performance was almost accurate in both planes. With further increase in frequency, the responses in the sagittal plane develop a peak and then get smaller, while the responses in the frontal plane remain essentially similar as before and then increase in amplitude, before the response becomes unstable at 0.65 Hz, where the robot lost contact with the support surface. Thus, notably, this task pushed the robot to its stability limits. This owed mainly to limitations of the actuators, which were not designed for high frequencies and torques (which also would exceed human capabilities). Interferences between the sagittal and the frontal plane are relatively small, which supports our notion that the two planes of the robot's body can be controlled by two independent control systems.

## DISCUSSION

The main aim of our neurorobotics approach is to investigate whether the networks of DEC modules so far used separately for the sagittal plane and frontal plane (Lippi et al., 2016; Ott et al., 2016) would adequately cooperate with each other without using

the threshold applied to <sup>α</sup>|˙ *n fs*) in (A,B), less so in (C).

a supervising full body model or some other form of software linkage between the two networks. In particular, we asked whether the mechanical coupling between the planes given by the physics of the robot's body would provide postural stability when the robot performs movements in the 45◦ plane intermediate to the sagittal and frontal planes, or when it balances external perturbations in this plane. We envisaged that the underlying DEC with both voluntary movements and postural reactions follows the rules of vector decomposition from the intermediate into the sagittal and frontal planes. In the experiments 1–5, the robot performed well within the stability margins based on the mechanical properties of the robot and the dynamics of the controls in the two planes, as ascertained empirically also in experiment 6 (**Figure 11**).

The described experiments and results considerably contribute to our aim of building a robotic system to further develop our human-derived DEC control with the ultimate goal of mirroring in robot experiments the human sensorimotor functions. This approach is complex and cannot be reached in one step, but requires several steps. The here-described steps comprise the cooperation between control modules in the frontal and sagittal planes of our modular control architecture, the testing of both reactive and proactive movement controls in view of control stability including mechanical aspects, and to provide measures such as frequency responses functions for robot-human comparisons. The current steps are important for further DEC developments that aim to implement in future steps more complex sensorimotor functions such as human-like walking.

The main computational challenge we expected in the present experiments was control stability in face of the feedback time delay of 20 ms, which is still much shorter than the known human time delay. Furthermore, the physical anisotropy of the robot's body in the sagittal and frontal planes represented a challenge. This arises among others from differences in body inertia and length of the base of support between the two planes, as reflected in the different responses to external disturbances and the different dynamic performance in voluntary squatting movements shown in the results section (**Figures 5**–**8**, **10**). Yet, the robot successfully mastered with the same set of control parameters the disturbance and movement scenarios we applied. Thus, our finding demonstrate that the modular DEC control architecture is able to coordinate the movements across the robot's 14 DoF without signal exchange between

an eccentric position (2.3 cm forward, 1.7 cm on the right), the robot was commanded to perform a rapid return of the COM to the upright position. The system response is shown in (A) and the COM trajectories are shown in (B) (for commands, see profiles of dashed and dotted step functions). (C,D) In order to visualize the coupling effects between the two body planes, additional experiments were performed where the robot moved only in the sagittal plane (C) and only in the frontal plane (D). Note that all plots are displayed with the same scales to facilitate the comparisons. The control was always active in both planes to keep the system stable.

the DEC module nets for the sagittal and the frontal planes or some higher order control mechanism. This is reminiscent of the so-called embodied approaches (e.g., Brooks, 1991) according to which the body or other loops through the physical world can mediate interaction effects directly, i.e., without the need of explicit connections at control levels. The frequency responses functions obtained in the present experiments give us a comparison basis for future experiments in robots and humans.

The following discussion firstly considers related issues in humanoid robotics and the relevance for robotic neuro-rehabilitation (section Related Issues in Humanoid Robotics and Relevance for Robotic Neuro-Rehabilitation), then insights for the modeling of the human postural control (section Insights for the Modeling of the Human Postural Control), and finally future steps and developments expected for the extended DEC concept (section Future Steps and Developments Expected for the Extended DEC Concept).

## Related Issues in Humanoid Robotics and Relevance for Robotic Neuro-Rehabilitation

Taking human bipedal control in standing and walking as a basis for comparison is an important research topic in humanoid robotics (Torricelli et al., 2014). One reason is that human postural and movement skills are still considered to be superior to those of robots, being more robust and efficient, and also covering a wider range of external conditions (Nori et al., 2014). Another reason is that humanoids acting in the human sphere may profit from human-like sensorimotor behavior when interacting with humans and their world.

Among the various tasks of sensorimotor control, maintaining balance is a primary task in the DEC concept as well as a basic rule that is followed in many fields of humanoid robotics. Secondary tasks can be performed in parallel with this primary task exploiting the kinematic redundancy of the robot, e.g., by projecting the secondary task into the null space of the

FIGURE 10 | Lucy performing squatting movements. The robot was commanded to perform knee bending at 0.17 Hz while balancing to keep the body upright. The observed coordination between knee, ankle, and hip movements (in terms of compensatory bending in hip and ankle joints) is a property emerging from the interaction between the control modules during the squatting movements (see text).

Jacobian of the balancing task (Sentis and Khatib, 2005). Care is generally taken to constrain secondary tasks such that they are not conflicting with the balancing task. In the present work, multiple tasks are achieved simultaneously, defined by different control variables for different modules. For example, in the squatting movement (section Voluntary Squatting Movements), the ankle module balances the body, while the knee module performs the vertical body motion and the other modules balance the COM of the supported body segments. In general, each module controls a joint (mechanical or virtual), along with all other segments supported by it. This works under the assumption that the controlled variable is affected directly by the controlled joint angle in that every rotation of the ankle joint produces the same rotation of the COM of the whole body around the ankle. For upright stance, the relation between COM sway and ankle joint is

$$\frac{\partial}{\partial \alpha^1\_{\,\,\,\,\ell}} \alpha^1\_{\,\,\,bs} = 1,\tag{17}$$

so that the servo controller can use the ankle control torque to control α 1 bs. The dynamic effects of joint movements on the supporting links are here neglected. The integration of tasks in which a joint is controlled in order to produce an effect on the supporting links, e.g., using hip movements to control sheer forces under the feet, is currently still an open issue in the DEC concept.

An often-used balance control based on sensory signals is in humanoid robotics the method of the zero moment point (ZMP) criterion (Vukobratovic and ´ Borovac, 2004). It allows for balancing against moderate disturbances, which do not require hand contact or a step. This method has been successfully applied in robots with stiff actuation to adjust actual to desired ZMP, e.g., for walking (Hirai et al., 1998; Sentis and Khatib, 2006). However, many robots nowadays use compliant joints, as is the case with Lucy and the human system. The compliance has advantages for robot-world interactions such as collisions and for robothuman interactions. However, the control of compliant joints is more complex due to higher complexity of the dynamics. On the other hand, an important advantage of compliant actuation based on passive stiffness and damping is its immediate response to impact, starting well before time-consuming sensory feedback mechanisms take action (Haddadin et al., 2007). This is comparable to the immediate passive stiffness "feedback loop" in humans from muscles, tendons, and connective tissues, where previous modeling and robot experiments suggested an improved control stability in face of considerable sensory, neural and muscular time delays (Antritter et al., 2014; Ott et al., 2016). While passive stiffness was implemented in Posturob I and II of this laboratory by using pneumatic muscles and in some experiments springs as tendons (e.g., Mergner et al., 2009), it has not yet been implemented in Lucy. This is one of the aims for future experiments with Lucy (see section Future Steps and Developments Expected for the Extended DEC Concept).

In the DEC-controlled robots including Lucy, compliant behavior and low energy consumption are positive side effects from the low control loop gain, which in humans appears to be mainly related to control stability in face of the biological time delays. Effects of the compliance showed up in the presented experiments in the form of residual body sway following external disturbances as well as overshoot and underdamped dynamics with fast voluntary movements. A particularly important beneficial side effect of using low actuation torques was that the robot's feet did not loose contact with the support surface in the experiments 1–5. The effects that passive stiffness has for reducing impact magnitude and improving energy efficiency has been an issue in actuator design (e.g., Ham et al., 2009) and recently has received considerable interest in humanoid robotics. The method has been combined for example with active compliance modulation in the robot COMAN (Li et al., 2012). In this robot, joint stiffness is modulated in response to an external disturbing force by shifting desired body COM such that the resulting center of pressure (COP) shift compensates for the disturbance. This mechanism is employed in combination with other assisting mechanisms that controlled upper body orientation and energy dissipation. Furthermore, flexible robots such a humanoids with compliant "ankle" and "hip" joint equivalents have been controlled with the so-called Reaction Null Space (RNS) formalisms that aim for a reactionless motion control via a feed forward mechanism and an error compensation via feedback (Nenchev, 2013).

Currently, our experiments do not include large perturbations; this topic remains to be addressed in future work (section Future Steps and Developments Expected for the Extended DEC Concept). For example, explicit inspirations from human balancing research for humanoids often address the "ankle strategy" and "hip strategy" (Nishio et al., 2006). In the "ankle strategy," ankle torque suffices to produce shifts of the body COP to compensate moderate perturbations, whereas the "hip strategy" becomes involved or dominates when the COP is expected to exceed the limits of the support base for balancing, be these limits determined by the feet or a restricted or compliant support surface. Another limiting factor is the maximal ankle torque. The hip movement then generates horizontal ground forces in order to keep the COM over the base of support (Nashner and McCollum, 1985). These bio-inspired dynamic balance mechanisms have been technically realized in a compliant humanoid robot by Hyon et al. (2007) and other groups. Measures related to COM and COP shifts were found to describe postural stability decision limits with increasing perturbation magnitude in three steps: from (1) "COP balancing" in the ankle joints via (2) "centroidal moment point, CMP, balancing" in the hip joints involving a moment about the COM, up to (3) a rescue step, often resulting in a double support (Stephens, 2007). Simulations showed that human "hip" and "ankle" balancing strategies tend to emerge in the presence of perturbations of different magnitudes from the same optimization criterion (Atkeson and Stephens, 2007). These robotic studies in turn provide inspirations from robotics back to human posture control research.

As stated in section Introduction, an important aspect of the DEC control is its modularity. Modular concepts have been widely applied before to both the analysis of human movement and the control of robots. In general, the concept of modular control implies the subdivision of the control problem into several sub-problems that can be managed by separate control modules. Such modules may interact in different ways, e.g., directly by exchange of signals or through the effects they produce on the system. Control modularity has been defined at different levels of abstraction. For example, in the "behaviorbased" control systems several sub-behaviors may operate in concert with each other, organized according to a hierarchical principle (Brooks, 1991). In the context of posture control for biped humanoids, such sub-behaviors can be used to control lowlevel tasks like joint torque profiles, or to process higher-level issues such as adapting control parameters to contextual situations, or performing specific reactive adjustments. For example, an implementation on a simulated humanoid is shown in Luksch (2010).

Higher-level functionalities can emerge from the complexity of behavior based systems. Recently, the integration of internal representations has been considered in behavior based system research; this allows for a direct solution of issues such as movement planning. It finds its basis in that several tasks in biological systems are considered to rely on internal representations for inverse modeling, forward models and sensor fusion (Schilling, 2011). Internal representations are different from a behavior in that they are not designed specifically as tasks, but instead have a general validity, as e.g., realized in the principle of mean of multiple computations (MMC). This method can represent body kinematics independently from the specific task and be generalized to arbitrary joint configurations (e.g., of a planar arm in Schilling, 2011 or of a hexapod in Schilling and Cruse, 2008). The DEC control, on the other hand, is specifically addressing posture control and disturbance rejection. Yet, although its bottom-up control design by the integration of locally acting modules shows interesting emerging properties such as segment coordination, it is not expected in general to cover higher level aspects of motor control. An integration of DEC with the above methods can be imagined, e.g., in a set up where high level processing receives sensory input preprocessed through the DEC sensor fusions and where the high level controller outputs into the DEC posture control, e.g., estimates of disturbances that are self-produced by voluntary movements.

Another modularity concept, the Modular Modality Frame model (Ehrenfeld and Butz, 2013), envisages multiple crosscoordinations between cortical sensorimotor representations for processing states of the body and the body parts and the integration of corresponding multisensory information with Bayesian optimality. Again, this concept addresses probabilistic mechanisms of the body schema for movement planning and commanding, whereas the DEC concept deals with the postural control that, on a lower level, helps movement execution by disturbance compensation. Yet, parallels to the DEC concept possibly exist concerning multisensory processing. Specifically, the DEC concept combines in a first step signals from a variety of transducers to obtain estimates of relevant physical variables. In a second step, from these estimated physical variables estimates then the four disturbances are derived. This two-step transformation of transducer signals into estimates of disturbances occurs without probability weighting. The processing in the DEC involves at least partially known transfer characteristics and noise properties and aims to fulfill the commanded task, using a direct reconstruction of the physical relationship between disturbing forces, joint positions and transducer stimulation.

The uncertainty of the results caused by the sensor noise (mainly vestibular noise; van der Kooij and Peterka, 2011) can partially be mitigated by thresholds (Mergner et al., 2009) and disturbance prediction (Mergner, 2010). Estimated disturbances smaller than a given threshold are excluded from the compensation and the control process. This prevents noise from sensory signals that are not participating in the control from influencing the system. To which extent this approach fulfills optimality criteria remains to be investigated.

Exchange of knowledge and inspirations between the robotic and the human domains is especially relevant when it comes to the use of robotic sensorimotor devices such as exoskeletons for neuro-rehabilitation and assistive devices in the development of new invasive or non-invasive stimulation methods to intervene with human sensorimotor functions and more. Knowing the neural control mechanisms behind human sensorimotor behavior will likely prove to be a key for ensuring an intuitive and safe use of such devices by patients.

A related consideration concerns the observation in the present experiments that the robot was able to balance across a broad range of scenarios and conditions using the same sets of control parameters, while automatically adjusting its motor performance to changes in stimulus modality and amplitude, this in combination with the afore discussed low loop gain, and soft mechanical compliance and low energy consumption. Versatility of the control is instrumental for the sensorimotor behavior in a rich and changing environment, which represents a major and still open challenge in the field of humanoid robotics and robotics-inspired assistive devices. The concept of a modular DEC architecture may provide a realistic solution in face of these challenges.

## Insights for the Modeling of the Human Postural Control

The coordination of disturbance compensation observed in our experiments when using support tilts in the intermediate plane resulted from mechanical forces that were produced by the controls for the sagittal and frontal planes. A human skeletal body equipped with sensors, actuators and DEC control modules can be expected to function similarly to Lucy. We take this as evidence that the DEC control can be considered as a valid control concept for human reactive postural control. This even applies if the realization in humans would differ in some respect from that of our robot. Differences may owe to the much larger number of sensory transducer and actuators in humans as compared to the robot, which may reflect the fact that humans are experiencing and producing a richer spectrum of sensorimotor behaviors than currently faced by Lucy.

One may object that here exists a conflict between the DEC concept and the notion of those neuroscientists, who assume that movement planning and commanding at the level of the cortical homunculus (compare section Introduction) entails a full-body control. Note, however, that the cortical motor commands, with few exceptions, do not reach the spinal motor centers directly. Rather, the commands pass through and interact with subcortical structures, mainly the basal ganglia and cerebellum, where they undergo modifications. Thereby, they may better fit to the DEC concept. In this view, one important role of these subcortical structures is to complement the voluntary movement commands by predicted-sensory disturbance estimates. These centrally generated estimates are thought to represent fast and low-noisy predictions of the sensory-derived disturbance estimates evoked by the movements, which they substitute for compensation. This allows for a distinction between self-produced and external disturbances and thus allows compensating both even when they occur in superposition (Maurer et al., 2006; Mergner, 2010).

Although the robot experiments provide no direct insights into human postural control, they may provide valuable support for one or the other hypothetical model when this is tested in robot experiments for "real world" robustness (see section Introduction). This neurorobotics approach will allow neuroscientists and roboticists to compare different models both with respect to specific and general performance criteria (e.g., versatility, failsafe robustness, etc.). For example, a concept of how humans may deal with inter-link coupling forces using Eigenmovements has recently been tested in a robot of this laboratory, where it successfully passed the "real world" test (Alexandrov et al., 2017). The DEC concept provides an alternative solution for the problem of inter-segmental force coupling in terms of a bundle of counter measures, which include the DEC mechanisms for Text and Tacc in **Figure 1**, modulation of passive stiffness and damping, and the use of waveforms for commanded movement trajectories, which are optimized for low acceleration and jerk (compare waveform in **Figure 6**).

In the previous robot experiments of Hettich et al. (2014), the DEC concept was restricted to the DIP scenario and used for the ankle and hip control estimated parameters, which were directly derived from the modeling of human reactive movements. The control parameters related to mass and COM heights of the robot's segments previously used were adjusted in the present experiments to Lucy accordingly. This does not apply, however, to the lumped time delay. Instead, a delay of 20 ms was used for all joints, which is clearly less than that used previously or and cannot be considered "human-like." Identified lumped delays for the ankle joint control in our previous experiments were ≥100 ms. We have not yet tested such lumped delays in Lucy, but plan to test them together with a number of yet pending robot implementations of related other issues such as the aforementioned envisaged solution for the inter-segmental coupling forces, the up-channeling of control parameters, and the passive stiffness and damping. A further aim is to replace the concept of a lumped time delay by the biologically more appropriate concept of "short latency reflexes" (local proprioceptive mechanisms with latencies of 20– 40 ms) and "long latency reflexes" (responses with latencies of 60–300 ms, which involve volition and intention; compare Mergner, 2010). With the integration of these mechanisms we expect stabilizing effects also for coupling forces and a better dynamic performance in faces of challenges that may occur during walking. This also may apply when horizontal body segment movements are included into the control, such as rapid head movements during gaze shifts (Falotico et al., 2017).

In future experiments, we also will address in more detail the emergence of conflict-free interactions between control constituents, as we observed it in our previous and present robot experiments. It allowed here the superposition of voluntary movements and reactive compensations of external disturbances, but also may allow superposition of two or more disturbances at a time. Compliance to the superposition law may represent an important criterion when judging a given robot performance as human-like.

## Future Steps and Developments Expected for the Extended DEC Concept

To develop the robotic model of the human sensorimotor system further, we consider the following steps important:


## AUTHOR CONTRIBUTIONS

VL and TM performed the experiment and collected and analyzed the data, performed the computer simulations. Both authors contributed to the interpretation of the data, drafted and critically revised the manuscript and finally approved the manuscript for submission.

## FUNDING

This research is supported by the by the European Commission (FP7 Grant 600698 H2R and 610454 EMBalance).

## SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fnbot. 2017.00049/full#supplementary-material

#### REFERENCES


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Lippi and Mergner. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Bioinspired Technologies to Connect Musculoskeletal Mechanobiology to the Person for Training and Rehabilitation

Claudio Pizzolato1, 2, David G. Lloyd1, 2 \*, Rod S. Barrett 1, 2, Jill L. Cook <sup>3</sup> , Ming H. Zheng<sup>4</sup> , Thor F. Besier <sup>5</sup> and David J. Saxby 1, 2

<sup>1</sup> School of Allied Health Sciences, Griffith University, Gold Coast, QLD, Australia, <sup>2</sup> Gold Coast Orthopaedic Research and Education Alliance, Menzies Health Institute Queensland, Griffith University, Gold Coast, QLD, Australia, <sup>3</sup> La Trobe Sport and Exercise Medicine Research Centre, La Trobe University, Melbourne, VIC, Australia, <sup>4</sup> Centre for Orthopaedic Translational Research, School of Surgery, University of Western Australia, Nedlands, WA, Australia, <sup>5</sup> Auckland Bioengineering Institute and Department of Engineering Science, University of Auckland, Auckland, New Zealand

Musculoskeletal tissues respond to optimal mechanical signals (e.g., strains) through anabolic adaptations, while mechanical signals above and below optimal levels cause tissue catabolism. If an individual's physical behavior could be altered to generate optimal mechanical signaling to musculoskeletal tissues, then targeted strengthening and/or repair would be possible. We propose new bioinspired technologies to provide real-time biofeedback of relevant mechanical signals to guide training and rehabilitation. In this review we provide a description of how wearable devices may be used in conjunction with computational rigid-body and continuum models of musculoskeletal tissues to produce real-time estimates of localized tissue stresses and strains. It is proposed that these bioinspired technologies will facilitate a new approach to physical training that promotes tissue strengthening and/or repair through optimal tissue loading.

#### Edited by:

Massimo Sartori, University of Göttingen, Germany

#### Reviewed by:

Sara Checa, Charité Universitätsmedizin Berlin, Germany Sergiy Yakovenko, West Virginia University, United States Andrew Phillips, Imperial College London, United Kingdom

#### \*Correspondence: David G. Lloyd david.lloyd@griffith.edu.au

Received: 03 March 2017 Accepted: 04 October 2017 Published: 18 October 2017

#### Citation:

Pizzolato C, Lloyd DG, Barrett RS, Cook JL, Zheng MH, Besier TF and Saxby DJ (2017) Bioinspired Technologies to Connect Musculoskeletal Mechanobiology to the Person for Training and Rehabilitation. Front. Comput. Neurosci. 11:96. doi: 10.3389/fncom.2017.00096 Keywords: biomechanics, mechanobiology, wearable devices, tissue strain, biofeedback, modeling

## INTRODUCTION

Musculoskeletal diseases, such as osteoarthritis and tendinopathy, impose substantial burden on individuals and health care systems. As a community of scientists and clinicians, we have been largely ineffective in managing musculoskeletal diseases, as current prevalence, incidence, and socioeconomic burden are at alarming levels and projected to increase sharply in coming decades (Hunter et al., 2014). In particular, we have a limited understanding of how physical behavior, i.e., whole-body mechanics, influences tissue state (Forwood and Burr, 1993), and this could underpin our failure to cure, or curb, these prevalent, harmful, and costly diseases. A case in point is the study of the effects of physical activity on cartilage morphology. Studies of animals (Kiviranta et al., 1987, 1988, 1992; Newton et al., 1997) and humans (Jones et al., 2000, 2003; Roos and Dahlberg, 2005) have reported increased physical activity to be associated with positive structural and biochemical adaptations in weight-bearing joints, while other studies have reported no effects of physical activity on bulk measures of cartilage morphology (Eckstein et al., 2002, 2006).

The failure to effectively treat musculoskeletal disease is frustrating for scientists and clinicians alike. We possess a wealth epidemiologic data detailing risk factors for many musculoskeletal diseases, e.g., increased age, female sex, body mass, prior joint trauma, obesity, abnormal physical activity levels, and joint structural deformity (Felson et al., 1997, 2000, 2013; Cooper et al., 2000; Coggon et al., 2001; Lohmander et al., 2004, 2007; Roemer et al., 2009; Andriacchi et al., 2015). At tissue- and sub-tissue levels, studies have explored the effect of loading on structure and biology (Radin and Paul, 1971; Simon et al., 1972; Radin et al., 1973, 1984; Rubin and Lanyon, 1985; Forwood and Turner, 1995; Wang et al., 2013, 2015; Joo Kim et al., 2016). However, integrating experimental results with whole-body-, tissue-, and cell-level computational models, and using these models to modulate physical behavior to affect musculoskeletal tissue health remains challenging (Erdemir et al., 2015). In a recent narrative review, Ng et al. (2017) proposed physical therapy to enhance and promote tissue regeneration, linking external mechanical stimuli to tissue mechanobiology. In line with Ng et al. (2017), we describe an approach to deterministically quantify the link between physical behavior and tissue mechanobiology, inspired by integration of biomedical technologies (i.e., wearable devices, contemporary motion capture, and medical imaging) coupled to computational models of joints and musculoskeletal tissues.

Wearable body sensors and systems for "Quantified-Self " are set to transform how people interact with their environment and may facilitate personalized training and rehabilitation programs in the future. Biofeedback is a psychophysical process to augment awareness of afferent signals from sensory receptors in the human body. In the case of musculoskeletal tissues, biofeedback can be used to increase awareness and modify physical behavior (Sigrist et al., 2013). However, current rehabilitation and training protocols which incorporate biofeedback to modulate physical behavior target external biomechanics, such as the knee adduction moments (Barrios et al., 2010; Shull et al., 2011, 2013a,b; Wheeler et al., 2011) or gait spatiotemporal parameters (Wrigley et al., 2009; Erhart-Hledik et al., 2017). Eternal biomechanics are readily measured or calculated, and thus viable for use in biofeedback paradigms. Unfortunately, external biomechanics have tenuous relationships with internal biomechanics, such as articular contact loads (Walter et al., 2010; Winby et al., 2013; Saxby et al., 2016b).

Musculoskeletal tissue stresses and strains are potentially superior to external biomechanics for use in biofeedback paradigms because they are physically coupled to the processes of mechanotransduction, whereby mechanical signals are registered as biologic stimuli, and result in cell- and tissue-level adaptations controlled by biologic regulatory mechanisms. However, musculoskeletal tissue stresses and strains have not been used in biofeedback technologies, because their computation is non-trivial, and depends on a complex interplay of multiple factors, including external biomechanics, neural control, tissue morphology and micro-architecture, and material properties (**Figure 1**). Importantly, recent advances in neuromusculoskeletal modeling have enabled real-time prediction of whole-body kinematics and external loading (Pizzolato et al., 2017a), as well as musculoskeletal tissue loading, such as muscle-tendon unit and articular contact forces during walking gait (Pizzolato et al., 2017b). Real-time musculoskeletal modeling can now be coupled to models of internal tissue mechanics and mechanobiology, and used to provide feedback to target training for tissue strengthening and repair.

In this narrative review we present an overview of (1) the known mechanical stimuli for promoting positive tissue adaptation in musculoskeletal tissues, (2) how local tissue stresses and strains can be estimated using computational methods, (3) an approach to estimating musculoskeletal tissue stresses/strains in real-time, and (4) challenges and future directions for research in this area.

## MECHANOBIOLOGY AND THE OPTIMAL MECHANICAL ENVIRONMENT FOR MUSCULOSKELETAL TISSUES

Mechanobiology is the study of the effect of mechanical stimuli on tissue biology. It is well-established that mechanical loading plays an essential role in (1) musculoskeletal tissue development throughout human maturation (Carter, 1987; Carter and Wong, 1988a,b, 1990; Wong and Carter, 1990; Carter et al., 1998, 2004; Beaupre et al., 2000), (2) maintenance of mature structures (Frost, 1988, 1990a,b,c,d), and (3) healing following injury, e.g., bone fracture (Pivonka and Dunstan, 2012). In particular, musculoskeletal tissues, such as articular cartilage, tendon, and bone, respond to strains by modulating tissue composition and organization. Generally, strains depend on the nature of applied loading, i.e., magnitude, location, orientation, duration, and frequency, as well as structural state of the object, i.e., morphology and material properties (**Figure 1**). Important to our study of musculoskeletal tissues, identical loads applied to different tissues (e.g., cartilage vs. bone vs. tendon), or same tissues but of different structural features (e.g., healthy vs. compromised, developing vs. mature), will produce different strains and eventually different biologic responses. Thus, to develop therapies targeting positive musculoskeletal tissue adaptations we must quantify relevant states. Equally important, if we wish evaluate therapeutic effectiveness we must also quantify changes to tissue states in response to those interventions.

## Estimating the State of Musculoskeletal Tissue

Musculoskeletal tissue state encompasses tissue morphology and function, both of which may be non-invasively assessed using medical imaging. Morphology, which encompasses all spatial descriptions of an object, can be measured using different medical imaging modalities, such as computed tomography (CT), magnetic resonance (MR), and ultrasound (US).

Computed tomography is well-suited to the study of bone and provides high-resolution images that can be automaticallyor semi-automatically segmented to render volumetric representations (Dufresne, 1998). Peripheral quantitative CT can be used to image cortical and trabecular bone microstructure (Lespessailles et al., 2017), which are important structural features to include in analysis of bone remodeling (Hambli, 2011). However, CT exposes tissues to ionizing radiation and may not be suitable for certain clinical or developing populations. state.

Magnetic resonance imaging is a powerful modality that does not produce ionizing radiation, and can be used to image a wide range of musculoskeletal tissues (Hunter et al., 2015). However, individuals with implanted medical devices (e.g., cardiac stimulators) or ferrous prosthetics cannot safely undergo MR imaging. Unlike CT, MR images require manual segmentation to produce three-dimensional reconstructions of musculoskeletal tissues. Currently, manual segmentation is time consuming, but advances in image auto-segmentation (Mimics, Materialize NV, Leuven, Belgium) will hopefully reduce labor demands. Once MR images can be rapidly segmented, this will make MR imaging a routine process to assess musculoskeletal tissue morphology.

Ultrasound is an inexpensive, non-invasive, and nonradiating modality to image musculoskeletal tissues. Importantly, US can accurately measure muscle morphology (Barber et al., 2009), track muscle fascicles during contractions (Cronin et al., 2011; Gillett et al., 2013), and measure in vivo tendon morphology at rest and under load in healthy (Obst et al., 2014a,b) and pathologic tendon (Nuri et al., 2017). In addition to muscle-tendon applications, US has been used to measure bone landmark coordinates (Peters et al., 2010; Passmore and Sangeux, 2016) and make in vivo clinical measurements of bone alignment (Passmore et al., 2016). However, limited signal penetration into the body means that many deep anatomic structures cannot be imaged using US. Furthermore, deformation of soft tissues out of the imaging plane impairs measurement fidelity. To summarize, CT, MR, and US are imaging modalities capable of measuring musculoskeletal tissue morphology, however, morphology is only one component of tissue state, and alone is an insufficient indicator of tissue function and integrity.

Tissue function is related to tissue mechanical properties, such as stiffness and strength. As many pathologic tissue changes are accompanied by changes in tissue elasticity (Ophir et al., 1991), measures of tissue mechanical properties may serve as surrogate measures of tissue health and integrity. Henceforth, we will refer to medical imaging modalities used to assess musculoskeletal tissue mechanical properties as "functional imaging." Elastography is a class of functional imaging, and is the study of elastic properties of materials. Elastography uses principles from the physics of wave propagation to quantify tissue mechanical properties (Ophir et al., 1991). In general, an internally- or externally- generated stimulus causes tissue deformation, which is measured and related to tissue elastic modulus (Yamakoshi et al., 1990). Relaxography is another class of functional imaging, whereby MR is used to indirectly assess tissue integrity by measuring time constants, e.g., T2, T<sup>∗</sup> 2 , and T1ρ, associated with the slow motion of water molecules. Relaxography has emerged as a potent method to study and detect early signs of articular cartilage degeneration (Baum et al., 2013). As cartilage degenerates, its extracellular matrix is disrupted and proteoglycan content is reduced, which results in increased water content and motility. Relaxographic imaging is sensitive to early degenerative changes, as T<sup>2</sup> relaxation times associated with healthy cartilages (∼25–45 ms) are lower than those associated with degenerated cartilages (Dunn et al., 2004). Overall, there are several imaging modalities capable of assessing musculoskeletal tissue state, thus enabling creation of personalized musculoskeletal tissue models as well as quantifying intervention outcomes. However, it is first necessary to identify the optimal mechanical environments of each musculoskeletal tissue, which will serve as targets for bioinspired rehabilitation and training.

Just as hyper-physiologic loading can cause musculoskeletal tissue damage (Bonnevie et al., 2015; Christiansen et al., 2015), load deprivation due to low-gravity (Lang et al., 2006), or spinalcord injury (Vanwanseele et al., 2002, 2003) causes tissue atrophy and weakening. More subtle changes in tissue loading can also affect tissue properties. For example a lower than normal knee contact force following orthopedic surgery has been associated with future onset of knee osteoarthritis (Wellsandt et al., 2016). Specifically, reductions of 10–20% of a body weight in the medial contact forces 6-months after anterior cruciate ligament reconstruction were associated with onset of medial knee osteoarthritis 5-years post-operation (Wellsandt et al., 2016). Similarly, animal experiments of unloading the weight-bearing limbs following knee ligament transection found subsequent muscle atrophy and loss of trabeculae (Anderson et al., 2016). Likewise, the human proximal tibia experiences substantial bone mineral density loss over the first year following anterior cruciate ligament reconstruction (Mundermann et al., 2015), which may be related to lower magnitude ambulatory tibiofemoral contact forces (Saxby et al., 2016a; Wellsandt et al., 2016). Overall, these results re-inforce the concept that inappropriate loading, due to over- and/or under-loading, precede articular tissue degeneration. It therefore follows that each tissue must have an optimal mechanical stimulus or "sweet spot" which maximizes anabolic tissue adaptation, where loads are neither too high to cause tissue damage, or too low to result in tissue degeneration (**Figure 2**).

#### Tendon Optimal Mechanical Environment

An illustrative example of a "sweet spot" in tissue regulation is drawn from in vitro studies of Achilles tendon. The Achilles tendon is a viscoelastic structure that links calf muscles, i.e., gastrocnemii and soleus, with the calcaneus bone of the foot, thus spanning the ankle joint. The Achilles tendon is crucial to common ambulatory tasks, such as walking, running, and jumping, through its role in biomechanical power generation and movement efficiency. When conditioned in a bioreactor, excised sections of Achilles tendons have shown optimal biomechanical response when subjected to ∼6% cyclic tensile strains (Wang et al., 2013, 2015). Cyclic 6% tensile strains, 0.25 Hz loading cycle, 8 h per day, maintained tendon homeostasis (Wang et al., 2013) and, importantly, regenerated injured tendon (Wang et al., 2015). Consistent with the idea of over- and underloading as mechanisms for tissue degeneration, tendon tensile strains below 3% or above 9% disrupted extracellular matrix (Wang et al., 2013), while tenocytes optimally responded to 4– 6% strains (Joo Kim et al., 2016). These results reinforce the need to target specific strain ranges to maintain and repair tissue.

#### Cartilage Optimal Mechanical Environment

Articular cartilage caps the terminal regions of long bones involved in synovial joints, and provides a smooth ultra-low friction bearing surface for articulation. Articular cartilage is considered biphasic, consisting of a solid phase composed primarily of organized collagenous extracellular matrix interposed with chondrocytes, highly charged macromolecules, and immersed in an ionized interstitial fluid phase. Interaction between solid and fluid phases causes the mechanical behavior of cartilage, i.e., anisotropy, strength, and viscoelasticity (Mow et al., 1980; Armstrong and Mow, 1982). Indeed, the network of collagens and macromolecules of the extracellular matrix provide enormous resistance to internal fluid motility, primarily through friction. Consequently, during rapid loading of cartilage, as occurs during sport and activities of daily living, cartilage behaves as a nearly incompressible isotropic material. Internal resistance to fluid flow is an essential mechanism by which cartilage resists externally applied compression. However, as

cartilage is avascular (Buckwalter, 2002; Buckwalter and Brown, 2004), large resistance to internal fluid flow prevents effective transport of materials and cells to injury sites. When cartilage deteriorates due to age, injury, or disease, it becomes more compliant (Setton et al., 1999). Consequently, collagen networks in the extracellular matrix experience larger, and potentially injurious, strains.

Limited interstitial motility combined with the avascular nature of cartilage, results in minimal regenerative capabilities (Newman, 1998; Buckwalter, 2002; Buckwalter and Brown, 2004). Understandably, research has focused on engineering cartilage implants and effective scaffolding to promote seamless uptake of implanted constructs into native cartilage. A recent review of literature found in vitro compressive strains of >20% applied at 0.5–1 Hz, were optimal for promoting cartilage cultivation (Natenstedt et al., 2015). A 20% strain is in the middle of the physiologic range experienced by cartilage (Grodzinsky et al., 2000), and 0.5–1 Hz loading rates are similar to the natural knee loading frequency during human gait. Thus, ∼20% and 0.5–1 Hz cartilage strain and loading rate, respectively, provide logical targets to condition cartilage to prevent future degeneration, and may be effective to attenuate, stop, and even reverse degeneration in cases of established disease.

## Bone Optimal Mechanical Environment

In vitro studies provide a rich source of direct measurements of strains that stimulate bone remodeling, as well as strains that injury or fracture bone. As many strain measures in the literature were acquired from experiments that did not incorporate bone dynamics, they should be considered timeindependent mechanical targets, that may stimulate bone remodeling (Lanyon et al., 1975; O'connor et al., 1982; Rubin and Lanyon, 1985; Ehrlich and Lanyon, 2002). Different bone components, i.e., cancellous and cortical bone, may have different optimal strain ranges required to elicit adaptive remodeling. However, a range of ∼200–1,000 µε (1µε = 1 microstrain; 1 µε = 0.0001% strain) represents everyday strains. During vigorous physical activities, such as sprinting, bone strains may reach peak values of ∼2,000–3,000 µε (Burr et al., 1996) and strain rates of ∼10,000–50,000 µεs −1 (Lanyon et al., 1975). Even during vigorous physical activities, bone strains and strain rates do not necessarily damage tissue, as injurious strains are ∼25,000 µε in tension or compression directed longitudinally (Reilly and Burstein, 1974). Bone durability is first due to its innate capacity to withstand large stresses at low strain rates, i.e., ∼80–170, ∼100–300, and ∼150–240 MPa in tensile, compressive, and bending modes, respectively (Reilly and Burstein, 1974). Second, bone is a viscoelastic material and its stiffness increases when subject to high strain rates, for example during running and jumping. Bone strains during strenuous physical activities have been reported to be ∼10% of ultimate failure, well below bone fracture threshold and therefore considered safe healthy for individuals (Burr et al., 1996). Low impact and activities such as walking do not appear to be oesteogenic.

## ESTIMATING THE MECHANICAL ENVIRONMENT OF MUSCULOSKELETAL TISSUES

Musculoskeletal tissue state varies between individuals, and is affected by disease processes. To personalize therapy, we must account for subject-specificity, such that training programs can be tailored to the individual. A further technical challenge is that we need to estimate musculoskeletal tissue mechanics in realtime, providing an appropriate form of biofeedback to enable individuals to volitionally modulate tissue mechanics during rehabilitation, recreation, or daily activities. To achieve this goal we must merge whole-body representations of human physical behavior with models of musculoskeletal tissue mechanics and mechanobiology within efficient computational frameworks.

Currently, there is no feasible method to directly measure in vivo loading applied to, and stresses/strains within, musculoskeletal tissues in native human joints. Articular contact forces can be measured in cadavers through a combination of robotic control and mathematical modeling (Wang et al., 2014) or by inserting pressure sensitive film between articulating surfaces (Ihn et al., 1993). However, a valid method of applying physiologic muscle, body, and inertial loads to cadavers has not been reported, thus casting doubt whether these measurements are representative of in vivo loading. Contact forces can also be measured by instrumenting prostheses used in arthroplasty, as has been done at knee (D'lima et al., 2005, 2006; Heinlein et al., 2007, 2009; Fregly et al., 2012), hip (Rydell, 1966; English and Kilvington, 1979; Bergmann et al., 2010), and shoulder (Bergmann et al., 2007) joints. Contact loads measured by instrumented prostheses provide critical information to implant designers regarding the nature of the mechanical demands placed upon these devices. Unfortunately, instrumented prosthetic implants are only appropriate for measuring contact loads in arthroplasty patients, who are typically elderly individuals with substantially degenerated joints and peri-articular muscle atrophy. Furthermore, arthroplasty, by definition, does not preserve the native joint and restricts the activity types that could be studied in these patients, e.g., it is unethical to ask an elderly knee arthroplasty patient to perform vigorous athletic movements. Consequently, contact loads sustained by implants are unlikely to be representative of contact loads in native joints of young physically active populations.

In addition to articular contact forces, muscle-tendon unit forces have also been directly measured in both animals (Walmsley et al., 1978; Hodgson, 1983; Herzog et al., 1992) and humans (Gregor et al., 1987; Komi et al., 1987; Fukashiro et al., 1995) by surgically implanting mechanical gauges. Proficient surgical implantation results in minimal inflammatory response, and instruments may left in situ in animals for days or even weeks. However, extrapolating in vivo animal muscletendon force measurements to humans is questionable and certainly of limited clinical relevance. In humans, surgical implantation of strain gauges into tendon may affect an individual's physical behavior, thus limiting ecologic validity of the measurements. Furthermore, muscle-tendon forces are subject-, task-, and state-specific, thus limiting applicability of measurements from an individual performing a specific task to another individual, movement or control task, or the same individual at a later date, i.e., following an intervention, suffering an injury, or onset of disease. Given the serious limitations of direct measurement of musculoskeletal tissue loads, researchers have used computational neuromusculoskeletal to predict musculoskeletal tissue loading.

## Neuromusculoskeletal Models to Estimate Musculoskeletal Tissue Loading

Neuromusculoskeletal models are bioinspired mathematical representations of specific neurologic, physiologic, and anatomic characteristics of an individual (Hatze, 1977; Buchanan et al., 2004, 2005; Lloyd et al., 2005). Neuromusculoskeletal models may be used to estimate muscle (Lloyd and Besier, 2003; Erdemir et al., 2007), ligament (Shelburne and Pandy, 1997; Pandy and Sasaki, 1998; Lloyd et al., 2005; Shelburne et al., 2005), and articular contact forces (Shelburne et al., 2005; Winby et al., 2009; Gerus et al., 2013; Manal and Buchanan, 2013; Erdemir et al., 2015; Walter et al., 2015; Saxby et al., 2016b; Smith et al., 2016; Konrath et al., 2017), and have been deployed across a wide range of scientific, industrial, and clinical applications, such as investigating fundamental properties of human motor control (Haeufle et al., 2014; Sartori et al., 2015), evaluating ergonomic demands of automotive operation (Rasmussen et al., 2009), and informing medical device designs by predicting in vivo loading conditions (Marra et al., 2015). Typically, structural characteristics used in a model are based on measurements from a small number of cadavers, and subsequently used as a generic template for each analysis. Bone dimensions and massinertia properties in a generic template are linearly scaled to match subject's dimensions (Delp et al., 1990), thus providing a basic level of model personalization. Using generic templates facilitates rapid and routine use of neuromusculoskeletal models, but scaled generic models are often poor representations of an individual's musculoskeletal anatomy, which may lead to inaccurate results, spurious conclusions, and potentially detrimental clinical decisions. For example, linear scaling of a generic model template may result in incorrect representation of muscle moment arms (Arnold et al., 2000; Scheys et al., 2008) and consequently erroneous joint contact force estimates (Lenaerts et al., 2009; Gerus et al., 2013; Wesseling et al., 2016).

Several aspects of neuromusculoskeletal models can be personalized to the individual to improve simulation results. Bone morphology and joint mechanics have been shown to influence kinematics and kinetics estimates (Brito da Luz et al., 2016; Kainz et al., 2016), and knee contact forces have been shown to be sensitive to tibiofemoral alignment (Lerner et al., 2015). Skeletal geometry also affects muscle-tendon paths and insertion points, which in turn define muscle-tendon lines of action, influencing both muscle-tendon lengths and moment arms. Overall, better representation of an individual's musculoskeletal anatomy has been shown to produce more realistic results, e.g., improved representation of muscle-tendon moment arms, improved knee (Gerus et al., 2013) and hip (Modenese et al., 2013) contact forces estimates.

Muscle activation patterns are known to vary between individuals and controls tasks (Tax et al., 1990; Buchanan and Lloyd, 1995), and are affected by training (Menegaldo and Oliveira, 2011) and pathology (Besier et al., 2009). Incorporating experimental measures of muscle activation patterns into neuromusculoskeletal models adds an important dimension of personalization. Electromyography (EMG)-informed neuromusculoskeletal models (Manal et al., 2002; Lloyd and Besier, 2003; Manal and Buchanan, 2013; Sartori et al., 2014; Pizzolato et al., 2015) are a class of neuromusculoskeletal models sensitive to variations in motor control. Specifically, EMGinformed neuromusculoskeletal models use experimentally measured muscle excitations and movement patterns to account for complex interplay between external biomechanics and muscle recruitment to estimate musculoskeletal tissue loadings, i.e., joint, muscular, ligamentous, and articular contact loads, that may serve as boundary conditions for continuum mechanics analysis.

## Finite Element Method to Estimate Musculoskeletal Tissues Mechanical Environment

The finite element method (FEM) is a well-established computational method used in many branches of engineering. In a FEM model, the real system is discretized into a field of elements of known geometries and material properties, from which constitutive equations may be developed. The model system dynamics are then equilibrated by imposing a set of boundary conditions, e.g., musculoskeletal tissue loads informed by a neuromusculoskeletal model. Halloran et al. (2010) applied this combined neuromusculoskeletal and FEM modeling to foot and ankle strains, while Besier et al. (2009) verified predicted patellofemoral stresses/strains using measurements of cartilage deformation acquired in a vertical bore MR unit. Recently, others have explored tibiofemoral cartilage stresses/strains during gait (Shim et al., 2016; Smith et al., 2016) and acetabulum stress distributions in relation to bone remodeling (Fernandez et al., 2014). These studies have shown the potential of the FEM models, but the models employed were not fully personalized.

Generating personalized FEM models of tissue requires both morphology and material properties. As previously described, different imaging modalities can be used to directly acquire tissue-specific morphology, but non-invasive methods to estimate material properties are scarce. Musculoskeletal tissues have a heterogeneous multiphasic structure, resulting in anisotropies and non-linear time-varying behavior (Freutel et al., 2014), thus making the estimation of material properties challenging. Relaxography (Labadie et al., 1994) is a powerful tool to assess tissue function, but it only provides measures which correlate with, but do not quantify, tissue material properties (Lammentausta et al., 2006). Elastography (Ophir et al., 1991) can provide direct measurement of musculoskeletal tissue stiffness by analyzing the response of tissue to external stimuli. In MR elastography, low frequency vibrations are externally introduced to the body by means of electromechanical devices, while multiple images are recorded to analyse tissue response at different time points and directions (Glaser et al., 2012). Ultrasound elastography works by the same principle as MR elastography, but the external stimuli can be provided by the US transducer itself. Ultrasound-based shear-wave elastography has recently been applied to musculoskeletal tissues (Eby et al., 2013) to quantify stiffness, but is limited to superficial tissues and subject to errors due to probe positioning (Brandenburg et al., 2014). Reverse FEM could also be used to estimate tissue material properties, whereby tissue is subject to multiple and explicitly known applied loading conditions that alter morphology. A numerical optimization then estimates a set of material properties best fitting the measured morphology change (Hansen et al., 2017).

Informing tissue material properties in FEM models through non-invasive imaging would provide a level of personalization well beyond current standard approaches, which typically apply literature values established through experiments performed on cadavers. Indeed, tissue material properties are specific to individuals and are affected by aging, training, injury, and disease (Arokoski et al., 2000; Buckwalter, 2002; Buckwalter and Brown, 2004). Different tissue stress and strain patterns will arise from FEM simulations that use different tissue material properties, even when composed of identical tissue morphology and subjected to identical boundary conditions. Finally, simulations of musculoskeletal tissue mechanics may use physiologic and personalized boundary conditions informed by neuromusculoskeletal models (Besier et al., 2005, 2009; Fernandez et al., 2014). Overall, when FEM models of musculoskeletal tissues are informed by measurements of subject-specific morphology, material properties, and boundary conditions, they are powerful tools to understand musculoskeletal tissue mechanics.

## Finite Element Method to Estimate Musculoskeletal Tissue Remodeling

Considerable research focus has been applied to studying relationships between applied tissue loading and morphology, with a fundamental assumption that tissue health may be assessed through structural analysis (e.g., thicker cartilage is indicative of healthy cartilage; Koo and Andriacchi, 2007). Rigid-body computational models have been used to determine external joint or articular loads, which in turn have been compared to measures of articular tissue structure using linear statistics (Koo and Andriacchi, 2007; Koo et al., 2011; Scanlan et al., 2013; Van Rossom et al., 2017). However, primary focus on applied loading may not be appropriate, as other biomechanical signals, such as extracellular fluid motion in bone (Zadpoor, 2013; Villette and Phillips, 2016) or bone strain energy (Kerner et al., 1999), are physically closer to cellular mechanisms of remodeling and have been shown to influence tissue adaptation. Simulations of trabecular remodeling have been performed whereby structural modifications were driven by local mechanical criteria, e.g., minimizing density of material anisotropy with respect to principle stresses (Fyhrie and Carter, 1986) or non-uniformity in local stresses (Adachi et al., 1997; Tsubota et al., 2002). Such simulations were able predict trabecular distributions consistent with experimental observations (Fyhrie and Carter, 1990), and results were highly sensitive to loading condition complexity. When complex loading patterns were applied to FEM models with embedded bone remodeling algorithms, predicted bone material property distributions were consistent with ex vivo imaging (Geraldes et al., 2016). Similarly, features such as bone cortical thickness and regional femoral trabecular density were better predicted when complex physiologic loads were applied compared to simple axially oriented compressive loads (Geraldes et al., 2016). When complex muscle loading patterns were included in FEM simulations of femoral bone remodeling in the context of prosthetic hip implants, simulations predicted bone retention patterns in regions of muscle attachment, which is not predicted by FEM models using simple idealized boundary conditions (Bitsakos et al., 2005). These results suggest incorporation of complex biomechanical loads into FEM models is required to predict correct spatial distribution and peculiar features of musculoskeletal tissue structure.

The complex biomechanical loads sustained by the human body are generated by non-linear muscle dynamics and their interaction with convoluted three-dimensional musculoskeletal architecture. Including muscle dynamics into FEM models directly affects spatial distribution of musculoskeletal tissue strains, and hence influences predictions of tissue remodeling (Duda et al., 1998). When pairing together computational rigidbody neuromusculoskeletal and FEM models, the degrees of freedom associated with the respective models must be consistent (Phillips et al., 2015). For example, the popular musculoskeletal modeling software OpenSim (Delp et al., 2007) enables users to define complex joint motions that are both arbitrarily bounded and computationally efficient (Seth et al., 2010). The OpenSim model may then be used to solve external joint and muscle loads, which can, in principle, be applied to FEM models. However, the FEM model must be constrained in an analogous manner to the OpenSim model to ensure model degree of freedom consistency. This is not a peculiar consideration of OpenSim models, rather, all hierarchical modeling frameworks which combine boundary conditions from a rigid-body simulation to a FEM model should respect this demand for consistency. In the context of bone remodeling simulations, Phillips et al. (2015) presented a method to ensure model degree of freedom consistency, but noted that the constraints of model displacement may limit scope of the analysis.

Optimal mechanical environments for musculoskeletal tissue adaptation have been provided from ex vivo, in vitro, and in silico studies. This knowledge, combined with an appreciation for modeling complexity required to estimate musculoskeletal tissue stresses and strains, leaves us well posited to move forward and apply these models in clinical contexts such as training or rehabilitation. If we can gain control of an individual's physical behavior through biofeedback paradigms, and target the optimal in vivo mechanical environment of their musculoskeletal tissues, we may be able to prevent tissue deterioration or restore health. We represent our vision in **Figure 3**, and its realization would be a breakthrough for rehabilitation science and medicine. However,

to realize this aim, current computation processes that are performed offline must be performed in real-time.

## REAL-TIME ESTIMATION AND BIOFEEDBACK OF MUSCULOSKELETAL TISSUE STRESS AND STRAIN

Behavioral movement changes, in the form of modulation of body kinematics and kinetics, have been used in rehabilitation to assist motor learning and improve function following injury or disease (Sigrist et al., 2013). Much research has focused on biofeedback technologies to improve movement and function in knee osteoarthritis patients (Barrios et al., 2010; Shull et al., 2013a,b; Van Den Noort et al., 2015). In these patients, larger magnitude walking knee adduction moments have been associated with structural progression of knee osteoarthritis (i.e., joint space narrowing; Miyazaki et al., 2002) and knee pain (Amin et al., 2004), making reduction of the magnitude of the knee adduction moment a logical target for physical therapy.

Numerous studies have combined biofeedback technologies with gait modification strategies to modify joint kinematics or external loads with the aim of improving health outcomes or reducing movement variability. For example, real-time visual biofeedback of upper-body posture (Hunt et al., 2011) and dynamic knee alignment (Barrios et al., 2010) have been used to reduce walking knee adduction moments in healthy individuals and knee osteoarthritis patients, respectively. Knee braces instrumented with auditory feedback have been used to reduce knee loading rates during walking (Riskowski et al., 2009), while instrumented footwear has been used to reduce lateral foot pressures through vibrotactile feedback (Dowling et al., 2010). Notably, Shull et al. (2013b) provided haptic feedback, delivered through body worn vibrating motors, to inform participants of their changes in foot progression and trunk sway during treadmill walking. This resulted in patients with knee osteoarthritis reducing their peak knee adduction moment magnitudes. However, the main limitation of modifying external kinematics and kinetics is their tenuous relationships with internal loads (Walter et al., 2010, 2015; Winby et al., 2013; Saxby et al., 2016b), which implies weaker still relationships to articular tissue stresses and strains. The reason for these poor relationships is external biomechanics cannot account for the direct effect of muscles on musculoskeletal tissue loading (Walter et al., 2010; Winby et al., 2013; Saxby et al., 2016b).

As previously discussed, neuromusculoskeletal models can provide FEM with appropriate boundary conditions to estimate musculoskeletal tissue stresses and strains (Besier et al., 2005, 2009; Fernandez et al., 2014). This may be done in an offline analysis, but real-time estimation of musculoskeletal tissue stresses and strains requires interfacing with, and enabling data flow from, external devices to modeling software to complete necessary computations within given time constraints. For neuromusculoskeletal models, this means solving kinematics, kinetics, and muscletendon forces in real-time. Muscle forces have been calculated in real-time using a static optimization method (van den Bogert et al., 2013), where an algorithm determined the minimized weighted sum of muscle forces that matched external joint moments (Van Der Helm, 1994). However, the real-time approach presented by (van den Bogert et al., 2013) was based on a generic anatomic model that could not be personalized. Model personalization, noted earlier in this review, is essential when coupling neuromusculoskeletal and FEM models of musculoskeletal tissue mechanics. Furthermore, many neuromusculoskeletal models rely on mechanical optimization to solve the muscle redundancy problem (Crowninshield, 1979; Crowninshield and Brand, 1981), however, mechanical optimization methods struggle to predict many empirical features of muscle coordination, such as patterns of muscle activation (Herzog and Binding, 1992), cocontraction (Herzog and Binding, 1993), and force sharing (Binding et al., 2000). To our knowledge, the first use of a real-time EMG-informed neuromusculoskeletal model was by Manal et al. (2002), and first applied to musculoskeletal tissue loading in Achilles tendon rehabilitation by Manal et al. (2012). These papers advanced the field and should be acknowledged as pioneering, but were limited in application to quasi-static movements and a single joint with few degrees of freedom. Recently, Pizzolato et al. (2017b) developed software, based on OpenSim (Delp et al., 2007), to calculate full-body kinematics and kinetics (Pizzolato et al., 2017a), as well as musculoskeletal tissue loading (Pizzolato et al., 2015, 2017b), in real-time. Their method is fully extensible to other joints and musculoskeletal tissues, but is currently limited to expensive and immobile laboratory-based stereophotogrammetry systems (Pizzolato et al., 2017b).

Wearable sensors that accurately estimate human kinematics are a promising alternative to laboratory-based measurement systems. Linear accelerometers have been used for many years to quantify movement patterns relative to the gravitational field and ambulatory temporal-spatial parameters (Kavanagh and Menz, 2008), but their estimates of joint kinematics are limited by signal drift caused by integration errors (Djuric-Jovicic et al., 2011). Improvements in microelectromechanical systems have enabled embedding tri-axial accelerometer, gyroscope, and magnetometer into a single sensor. These integrated sensors are known as inertial measurement units and are able to estimate spatial orientation (Sabatini, 2006; Madgwick et al., 2011) and, when used in combination with anatomic models, joint angles. Strain sensors are another class of promising wearable sensors that can be used to estimate joint angles (Nakamoto et al., 2016). Strain sensors are low profile, flexible, and can be easily embedded into garments or mounted on the skin (Amjadi et al., 2016). To date, strain sensors have been used in biomechanics primarily to classify movement (Mattmann et al., 2007) or estimate single joints angles (Nakamoto et al., 2016). However, continuous technologic improvements in smart textiles (Honarvar and Latifi, 2017) may soon lead to advanced garments capable of estimating full-body kinematics.

Measuring or estimating reaction forces between body and ground is required to correctly estimate load applied to specific musculoskeletal structures, such as joints and ligaments. In laboratory conditions, ground reaction forces are acquired via ground mounted force plates, but alternative solutions are required for applications in the real-world. Pressure-sensitive insoles can be used to estimate the normal component of the ground reaction force, but they neglect shear components (Chesnin et al., 2000). Conversely, shoes instrumented with triaxial force sensors have shown agreement with force plates for all components of the ground reaction force vector (Liedtke et al., 2007). Alternative to measurements, deep learning algorithms have been shown to correctly estimate ground reaction forces during walking (Oh et al., 2013). However, these data-driven models require big data as training sets. Mechanical approaches can be used to solve dynamics of motion and estimate ground reaction forces without body-worn force sensors. For example, the zero-point moment is an algorithm developed for humanoid robots (Xiang et al., 2009) that has also been successfully applied human biomechanics (Fluit et al., 2014; Dijkstra and Gutierrez-Farewik, 2015). However, to correctly estimate ground reaction forces, full-body kinematics and subject-specific musculoskeletal models are required (Fluit et al., 2014).

Overall, advances in wearable sensors, i.e., smaller, lighter, low-power, and integrated sensor systems, will enable novel real-world applications (Brodie et al., 2008). Currently, intrinsic limitations and measurement inaccuracies associated with these devices prevent their use in advanced biomechanical analysis. Combining wearable sensors with sophisticated biomechanical models may help to minimize the limitations associated with wearable sensors. Realistic musculoskeletal models, such as those offered by OpenSim (Delp et al., 2007), associated with probabilistic frameworks that adequately model wearable sensor inaccuracies (Latella et al., 2016) and computationally efficient real-time software architectures (Pizzolato et al., 2017a,b), have the potential to accurately estimate human motion in real-world setting free from the laboratory.

As previously stated, boundary conditions for subsequent FEM model simulations may be computed in real-time using neuromusculoskeletal models. However, even if appropriate boundary conditions are provided, real-time solutions to continuum mechanics problems is an ongoing computational challenge. When implementing entire musculoskeletal structures in FEM models (e.g., complete bones), computational demand may be substantially reduced by spatially averaging many microstructural features, such as trabecular and cortical bone architecture. However, spatial averaging neglects analysis of tissue anisotropy and micro-architecture, which are known to influence tissue function (Stein et al., 2010). Generally, FEM models are computationally demanding and not solvable in real-time. Thus, FEM models must be reduced to surrogates by a process known as "Kriging" (Matheron, 1963), whereby the continuum model is first solved offline for all possible, or physiologic, configurations (Wu et al., 2014; Eskinazi and Fregly, 2016), and simulation results are then be stored for future realtime use. However, it is computationally expensive to establish robust surrogates of musculoskeletal tissue continuum models, given the large data throughput imposed by performing many multi-scale simulations (Erdemir et al., 2015). One potential strategy we are pursuing is use of high-performance computing, whereby a large number of simulations are managed and solved by a remote computing cluster.

## CHALLENGES AND FUTURE DIRECTIONS

Our proposed framework to modify an individual's physical behavior to optimize musculoskeletal tissue mechanobiology (**Figure 3**) is feasible and currently being developed. In **Table 1** we have summarized several challenges and possible future directions discussed in the text. To move these bioinspired technologies to clinical settings we need to direct our efforts toward: (1) rapid generation and seamless integration of personalized neuromusculoskeletal and FEM models, (2) use of wearable sensors, and (3) improvement of biofeedback modalities for stress and strain modulation.

Currently, creating personalized anatomic models from medical imaging is expensive (i.e., involves costly image acquisition and numerous man-hours to process raw medical imaging into high-fidelity computational models). However, improvements in image processing software, such as automatic segmentation and statistical shape modeling (Zhang et al., 2014; Zhang and Besier, 2017) may greatly accelerate model generation. Statistical shape modeling is promising as it may limit the need to acquire expensive medical imaging, relying instead upon a musculoskeletal atlas database to characterize an individual's anatomy from sparse or meta-data (Zhang and Besier, 2017). However, it is unclear whether current publically available medical imaging databases are sufficient to represent the variability in musculoskeletal anatomy in clinical populations, or those with traumas or implants. This is a limitation that will eventually be addressed by data sharing amongst research teams, which is an effort we thoroughly support.

Another limitation is that motion capture systems typically used in research gait laboratories are seldom used into clinical settings, because of their complexity, space requirements, and high purchase and operational costs. For bioinspired technologies to be broadly adopted, we need to free ourselves of traditional motion capture systems and look to wearable sensors to measure movement, external loads, and muscle excitation. A promising and relatively inexpensive example of wearable sensors that could help us on this mission are inertial measurement units, which provide a wealth of data that may be used to determine whole-body motion. Currently, inertial measurement units are limited by issues such as insufficient shielding from electromagnetic interference (i.e., while walking on treadmills or near informatics cabling) and registration of body-worn sensor positions to anatomic models. Wireless EMG systems have been used effectively in research and clinical settings for many years, and are now being integrated with inertial measurement units and other sensors as standalone devices or embedded into garments. Future research should aim to first establish if these wearable sensor systems can match the performance of traditional motion capture systems, and then minimize


TABLE 1 | Summary of the various challenges faced in modeling tissue mechanobiology and using biofeedback to modulate in vivo tissue strains in real-time.

the number of sensors required to drive neuromusculoskeletal models.

Very little is currently known about the ability of individuals to volitionally modulate musculoskeletal tissue stresses and strains in response to real-time feedback. To our knowledge, musculoskeletal tissue stresses and strains have never been estimated in real-time, let alone used to modify physical behavior. To date, only two research groups (Manal et al., 2012; Pizzolato et al., 2017b) have provided real-time biofeedback of musculoskeletal tissue loads, but their work has been limited to muscle-tendon and rigid articular contact forces, and did not model tissue stresses and strains.

We know from previous studies people can use visual biofeedback to manipulate external biomechanical variables, muscle excitations, and tibiofemoral contact forces (Manal et al., 2012; Pizzolato et al., 2017b). Future research should strive to identify the biofeedback modality optimal for modulating musculoskeletal tissue stresses and strains through changes in human movement and muscle activation. Further, it may be possible to draw inspiration from a variety of native mechanoreceptors in the human body to provide enhanced visual biofeedback of stresses and strains. We imagine a technology whereby estimates of musculoskeletal tissue loading (i.e., forces or stresses and strains) could be transformed according to golgi organelle and muscle spindle transfer functions to provide more intuitive biofeedback.

By optimizing the mechanical environment it may be possible to regulate musculoskeletal tissue mechanobiology, potentially preventing disease, or restoring degraded tissue to health.

#### REFERENCES


Consequently, modeling and controlling physical behavior of individuals has enormous implications for development and management of chronic musculoskeletal diseases such as osteoarthritis or tendinopathies. We have presented a framework to move from in vitro and ex vivo studies of tissue mechanobiology to personalized in silico real-time models of musculoskeletal tissue loading. Integrating and translating these bioinspired technologies to clinical settings will prove challenging and resource intensive. Skepticism from clinicians accustomed to generic recommendations based on linear statistics is anticipated and will need to be overcome by demonstrating the efficacy and clinical utility of the proposed new approach. However, there awaits a wide spectrum of important clinical conditions to which these bioinspired technologies could be applied with the goal of reducing the socio-economic burden of musculoskeletal diseases.

#### AUTHOR CONTRIBUTIONS

CP, DL, and DS contributed to conceptualize, draft, critically revise, and approve the final version of the article. RB, JC, MZ, and TB contributed to conceptualize, critically revise, and approve the final version of the article.

#### ACKNOWLEDGMENTS

This work was supported by the Australian Research Council Linkage Project scheme (LP150100905).


osteoarthritis in later life. Arthritis Rheum. 43, 2543–2549. doi: 10.1002/1529- 0131(200011)43:11<2543::AID-ANR23>3.0.CO;2-K


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Pizzolato, Lloyd, Barrett, Cook, Zheng, Besier and Saxby. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Estimation of Neuromuscular Primitives from EEG Slow Cortical Potentials in Incomplete Spinal Cord Injury Individuals for a New Class of Brain-Machine Interfaces

Andrés Úbeda1,2, José M. Azorín<sup>2</sup> , Dario Farina<sup>3</sup> and Massimo Sartori <sup>4</sup> \*

<sup>1</sup> AUROVA Group, Department of Physics, Systems Engineering and Signal Theory, University of Alicante, San Vicente del Raspeig, Spain, <sup>2</sup> Brain-Machine Interface Systems Lab, Miguel Hernández University, Elche, Spain, <sup>3</sup> Chair in Neurorehabilitation Engineering, Department of Bioengineering, Imperial College, London, United Kingdom, <sup>4</sup> Department of Biomechanical Engineering, University of Twente, Enschede, Netherlands

One of the current challenges in human motor rehabilitation is the robust application of Brain-Machine Interfaces to assistive technologies such as powered lower limb exoskeletons. Reliable decoding of motor intentions and accurate timing of the robotic device actuation is fundamental to optimally enhance the patient's functional improvement. Several studies show that it may be possible to extract motor intentions from electroencephalographic (EEG) signals. These findings, although notable, suggests that current techniques are still far from being systematically applied to an accurate real-time control of rehabilitation or assistive devices. Here we propose the estimation of spinal primitives of multi-muscle control from EEG, using electromyography (EMG) dimensionality reduction as a solution to increase the robustness of the method. We successfully apply this methodology, both to healthy and incomplete spinal cord injury (SCI) patients, to identify muscle contraction during periodical knee extension from the EEG. We then introduce a novel performance metric, which accurately evaluates muscle primitive activations.

Keywords: brain-machine interface, muscle primitives, corticospinal mapping, linear decoders, gait rehabilitation, lower-limb exoskeletons

## 1. INTRODUCTION

A brain-machine interface (BMI) is a tool that can translate brain activity into device control commands, thus enabling an alternative pathway for the brain to physically act upon the environment (Wolpaw et al., 2002). In a rehabilitation context, BMIs are conveniently combined with wearable robots such as exoskeletons (Contreras-Vidal et al., 2016). One of the main challenges is the ability of restoring ambulatory functions in paraplegic patients with neurological conditions including incomplete spinal cord injury or stroke (del Ama et al., 2012). Within this scope, the combination of BMIs and lower limb exoskeletons can may exploit the concept of neuroplasticity, i.e. linking descending neural commands and peripheral somatosensory feedback to promote the reorganization of central nervous system damaged pathways in charge of motor control (López-Larraz et al., 2016).

#### Edited by:

Florentin Wörgötter, University of Göttingen, Germany

#### Reviewed by:

Robert H. Lee, Emory University, United States Jan-Matthias Braun, University of Southern Denmark Odense, Denmark

#### \*Correspondence:

Massimo Sartori m.sartori@utwente.nl

Received: 28 February 2017 Accepted: 04 January 2018 Published: 25 January 2018

#### Citation:

Úbeda A, Azorín JM, Farina D and Sartori M (2018) Estimation of Neuromuscular Primitives from EEG Slow Cortical Potentials in Incomplete Spinal Cord Injury Individuals for a New Class of Brain-Machine Interfaces. Front. Comput. Neurosci. 12:3. doi: 10.3389/fncom.2018.00003

BMI-extracted neural commands, which encode user's motor intentions, should be subsequently translated into control commands to the exoskeleton in real-time during the rehabilitation procedure. Reliable decoding of motor intentions and accurate timing of the robotic device actuation is fundamental to optimally enhance the patient's functional improvement (López-Larraz et al., 2016). It has been shown that motor intentions can be detected from electroencephalographic (EEG) signals and used to trigger an ankle exoskeleton so that the assisted movement was perceived as voluntary (Mrachacz-Kersting et al., 2012).

Current research has also focused on the relationship between low-frequency cortical modulations and motor tasks. Slowcortical potentials (SCPs) reflect shifts in the cortical electrical activity lasting from several hundreds milliseconds to several seconds (Birbaumer et al., 1990; Shibasaki and Hallett, 2006). An example of this paradigm are movement-related cortical potentials (MRCPs) (Jiang et al., 2006; Shakeel et al., 2015). SCPs are triggered naturally as a person commences or imagines the onset of a movement. Moreover, there have been studies proposing the use of global cortical activity to extract kinematic information of upper and lower limb movements. In these studies, kinematic parameters were directly decoded from the activity of larger regions of the scalp by applying linear decoders to SCPs for decoding both upper and lower limb joint movements (Bradberry et al., 2010; Presacco et al., 2011), sitting and standing states (Bulea et al., 2014), finger movements (Paek et al., 2014), and types of grasping (Agashe et al., 2015). Other studies have dealt with the characteristics of the performed movement, showing that hand kinematics are better decoded when continuous and linear movements are performed (Úbeda et al., 2015) and exploring the possibility of using them to classify reaching directions (Úbeda et al., 2017).

In general, the use of linear decoders applied to SCPs are subject of controversy. Mechanical artifacts strongly affect the EEG low-frequency range during cyclic motion activities. This is suggested to directly influence the reliability of these decoders (Castermans et al., 2014; Costa et al., 2016). Moreover, other studies show that performance is not statistically different from chance levels due to the inherent properties of linear regression (Antelis et al., 2013). As a result, there is general consensus suggesting that current techniques are still far from being systematically applied to an accurate real-time control of rehabilitation or assistive devices (Úbeda et al., 2017). Indeed, only a few attempts reported to have obtained a reliable realtime decoder of movement kinematics (Bradberry et al., 2011). This study has been as well criticized for the way results are assessed which lead to performance similar to chance level (Poli and Salvaris, 2011).

The present study seeks to establish a reliable procedure to be applied in future real-time environments. Previous works are based on a single macroscopic regression function to directly map neural activity into the emerging/desired limb kinematics. In this, a single regression function may not be sufficient to capture all intermediate neuro-mechanical processes, thus only partly representing the mechanisms underlying movement. We suggest a possible solution to these major problems that is based on the combined use of linear decoders (for extracting high-level neural information) and multi-muscle electromyography dimensionality reduction (for capturing the basic spinal primitives of muscle control). Motor primitives encode information of the neural drive and have shorter pathways with respect to the cortical activity. As a consequence, our proposed method may be intrinsically robust (better signal to noise ratio) because it enables reconstructing a shorter neuromechanical gap (from brain activity to spinal cord activity) and applies the regression to a lower dimensional space (lowdimensional muscle primitives). Importantly, primitives have a lower dimensionality than lower limb kinematics, i.e., 12 degrees of freedom are needed to control 2 legs but only 4 primitives are needed to represent lower limb locomotion. To explore this methodology, we propose a novel approach that consists of detecting knee extensions from SCPs through the decoding of EMG primitives extracted from the recorded activity of the quadriceps femoris group.

## 2. MATERIALS AND METHODS

## 2.1. Experimental Setup

Four patients (P01–P04) (2 males and 2 females, age: 43.5 ± 12.4 years old) were recruited from the patients services at the National Hospital for Spinal Cord Injury in Toledo. Only adults with incomplete spinal cord injury (iSCI) lesion above D7-D8, with ASIA C or D were selected. All patients were able to maintain standing position and ambulate for 30 m without external assistance and had enough functionality and strength in the upper limbs to use a walker or crutches. Additionally, four healthy subjects (H01–H04) (3 males and 1 female, age: 33.5 ± 7.9 year old) participated in the study. This study was carried out in accordance with the recommendations of the ethical committee of the National Hospital for Spinal Cord Injury and Miguel Hernández University of Elche, with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by both committees.

Subjects seated comfortably on a chair and were asked to perform self-paced knee flexion-extension movements from full flexion (90◦ ) to full extension (0◦ ) (**Figure 1**, left). For each subject, data were recorded for 3 min, divided into 30-s runs with a 15-s rest period between them. Subject P04 only performed five runs due to fatigue. In the case of healthy subjects, the dominant leg was used to perform the movements. In the case of SCI patients, the movements were performed with the leg most affected by the lesion. All patients were capable of performing the knee flexion-extension movements, although more resting time between runs was given when necessary. During the performance of the knee flexionextension movements, electroencephalographic (EEG) signals were recorded with two gUSBamp amplifiers (g.Tec, GmbH, Austria) at 1,200 Hz from 32 electrodes placed over the central and parietal cortex according to this distribution: FZ, FC5, FC1, FCZ, FC2, FC6, C3, CZ, C4, CP5, CP1, CP2, CP6, P3, PZ, P4, PO7, PO3, PO4, PO8, FC3, FC4, C5, C1, C2, C6, CP3, CPZ, CP4,

P1, P2 and POZ. Subjects were asked to avoid blinks and head movements during each run. EMG signals were also recorded at 2,000 Hz from bipolar electrodes placed on 16 different muscles (Wave Wireless EMG, Cometa SRL, Italy). Additionally, knee angles were measured at 30 Hz using two inertial sensors (Technaid SL, Spain) placed on the thigh and on the leg.

### 2.2. Signal Preprocessing

First, EMG signals were resampled to match EEG signal time stamps. Raw EEG signals were analyzed to reject blinks. To that end, sections of EEG data with abnormal amplitude were rejected. Afterwards, EEG signals were low-pass filtered with a zero-phase 2nd-order Butterworth filter (2 Hz). Finally, EEG data from each electrode were standardized by subtracting, for each time sample (t), the mean (V¯ ) of the signal and dividing the result by the standard deviation (SDV) as shown in Equation (1). This standardization was computed for each individual run.

$$EV[t] = \frac{V[t] - \bar{V}}{SD\_V} \tag{1}$$

Raw EMG recordings were band-pass filtered (30–100 Hz), full-wave rectified, and low-pass filtered (6 Hz) using a zerophase second-order Butterworth filter. For each subject and muscle group, the resulting linear envelopes were normalized with respect to the overall peak amplitude for that muscle. This was selected as the maximum value of a 50 ms movingaverage window applied to the muscle linear envelopes across each recorded run (Gonzalez-Vargas et al., 2015).

#### 2.3. Muscle Excitation Primitives

Non-negative matrix factorization (NNMF) (Lee and Seung, 2001) was performed for the set of consecutive extension cycles of each subject. Muscle activations are inherently non-negative. NNMF decomposes a data matrix (EMG activity) into a synergy matrix, W, and a command matrix, A, such that EMG = W∗A, where the components of EMG, W, and A are all non-negative. During the knee flexion-extension exercise two primitives were identified, one active during the knee flexing phase and one during the knee extending phase. The knee extending phase was performed against gravity, resulting in pronounced extension primitives when compared to those extracted during the flexing phase. As a consequence, extension primitives (non-negative factors) were selected as major determinants of periodical multimuscle contractions during the self-paced knee movements. An example of this behavior can be observed in **Figure 1**, right, where a representative subject performs four consecutive knee extensions. Each extension is commanded by an almost equal activation of all the muscles included in the quadriceps femoris group and can be explained by the extension primitive.

#### 2.4. Linear Decoder

To decode the muscle primitive in charge of knee extension, a multidimensional linear regression has been applied in a similar way to Úbeda et al. (2017) and according to the formula:

$$\chi\left[t\right] = a + \sum\_{n=1}^{N} \sum\_{k=0}^{L} b\_{nk} \mathbb{S}\_n \left[t - G \ast k\right] \tag{2}$$

where x[t] is the non-negative factor of the primitive at time t and Sn is the voltage measured at electrode n. L is the number of lags (past voltage samples), G is the gap between lags, N the number of electrodes and a and b are the weights of the linear regression. N corresponds to 16 (number of electrodes introduced in the decoder). L was fixed to 10, meaning that 10 time samples per electrode are selected to feed the decoder.

#### 2.5. Automated EEG-Based Detection of Periodical Muscle Contractions 2.5.1. Electrode Selection

Several distributions of electrodes have been evaluated to extract valuable information of the activation of different cortical regions during the performance of the movements (**Figure 2**). The first distribution covers all the recorded electrodes (global activity). This is in line with previous decoding studies where it is suggested that regions not located over the motor cortex have a significant contribution in decoding performance and, thus,

they should not be discarded in the analysis (Agashe et al., 2015). However, under a classical electrophysiological basis, cortical modulations in charge of lower-limb motor control should be mainly located over the motor and premotor cortex midline (Brouwer and Ashby, 1990). Two distributions (midline region and midline) have been selected based on the assumption that motor cortex regions will provide a better performance than the activity of the whole cortex. The final distribution (lateral regions) has been selected to show if regions apparently not related to lower-limb motor activity have a significant contribution in the decoding performance.

#### 2.5.2. Decoding Process

The proposed linear decoder has been applied for each electrode distribution. To improve decoding performance, the parameter G (gap) has been swept to evaluate a processing time interval from 100 ms to 2.5 s. Processing time interval has been limited to 2.5 s to minimize the effect of previous cycles in the decoding process. For each subject, a cross-fold validation (6-folds) has been applied (5-folds in the case of subject P04 who only performed 5 runs). For each fold, the training data was used to compute the weights of the linear regression that are then applied to the test data to obtain the decoded non-negative factors. We computed the Pearson correlation coefficient between the real and decoded primitives for each testing fold and reported the performance in terms of average correlation. All electrode distributions have been then compared to select the one with higher performance. From the selected distribution, the processing time interval with the higher correlation has been fixed for further analysis.

#### 2.5.3. Significance Analysis

Shuffled data have been used as input to assess if the decoding accuracy was above chance levels. Shuffled data was obtained by randomly mixing trials of real cortical data and the associated non-negative factors to keep the temporal structure of the EEG signals in a way similar to Agashe et al. (2015). Shuffled data were filtered and standardized in the same way as the actual experimental data. Shuffled data decoding coefficients were computed for each subject with the previously selected best processing time interval for each electrode distribution. This means that, for each electrode distribution and subject, the crossfold validation was applied to obtain a total of 96 correlation coefficients for healthy subjects and 92 for SCI subjects. This helps to avoid chance effects due to the stochastic nature of the process and also reduces the possible bias of a particular electrode distribution or subject.

#### 2.5.4. Identification of Muscle Contractions

The EEG-decoded muscle primitive was then compared to the one extracted from EMGs. To that end, peaks of maximum contraction were computed for both the original and decoded signal to obtain similarity metrics. Peaks were detected by looking for downward zero-crossings in the first derivative that exceeded a slope threshold and an amplitude threshold. The slope threshold was fixed to a very low value (10−<sup>6</sup> ) while the amplitude threshold was fixed to 0.3 in the case of the original primitive and 0.05 in the case of reconstructed primitives, which were usually decoded with lower amplitudes.

True positive rate (TPR) was computed as the number of positive matches between the peaks extracted from both signals divided by the total number of extracted peaks. Only reconstructed peaks, which were closer than M times the average peak-to-peak distance in the original signal, were considered as positive. Detection rate (DR) was computed as the number positive matches divided by the number of peaks extracted from the original signal. Finally, time shift (TS) was computed as the average time shift between all the positive extracted peaks and their corresponding peak in the original signal. For comparison purposes, all the similarity metrics were computed for three different values of the parameter M: 0.1, 0.25, and 0.5. Additionally, the previously generated shuffled data was processed in the same way and compared to real data to evaluate the significance of the identification.

## 3. RESULTS

We performed three tests to evaluate the performance of the proposed methodology. The **first test** assessed decoding performance trends across subjects and conditions. **Figure 3** shows average decoding performance across subjects and cortical

regions. Each plot shows four curves that correspond to each preselected electrode distribution including: global activity, midline region, midline, and lateral regions. Each curve evaluates decoding performance for different processing time windows ranging from 100 ms to 2.5 s. Results show that decoding performance steadily increases from the minimum time window (i.e., 100 ms) and peaks at a subject-specific processing time interval (see **Table 1**). Then, decoding performance steadily decreases to levels similar to the starting point. This behavior is particularly evident for subjects H01, H02, H04, and for patient P03. Across all subjects and conditions decoding performance peaks approximately in the time frame 2–3 s (**Figure 3**). Subject H04 and patient P03 have however earlier peak decoding performance values, i.e., 0.66 s. Interestingly, lateral areas generally show worst decoding performance than global and midline areas. This is particularly visible in most of the subjects and patients, i.e., P01, P02, P04, H01, and H02. This is however not so clear in subjects H03 and H04 where lateral area decoding performance is most favorable.

The **second test** (**Figure 4**) identified the best decoding performance levels for each subject. These correspond to the processing time interval peaks in a particular electrode distribution (see **Table 1**). Additionally, chance levels (mean and STD) are represented for both healthy and SCI patients. The results obtained for all subjects are significantly different from chance levels (Wilcoxon Sum-Rank Test, p < 0.05). For most of the subjects and patients average decoding performance is >0.3 (subjects H04 and P04). Decoding performance for subjects H02 and P01 is >0.5. Interestingly, there is no significant difference between healthy and SCI subjects (Wilcoxon Sum-Rank Test, p > 0.05). **Figure 5** shows a representative example of how decoding performance influences the behavior of the reconstructed signals. It presents the original muscle primitive (activation coefficients) and its reconstruction for 4 representative folds. **Figure 5** (topleft and top-right) shows a similar behavior of the reconstructed signal despite the fact that decoding performance largely differs between these 2-folds. In the case of patient P02 (**Figure 5**, bottom-left), the decoding performance is high but the amplitude level mismatches between the original and the reconstructed signal. Finally, a poor reconstruction is shown in **Figure 5** (bottom-right), where the reconstructed primitive does not correctly match the original signal.

The **third test** assessed the ability of detecting periodical muscle contraction patterns based on reconstructed primitives and subsequently investigate how well they matched with the original primitives, i.e., those experimentally derived from EMG information. True positive rate (TPR), detection rate (DR), and time shift (TS) are presented (mean and STD) in **Table 2** for each selected detection margin (M). From the table, we can see that TPR and DR increase with higher margins. Unsurprisingly, TS also increases with M. In this situation, the matching peaks rise in number, as there is a wider window of detection. TPR is generally consistent with previously obtained decoding performance. Subjects with higher decoding performance, such as H02 and P01, obtain the best results. Good TPR is also achieved for subjects H01 and P03 who, on the contrary, have lower correlation levels. Significant TPRs have been highlighted TABLE 1 | Selected electrode distribution and processing time interval (PTI) for each subject.


The corresponding decoding performance in terms of correlation coefficient (CC) is also shown.

in the table after comparing them to chance levels (Wilcoxon Sum-Rank Test, p > 0.05). Interestingly, TPR computed for M = 0.5 is always above chance, while the remaining values are not always significantly different. To illustrate how the peaks are detected on both the original and the reconstructed signal, **Figure 6** shows an example for the same representative subjects and folds shown in **Figure 5**. Peak detection performance on each of the graphs is clearly consistent with previous results on signal reconstruction. **Figure 6** (top-left), with the best decoding performance of all four, presents a very good detection of original peaks (13 correct detections, 1 false detections, and 3 no detections). **Figure 6** (top-right) also shows a high number of accurate detections (11 correct detections, 2 false detections and 3 no detection). The number of detected peaks is quite lower for **Figure 6** (bottom-left), although peaks are identified with very good precision (9 correct detections, 0 false detections, 9 no detections). This is probably due to the bad scaling of the decoded primitive whose amplitude was comparatively lower than the rest. Finally, the fold with the worst decoding performance (**Figure 6**, bottom-right) shows, consequently, a

quite low identification accuracy (6 correct detections, 3 false detections, 5 no detections).

## 4. DISCUSSION

The main goal of this study was to obtain a robust way to translate brain signals into control commands provided to assistive devices such as robotic exoskeletons. For this purpose we proposed to identify motor primitives from SCPs. This enabled extracting high-level motor-related neural information and capturing the basic spinal primitives of multi-muscle control. As many motor processes are usually rhythmic, this methodology can provide a framework to identifiy periodical muscle activation from brain modulations that could be later applied to map full lower-limb mechanical information.

Current EEG-based continuous decoding techniques measure performance based on the cross-correlation between the original and the reconstructed signal. This correlation (Pearson correlation coefficient) is reported to be generally <0.4, which leads to difficulties in robustly translating the approach to a real-time assistive or rehabilitation scenario (Bradberry et al., 2010; Paek et al., 2014; Úbeda et al., 2017). One possible reason for low correlation metrics is the fact that current paradigms create direct mappings to body kinematics as a pure function of brain activity, thus bypassing all intermediate non-linear transformations, i.e., transmission pathways at the spinal and at the muscular level. As a consequence, important information may not be captured by a single macroscopic mapping. In our study we have decoded muscle primitives from brain activity instead of the direct kinematics of the lower limb. Results showed high decoding performance significantly above chance-level for all participants (**Figure 4**), with correlation coefficients being on average between 0.3 and 0.4, and reaching higher standard deviation values of up to 0.7. The obtained decoding performance was in line or higher than what was obtained in previous studies (Bradberry et al., 2010; Paek et al., 2014; Úbeda et al., 2017).

Moreover, our proposed approach offers the possibility to link decoded primitives to neuromusculoskeletal (NMS) models. In this context, it is not important to decode the exact shape and amplitude of activation primitives but just their timings. These would represent, in our formulation, the descending neural burst produced by the central nervous system (CNS) in the control of a group of muscles. In combination with modeling we propose in the near future to translate this burst into precisely timed mechanical function.

Our study resulted into three main findings: (1) decoding performance generally increases when only taking into account the information from motor cortex areas related to lowerlimb movements and, thus, our approach is physiologically consistent with previous results of cortical motor control (Brouwer and Ashby, 1990), (2) we provide evidence that the processing time interval should be increased to achieve the

TABLE 2 | Similarity metrics (mean ± STD) for different detection margins (M).


True positive rates (TPR), detection rates (DR) and time shifts (TS) are presented for each subject and condition. In gray, conditions that are significantly different from chance levels (Wilcoxon Sum-Rank Test, p < 0.05).

optimal performance in the decoding process and that this time interval is generally in the proximity of 2 s which is consistent with the generation of anticipatory low-frequency potentials (Jahanshahi and Hallett, 2003); and (3) our proposed method to identify muscle contractions from the decoded primitives is less dependent on amplitude and phase variations compared to other correlation metrics such as the correlation coefficient.

This study was based on a small subject size so caution must be applied in the interpretation of results. However, it is worth stressing that our method proved to operate on individuals with spinal cord injury and with disrupted neuromusclar control. This itself is an important element providing initial evidence that our approach could be further extended and translated to larger clinical scenarios.

## 4.1. Evaluation of Cortical Involvement in the Decoding

Cortical modulations in charge of lower-limb motor control are mainly located over the motor and premotor cortex midline (Brouwer and Ashby, 1990). For this reason, we have hypothesized that decoding performance of knee extension muscle primitives should increase if these particular areas are taken into consideration (test 1). We have compared decoding performance for different cortical regions to evaluate which distribution of electrodes increase the accuracy of the decoding (**Figure 2**). Closer inspection of the graphs in **Figure 3** reveals that most of the subjects obtained a lower performance when taking into account lateral regions. This is consistent with our hypothesis, suggesting that regions not related to the lower-limb motor cortex have less influence in the decoding performance. For most of the subjects, the optimal electrode distribution is centered on the cortex midline (H01, P01–P04) or on the midline area (H02), which again suggests that these areas are more relevant when decoding lower-limb activity.

In contrast, in the specific case of subject H03, when the processing time interval increases, performance levels for lateral areas increased compared to the other distributions. Subject H04 also obtained better results from the global activity of the whole cortex. A possible explanation of this behavior may be found in the variability of the cortex modulations across subjects. Indeed, EEG analysis is highly subject-specific and, for singular individuals, certain regions, different from the motor cortex, could contribute to motor control as previously suggested in Agashe et al. (2015).

Another possible reason could be related to the presence of motion artifacts affecting the global activity of the whole cortex. Motion artifacts are a key limitation in the application of BMIs under ambulatory conditions, particularly during gait rehabilitation procedures (Costa et al., 2016). This fact could also explain why all the configurations showed a similar behavior in subjects P03 and P04. Even so, our results suggest that artifact influence, if any, is limited due to the experimental conditions: the experimenter permanently monitored head movement and the proposed task did not involve important movement transmission through the body affecting the head of the participants (subjects sitting during the performance of knee extensions). Also, the lower decoding performance that many subjects obtained for lateral areas in comparison to other configurations indicates that artifact activity is not dominating the decoded output, being more dependent on actual cortical modulations. However, a future application of the proposed methodology in more complex conditions, such as the decoding of locomotion, should consider this element, as it may hinder the translation of the BMI system to the technology level in a realistic environment.

## 4.2. Analysis of the Processing Time Interval

Previous works used short processing time windows (i.e., 100 ms) in the application of linear decoders to SCPs, (Bradberry et al., 2010; Presacco et al., 2011; Úbeda et al., 2015). By using this approach, it is possible to obtain significant performance (in terms of signal-to-signal correlation) in the decoding of upper and lower limb kinematics. However, from the point of view of signal analysis, there is little variation of the signal amplitude in such a short time window, as the signals of interest are previously filtered below 2 Hz. This is even more critical in the case of

EEG modulations, where the signal to noise ratio is particularly low. Our proposed approach provides robustness in this aspect. As a result, we could enlarge the processing time interval (i.e., increasing it up to 2.5 s), thus enabling information extraction from larger low-frequency EEG modulation windows.

The analysis of the processing time window revealed that decoding performance peaks were associated to larger processing time intervals, i.e., generally between 2 and 3 s (**Table 1**). Our results showed that longer processing time intervals not only carried more information of low-frequency modulations, but also have electrophysiological consistency, e.g., previous studies reported anticipatory SCPs initiating around 2 s prior to movement onset (Jahanshahi and Hallett, 2003). These findings are limited to the small size of the population (8 subjects) so the assessment of larger populations is necessary to validate this conclusion.

One of the limitations of our experimental setup is the requirement of periodicity of the knee extensions as the synergistic analysis generates primitives for cyclic movements. However, important functional tasks in daily life are cyclic, e.g., locomotion, stairs climbing, ramp ascending, etc. Therefore, our approach is expected to have important implications despite the cyclic constraint imposed by muscle primitives analysis. Indeed, this fact can explain why, for particular subjects and channel distributions, decoding performance curves do not peak (for instance, subject H3 for lateral regions or P1 for midline area) (**Figure 3**). If the processing time interval is longer than one cycle, cortical modulations responsible for previous cycles can sum their influence into the decoding performance. To minimize this effect, we limited the selected processing time interval to 2.5 s. We also believe that a longer resting period between extensions (instead of continuous movements) will provide a better analysis of the proper processing time interval. In this sense, further evaluation should analyze how well this method adapts to pauses or absence of the periodical activity. This is a critical aspect to be considered in future experiments that assess similar singlejoint lower limb movements as well as those related to human locomotion.

#### 4.3. Identification of Muscle Contractions

It is worth stressing that current correlation metrics may be limited in determining the true performance of our proposed system. Correlation is invariant to scale and location (baseline) but very dependent on phase. This can be clearly seen in our results. High correlation coefficients could be obtained from a very good reconstruction (**Figure 5**, top-left) or with important differences in amplitude (**Figure 5**, bottom-left). On the contrary, a low correlation did not always translate into a poor reconstruction, as it happened in **Figure 5**, top-right, where the reconstructed signal was only slightly shifted but accurate. Future work will determine the proper performance metric to be applied, which will eventually depend on the final goal of the study. In this sense, recent works have already discussed about the effects of applying different performance metrics (Spuler et al., 2015), from the more typical Correlation Coefficient (CC) applied in our study, to other methods such as Normalized Root Mean Squared Error (NRMSE) or Signal to Noise Ratio (SNR), among others.

As a result, our study employed additional metrics for evaluating decoding performance by extracting peaks in the reconstructed signals that match the original muscle contractions. This is more suitable for detecting neural bursts to feed NMS models, as this method solves some of the limitations of the previous performance metric, e.g., the high dependence on phase. As an example, the reconstructed primitive for Subject H03 in **Figure 6**, top-right, which had a quite low correlation, achieved a very high identification accuracy, while folds with high correlation kept a very low detection error (**Figure 6**, top-left and bottom-left). In the case of Subject P02, the bad scaling in the reconstruction increased no detections, but did not affect the number of false detections.

Our proposed identification method is still dependent on the tuning of internal parameters including the margin of detection (M) or the amplitude threshold (Section 2.5.4). When parameter L was increased, the true positive rate importantly increased (**Table 2**). This is somehow misleading and does not represent a proper identification of decoded peaks because of the already mentioned continuous periodicity of the knee extensions. In fact, a very wide margin can lead to the misidentification of many detected peaks that are not really close to one of the peaks in the original signal. On the other hand, short margins failed to detect most of the peaks. To evaluate this issue we have a applied a paired test between our identification results and chance levels showing that both low (M = 0.1) and high (M = 1) margins reduce the significance of the identification accuracy (true positive rate) and, that an average length of this margin (M = 0.5), which corresponds to half of the peak-to-peak distance in the original signal, is a more suitable tuning for parameter M. This tuning is a critical aspect in the timing of actuated gait-assisting devices in realistic scenarios.

## 4.4. Further Application of Corticospinal Mapping

Our proposed procedure accurately extracts the activation onsets of muscle primitives and, thus, reduces the dimensionality of the decoding by directly mapping corticospinal transmission. Extracting muscle primitives from EEG signals may be more physiologically plausible than directly decoding joint kinematics as EMG extracted motor primitives encode alpha motor neuron discharges and have shorter pathways with respect to the cortical output.

Another important advantage of the proposed method is the reduction of dimensionality in the decoding procedure. In cyclic movements, such as locomotion, up to 12 different variables are needed to define movement, e.g., during gait, while with this procedure it is possible to reduce this output to just 4 primitives.

In addition, we evaluated new metrics that may be more suitable to trigger, for instance, an exoskeleton during gait assistance, as they are more sensitive to cyclic muscular activations. In this regard, the influence of mechanical artifacts affecting corticospinal mapping should be evaluated and removed to increase the robustness of the method and make it feasible to be applied in a realistic scenario.

In the future, corticospinal mapping may be combined with explicit models of the composite musculo-skeletal system. This will enable extracting whole-limb mechanical information from decoded muscle primitives, as previously proposed in, Sartori et al. (2013, 2016, 2017). This novel approach may open new avenues for the clinically viable interfacing with an individual's nervous system and the concurrent reconstruction of the intended musculoskeletal function. This methodology has the potential of, in the future, establishing man-machine interfaces that are robust and intuitive.

## AUTHOR CONTRIBUTIONS

AU: processed and analyzed the data and wrote the manuscript; MS: provided the tools for muscle primitives extraction, supervised the analyses and collaborated with the writing of the paper; JA, DF, and MS: discussed the results and implications of the experimental analysis and commented on the manuscript at all stages.

## FUNDING

This research has been supported by Conselleria d'Educació, Cultura i Esport of Generalitat Valenciana of Spain through grant APOSTD/2015/104, by the European Research Council Advanced Grant DEMOVE (grant agreement 267888), by the European Commission as part of the project BioMot (FP7-ICT-2013-10, Grant Agreement 611695) and by the Spanish Ministry of Economy and Competitiveness and the European Union through the European Regional Development Fund (ERDF) "A way to build Europe" as part of the project Associate–Decoding and stimulation of motor and sensory brain activity to support long term potentiation through Hebbian and paired associative stimulation during rehabilitation of gait (DPI2014-58431- C4-2-R).

## ACKNOWLEDGMENTS

Authors would like to thank Antonio J. del Ama and Ángel Gil-Agudo, from the Biomechanics and Technical Aids Department of the National Hospital for Paraplegics in Toledo (Spain) for their support and advice in the selection of the patients that participated in the study.

## REFERENCES


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Úbeda, Azorín, Farina and Sartori. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.