# FROM PERCEPTION TO ACTION: THE ROLE OF AUDITORY AND VISUAL INFORMATION IN PERCEIVING AND PERFORMING COMPLEX MOVEMENTS

EDITED BY : Mauro Murgia, Tiziano A. Agostini and Penny McCullagh PUBLISHED IN : Frontiers in Psychology, Frontiers in Neuroscience and Frontiers in Neurology

#### Frontiers eBook Copyright Statement

The copyright in the text of individual articles in this eBook is the property of their respective authors or their respective institutions or funders. The copyright in graphics and images within each article may be subject to copyright of other parties. In both cases this is subject to a license granted to Frontiers. The compilation of articles constituting this eBook is the property of Frontiers.

Each article within this eBook, and the eBook itself, are published under the most recent version of the Creative Commons CC-BY licence. The version current at the date of publication of this eBook is CC-BY 4.0. If the CC-BY licence is updated, the licence granted by Frontiers is automatically updated to the new version.

When exercising any right under the CC-BY licence, Frontiers must be attributed as the original publisher of the article or eBook, as applicable.

Authors have the responsibility of ensuring that any graphics or other materials which are the property of others may be included in the CC-BY licence, but this should be checked before relying on the CC-BY licence to reproduce those materials. Any copyright notices relating to those materials must be complied with.

Copyright and source acknowledgement notices may not be removed and must be displayed in any copy, derivative work or partial copy which includes the elements in question.

All copyright, and all rights therein, are protected by national and international copyright laws. The above represents a summary only. For further information please read Frontiers' Conditions for Website Use and Copyright Statement, and the applicable CC-BY licence.

ISSN 1664-8714 ISBN 978-2-88963-371-5 DOI 10.3389/978-2-88963-371-5

#### About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

#### Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

#### Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

#### What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office: researchtopics@frontiersin.org

# FROM PERCEPTION TO ACTION: THE ROLE OF AUDITORY AND VISUAL INFORMATION IN PERCEIVING AND PERFORMING COMPLEX MOVEMENTS

Topic Editors: Mauro Murgia, University of Trieste, Italy Tiziano A. Agostini, University of Trieste, Italy Penny McCullagh, California State University, East Bay, United States

Citation: Murgia, M., Agostini, T. A., McCullagh, P., eds. (2020). From Perception to Action: The Role of Auditory and Visual Information in Perceiving and Performing Complex Movements. Lausanne: Frontiers Media SA. doi: 10.3389/978-2-88963-371-5

# Table of Contents


Shashank Ghai, Mircea Stoica, Alexander Maye, Holger Blume and Alfred O. Effenberg

*34* In dubio pro silentio *– Even Loud Music Does not Facilitate Strenuous Ergometer Exercise*

Gunter Kreutz, Jörg Schorer, Dominik Sojke, Judith Neugebauer and Antje Bullack


A. A. M. (Daphne) van Opstal, Niek H. Benerink, Frank T. J. M. Zaal, Remy Casanova and Reinoud J. Bootsma


Robin C. Jackson, Hayley Barton, Kelly J. Ashford and Bruce Abernethy

*130 Collision Avoidance With Multiple Walkers: Sequential or Simultaneous Interactions?*

Laurentius Antonius Meerhoff, Julien Pettré, Sean Dean Lynch, Armel Crétual and Anne-Hélène Olivier

*144 Watching or Listening: How Visual and Verbal Information Contribute to Learning a Complex Dance Phrase*

Bettina E. Bläsing, Jenny Coogan, José Biondi and Thomas Schack


Nina Schaffert, Thenille Braun Janzen, Klaus Mattes and Michael H. Thaut


Joan N. Vickers, Joe Causer and Dan Vanhooren

# Editorial: From Perception to Action: The Role of Auditory and Visual Information in Perceiving and Performing Complex Movements

#### Mauro Murgia<sup>1</sup> \*, Tiziano Agostini <sup>1</sup> and Penny McCullagh<sup>2</sup>

<sup>1</sup> Department of Life Sciences, University of Trieste, Trieste, Italy, <sup>2</sup> Department of Kinesiology, California State University East Bay, Hayward, CA, United States

Keywords: perception, action, complex movements, sport, rehabilitation

**Editorial on the Research Topic**

#### **From Perception to Action: The Role of Auditory and Visual Information in Perceiving and Performing Complex Movements**

Edited and reviewed by: Ana Bengoetxea, Université libre de Bruxelles, Belgium

> \*Correspondence: Mauro Murgia mmurgia@units.it

#### Specialty section:

This article was submitted to Movement Science and Sport Psychology, a section of the journal Frontiers in Psychology

Received: 14 October 2019 Accepted: 15 November 2019 Published: 29 November 2019

#### Citation:

Murgia M, Agostini T and McCullagh P (2019) Editorial: From Perception to Action: The Role of Auditory and Visual Information in Perceiving and Performing Complex Movements. Front. Psychol. 10:2696. doi: 10.3389/fpsyg.2019.02696 In this introduction to the Research Topic "From Perception to Action. The Role of Auditory and Visual Information in Perceiving and Performing Complex Movements," we would like to thank all the contributors for their valuable efforts. In addition, we are extremely grateful to the reviewers for their constructive comments and for helping us to improve the overall quality of the articles of this special issue. Finally, we thank the staff of the editorial office for their advice in the management of the review process and for their technical support.

This article collection extends our knowledge on the influence of sensory information on the perception and execution of movements, with a special focus on movement-related auditory and visual information.

In recent years, various studies investigated auditory information in complex movements, in terms of how sounds can affect movement execution (e.g., Thaut et al., 2015; Pizzera et al., 2017; Bailey et al., 2018; Murgia et al., 2018) and of how biological motion perception is affected by sounds (e.g., Allerdissen et al., 2017; Camponogara et al., 2017; Murgia et al., 2017; Sors et al., 2018). In our Research Topic, we include several contributions both on perception and motor execution, using in some cases purely auditory stimuli or combining/comparing auditory and visual stimuli. Interestingly, the contributions to this line of research cover a relative wide range of domains (sports, rehabilitation, music, and dance), providing readers with a quite large overview of applications of these studies.

As for the visual modality, many studies investigated the role of visual information on the perception of other's movement, with applications in sport anticipation (Williams and Jackson, 2019), perceptual training (Abernethy et al., 2012), and interpersonal coordination dynamics (Travassos et al., 2011; Nalepka et al., 2017). In this special issue, we host contributions which provide new knowledge in these fields, showing empirical findings deriving from a quite large variety of paradigms and research methods (e.g., point-light displays, spatial and temporal occlusions, virtual reality, eye tracking).

The articles of this collection are grouped in four sections.

# AUDITORY INFORMATION IN SPORT, EXERCISE AND REHABILITATION

The first contribution of this section is a review by Schaffert et al., describing the mutual influences between complex movement and sound. The authors critically analyze the studies on ecological sounds and movement sonification in sports and those on rhythmic auditory information and sonification in rehabilitation. The next two contributions address two methodological issues. The contribution by Schmitz et al. proposes a new method based on movement sonification for the rehabilitation of patients with stroke. In particular, the authors contribute a "Clinical study protocol article," describing a protocol that provides auditory real-time feedback on upper limb movement, aimed at helping patients participating to a motor rehabilitation program after stroke. The work by Ghiselli et al. illustrates three clinical cases of children with congenital hearing impairment engaged in non-instrumental musical training. The authors describe this training and its effects on cognitive and motor skills, discussing the preliminary evidence of this method and its potential clinical relevance.

The last two contributions of this section are original research articles. The study by Ghai et al. investigates the effects of auditory feedback in real-time to facilitate knee proprioception. The authors provide empirical evidence that the use of auditory feedback improves the accuracy of knee re-positioning and that this effect can be modulated with step-wise transposition of frequency. The authors discuss the potential applications of their finding in rehabilitation settings. Conversely, the last work of this section—by Kreutz et al. concerns the effects of loud music in sports, and in particular on ergometer exercise. The authors investigate the effects of electronic music, manipulating the intensity levels, and evaluated the ergometer performance, the perceived fatigue and the heart rate in university students with relatively high and low levels of training.

# AUDITORY AND VISUAL INFORMATION IN MOTOR LEARNING AND IMAGERY

This section includes those contributions examining the effects of auditory and visual cues (either compared or combined) on movement or imagery. The study by Bienkiewicz et al. investigates whether auditory and visual cues regarding the kinematic of experts can enhance motor learning in golf, demonstrating that both auditory and visual cues can be beneficial for novices. Bläsing et al. focus on motor learning in dance. They compare visual cues and verbal instructions and show that the latter are more effective than the former, when learning dance movements. Finally, Yu et al. investigate the lower limb imagery alone or combined with visual or audiovisual stimuli, using neurophysiological measures. They find that the visual-auditory stimuli produce the most valuable effects, with important implications for motor learning and rehabilitation.

# VISUAL INFORMATION AND MOTOR EXPERTISE

This section starts with the contribution by Kurz and Munzert, who present a mini review on football penalty takers and eye movements. In particular, they analyze how experimental artificial conditions influence gaze behavior. The second contribution of this section is an original study by Vickers et al.. The authors investigated the role of quiet eye in basketball, and in particular they focused on the timing and the location of fixations, and on the effect of the defender on performance, in three-point shots. The next contribution—by Jackson et al.—further analyzes the role of visual perception in football. In this case, the authors used the spatial and temporal occlusion paradigms to investigate the ability to discriminate between genuine and deceptive actions, and examined the sensitivity to different sources of visual information of the opponent.

The third article of this section is by Bläsing and Sauzet and investigates the perception of action in the domain of dance. In particular, the authors analyze the participants' ability to recognize point-light displays of dance-like actions, previously performed by the same participants. The next article is by Marchal-Crespo et al. and deals with different training strategies to enhance motor learning. In their work, the authors focus on the learning process of a modified gait pattern, and compared the haptic error modulation and the visual error amplification strategies. Finally, this section ends with the contribution by Castañer et al., who study the laterality profile and the approach of young athletes on a novel perceptual-motor situation. In particular, they examine how the athletes use the limbs and investigated their spatial orientation.

# INTERPERSONAL COORDINATION AND SENSORY INFORMATION

The last section of this special issue is dedicated to original studies on interpersonal coordination, interactions among actors, and perception of others' point of view. The first contribution of this section is by Hwang et al., who examine the social coupling between two individuals in a collaborative task. They manipulate the perceptual information available, by combining visual information with different types of auditory feedback. The next study by van Opstal et al. focuses on the investigation of interception, using a doubles-pong task. In particular, the authors study how teams intercept approaching balls, when teams are composed of two different level players. The third study of this section is by Meerhoff et al., who focus on collision avoidance. In their study, the authors examine the strategies of dyadic avoidance compared to triadic avoidance, and how locomotor interactions are influenced by the dynamics of a passable gap between two walkers. Finally, the last contribution of this section (and of the entire article collection) is by Cook et al.. In this study, the authors investigate how naturally produced virtual motion can affect postural regulation. Moreover, they study the response to different types of optical flow, which was produced by other individuals.

## FINAL REMARKS

As editors, we are fully satisfied with this collection of articles and are convinced that most of them will have a high impact on research in this field. We hope that these works will stimulate new ideas, and contribute to the development of research on the

#### REFERENCES


mutual influences between auditory and visual perception and complex movements.

#### AUTHOR CONTRIBUTIONS

MM, TA, and PM contributed equally to the development of the outline of this editorial. MM wrote the first draft, which was revised and edited by TA and PM. All the authors approved the final version of the manuscript.


**Conflict of Interest:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Murgia, Agostini and McCullagh. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Auditory Proprioceptive Integration: Effects of Real-Time Kinematic Auditory Feedback on Knee Proprioception

#### Shashank Ghai\*, Gerd Schmitz, Tong-Hun Hwang and Alfred O. Effenberg

Institute of Sports Science, Leibniz University Hannover, Hannover, Germany

The purpose of the study was to assess the influence of real-time auditory feedback on knee proprioception. Thirty healthy participants were randomly allocated to control (n = 15), and experimental group I (15). The participants performed an active knee-repositioning task using their dominant leg, with/without additional real-time auditory feedback where the frequency was mapped in a convergent manner to two different target angles (40 and 75◦ ). Statistical analysis revealed significant enhancement in knee re-positioning accuracy for the constant and absolute error with real-time auditory feedback, within and across the groups. Besides this convergent condition, we established a second divergent condition. Here, a step-wise transposition of frequency was performed to explore whether a systematic tuning between auditory-proprioceptive repositioning exists. No significant effects were identified in this divergent auditory feedback condition. An additional experimental group II (n = 20) was further included. Here, we investigated the influence of a larger magnitude and directional change of step-wise transposition of the frequency. In a first step, results confirm the findings of experiment I. Moreover, significant effects on knee auditory-proprioception repositioning were evident when divergent auditory feedback was applied. During the step-wise transposition participants showed systematic modulation of knee movements in the opposite direction of transposition. We confirm that knee re-positioning accuracy can be enhanced with concurrent application of real-time auditory feedback and that knee re-positioning can modulated in a goal-directed manner with step-wise transposition of frequency. Clinical implications are discussed with respect to joint position sense in rehabilitation settings.

Keywords: perception, rehabilitation, sonification, coordination, joint position sense

# INTRODUCTION

Real-time kinematic auditory feedback can be effective in enhancing motor perception, control, and learning (Effenberg, 2005, 2014; Sigrist et al., 2015; Effenberg et al., 2016; Dyer J. et al., 2017). The perception of additional real-time acoustic feedback driven by dynamic or kinematic movement parameters obviously supports sensory/perceptual-motor representations (Effenberg, 2005; Schmitz et al., 2013) by enhancing cross-modal stimulation (Scholz et al., 2015; Ghez et al., 2017), multisensory integration (Schmitz et al., 2013; Effenberg et al., 2016), internal motor

Edited by:

Mauro Murgia, University of Trieste, Italy

#### Reviewed by:

Federica Corona, Università degli Studi di Cagliari, Italy Matthew Rodger, Queen's University Belfast, United Kingdom

> \*Correspondence: Shashank Ghai shashank.ghai @sportwiss.uni-hannover.de

#### Specialty section:

This article was submitted to Perception Science, a section of the journal Frontiers in Neuroscience

Received: 15 December 2017 Accepted: 22 February 2018 Published: 08 March 2018

#### Citation:

Ghai S, Schmitz G, Hwang T-H and Effenberg AO (2018) Auditory Proprioceptive Integration: Effects of Real-Time Kinematic Auditory Feedback on Knee Proprioception. Front. Neurosci. 12:142. doi: 10.3389/fnins.2018.00142 simulation (Schmitz and Effenberg, 2017), and neural plasticity (Altenmüller et al., 2009; Ghai et al., 2017c). Literature indicates strong associations between auditory and motor areas for enhancing the performance in music (Lahav et al., 2013), breathing (Murgia et al., 2016), writing (Effenberg et al., 2015; Danna and Velay, 2017), sports (Sigrist et al., 2013, 2015; Effenberg et al., 2016), and rehabilitation (Altenmüller et al., 2009; Murgia et al., 2015; Pau et al., 2016; Scholz et al., 2016; Ghai et al., 2017c; Mezzarobba et al., 2017). Strong auditory motor couplings have also been confirmed in neuroimaging studies, where enhanced activation in cortical and sub-cortical structures associated with biological motion perception were reported (Scheef et al., 2009; Schmitz et al., 2013). Several underlying theories have been suggested to ascertain the beneficial effects of concurrent auditory feedback on motor performance. For instance, the concurrent auditory feedback is thought to amplify the brain's ability to integrate multiple congruent perceptual streams, leading to formation of stable internal feed-forward models (Wolpert and Miall, 1996; Calvert et al., 2000; Shams and Seitz, 2008; Van Vugt, 2013). Moreover the real-time availability of feedback can serve as an external guidance for motor execution (Dyer J. F. et al., 2017) as well as an error feedback (Altenmüller et al., 2009; van Beers, 2009; Sigrist et al., 2015; van Vugt and Tillmann, 2015), and can enhance motor imagery (Sigrist et al., 2013), cognitive-emotional functioning (Eschrich et al., 2008; Sihvonen et al., 2017; see also Sigrist et al., 2013).

A strong influence of real-time auditory feedback on motor performance (Eriksson and Bresin, 2010; Schmitz et al., 2014; Scholz et al., 2015; Sigrist et al., 2015; Danna and Velay, 2017; Dyer J. F. et al., 2017), indicates a proportional influence of auditory domain over proprioception (Pantev et al., 2001; Scholz et al., 2015; Effenberg et al., 2016; Danna and Velay, 2017; Sihvonen et al., 2017), and it becomes effective as an integral component of motor control and coordination process (Proske, 2005; Ghai et al., 2017a). Scholz et al. (2015) mentioned that spatio-temporal associations generated by realtime kinematic auditory feedback during motor execution might allow substitution of proprioceptive deficits, possibly by closing the sensorimotor loop (Altenmüller et al., 2009; Särkämö et al., 2016; Scholz et al., 2016). Dyer J. et al. (2017) and van Vugt and Tillmann (2015) further added that the concurrent auditory feedback might supplement the low temporal-perceptual resolution of the proprioceptive domain (Tinazzi et al., 2002). Danna and Velay (2017) in their recent study proposed auditory-proprioceptive substitution for the enhancements the authors reported in handwriting performance for deafferented subjects receiving concurrent auditory feedback. These findings draw inferences from literature pertaining to cross-modal stimuli processing (Stein and Meredith, 1993; Calvert, 2001; Bavelier and Neville, 2002). For instance, sensory convergence from different sensory modalities have been reported to provoke cross-modal interactions (Macaluso et al., 2000; Macaluso and Driver, 2001). Furthermore, these claims are supported by neuroanatomical studies, reporting the presence of long range cortico-cortical connections in between sensory cortices (Falchier et al., 2001; Foxe, 2009; Keniston et al., 2010; Butler et al., 2012), and multisensory integration sites (Chabrol et al., 2015; for a detailed review see Calvert, 2001). This might suggest the possibility of a level of interdependency that the sensory modalities might share with each other to generate an integrated multimodal percept (Macaluso et al., 2000; Macaluso and Driver, 2001; Bavelier and Neville, 2002; Butler et al., 2012). In addition, several psychophysical studies have reported strong associations between the auditory and motor areas (Jokiniemi et al., 2008; Chen et al., 2009; Yau et al., 2009; Wilson et al., 2010b; Butler et al., 2012). These findings are further supplemented by the neuroimaging studies, reporting shorter pathways between the auditory and motor cortices, especially for multisensory integration (Lang et al., 1990; Zatorre et al., 2007; Foxe, 2009; Keniston et al., 2010; Butler et al., 2012; Chauvigné et al., 2014; Ishikawa et al., 2015). This might explain the strong influence of such audio-tactile cross-modal stimuli in terms of processing temporal (Fujisaki and Nishida, 2009), and certain impact on spatial information (Belardinelli et al., 2009; Jimenez and Jimenez, 2017; for a review see Lu et al., 2013). Nevertheless, despite the vast amount of literature indicating a strong influence of the audio-motor coupling for sensorimotor processing (Ghai et al., 2017c,d,e, 2018), a gap in literature persists concerning its applicationsin rehabilitation (Danna and Velay, 2017; Ghez et al., 2017), and/or sports (Ghai et al., 2017c).

As mentioned before, proprioception is an integral component of the coordination processes of the body (Gentilucci et al., 1994; Laskowski et al., 2000; Smith et al., 2012; Aman et al., 2014; Ghai et al., 2016, 2017a). Deficits in proprioceptive perception are directly linked with poor sensorimotor and somatosensory functioning (Aman et al., 2014; Ghai et al., 2016), characterized by a wide range of musculoskeletal and neuromuscular disorders (Sacco et al., 1987; Jensen et al., 2002; Ribeiro and Oliveira, 2007; Gay et al., 2010; Konczak et al., 2012; Ghai et al., 2017a). Its predominant role in rehabilitation has been emphasized in several studies (Lephart et al., 1997; Laskowski et al., 2000; Ribeiro and Oliveira, 2007; Rosenkranz et al., 2009; Gay et al., 2010; Aman et al., 2014). Therefore, exploring the possible influences of concurrent auditory feedback on proprioception might provide multifaceted benefits. First and foremost, the outcomes might provide a better understanding of intervention designs in rehabilitation, and sport settings with auditory feedback. Moreover, the evaluation of audio-proprioceptive coupling during an arbitrary action (knee-joint proprioception) might allow a better understanding of trans-modal activity of auditory and motor domains beyond music and language (Altenmüller et al., 2009). Finally, a better comprehensive understanding might be developed to support the psychophysical (Butler et al., 2012), neurophysiological (Ishikawa et al., 2015), studies analyzing the multisensory and cross modal integration between auditory and proprioceptive domains. Till this date, only a handful of researchers have attempted to answer the possible effects of real-time auditory feedback on proprioception (Van Vugt, 2013; Scholz et al., 2016; Danna and Velay, 2017; Dyer J. et al., 2017; Ghez et al., 2017). However, their interpretations on proprioceptive-auditory substitution are mostly speculative. For instance, none of the performed studies excluded vision during the performance of the motor task. As a result, possible influences from the visual modality during multisensory or cross modal integration processes can be expected (Plooy et al., 1998; Verschueren et al., 1998; Lönn et al., 2000). Research indicates the importance of isolating inputs from specific sensorimotor structures to provide a better understanding of direct influence over proprioception (Gay et al., 2010).

In a first attempt we tried to analyse the effects of real-time auditory feedback on clinical aspects of knee joint proprioception in a joint position sense test (Sherrington, 1907; Dover and Powers, 2003; Van Vugt, 2013). Based on interpretations drawn from state feedback control theory (Wolpert and Miall, 1996; Shadmehr and Krakauer, 2008), we expected realtime auditory feedback to cause enhancements in knee-joint proprioception or. Moreover, in a second step, we tried to analyze the effects of subliminal transposition of real-time auditory feedback's frequency on auditory-proprioceptive perceptions. The motivation of this part of study was derived from psychophysical studies revealing strong evidence of convergence between auditory and motor systems for computing frequency (Pantev et al., 2009; Wilson et al., 2009, 2010a), especially within well matched stimuli reflecting a similar event (Foxe, 2009). We expected that if auditory feedback could influence proprioception, understanding the role of frequency in this attained effect could allow a better understanding of the results. We therefore, evaluated influence of any divergent step-wise transposition of frequency with real-time auditory feedback would allow directed modulation of proprioceptive perceptions in terms of knee position.

In this article two experiments are mentioned. The second experiment is an extension of the first study, which was conducted after the analysis of results. The experiment II follows the same design and protocol but differs in terms of the magnitude and direction of step-wise transposition of the frequency of the feedback. se experiments differ based on magnitude and direction of step-wise transposition. We expect the outcomes from this study to provide novel practical implications in rehabilitation and sports settings.

# METHODS

# Experiment I

#### Experimental Design

This whole CCT was carried out between August 2016 and February 2017. Participants were randomly allocated to experimental or control group. In each group, participants carried out the active (knee-joint) repositioning task with their dominant legs. The experimental group concurrently received real-time and transposed (0.25◦ /repetition) auditory feedback while performing the active knee re-positioning tasks. The control group received white noise. The experiment consisted of five treatment blocks. Re-positioning tasks without any auditory feedback were performed on the odd numbered blocks. Auditory feedback (real-time, modulated, white noise) was provided in the even treatment blocks. The participants performed 15 repetitions per angle in a block i.e., 30 repetitions per block. The target angle for the repositioning task was 40 and 75◦ .

#### Participants

Thirty participants, randomly divided in control [8 males/7 females; mean ± SD (age): 23.5 ± 2.5 years], and experimental group I (7 male/8 female; 24.2 ± 3.7 years) volunteered to participate in the study. All participants self-reported as healthy with no history of significant hip, knee, or back injury. Written informed consent was obtained from each participant, and ethical approval was obtained from the Ethics Committee of the Leibniz University Hannover. All participants underwent a baseline test for auditory capabilities (HTTS Audiometry) and were asked to fill a self-reported questionnaire post the experiment. All participants received eight Euros for their participation.

#### Experimental Procedure

Participants were comfortably seated with their feet on the floor, their back resting against a wall, and their pelvis stabilized (Tiggelen et al., 2008; Ghai et al., 2016). During the sitting position, the knee joint was maintained at the right angle. This position of the knee joint was considered as 0◦ and further extension from this position onwards was referred as positive angles from this value (Supplementary File 1). Participants wore wireless headphones (Sennheiser, Wedemark, Germany), and were blindfolded to eliminate visual cues. The experimenter passively moved the dominant leg to a previously identified target position (40 or 75◦ ) in an open kinetic chain and held at the target angle to allow the participant to memorize the position (Selfe et al., 2006; Ghai et al., 2016). The experimenter, a physiotherapist, checked and rechecked the angle while using a handheld goniometer, and motion capture reading to confirm the target angle. The leg was then returned to the initial position, and following a 5 s interval, the participant attempted to reposition the leg at the same joint angle. The participant was instructed to repeatedly re-position the leg to the instructed angle with an instruction "please re-position your leg to the performed angle hold the angle for 2 s and then return it to the starting position." The experimenter counted 15 repetitions and asked the participants to stop. This protocol was repeated for both the target angles (40 and 75◦ ), across 5 treatment blocks. During the first, third, and fifth treatment blocks no auditory feedback was provided to the participants. However, during the second treatment block the same protocol was followed with real-time auditory feedback i.e., the experimenter initially took the dominant leg to the target angles with real-time auditory feedback. Thereafter, the participants performed the same target angles with real-time auditory feedback. During the fourth block, the experimenter initially positioned the dominant leg passively with real time auditory feedback, after which participants re-positioned their knee unaware of the modulation in frequency of auditory feedback (Supplementary File 2). Dynamic repositioning accuracy was computed to determine discrepancies while consecutively repositioning the knee joint. For instance, the repositioning performance of 40, 38, 43, 37◦ . . . the computation of repositioning error was performed by subtracting the performed angle with the previous angle i.e., 38◦–40◦ , 43◦–38◦ , 37◦–43◦ . . . and so on. After the experiment was concluded, the participants were asked to fill a four-point questionnaire. The questionnaire enquired about the perceived duration of the experiment, the fatigue level, the excerptions perceived if any in the quality of the auditory feedback (for identifying whether participants were consciously able to detect changes in the frequency of the real-time auditory feedback), and subjective rating for compliance with auditory feedback on a 10-point Likert scale. The experimental protocol lasted approximately for 45 min.

#### Real-Time Auditory Feedback Mapping

Real-time auditory feedback was generated using Python (version 2.7) and Csound version 6.0. Sound synthesis was based on a band-limited oscillator bank with lowpass filtering. Knee joint angle and angular velocity are mapped onto pitch and amplitude of the auditory feedback, respectively. During sitting the right angle at the knee joint is regarded 0◦ , and any extension from this point onwards is referred in positive values from this angle. The changes in angles from 0 to 90◦ of full extension is configured from 120 to 300 Hz of frequency change, respectively. Here, amplitude is a function of square of knee angular velocity which is relevant to kinematic energy. For the amplitude function, exaggerated representation of the angular position was added because, as the frequency increases, human ear gets less sensitive in identifying the same pitch differences. The exaggeration in amplitude can therefore complement the lack of sensitivity, which properly stimulates the human ears. These mapping functions are also provided as a mathematical equation for clarity.

$$\begin{aligned} \text{Pit} &= \text{2} \times \theta\_{kme, joint} + 120 \text{ (Hz)} \,\text{.}\\Amp &= \,\alpha\omega\_{kme, joint}^2 + \beta \left(\cos\left(90^\circ - \theta\_{kme, joint}\right) - k\right) \end{aligned}$$

In the equations, Pit is pitch (audio frequency), θ knee.joint is the knee joint angle, Amplitude is Amp, ωknee.joint is joint angular velocity. The equation also includes coefficients α, β as well as a constant value, k.

Modulation of real-time auditory feedback was subtle and provided in an under-transposition manner. Here, the mapping information between audio frequency and knee angle was manipulated during repetitions. For example, 15 repetitions in a step-down transposition by −0.25◦ (−0.5 Hz/rep) at the target angle. Frequency was changed per repetition, for instance from 180 to 193 Hz which would be is equivalent to a change of the knee angle from 40 to 36.5◦ in the constant original mapping (Supplementary Files 3, 6) for 15 repetitions. A sample for both the real-time auditory feedback (Supplementary File 5) and modulated auditory feedback (Supplementary File 6) have been provided.

#### Kinematic Analysis

Repositioning error (RE) was assessed in each trial using XSENS MVN Biomech (XSENS Technologies B.V, Netherlands), in a configuration mode limited to the lower body. High reliability and validity of this inertial sensor based motion analysis device has been previously reported (Cooper et al., 2009; Zhang et al., 2013). Seven pre-identified inertial measurement units (IMUs) were placed by a physiotherapist on sacrum, lateral side of femoral shaft, medial surface of tibia, and tarus using velcro straps (Supplementary File 1; Zhao et al., 2016). The angular repositioning data, expressed in sensor coordinate frame was wirelessly recorded with a sampling frequency of 60 Hz in a laptop (Lenovo INC, Hongkong) and saved in MVN file format. Thereafter, the saved file was converted to XML format (MVNX) and imported in a Microsoft Excel spreadsheet. This format incorporates information concerning sensor data, segment kinematics and joint angles. Marked data points (highlighted in MVN file during recording) were matched with MVN recording graphs and the data was manually extracted by two researchers for further calculations. Absolute and constant error were then computed for characterizing the repositioning error in both the magnitude and direction of error, by considering the target angle as the previous consecutive angle to the current performance by the participant.

#### Statistical Analysis

Statistical analyses were performed using Statistical Package for Social Science (V. 23.0, SPSS Inc., Chicago, IL). In 2 separate analysis for absolute and constant errors. We analyzed Repositioning Error (the dependent measure), by conducting a Group (Experimental/control) × block (1–5) × Angles (40/75◦ ) RM-ANOVA with repeated measures on the last two factors. Effect sizes of the independent variables were expressed using partial eta squared (η<sup>p</sup> 2 ), with effect sizes <0.01 considered to be small, effect sizes between 0.01 and 0.06 considered to be medium and effect sizes >0.14 considered to be large (Sedlmeier and Renkewitz, 2008). Post-hoc comparisons were performed using stepwise Bonferroni holm corrections. The overall significance level was set to 5%.

## Results

.

#### Absolute Error

**Figure 1** illustrates the absolute repositioning accuracy in both groups. The experimental group I, with real-time auditory feedback performed significantly better than the control group without auditory feedback as confirmed by the significant main effect of group [F(1, 28) = 6.92, p = 0.014, η<sup>p</sup> <sup>2</sup> = 0.20]. Furthermore, repositioning accuracy depended on block [F(4,112) = 10.16, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.27]. Differences between block were mainly caused by the auditory feedback in the experimental group I as shown by the interaction block<sup>∗</sup> group [F(4,112) = 8.34, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.23]. A post-hoc test confirmed significant differences between the first and second block in the experimental group I (p < 0.001), but not in the control group (p > 0.999). Furthermore, the second (p < 0.001), but not the first (p > 0.999) block differed significantly between groups. After the removal of feedback this effect diminished. Accordingly, both groups performed in block 3 not significantly different than in block 1 (experimental group I: p > 0.999; control group: p > 0.999). Differences between angles were not significant [angles: F(1, 28) = 3.39, p = 0.076, η<sup>p</sup> <sup>2</sup> = 0.11; angle<sup>∗</sup> group; F(1, 28) = 3.65, p = 0.066, η<sup>p</sup> <sup>2</sup> = 0.12; angle∗block: F(4,112) = 0.46, p = 0.714, η<sup>p</sup> <sup>2</sup> = 0.02; angle∗block<sup>∗</sup> group: F(4,112) = 0.49, p = 0.690, η<sup>p</sup> <sup>2</sup> = 0.02].

represents experimental group I, T: Proprioceptive test without auditory feedback, RT: Real-time auditory feedback, MAP: Acoustic mapping, CT: Control group, EXP:

#### Constant Error

Experimental group). \*Represents significant differences.

**Figure 2** illustrates the constant repositioning error in both groups. The experimental group I with real-time auditory feedback performed significantly better than the control group without auditory feedback, as confirmed by the significant main effect of group [F(1, 28) = 6.150, p = 0.019, ηp <sup>2</sup> = 0.18]. Furthermore, a main effect was observed for block [F(4,112) = 4.320, p = 0.030, η<sup>p</sup> <sup>2</sup> = 0.13]. Differences between blocks were mainly caused by the auditory feedback in the experimental group I as shown by the interaction block<sup>∗</sup> group [F(4,112) = 4.560, p = 0.002, η<sup>p</sup> <sup>2</sup> = 0.140]. A post-hoc test confirmed significant differences between the first and second block in the experimental group I (p < 0.001), but not in the control group (p = 0.360). Furthermore, the second (p < 0.001), but not the first (p = 0.810) block differed significantly between groups. After the removal of feedback this effect diminished. Accordingly, both groups performed in block 3 not significantly different than in block 1 (experimental group I: p > 0.999; control group: p > 0.999).

In the 4th block, modulation in frequency of real-time feedback were introduced. We observed significant differences between the 3rd and 4th block of experimental group I (p = 0.001), and as compared to the 4th block control group (p < 0.001). No such differences were observed between 3rd and 4th block in control group (p = 0.660). Likewise, in 5th block both groups performed not significantly different than in 1st and 3rd block (all p's > 0.05). Significant differences were also not evident when the 4th block was compared with the 2nd block (p > 0.999) i.e., modulated feedback with un-modulated feedback. Constant error was significantly larger for angle 40◦ as compared to 75◦ [F(1, 28) = 21.80, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.44]. However, none of the interactions with the effects of the angles

were significant, but not for angle<sup>∗</sup> group; [F(1, 28) = 0.40, p = 0.532, η<sup>p</sup> <sup>2</sup> = 0.01]; angle∗block [F(4,112) = 0.36, p = 0.838, ηp <sup>2</sup> = 0.01] angle∗block<sup>∗</sup> group [F(4,112) = 0.20, p = 0.941, ηp <sup>2</sup> = 0.01].

#### Experiment II Experimental Design

This whole trial was carried out between March 2017 and September 2017. Participants were allocated to experimental group II. Due to the identical experimental design as experiment I data from the same control group was utilized for comparison and the data from control group of first experiment was utilized. Here, the participants carried out the active (kneejoint) repositioning task with their dominant legs. The experimental group concurrently received real-time, modulated (±1.3◦ /repetition) auditory feedback while performing the repositioning tasks. The control group received white noise. The experiment consisted of five treatment blocks. Re-positioning tasks without any auditory feedback were performed on the odd numbered blocks. Auditory feedback (real-time, modulated, white noise) was provided in the even treatment blocks. The participants performed 15 repetitions per angle in a block i.e., 30 repetitions per block. The target angle for the repositioning task was 40 and 75◦ .

#### Participants

Twenty healthy participants were included in experimental group II [10 females/10 males; mean ± SD (age): 26.8 ± 3.5 years]. All participants underwent a baseline test for auditory capabilities (HTTS Audiometry). All participants received eight Euros for their participation.

Experimental group). \*Represents significant differences.

#### Experimental Procedure

Same as experiment I.

#### Real-Time Auditory Feedback Mapping

Real-time auditory feedback was generated using Python (version 2.7) and Csound version 6.0. Sound synthesis was based on a band-limited oscillator bank with lowpass filtering. Knee joint angle and angular velocity are mapped onto pitch and amplitude of the auditory feedback, respectively. During sitting the right angle at the knee joint is regarded 0◦ , and any extension from this point onwards is referred in positive values from this angle. The changes in angles from 0 to 90◦ of full extension is configured from 120 to 300 Hz of frequency change, respectively. Here, amplitude is a function of square of knee angular velocity which is relevant to kinematic energy.

The modulation of real-time auditory feedback was subtle and provided in an over/under-transposition manner. Here as well, the frequency of the auditory feedback was manipulated per repetition, for 15 repetitions. However, the gradient of change was larger i.e., ±2.6 Hz (equivalent to ±1.3◦ change). Here during step down-up the change in frequency was equivalent as a change from 180 Hz (40◦ ) to 167 Hz (34.8◦ ) in the 5th repetition, and then to 182.6 Hz (41.7◦ ) for the 10th repetition, and finally to 167 Hz (34.8◦ ) for the 15th repetition. For instance, in step updown manner 15 repetitions were accounted in three continuous steps: first five repetitions i.e., 1–5 transposition were performed in step-up manner i.e., 40, 41.3, 42.6, 43.9, 45.2◦ . Thereafter, for repetitions 6–10 continuously the direction of transposition was changed in step-down manner i.e., 43.9, 42.6, 41.3, 40, 38.7◦ . Lastly, for the final 11–15 repetitions the transposition was again changed to step-up manner i.e., 40, 41.3, 42.6, 43.9, 45.2◦ . This transposition change was randomized with step down-up approach during the study. For better clarity see Supplementary Files 4, 7.

The application of transposition was counterbalanced across four sub-groups i.e., sub-group I (40◦ : under-over-under, 75◦ : over-under-over), sub-group II (40◦ : over-under-over, 75◦ : overunder-over), sub-group III (40◦ : over-under-over, 75◦ : underover-under), and sub-group IV (40◦ : under-over-under, 75◦ : under-over-under). Therefore, the number of participants was balanced across the conditions and increased to 20 i.e., 5 in each sub-group. A sample for both the real-time and modulated auditory feedback (Supplementary Files 6, 7) have been provided.

#### Kinematic Analysis

Same as experiment I.

#### Statistical Analysis

Like experiment I, in 2 separate analysis absolute and constant errors were compared with control group. Here, the control group from experiment I was utilized. We analyzed Repositioning Error (the dependent measure), by conducting a Group (Experimental/control) × blocks (1-5) × Angles (40/75◦ ) RM-ANOVA with repeated measures on the last two factors. Additionally, data were decomposed for the 4th block, where the frequency was modulated, across four different sub-groups. Here, the data were normalized on an individual level to the real-time non-modulated auditory feedback by subtraction. The four subgroups differed in performance of episodes of transposition i.e., sub-group I (40◦ : under-over-under, 75◦ : over-under-over), subgroup II (40◦ : over-under-over, 75◦ : over-under-over), sub-group III (40◦ : over-under-over, 75◦ : under-over-under), and sub-group IV (40◦ : under-over-under, 75◦ : under-over-under). Here, each episode represented the mean of five subsequent movements.

For the analysis the values for the over-transposition were inverted. Here, analysis of variance was performed on normalized repositioning errors as dependent variable and between subject factor sub-groups (I, II, III, IV) and within subject factor episodes (1–3) and angles (40/75◦ ). Here, each episode represented the mean of five subsequent movements. Post-hoc comparisons were performed using step wise Bonferroni holm corrections.

#### Results

#### Absolute Error

**Figure 3** illustrates the absolute repositioning error in both groups. Significant differences were observed in between blocks [F(4, 132) = 38.3, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.54] and interaction was evident for block<sup>∗</sup> group [F(4,132) = 4.4, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.12]. A post-hoc test confirmed significant differences between the first and second block in the experimental group I (p < 0.001), but not in the control group (p = 0.940). Furthermore, the second (p < 0.001), but not the first (p = 0.30) block differed significantly between groups. After the removal of feedback this effect diminished. Accordingly, both groups performed in block 3 not significantly different than in block 1 (experimental group I: p > 0.999; control group: p > 0.999). None of the other results were significant group [F(1, 33) = 2.0, p = 0.15, ηp <sup>2</sup> = 0.06], angles [F(1, 33) > 0.01, p = 0.970, η<sup>p</sup> <sup>2</sup> < 0.001], angle<sup>∗</sup> group [F(1, 33) = 0.01, p = 0.920, η<sup>p</sup> <sup>2</sup> < 0.001], angle∗block [F(4,132) = 0.3, p = 0.780, η<sup>p</sup> <sup>2</sup> = 0.01], angle∗block<sup>∗</sup> group [F(4,132) = 0.77, p = 0.490, η<sup>p</sup> <sup>2</sup> = 0.02].

#### Constant Error

**Figure 4** illustrates the constant repositioning accuracy in both groups. The repositioning accuracy depended on block [F(4,132) = 14.2, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.3]. Differences between conditions were mainly caused by the auditory feedback in the experimental group I as shown by the interaction block<sup>∗</sup> group [F(4,112) = 4.56, p = 0.002, η<sup>p</sup> <sup>2</sup> = 0.14]. A post-hoc test confirmed significant differences between the first and second block in the experimental group I (p = 0.003), but not in the control group (p = 0.730). Furthermore, the second (p = 0.001), but not the first (p > 0.999) block differed significantly between groups. After the removal of feedback this effect diminished. Accordingly, both groups performed in block 3 not significantly different than in block 1 (experimental group I: p > 0.999; control group: p > 0.999). In the fourth block, subliminal modulation in frequency of real-time feedback were introduced. We observed no significant differences in the 4th block of experimental group II (p = 0.220), control group (p = 0.770) as compared to the 3rd block. This difference was however, significant when compared to the control group (p = 0.010). Likewise, both groups performance in 5th block did not significantly different than in block 1, and 3 (experimental group II: p > 0.999; control group: p > 0.999). Significant differences were not evident when modulated feedback in 4th block was compared with un-modulated feedback in the 2nd block (p > 0.999). Differences were significant in between the angles [F(1, 33) = 19.6, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.37] i.e., constant errors were larger for 40◦ as compared to 75◦ and for angle<sup>∗</sup> group; [F(1, 33) = 14.5, p = 0.001, η<sup>p</sup> <sup>2</sup> = 0.31], but not for group [F(1, 33) < 0.01, p = 0.990, η<sup>p</sup> <sup>2</sup> < 0.01], angle∗block [F(4,132) = 0.6, p = 0.650, ηp <sup>2</sup> = 0.02], angle∗block<sup>∗</sup> group [F(4,132) = 0.89, p = 0.470, ηp <sup>2</sup> = 0.03].

#### Transposition Condition

For specifying the effect of transposition, we decomposed the data from the 4th block. We computed constant errors

separately for every five repetitions with transposition in the same directions. Each episode began with either over-under-over or under-over-under transposition. **Figure 5** shows the constant errors separately for participants with different episodes. Here, four sub-groups were distinguished with five participants each i.e., sub-group I performed for (40◦ : under-over-under, 75◦ : over-under-over), sub-group II (40◦ : over-under-over, 75◦ : overunder-over), sub-group III (40◦ : over-under-over, 75◦ : underover-under), and sub-group IV (40◦ : under-over-under, 75◦ : under-over-under). **Figure 5** indicates that the re-positioning performance tended to compensate in the opposite direction in which the auditory feedback was manipulatively directed i.e., the participants knee flexion when the feedback was over transposed and vice versa for the under transposition. For the analysis, the over transposition repositioning errors were multiplied with −1.

The data were normalized for the analysis according to individual real-time auditory feedback performance of each participants. Further, step-up transposition findings were multiplied with −1 to allow the direction of transposition to be similar for all episodes (1–3). The statistical analysis revealed that episodes had no significant effect [Episode: F(3.16) = 1.51, p = 0.414, η<sup>p</sup> <sup>2</sup> = 0.16; angle<sup>∗</sup> episode: F(3.16) = 0.72, p = 0.556, ηp <sup>2</sup> = 0.12; block<sup>∗</sup> episode: F(6.32) = 1.43, p = 0.233, η<sup>p</sup> <sup>2</sup> = 0.22; angle<sup>∗</sup> episode<sup>∗</sup> group: F(6.32) = 1.04, p = 0.420, η<sup>p</sup> <sup>2</sup> = 0.16] indicating that over- and under-transpositions did not differ in their impact. However, the transpositions were more effective in the second compared to the first episode (p = 0.002) as confirmed by post-hoc comparisons to the main effect of episode [F(2, 32) = 7.39, p = 0.002, η<sup>p</sup> <sup>2</sup> = 0.32]. Differences between the first and the third (p = 0.267) or the second and the third episode (p = 0.090) were not significant.

To scrutinize whether the altered mapping between auditory feedback and angle changed the repositioning error we performed t-tests against zero separately for episodes (1–3). The results confirmed significant differences to zero in episode 2 (p < 0.001) and episode 3 (p = 0.029) but not block 1 (p = 0.208).

# DISCUSSION

Results from the current experiment demonstrate beneficial effects of real-time auditory feedback on knee re-positioning accuracy. Significant enhancement in re-positioning accuracy was observed for both absolute (p < 0.001) and constant error (p < 0.01) and both within and across the experimental I and II (For clarity see **Figures 1**–**4**), with real-time auditory feedback. These findings agree with previous literature indicating strong associations between the auditory and motor domains (Foxe, 2009; Butler et al., 2012; Schmitz et al., 2013; Ishikawa et al., 2015), and support the possibility of the auditory-proprioceptive substitution hypothesis raised by Altenmüller et al. (2009), Danna and Velay (2017), and Scholz et al. (2015). In this experiment, the enhancement in re-positioning accuracy with real-time auditory feedback could possibly be associated with the "guidance hypothesis" (Schmidt, 1991; Park et al., 2000). The auditory feedback could have made it easier for the participant to identify the target angles, reduce errors, and re-produce the instructed target angles more precisely. This enhancement in re-producibility of target angles could also be due to high spatio-temporal precision of combined audio-motor domains (Hancock et al., 2013; van Vugt and Tillmann, 2015; Dyer J. et al., 2017), which also might have lowered the somatosensory

mismatch negativity (Butler et al., 2012). These changes were also affirmed by Fujioka et al. (2012a). The authors reported modulations in the functional reorganization of spatio-temporal patterns of neuromagnetic β activity (between auditory and sensorimotor modalities; Fujioka et al., 2012a,b). Moreover, the enhanced activation in multisensory integration sites (such as neocortex, superior colliculi, striatum, and cerebellum) and action observation system (Superior temporal sulcus, BA 44, 45) might have aided in enhancing the saliency of executed movement patterns (Schmitz et al., 2013; Stein et al., 2014; Chabrol et al., 2015).

These enhancements in re-positioning accuracy however, were not as stable. Once the auditory feedback was removed in the third treatment block, the re-positioning errors returned to their initial levels. This lack of retention in re-positioning accuracy might be linked with over dependency of the participants with the concurrent feedback (Schmidt, 1991). Park et al. (2000) reported that the concurrent feedback can make the learners dependent on the feedback for maintaining their performances, possibly by bypassing the important internal correction and/or error detecting mechanisms (Schmidt, 1991). Moreover, the concurrent feedback might also limit a performer's initial movement error's (Winstein and Schmidt, 1990), which are thought to represent internal variability of the motor system and are considered as essential for the learning process (see dynamic system theory; Clark and Phillips, 1993). Similarly, the rapid change in knee re-positioning accuracy with substitution of auditory feedback could be affirmed with changes in attentional resources. Recently, Ghai et al. (2016) demonstrated that proprioception is adversely impacted under the influence of higher information processing constrains. However, Hopkins et al. (2017) suggested that cross modal cueing can avoid information overload in the native sensory modality by directing task-irrelevant information toward the underused sensory modality (Hameed et al., 2009). Here as well, the introduction of auditory feedback could have possibly allowed enhancements in re-positioning accuracy by transferring excess information in the sister domain (Lohnes and Earhart, 2011; Ghai et al., 2017b).

Furthermore, we analyzed modulations in knee repositioning performance with modulations in frequency of the auditory feedback. We confirmed with a self-reported questionnaire that participants were not able to consciously perceive any differences introduced in the frequency of the auditory feedback in both group I and II. However, our results demonstrate that these modulations were dependent on the magnitude of modulation introduced in the frequency. In experiment group I, the stepwise modulations were produced in a step-down transposition by 0.5 Hz/repetition (0.25◦ or 0.2%/rep). Although a trend toward step-wise modulation was observed for some individual participants, possibly due to their different inherent auditory perceptual capabilities (Kagerer et al., 2014), these differences could not be proven statistically (p > 0.05), when compared with real-time auditory feedback condition. Thereafter, upon deliberate examination in multiple pilot trials, a step-wise modulation by 2.6 Hz/repetition (1.3◦ or 1.1%/rep) was identified and included. The step-wise modulation was performed in three steps, across both the directions i.e., under, over, under transposition across 15 repetitions and vice versa. The direction was changed after five repetitions to avoid conscious perceptions i.e., five repetitions accounted for 6.5◦ change in one direction, and 19.5◦ overall change 15 repetitions. On the contrary, in experiment I only 3.5◦ change was evitable across 15 repetitions. During the initial analysis, no significant differences in knee repositioning accuracy were observed, possibly due to the negation of directional errors in perceptions across the blocks by step-up/down transposition. Therefore, upon factorial re-analysis of decomposed data for directional changes for knee repositioning, we observed significant effect of modulated auditory feedback as compared to real-time auditory feedback. The participants tried to compensate their knee re-positioning by tending to either extend or flex their knee's more with step-down and step-up transposition in frequency (**Figure 5**), respectively. In our analysis we observed a significant effect of transposition as compared to real-time auditory feedback and demonstrated a combined effect of the transposition to manipulate knee repositioning. As demonstrated in **Figure 6**, the participants could have taken time to adjust their repositioning according to the dynamically transposed auditory feedback, or the significance in the next two episodes might be due to practice effect. Previously, published literature has demonstrated the effectiveness of audio-motor coupling due to subliminal changes in rhythmic auditory feedback (Repp, 2000, 2001; Tecchio et al., 2000; Kagerer et al., 2014). These findings also build up on psychophysical studies demonstrating the cross-sensory impacts of frequency modulation between auditory and motor domains (Foxe, 2009; Butler et al., 2012). We demonstrate that subliminal modulation of frequency can lead to goal-directed changes in knee repositioning. To the best of our knowledge, this study for the first time demonstrates modulation in knee repositioning due to subliminal changes in frequency of real-time auditory feedback. Previously, published literature has only demonstrated this association of audio-motor coupling with subliminal changes in inter stimulus interval for rhythmic auditory feedback (Repp, 2000, 2001; Tecchio et al., 2000; Kagerer et al., 2014).

Finally, building upon the strong correlation suggested for proprioceptive, re-positioning tasks (Vidoni and Boyd, 2009; Van Vugt, 2013), and similar open kinetic chain training regimes in rehabilitation (Tagesson et al., 2008; Fukuda et al., 2013; see review Glass et al., 2010), we believe enhancements observed in this experiment can have a range of practical implications in both rehabilitation and sports settings. Fukuda et al. (2013), for instance reported considerable enhancement in quadriceps, hamstrings strength recovery in patients with ACL reconstruction while performing similar non-weight bearing open kinetic chain movements at the knee joint. Moreover, changes in movement patterns associated with subliminal changes in frequency can also have practical implications. For instance, enhancement in breathing (Murgia et al., 2016), music learning (Hol, 2011; Lahav et al., 2013), arm reaching (Maulucci and Eckhouse, 2001; Schmitz et al., 2014; Scholz et al., 2016), gait (Maulucci and Eckhouse, 2011; Zhang et al., 2013; Mezzarobba et al., 2017), sports (Eriksson and Bresin, 2010; Sigrist et al., 2013), performance with real-time auditory feedback has been demonstrated in a few studies. Here, subliminal modulation in frequency during training can be introduced to enhance variability in movement patterns, which further can lead to a dynamic learning pattern (Stein et al., 2014). Moreover, introduction of subliminal changes can be used to prompt the patient or sports person to exceed their performance parameters without consciously perceiving them i.e., possibly reducing movement re-investment (see Masters and Maxwell, 2008). Future studies can evaluate these aspects of modulation in training paradigms in both sports and rehabilitation settings. Finally, the subjective rating of the compliance of auditory

feedback in the experiment revealed higher rating for the auditory feedback (6.1 ± 1.0) as compared to the control condition (3.5 ± 1.5). A higher compliance with auditory feedback in past has been associated with enhanced motivation, attention and arousal (Menon and Levitin, 2005; Cha et al., 2014). Thereby, possibly supporting the applications of such type of concurrent auditory feedback in rehabilitation settings.

### AUTHOR CONTRIBUTIONS

AE, GS, and SG developed the research question; SG, AE, and GS developed the research paradigm; SG conducted the experiment, collected the data, and wrote main parts of the paper; GS performed the statistical analysis supported by AE; SG contributed to the results section; T-HH was responsible for technical implementing and customization of the sonification system; AE supervised the project. All authors critically revised the paper.

#### REFERENCES


#### FUNDING

The work was supported by EC H2020-FETPROACT-2014 No. 641321.

#### ACKNOWLEDGMENTS

The publication of this article was funded by the Open Access funds of Leibniz Universität Hannover. The co-author would like to thank student assistant Mr. Pascal Moszczynski, Christian Speckelmeyer for their assistance during the experimental procedure.

#### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnins. 2018.00142/full#supplementary-material


improving gait in people with Parkinson's disease. J. Neurol. 263, 1156–1165. doi: 10.1007/s00415-016-8115-2

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Ghai, Schmitz, Hwang and Effenberg. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Effect- and Performance-Based Auditory Feedback on Interpersonal Coordination

Tong-Hun Hwang1,2, Gerd Schmitz <sup>1</sup> , Kevin Klemmt <sup>1</sup> , Lukas Brinkop<sup>1</sup> , Shashank Ghai <sup>1</sup> , Mircea Stoica<sup>3</sup> , Alexander Maye<sup>3</sup> , Holger Blume<sup>2</sup> and Alfred O. Effenberg<sup>1</sup> \*

1 Institute of Sports Science, Leibniz University Hannover, Hannover, Germany, <sup>2</sup> Institute of Microelectronic Systems, Leibniz University Hannover, Hannover, Germany, <sup>3</sup> Department of Neurophysiology and Pathophysiology, University of Hamburg, Hamburg, Germany

When two individuals interact in a collaborative task, such as carrying a sofa or a table, usually spatiotemporal coordination of individual motor behavior will emerge. In many cases, interpersonal coordination can arise independently of verbal communication, based on the observation of the partners' movements and/or the object's movements. In this study, we investigate how social coupling between two individuals can emerge in a collaborative task under different modes of perceptual information. A visual reference condition was compared with three different conditions with new types of additional auditory feedback provided in real time: effect-based auditory feedback, performancebased auditory feedback, and combined effect/performance-based auditory feedback. We have developed a new paradigm in which the actions of both participants continuously result in a seamlessly merged effect on an object simulated by a tablet computer application. Here, participants should temporally synchronize their movements with a 90◦ phase difference and precisely adjust the finger dynamics in order to keep the object (a ball) accurately rotating on a given circular trajectory on the tablet. Results demonstrate that interpersonal coordination in a joint task can be altered by different kinds of additional auditory information in various ways.

Keywords: auditory feedback, collaborative task, interpersonal coordination, movement sonification, sensorimotor contingencies theory

## INTRODUCTION

Researchers have recently focused on different modes of non-verbal communication concerning interpersonal coordination (e.g., mimicry, gestures, and facial expressions) as a basis of social interaction (Vicaria and Dickens, 2016). These kinds of nonverbal behavior can cause spatiotemporal coordination and support affective entrainment between two or more individuals (Phillips-Silver and Keller, 2012). Although it can be helpful to verbally share action plans and strategies, verbal communication might be too slow when one needs to instantly react to others' actions on a joint task (Knoblich and Jordan, 2003). Even in basic communication, concerning mother-child-dyads, it is important that two individuals immediately mediate information to drive entrainment (Phillips-Silver and Keller, 2012). Nonverbal communication can be realized via a broad spectrum of perceptual modalities, like visual, kinesthetic, tactile, or auditory systems, to support emergent coordination (Marsh et al., 2009). For example, Waterhouse et al. (2014) reported that two dancers nonverbally coordinated during their choreography performance. They synchronized the same movements or aligned the onset of different movements, relying on visual cues from their body movement as well as on auditory cues from breath and stepping sounds.

Edited by:

Tiziano A. Agostini, University of Trieste, Italy

#### Reviewed by:

Donatella Di Corrado, Kore University of Enna, Italy Bettina E. Bläsing, Bielefeld University, Germany

#### \*Correspondence:

Alfred O. Effenberg effenberg@sportwiss.uni-hannover.de

#### Specialty section:

This article was submitted to Movement Science and Sport Psychology, a section of the journal Frontiers in Psychology

Received: 15 December 2017 Accepted: 12 March 2018 Published: 29 March 2018

#### Citation:

Hwang T-H, Schmitz G, Klemmt K, Brinkop L, Ghai S, Stoica M, Maye A, Blume H and Effenberg AO (2018) Effect- and Performance-Based Auditory Feedback on Interpersonal Coordination. Front. Psychol. 9:404. doi: 10.3389/fpsyg.2018.00404

Commonly, if the amount of information is enhanced within a certain perceptual modality, interpersonal coordination will benefit from temporal synchronization (Knoblich and Jordan, 2003; Schmidt and Richardson, 2008). This is also given for the auditory domain: Musicians, performing in a joint action setting (e.g., orchestra, musical ensemble), regularly monitor auditory performance of their own, their co-performers' and the joint action outcomes to allow a smooth performance (Loehr et al., 2013). Likewise, Goebl and Palmer (2009) reported that auditory and visual information might function in a complementary fashion to support each other: During a joint action task, pianists produced exaggerated cues for their co-performers by finger movements when the auditory feedback was reduced or removed, which is possibly a compensatory mechanism in the visual domain to align co-performers actions (Repp and Keller, 2004). The important role of the auditory feedback for managing temporal synchrony during interpersonal coordination has been reported repeatedly (Goebl and Palmer, 2009; Demos et al., 2017; Vicary et al., 2017). Demos et al. (2017) compared asynchrony in the tone onset of expert pianists during a recorded and joint performance. The authors reported increased asynchronies once the auditory feedback was removed during the duet performance, confirming strong effects of auditory feedback on temporal synchronization in a joint task.

Demos et al. (2012) compared spontaneous interpersonal coordination under different combinations of auditory and visual information during a rhythmical rocking chair task. The authors reported that instantaneous coordination was enhanced with audio information alone (moving-chair sound, non-task-related music), compared to the condition with neither audio nor vision. In the audio-visual condition, the authors showed that the benefits of moving-chair attendant sound were much higher than in all other conditions, indicating enhanced spontaneous coordination compared to both vision-only and audio-only conditions (see also Schmidt and Richardson, 2008). However, Demos et al. (2012) observed less interpersonal coordination with non-task-related music compared to the moving-chair attendant sound condition, and even to the vision alone condition. Authors indicated that audio-visual feedback does not always lead to a positive effect, but it can cause interference. In an experiment on predictions of opponent's fencing attacks, Allerdissen et al. (2017) also reported that novices showed less performance in the audio-visual condition than in the visual-only condition. Allerdissen et al. (2017) explained that the meaningless additional auditory information might induce cognitive overload. Demos et al. (2012) reasoned that the spontaneous coordination would result from emergent perceptuo-motor couplings in the brain (Kelso, 1995). This can induce co-activation between auditory and motor cortices, so that additional auditory information can enhance synchronization (Bangert et al., 2006; Schmitz and Effenberg, 2017).

Research on additional auditory information related to motion has been reported recently. Vesper et al. (2013), for instance, asked a pair of participants to perform forward jumps next to each other, providing auditory and visual information about the partner's landing positions. Authors showed that the information aided participants to coordinate with each other, supporting both inter- and intra-personal coordination. In a study on audio-based perception of movements, Murgia et al. (2012) showed that participants are able to identify their own golf swing sounds. This study highlights the importance of temporal factors on self-other-discrimination because participants wrongly recognized golf swing sounds from others as their own sounds when the relative timing and the overall duration of movements are similar. On the other hand, a study from Kennel et al. (2014) found no effect of movement rhythm on self-other-discrimination in hurdling performance. The authors concluded that selfother discrimination of movement sounds is achieved by the individuality of sounds that activates one's own sensorimotor memory. They also argued that the larger number of appropriate internal models (e.g., sensorimotor, visual, auditory) enable participants to more accurately reproduce their movements. Furthermore, Keller (2012) suggested that online perceptual information might enhance the anticipation of one's own action as well as the co-performer's action in terms of developing common predictive internal models (Keller and Appel, 2010; Keller, 2012). From a neurophysiological aspect, it was suggested that auditory information possibly allows phase correction through a neural pathway across subsections of the cerebellum, which are connected to motor and auditory cortices (Keller et al., 2014). Periodic correction is, furthermore, enhanced with auditory feedback by additional recruitment of a corticothalamic network which includes the basal ganglia, prefrontal cortex, medial frontal cortex, and parietal cortex (Repp and Su, 2013; Keller et al., 2014).

Furthermore, recent studies have demonstrated the beneficial effects of real-time kinematic auditory feedback for enhancing motor control and learning (Effenberg, 2005; Effenberg et al., 2016). Even though it was in an individual setting, Effenberg et al. (2016) suggested that additional real-time auditory feedback enhances motor learning precisely in terms of a steeper temporal course for the development of motor representations. When mapped onto the kinematic and dynamic movement patterns, the additional real-time movement information might enhance the development of sensorimotor representation below the level of consciousness (Effenberg, 2005; Effenberg et al., 2016). This auditory feedback can be implemented in terms of both effectbased auditory feedback (EAF) and performance-based auditory feedback (PAF). Additional performance-based information provides feedback related to the quality of movement, whereas the effect-based information relays feedback of the result (Magill and Anderson, 2007; Schmidt and Wrisberg, 2008). Both the "knowledge of performance" (KP) and the "knowledge of result" (KR) are important for motor learning (Schmidt and Wrisberg, 2008). Several studies have reported the benefits of performancebased information on learning (Weeks and Kordus, 1998; Nunes et al., 2014; Sharma et al., 2016). Nevertheless, in situations when the feedback of performance is reduced, the impact of effectbased information is usually increased (Winstein, 1991; Schmidt and Wrisberg, 2008; Sharma et al., 2016). These types of feedback have been compared in the context of motor learning. We apply both types of feedback to the cooperative task in our study in order to explore their impact on interpersonal coordination.

In this study, we developed a novel paradigm which we call the tetherball paradigm. The paradigm was implemented on a tablet computer (hereinafter called "tablet") as shown in **Figure 1**. With rhythmical tilt-movements, a pair of participants had to accelerate a bound metal ball to revolve around the center of the scene (**Figure 1**). This task allows the analysis of joint performance by measuring the spatial error between the ball trajectory (controlled by both participants) and the circular target trajectory. Apart from visual information about the performance of both co-actors (the realized tilt in their own and their co-actor's axis) and about its effect (the deviation of the revolving ball from the target trajectory), we added different kinds of acoustic information to the paradigm. The feedback types correspond to the information about the performance. Although the action effect that is usually only available in the visual domain, PAF was generated from the tilt of the axes of the tablet and EAF was generated from the trajectory of the ball. The auditory information was based on the same features as the visual information (performance: tablet tilt; effect: ball trajectory). It may, nevertheless, affect the participants' perception in a different way because the auditory system is especially powerful in the temporal analysis of acoustic events, as well as in pace and rhythm specification and discrimination (Collier and Logan, 2000; Murgia et al., 2017). Furthermore, it is highly effective not only in the assessment of smoothness and regularity, but also in the synchronization and phase couplings and the adjustments of actions to external events (Repp and Penel, 2002). Therefore, we expect a better task performance, a stronger interpersonal

coordination and a higher level of collaboration experience due to the additional involvement of the auditory perceptual system.

We compared three different audio-visual conditions [EAF, PAF, combined EAF and PAF (CAF)] to a visual condition (VF; no audio). For the PAF condition, we used a rhythmical sound which is in line with a recent research by Demos et al. (2012). EAF is a melodic sound (non-rhythmical sound) of integrated dynamics, which is created when two agents' joint actions result in a rotation of the ball. We intended to avoid a rhythmical feature in EAF because this might have allowed participants to identify the effect of their own movement effect within the effect sound. We decided to positively hypothesize according to previous literature (Vesper et al., 2013; Effenberg et al., 2016). In each condition, we evaluated the reduction of the trajectory error as a measure of task performance with on-going training as well as the cross-correlation of two participants' actions as a measure of their temporal synchronization. Participants were also asked to report their subjective experience of the coordination. With respect to these data, the following hypotheses were tested:

H1: Faster error reduction in the task is achieved when participants are provided with additional (a) effect-based, (b) performance-based, and (c) both combined auditory feedback.

H2: Cross-correlation in the participants' actions is stronger when participants are provided with (a) effect-based, (b) performance-based, and (c) both combined auditory feedback.

H3: Subjective ratings of the sense of interpersonal coordination are more positive when participants are additionally provided with (a) effect-based, (b) performance-based, and (c) both combined auditory feedback.

## MATERIALS AND METHODS

#### Participants

We tested 72 healthy participants (30 females and 42 males; 24.8 ± 3.3 years) for normal eyesight and hearing abilities. Thirty-six pairs of participants were divided into four groups, corresponding to the four different conditions, so that each group consisted of nine pairs. Participants were randomly assigned to couples and the only criterion was "same-sex pair." We also instructed them to use the dominant hand. The study was ethically approved by the Ethics Committee of Leibniz University Hannover.

## System Specifications

The paradigm was implemented in Objective-C for iOS 10.2 on an iPad Air (Apple Inc.). Screen resolution was 1,024 × 768 at 60 Hz refresh rate. Accelerometers in the iPad were also sampled at 60 Hz. We used the Csound 6.0 (open-source code under LGPL) and Chipmunk2D Pro (Howling Moon Software) for the auditory feedback and physical implementation, respectively. The participants wore the headphones, Beyerdynamic DT-100. The audio signal was divided by a 4-channel stereo headphone amplifier, Behringer MicroAMP HA400.

# Design and Stimuli

**Figure 1** shows the main screen of the tablet application. The main components are the ball that is connected to the center by an invisible spring, the circular target trajectory continuously displayed on the tablet screen, and the levers fixed on both sides of the tablet. The tablet displays the components at XGA resolution, in which the ball radius is 30 pixels (px) and the radius of the target circle is 232.5 px (thickness: 15 px). The ball position refers to the center of the ball, expressed in x-y Cartesian coordinates. The ball is connected to the center anchor with an invisible elastic spring. The spring force is strong enough to pull the ball to the center when the tablet is flat. Participants have to tilt the tablet to rotate the ball around the center. Each participant controls only one axis, either x or y, by moving the index finger up and down. The lever on the x-axis is longer in order to compensate the different edge lengths of the tablet. The tablet is limited to two degrees of freedom (DOF) and prevents any rotation (see **Figure 2**). The task for the participants is to rotate the ball around the center while following the circular target trajectory as precisely as possible. The ball's circular movement can be realized when both axes of the tablet are tilted in a certain pattern and with a certain amplitude of frequency. Optimal performance is achievable with synchronization of the finger movements with a 90◦ phase difference (see Video 1 in the Supplementary Material).

**Figure 2** shows the side view of the experimental setup. Participants tilt the tablet up and down through the levers that are attached to the casing. The tablet is supported by a universal joint that allows rotations on the x- and y- axis (roll and pitch), but prevents rotations around the z-axis (yaw). To avoid hand movements other than up-and-down movements of the index finger, participants were asked to hold the handle that was fixed to a wooden frame which is shown in **Figure 2**. Participants can comfortably rest their elbows on the layer 2 of the wooden frame.

**Figure 3** shows the top view of the tetherball paradigm including the wooden frame. Participants sit to control the tablet by using their dominant hand. Right-handed (RH) participants sit on the left of a wooden frame's wing and left-handed (LH) participants sit on the right of a wing. The participants stay on their seats during the whole task and do not swap position. The handles can be adjusted to the dominant hand and to the hand size of each participant. Participants can see the screen from nearly the same distance, which establishes the same condition for visual feedback. They wear headphones for auditory feedback. The audio output of the tablet is connected to an audio splitter, and the participants hear the same sound at the same time. They hear their own and their partner's auditory feedback.

**Figure 4** shows the perceptual information flows including visual, auditory, tactile, and kinesthetic information. Effect-based visual (VF), EAF, and PAF are digitally treated as experimental variables, whereas the kinesthetic, tactile, and visual feedback of finger movements are independent variables in this paradigm. The ball moving through the scene constitutes VF. Effectbased auditory feedback is driven by the position of the ball, which is congruent to VF. For EAF, "synthesized violin" is used to create continuous string instrument sound so that it is appropriate to sonify the ball's continuous movement pattern. Two distinguishable violin sounds can also be converted from two spatial parameters, the x- and y-position. The sound is, furthermore, familiar to human ears because it can mimic the human voice in terms of range of spectrum and vibration, wherein participants can hear the sound for a relatively long time. To be specific, EAF is represented by pitch and amplitude of the sound. The pitch of the sound corresponds to the x- and yposition, whereas the amplitude depends on the ball's velocity. Depending on the ball position on the tablet's screen, the base audio frequency is modified from 250 to 427 Hz along the x axis and from 600 to 835 Hz along the y axis.

Performance-based auditory feedback represents the angular velocity of the tablet measured by the built-in gyroscope. When the tablet is tilted, the resulting angular velocity affects PAF as additional auditory feedback about the participant's actions—convergent with their kinesthetic finger perception. The sound of PAF is created by a noise generator with a band-pass filter, which is a "broom sweeping sound." We decided to use this sound because it is suitable to express accelerating up-and-down finger movements of participants. Spectra of both tilt sounds are easily distinguished because they were located within different frequency bands. This timbre

FIGURE 3 | Illustrations of (V1) top view of the task apparatus and the seat plan for right-handed (RH) and left-handed (LH) participants (P1 and P2), as well as (V2) positions of two right-handed participants and a tester during the experiment.

is closely related to natural sounds so that participants can hear it comfortably during the task. The PAF sound also allows the participant to clearly distinguish it from EAF in the CAF condition. Higher velocity of finger movements generates a higher amplitude and frequency of the PAF sound. Depending on the centrifugal force from accelerometer data, the base frequencies (f<sup>b</sup> ) are 700–1,700 Hz for lever 1's tilt and 100–1,100 Hz for lever 2's tilt, respectively. We obtained the sound from the white noise after using the band-pass filter (cutoff frequency: f<sup>b</sup> ±25 Hz). Together, the auditory feedback generates rhythmical sounds corresponding to the periodic finger movements with altering velocities and short phases of silence at the turning points. Besides these two types of augmented auditory feedback (PAF, EAF), participants also had natural kinesthetic, tactile and visual feedback to solve the experimental task. A sample video of the tetherball paradigm with additional auditory feedbacks is provided in Video 1 (in the Supplementary Material).

#### Procedure

Before the experiment, participants were asked to complete a questionnaire regarding their personal backgrounds including previous experiences in music and sports settings. Two pre-tests were administered to confirm that the participants have a normal range of eyesight and hearing abilities, which were tested with the Landolt rings chart (Jochen Meyer–Hilberg) and HTTS audio test (SAX GmbH). The third pre-test was carried out to classify participants depending on their ability to handle the ball on the screen, which might decide their performance in the pretest shown in **Figure 5**. Participants have to keep the balls on randomly moving targets. Each participant handled a separated ball moving along the corresponding axis.

The participants performed the visuo-motor pre-test for 2 minutes. For each participant, the mean absolute error (the distance between the target and ball position) was measured during the last 30 s. The performance of this task and participants' gender was used for parallelization between groups: The first four pairs were randomly assigned to four groups. The visual group (VFG) received VF without auditory feedback as a reference condition. The EAF group (EAFG), the PAF group (PAFG), and the CAF group (CAFG) additionally received EAF, PAF, and CAF, respectively. All groups also received natural kinesthetic, tactile, and visual feedback which was not modified in the experiment. Group assignment of all other pairs considered their mean error in the pre-test. Thereby, it was possible to compose four groups with nearly the same level and without statistically different visuo-motor pre-test performances [VFG: 75 ± 23 px, EAFG: 72

± 29 px, PAFG: 70 ± 20 px, CAFG: 78 ± 16 px; F(3, 36) = 0.24, p = 0.872, η<sup>p</sup> <sup>2</sup> = 0.02].

Couples of participants performed 15 trials of 1 min each. After every five trials, a 2-min break was administered, resulting in three sets. During the trial, participants abstained from talking and discussing about possible strategies, so that they could focus on the task. Participants were also instructed to initiate the revolving of the ball in clockwise direction (CW). After the experiment, the participants were asked to answer the second questionnaire that assessed subjective ratings of participants' experience in terms of interpersonal coordination at solving the task. The questionnaire consisted of four questions subjectively evaluating their personal, their partners', and the joint performance during the experiment.

#### Data Analysis

The tablet recorded the path of the ball (from screen) and the angular velocity (from gyroscope) at the sampling rate of 60 samples per second. For statistical analysis, absolute tracking errors as well as mean peak values from the cross correlations were submitted to three-way analyses of variance with a betweensubject factor Group (VFG vs. EAFG, VFG vs. PAFG, VFG vs. CAFG) and the within-subject factors Set (I–III) and five Trials in each set. The sphericity assumption was tested with the Mauchley's test, and in case of significance, ANOVAs were adjusted according to the Huynh–Feldt procedure. Levene's test was applied to analyze homogeneity of variances. Posthoc comparisons were performed with Tukey's post-hoc tests. Subjective ratings of interpersonal coordination were compared across groups with Mann–Whitney-U-Tests and within groups with a Wilcoxon test. The overall significance level was set to 5%.

#### RESULTS

Sport-, music-, and computer-game-expertise, as well as pretest performance were taken into account because they could influence performance in the tetherball paradigm. Comparing these variables of groups with those of the VFG, we found no significant differences in these variables except for sport expertise between the VFG and PAFG [F(1, 16) = 6.38, p = 0.022, η<sup>p</sup> 2 = 0.29]. Therefore, we considered sport specific expertise as a possible covariate in the subsequent analyses.

The performance was measured by the absolute error between the radius of the target circular trajectory and the ball's trajectory. An average value of the absolute error during a 1-min trial was calculated; however, data of the first 8.3 s (500 samples at 60 Hz) in every 1-min trial were omitted because the circling ball's movement had to be initiated. With the average absolute error, we calculated across subject means and standard deviations for each trial and in each group (**Figure 6**).

The mean absolute errors of four groups are shown in **Figure 6**. Comparing the results of VFG and EAFG across trials, the absolute error decreased significantly from Set I to Set II and Set III as confirmed by the significant effect set [F(2, 30) = 3.95, p = 0.043, η 2 <sup>p</sup> = 0.21] and significant differences between Set I to Set II (p < 0.001) as well as Set I to Set III (p < 0.001) in the post-hoc test. Furthermore, within each set, the error decreased

the CAFG. Illustrated are between-subject means and standard deviations. The first 8.3 s (500 samples) in every 1-min trial was eliminated.

from Trial 1 to 5 [F(4, 60) = 4.58, p = 0.005, η<sup>p</sup> <sup>2</sup> = 0.23]. A posthoc comparison confirmed significant differences between Trial 1 and all the other trials (each p < 0.001) and between Trial 2–4 and 5 (both p < 0.01). For the error reduction across trials, sport specific expertise was the significant covariate [F(4, 60) = 3.84, p = 0.013, η<sup>p</sup> <sup>2</sup> = 0.20].

The error reduction differed between groups as confirmed by the three-way interaction Set∗Trial∗Group [F(8, 120) = 2.63, p = 0.030, η<sup>p</sup> <sup>2</sup> = 0.15]. The participants in EAFG predominantly increased their performance within the first four trials and then reached a stable plateau. Accordingly, a post-hoc test showed significant differences from the first three trials to the last trial of the task (at least p < 0.05), but no significant differences from Trial 4 onwards (all p > 0.05). The error of the VFG reached a plateau at the same level as that of the EAFG but at a later trial. Thus, the post-hoc test confirmed significant differences between the first six trials (Trial 1–6) and the last three trials (Trial 13–15) in the task (at least p < 0.05). Levene's test revealed that variances differed significantly between groups in Trials 4–8 and Trial 12 (at least p < 0.05).

In contrast to the EAF, the PAFG did not show a significant difference in performance, compared to the VFG. A comparison of the absolute error with the VFG neither resulted in significant group differences nor interactions. Across groups, however, became significant in terms of the main effects, set [F(2, 32) = 56.66, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.78] and trial [F(4, 64) = 40.81, p < 0.001, ηp <sup>2</sup> = 0.72] as well as their interaction [F(4, 64) = 10.19, p < 0.001, ηp <sup>2</sup> = 0.39]. A post-hoc test to the latter interaction confirmed significant differences from Trial 1–2 to Trial 3–5 in Set I (at least p < 0.05), significant differences from Trial 6 to Trial 9–10 in Set II (at least p < 0.05), and no significant difference between the trials in Set III (all p > 0.05). This indicated that the performance increased predominantly in Set I and reached a plateau in Set III. The Levene's test was not significant in any the other trials.

An ANOVA for VFG and CAFG yielded the same overall effects as the other ANOVAs [Set: F(2, 32) = 67.26, p < 0.001, ηp <sup>2</sup> = 0.81; Trial: F(4, 64) = 35.76, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.69] as well as a significant interaction in Set∗Trial [F(4, 64) = 10.56, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.40]. Furthermore, the CAF had a significant effect on the progress of error reduction, which is confirmed by significant interactions in Trial∗Group [F(4, 64) = 3.70, p = 0.021, η<sup>p</sup> <sup>2</sup> = 0.19]. Here, a post-hoc test confirmed that the CAF allowed the participants to further increase their performance from the second last to the last trial (p = 0.02). This was not the case in VFG (p > 0.05). Furthermore, the significant three-way interaction in Set∗Trial∗Group [F(8, 128) = 2.45, p = 0.031, η<sup>p</sup> 2 = 0.13] indicated that the error reduction progressed differently between groups. In CAFG, the performance reached a plateau earlier than in VFG. According to Tukey's post-hoc test, the first five trials (Trial 1–5) in CAFG differed significantly from the last trial (all p < 0.001). In VFG, the first six trials (Trial 1–6) differed significantly from the last trial (all p < 0.001). Levene's test was not significant in any of the trials.

Regarding the level of temporal synchronization, we calculated the cross correlation between the angular velocities of a pair of participants' up-and-down finger movements, which is applied to all other pairs (**Figure 7**). Cross-correlation was calculated with 1,000 samples, and then this was divided into three periods in each 1-min trial (3,610 samples, 60.2 s). A calculation of the cross-correlation resulted in coefficients along with lags (n = ±50). Parts of coefficients were considered, especially when the lags were between 8 and 15 samples. These

values were empirically determined as a standard, regarding quarter-phase synchronization. To decide the optimal lag values, we selected the best 12 pairs (three pairs per group) who achieved the lowest average error of ball trajectory during the last five trials (Trial 11–15). We measured an average time difference equivalent to a 90◦ phase difference between a pair of participants' angular velocities. The time difference was 194.2 ± 75.5 ms corresponding to 11.65 (±4.53) samples of the lag. According the calculations, the highest coefficient was extracted between 133.3 ms (n = 8) to 250.0 ms (n = 15). Then, we had three coefficients (n = 500–1,500, 1,500–2,500, and 2,500–3,500) in every 1-min trial (n = 3,610) for every pair. The largest coefficient in a 1-min trial was regarded as a representative value for the trial. This allowed us to record the best performance of pairs in each trial. This is because we can avoid the average effect of participant's mistakes. The first 8.3 s (n = 500) were eliminated, because it was before the ball was released. From these three sections, the maximum coefficient for a single trial was selected. According to across subject means and standard deviations of the coefficients shown in **Figure 7**, the correlations improved over time. This was statistically confirmed by significance of the factor "set" in the ANOVAs which analyzed the data of the VFG and audio-visual groups [VFG & EAFG: F(2, 32) = 26.81, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.63; VFG & PAFG: F(2, 32) = 21.17, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.57; VFG & CAFG: F(2, 32) = 26.82, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.63] as well as the significant effects of trial [VFG & EAFG: F(4, 64) = 5.49, p = 0.003, η<sup>p</sup> <sup>2</sup> = 0.26; VFG & PAFG: F(4, 64) = 5.48, p < 0.001, <sup>p</sup> <sup>2</sup> = 0.26; VFG & CAFG:

F(4, 64) = 8.68, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.35]. The improvement of cross-correlation changed over time as shown by the significant interactions Set∗Trial in these groups [VFG & EAFG: F(8, 128) = 5.25, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.25; VFG & PAFG: F(8, 128) = 4.68, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.23; VFG & CAFG: F(8, 128) = 6.20, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.28]. Furthermore, cross correlations increased significantly faster with CAF than without auditory feedback (VF). This is confirmed by the significance of the three-way interaction Set∗Trial∗Group [F(8, 128) = 2.53, p = 0.014, η<sup>p</sup> <sup>2</sup> = 0.14]. Accordingly, a Tukey's post-hoc test results in significant differences between the first three trials and the last trial within VFG (each p at least <0.05), whereas in CAFG only the first two trials differed significantly from the last (each p<0.05).

Results of the questionnaire are shown in box and whisker plots in **Figure 8**. Participants were asked to choose an integer between 1 (not at all) and 7 (very much), when answering the first question "How much did you feel your movement helps the collaborator's performance?" All participants answered without significant difference between VFG and audio-visual groups (EAFG, PAFG, CAFG) according to Mann–Whitney U-tests (VFG vs. EAFG: U = 150.0, p = 0.719; VFG vs. PAFG: U = 161.0, p = 0.988; VFG vs. CAFG: U = 120.0, p = 0.192). Participants normally scored between 4 and 7. Medians of all groups were between 5 and 6. The second question "How much did you feel the collaborator's movement helps your performance?" also resulted in no significant differences (VFG vs. EAFG: U = 159.5, p = 0.938; VFG vs. PAFG: U = 136.0, p = 0.424; VFG vs. CAFG: U = 159.0, p = 0.938). However, in the third question "How did

you experience the collaboration with your partner?" participants were asked to mark from 1 (unpleasant) to 7 (very pleasant). The ratings audio-visual groups showed significant differences to the VF group (VFG vs. EAFG: U = 66.0, p = 0.002; VFG vs. PAFG: U = 60.0, p = 0.001; VFG vs. CAFG: U = 90.5, p = 0.022). The fourth question "How effectively did you feel that you managed to do the task?" was asked to be marked from 1 (not effectively at all) to 7 (very effectively) for their feeling at the beginning and at the end of experiment. Results of rating by EAFG showed a higher median value at the initial time than VFG, and there was a tendency of difference between VFG and EAFG (U = 103.0, p = 0.064). However, neither this nor other differences between groups were significant (VFG vs. PAFG: U = 151.0, p = 0.743; VFG vs. CAFG: U = 156.0, p = 0.864). In comparison to the beginning of the experiment, participants felt that they managed the task more effectively at the end as shown by a significant effect in the Wilcoxon-Test (z = −7.28, p < 0.001). Noteworthy, the progress from the initial time to the end, calculated as pre-post difference, was not significantly different between groups (VFG vs. EAFG: U = 112, p = 0.118; VFG vs. PAFG: U = 156, p = 0.864; VFG vs. CAFG: U = 128, p = 0.293).

## DISCUSSION

In the tetherball paradigm, participant pairs were asked to tilt the tablet together for the task. We compared three different audio-visual conditions with the visual condition in terms of error reduction, cross correlation and subjective ratings in a selfreport questionnaire. Results demonstrate that error reduction was faster with EAF and CAF than the visual condition; however, no statistical difference was observed with PAF. This confirms H1(a) and H1(c), but not H1(b). Regarding H2, only H2(c) is supported by our results, because only CAF showed a significant effect on the cross correlation between participants compared to the visual condition. In terms of H3, participants hardly perceived that their actions affected their partner's action and vice versa. Nonetheless, participants with auditory feedback felt more pleased in the collaborative task than those without auditory feedback. Across groups, participants felt progress in collaboration; however, differences between visual and audiovisual groups were not significant. Therefore, H3 can be partially confirmed in terms of pleasant feeling during the task by the present study.

The task required the participants to predict their partner's actions as well as the combined effect of their joint actions. Our results suggest that real-time audio-visual feedback improved performance. According to Stein and Stanford (2008), perception can be usually enhanced if visual and auditory information are integrated within multisensory areas of the central nervous system (CNS). This might enhance participants' understanding of their own and their partner's actions as well as joint actions, which positively affects interpersonal coordination. In addition, previously published literature (Schmidt and Richardson, 2008; Keller et al., 2014; Lang et al., 2016; Loehr and Vesper, 2016) highlights the significance of rhythmical movement components in interpersonal coordination. Additionally, there is evidence that the rhythmic component during interpersonal coordination reduces practice effort and errors (Lang et al., 2016; Loehr and Vesper, 2016). When rhythmical information of the movement is shared between two or more individuals by visual or auditory cues, usually spatiotemporal entrainment is supported by the same dynamical principles of the movement (Knoblich et al., 2011; Phillips-Silver and Keller, 2012). According to Schmidt and Richardson (2008), moreover, additional perceptual information can increase the level of action coupling, possibly enabling coactors to align their actions. In our setting, EAF contained nonrhythmical sound; however, it provided a temporally structured melody. This sound could have aided the participants to predict the ball dynamics, to estimate the achieved precision, and to adapt further actions. Furthermore, after reaching the plateau level of performance until the end of the task, the absolute error in EAFG showed significantly lower standard deviations than VFG. This might indicate that participants maintained interpersonal coordination more consistently after establishing a task-specific audio-visual-motor network in the brain.

However, PAF alone caused no significant effect on error reduction and cross correlation. This result indicates different effects of various types of auditory information on interpersonal coordination. A plausible explanation is the integration of auditory information with perceptual information of other modalities in terms of multisensory integration. For example, Allerdissen et al. (2017) reported that fencing experts showed nearly the same pattern of results in both audio-visual and visual conditions. A similar suggestion had been made by Demos et al. (2012): The authors showed that the level of coordination can be enhanced by audio-visual information, but can be reduced by non-task-related auditory stimuli like music. In our setting, the characteristics of the chosen sounds may also have influenced the results. The EAF sound ("synthesized violin") was a more dominant auditory cue than the PAF sound because it was a continuous sound with high pitch and bright timbre. If we used other sounds similar to the "synthesized violin" of the EAF condition, PAF could have enhanced interpersonal coordination. Of course, not only the chosen timbre of the sound can change the way it is perceived, but also the determined level of volume as well as masking effects between both sounds. We nevertheless, tried to find well balanced compositions where both sounds were equally perceivable well. Finally, PAF had neither positive nor negative effects on interpersonal performance compared to the visual group (VFG) in our study.

As CAF, we used PAF and EAF together to investigate the effect when more types of additional auditory information were applied additionally to VF, expecting enhanced performance without the need of conscious attention (see Effenberg et al., 2016). Interpersonal coordination was significantly affected by CAF in terms of enhanced joint performance and temporal synchronization. The effect on joint performance can be explained by the presence of EAF because PAF did not show an effect. The effect on temporal synchronization, nevertheless, might be supported by the combination of EAF and PAF. Although PAF alone does not affect interpersonal coordination, it seems there is a synergy between PAF and EAF.

Our results suggest that additional auditory feedback can make collaboration easier and more pleasant. Effect-based auditory feedback can increase motivation for the task because participants in audio-visual groups reported that they felt more pleased during interpersonal coordination. Most interestingly, PAF also resulted in a similar pleasing effect. Demos et al. (2012), for instance, reported that music irrelevant to vision and movement made participants feel connected with their partners. This might suggest that the pleasant feelings are rather related to the auditory task component than to task performance. For future research, it might be interesting to investigate whether participants feel pleased during the task with non-task-related auditory feedback (rhythmical, non-rhythmical). This would be in line with a study of Phillips-Silver and Keller (2012) on affective entrainment when the authors investigated the relations between the task-relatedness of a sound and the pleasantness of the participants' feelings in the synchronization with others.

In future, auditory feedback might be applied to facilitate interactions between humans and machines. Humans possess an ecological acoustic-motion mapping background based on every-day experiences (Carello et al., 2005): For example, when driving a car, the engine sound correlates with its speed. Such movement sounds like a washing machine, a vacuum cleaner, and a printer might be regarded as performance-based feedback. Other examples suggest that many humans are also experienced with ecological or artificial effect-based auditory feedback: A modern car provides the driver with artificial auditory feedback about the distance to objects during parking, and a radar sonifies the distance and velocity of approaching objects. In these scenarios, machines mediate information via audition to humans. As the present study represents a first step in the case of human-human interaction, future studies might investigate which sounds support human-machine interactions best. The adequate choice of an appropriate auditory coding of physical performance and events is important. As already stated, out results suggest that certain kinds of human-human interaction benefit from effect-based auditory information, at least, if the common goal is already known. In the case of humanoid human-robot interaction scenarios it might not be possible to predict joint effects as long as referenced actions have not been experienced before. For such underdetermined, novel interaction scenarios it might be useful to apply a performancebased acoustics in a first step. Although we did not find benefits of exclusively performance-based auditory information in our study, humanoid robot-human interactive settings might benefit from additional performance-based kinematic real-time acoustics: With reference to Schmitz et al. (2013), auditory information about humanoid robotic movements might be suitable to address biological motion perception mechanisms in the human brain, if configured adequately. Biological motion perception mechanisms are usually not addressed by artificial agents with non-human motions.

# CONCLUSION

Additional artificial auditory information can be synthesized in many different ways for interpersonal coordination. In this study, we referred to the feedback research in the motor domain with a basic reference to the both categories of "knowledge of performance" (KP) and "knowledge of result" (KR), wellestablished in motor learning research. In future, it might be interesting to investigate relationships between sounds and movements in various situations with more difficult levels of joint tasks with long-term period (e.g., shape-changing trajectory). An important aspect of further research is how motor learning and the emergence of interpersonal coordination are related to each other. Undoubtedly both are referring closely to the perception of kinematics—mainly dedicated to human movements or to the referenced object's movements (e.g., a sofa, a tetherball). To support the perception of kinematics might be a key issue on many places in future related to the support of individual behavior as well as of interpersonal coordination. Nevertheless, it is a challenging approach—related to motor learning and to interpersonal coordination.

## AUTHOR CONTRIBUTIONS

T-HH together with AE and GS developed the paradigm and the experimental design. T-HH realized the software development supported by HB. T-HH organized the database and wrote the technical parts of the paper. AE and SG wrote main parts of

#### REFERENCES


the behavioral sections of the paper. AE and GS supervised the data collection. KK and LB performed the experiment. Statistical analysis and major parts of the results were realized by GS, supported by T-HH. The software development of the visuomotor pre-test as well as the development of the mechanical parts of the apparatus were realized by MS and AM. All authors critically revised the manuscript.

#### FUNDING

The authors acknowledge support by European Commission HORIZON2020-FETPROACT-2014 No. 641321.

#### ACKNOWLEDGMENTS

The publication of this article was funded by the Open Access fund of Leibniz Universität Hannover.

#### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg. 2018.00404/full#supplementary-material


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Hwang, Schmitz, Klemmt, Brinkop, Ghai, Stoica, Maye, Blume and Effenberg. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# In dubio pro silentio – Even Loud Music Does Not Facilitate Strenuous Ergometer Exercise

Gunter Kreutz<sup>1</sup> \*, Jörg Schorer<sup>2</sup> , Dominik Sojke<sup>1</sup> , Judith Neugebauer<sup>2</sup> and Antje Bullack<sup>1</sup>

<sup>1</sup> Department of Music, Carl von Ossietzky University of Oldenburg, Oldenburg, Germany, <sup>2</sup> Institute of Sport Science, Carl von Ossietzky University of Oldenburg, Oldenburg, Germany

Background: Music listening is wide-spread in amateur sports. Ergometer exercise is one such activity which is often performed with loud music.

Aim and Hypotheses: We investigated the effects of electronic music at different intensity levels on ergometer performance (physical performance, force on the pedal, pedaling frequency), perceived fatigue and heart rate in healthy adults. We assumed that higher sound intensity levels are associated with greater ergometer performance and less perceived effort, particularly for untrained individuals.

#### Edited by:

Penny McCullagh, California State University, East Bay, United States

#### Reviewed by:

Fabrizio Sors, University of Trieste, Italy Alfred Oliver Effenberg, Leibniz University of Hanover, Germany

\*Correspondence: Gunter Kreutz gunter.kreutz@uni-oldenburg.de

#### Specialty section:

This article was submitted to Movement Science and Sport Psychology, a section of the journal Frontiers in Psychology

Received: 12 December 2017 Accepted: 09 April 2018 Published: 07 May 2018

#### Citation:

Kreutz G, Schorer J, Sojke D, Neugebauer J and Bullack A (2018) In dubio pro silentio – Even Loud Music Does Not Facilitate Strenuous Ergometer Exercise. Front. Psychol. 9:590. doi: 10.3389/fpsyg.2018.00590 Methods: Groups of high trained and low trained healthy males (N = 40; age = 25.25 years; SD = 3.89 years) were tested individually on an ergometer while electronic dance music was played at 0, 65, 75, and 85 dB. Participants assessed their music experience during the experiment.

Results: Majorities of participants rated the music as not too loud (65%), motivating (77.50%), appropriate for this sports exercise (90%), and having the right tempo (67.50%). Participants noticed changes in the acoustical environment with increasing intensity levels, but no further effects on any of the physical or other subjective measures were found for neither of the groups. Therefore, the main hypothesis must be rejected.

Discussion: These findings suggest that high loudness levels do not positively influence ergometer performance. The high acceptance of loud music and perceived appropriateness could be based on erroneous beliefs or stereotypes. Reasons for the widespread use of loud music in fitness sports needs further investigation. Reducing loudness during fitness exercise may not compromise physical performance or perceived effort.

Keywords: music listening, ergometer, loudness, perceived effort, hearing prevention

# INTRODUCTION

Music listening during every-day activities is a global phenomenon in present-day leisure and sports cultures (Kurmaeva, 2011). Background music appears to play an ambiguous role as a distractor that can interfere with cognitive tasks (e.g., Cho, 2015) or enhance physical performance (e.g., Copeland and Franks, 1991). Therefore, the overall effectiveness of background music in mediating psychological processes has been questioned (Behne, 1999), pointing to the importance of psychological attributions such as liking or preference (e.g., Stratton and Zalanowski, 1984;

Kreutz et al., 2007; Zatorre and Salimpoor, 2013) and prior exposure (Crust, 2004) on the one hand, and musical features such as tempo, sound intensity, and loudness (e.g., Copeland and Franks, 1991; Waterhouse et al., 2010; Thompson et al., 2012; Metcalfe, 2016) on the other. Here, we investigate some of these issues in the context of physical ergometer exercise, in which participants were exposed to background music of varying sound intensity levels.

Audio-based interventions have become a much debated topic in sport science approaches to enhance performance in a variety of domains (Sors et al., 2015). Specifically, auditory action–perception coupling as part of more general research on the role of natural movement sounds in sports has been studied across various sports domains including basketball (Camponogara et al., 2017), fencing (Allerdissen et al., 2017), elite rowing (Schaffert et al., 2011), ball sports (Sors et al., 2017, 2018), and tennis (Cañal-Bruland et al., 2018). For example, auditory information can improve fencers' prediction of attack movements (Allerdissen et al., 2017), enhance the performance in hammer throwing (Agostini et al., 2004), or facilitate long-term storage of individual movement patterns in hurdling (Pizzera et al., 2017).

Researchers have pointed out the importance of self-generated movement sounds in action–perception coupling in the sense that such sound cues can help to discriminate between one's own movements and movements from other sources (Murgia et al., 2012; Kennel et al., 2014a,b). Note that loudness is one auditory attribute that seems of particular relevance in sports as greater loudness can improve reaction times (Brown et al., 2008), or influence referees' judgments in team sport games (Unkelbach and Memmert, 2010).

Kämpfe et al. (2011) conducted a review and meta-analysis of the psychological and behavioral effects of background music across a wide range of cognitive and physical tasks. Generally, the hypothesis of a modulating effect of background music was not confirmed. However, background music in sports was one of few domains showing a small but positive impact on performance. By contrast, Brooks and Kristal (2010) concluded from their review of studies on music listening exclusively in the field of sports that the evidence of motivational effects of music listening in sports was mixed. This means that music can also be perceived as disturbing or interfering with sports activities. Therefore, the perceived appropriateness and objective effectiveness of music listening in sports activities can be modulated by a range of variables. Moreover, the individual level of training status can also influence the psychological effects of music listening during sports exercises (Baldari et al., 2010).

Studies showing that music listening may have motivating effects to enhance physical performance and reduce perceived effort have focused on individual sports (Karageorghis et al., 2006; Terry et al., 2012). For example, Karageorghis et al. (2008) found that music listening in running could be motivating, but self-selection, preference, and tempo were important moderating variables. Waterhouse et al. (2010) showed that manipulating the musical tempo during ergometer cycling also modulated performance in the sense that increasing the tempo led to greater distances covered and more positive affective experience. Barwood et al. (2009) found that music could distract gym users from bodily perceptions and provide motivation to enhance performance during motorized treadmill exercise. These authors showed that runners covered significantly more distance in a motivational music (and video) condition as compared to non-motivational and control conditions.

Most study designs entail participant's exposure to recorded or live music. Fritz et al. (2013), however, tested a novel music agency concept, in which fitness devices were equipped with sound processing software such that movement of the devices during exercise controlled the production of synthesized sound and thus provides a music feedback. The authors compared psychophysiological responses to the music listening with feedback versus music listening without such feedback. They observed that the ratio of performance and subjective exertion was significantly more favorable in the feedback condition and concluded that music agency was an efficient strategy to enhance pleasantness of strenuous exercise (Fritz et al., 2013).

Listening to background music in sports and fitness contexts is not without risk. There is controversial debate as to whether induced hearing loss may be attributable to music listening for leisure purposes (Zhao et al., 2010; Carter et al., 2014). Some authors maintain that prolonged exposure to high sound pressure levels might pose a threat to hearing especially for younger people (Vogel et al., 2007; Petrescu, 2008). Specifically, fitness instructors were found particularly prone to attract hearing problems through their profession (Nie and Beach, 2016). Consequently, attendance at fitness studios has been explicitly included in a portfolio of potentially harmful activities for adolescents' and young adults' hearing (Beach et al., 2013).

The motivations for listening to loud music and the preference for higher as opposed to lower volume levels are unclear. Todd and Cody (2000) found that high volume levels of dance music were associated with vestibular responses to low-frequency beats. They assume that such responses could reach the pleasure centers of the brain via the thalamus. Studies of the behavioral characteristics of loud music consumers reveal indications of addiction in a proportion of excessive listeners (Florentine et al., 1998). The marginal evidence supporting favorable psychological effects of loud music notwithstanding, production and dissemination strategies in the music and broadcasting industries seem to adhere to the notion that music listeners under most circumstances might prefer louder over the softer music of the same kind (Vickers, 2010; Katz, 2015).

Metcalfe (2016) undertook one of the few studies to investigate the influences of different intensity levels (45 and 75 dB) on walking speed but found no systematic influence of this variable. However, this study did not include a silent condition and participants' subjective levels level of exertion were not assessed. In another study comparing the differential effects of loud vs. soft music on subjective experience during a treadmill exercise, Edworthy and Waring (2006) observed that music per se had a significant impact on positive affect, but not on perceived exertion. However, based on their findings, these authors recommend loud music to optimize the affective experience of work-out in the gym.

The present study used a broader range of intensity levels, included participants with varying sports experience, and also entailed measures of physical performance and perceived effort during a rigorous ergometer exercise. Therefore, despite the negative findings by Metcalfe (2016), increases in performance and decreases in the perceived effort were expected to be associated with higher intensity levels as compared to lower levels. We also took measures to ascertain the appropriateness of the music from the participant's point of view. Finally, a physiological measure (heart rate) during task performance was used as a proxy for the participants' fitness levels.

# Aim, Research Questions, and Hypotheses

The central aim of the study was to investigate the influences of electronic dance music of different loudness levels on physical, behavioral, and physiological responses in trained and untrained healthy adults during ergometer exercise. Hence, we ask to what extent loudness modulated an aerobic ergometer performance. We further were interested in how the presence of music per se was perceived as appropriate in terms of loudness and tempo, preferable, and motivating during the exercise for two groups of different skill levels. Despite the mixed evidence in favor of positive effects of music listening during sports exercise, we nevertheless assumed that louder music leads to (a) significantly greater output and (b) significantly reduced perceived effort as compared to both exposure to softer music or no music.

# MATERIALS AND METHODS

# Participants

Forty males at the age between 19 and 35 years (M = 25.25 years; SD = 3.89 years) were recruited from the University of Oldenburg. These participants were classified into two different groups. Group 1 consisted of male handball players that played on a medium level of skill and can be categorized as advanced players with at least 4 h of training per week. For the second group, students were recruited that did no sports on a regular basis. All participants reported normal hearing conditions, no cardio-vascular diseases, impairments of the locomotor system, or intake of mind-altering medication. Before testing, every individual participant provided written consent to participate in the study.

## Stimulus Material

A selection of three music pieces was used in this study: (1) Roxfield "Freak Out" (stone mix), (2) Robbie Moroder featuring Anna Carels "Fucking hands up," and (3) Paranoja Crank House Stage "Infinity." The selection was strategic as representing modern electronic dance music that is typical for functional use in sports contexts. As expected, the pieces were unfamiliar to the majority of participants. The tempi and dynamics of these songs were adjusted to 128 beats per minute with a standard software (Audacity and logic pro X). The stimuli were presented by an Apple© "MacBook Pro Notebook" via dB Technologies© "Twin 128" stereo-loudspeakers. Sound emission was measured in the vicinity of participant's head by using a Testo© "816-1" sound pressure level meter. Sound intensity was adjusted such that it represented the average intended dB-level in the music conditions.

# Measurement Instruments Equipment

The Cyclus 2 <sup>R</sup> ergometer was used to evaluate physical performance in Watt (W), the force on the pedal in Newton meter (Nm), and the pedaling frequency as revolutions of the crank per minute (rpm). The ergometer was combined with a frame of a Felt <sup>R</sup> racing cycle (size 56) equipped with a Shimano <sup>R</sup> 'Sora' gear change. Data were read out from the ergometer via a USB-port and transferred to a desktop computer.

The Polar <sup>R</sup> 'RS400' heart rate monitor watch in connection with a chest belt 'Wearlink 31' was used to examine heart rate measured in beats per minute (bpm). Data were transferred to a computer via a USB-port and analyzed using the 'ProTrainer5' software package provided by Polar <sup>R</sup> . Subsequently, the data were exported and combined with the ergometer data file.

#### Questionnaires

A brief questionnaire was developed to collect information about the age of the participants and the regularly performed sportive activities. The health status was ascertained with the 'Health check questionnaire,' developed by the German Society for Sports Medicine and Prevention [Deutsche Gesellschaft für Sportmedizin und Prävention (DGSP)]. During the experiments, participants rated their current perception of fatigue and acoustical environment. Fatigue ("How exhausted are you in this moment?") was evaluated on a nine-point likert scale with 1 equalling "I feel really exhausted now" to 9 "I do not feel exhausted now at all." The second question concerned their perception of the acoustical environment ("How pleasant do you perceive your acoustical environment?") was rated again on a nine-point likert scale, with 1 equalling 'really unpleasant' and 9 being 'really pleasant.' Furthermore, after the experiment, participants gave information about the perceived loudness ("Was it too loud during the experiment?") and the familiarity with the music pieces ("Were you familiar with one of the presented musical pieces during the experiment?"). In addition, they indicated their subjective perception during the experiment based on three questions rated on five-point likert scales. The motivating effect of the music ("I found the music. . .") was rated on a scale labeled with 'disturbing' (1), 'rather disturbing' (2), 'irrelevant' (3), 'rather motivating' (4), and 'motivating' (5). Categories for the appropriateness of the genre ("The music genre was for this sport. . .") were labeled with 'inappropriate'(1), 'rather inappropriate' (2), 'suitable' (3), 'rather appropriate' (4), and 'appropriate' (5). Ratings of the general impact and the perception of the music genre averaged with higher values representing greater impact of the respective measure. Further, the tempo was assessed on a scale labeled 'too slow' (1), 'slow' (2), 'appropriate' (3), 'fast' (4), and 'too fast' (5). These ratings were also averaged. A response in the middle of the scale indicates an ideal tempo.

## Procedure

fpsyg-09-00590 May 3, 2018 Time: 17:36 # 4

Participants were tested in single sessions in the sports science lab of the University of Oldenburg. Upon arrival, they gave informed consent and filled the demographic questionnaire as well as the DGSP health questionnaire. The latter was immediately evaluated by a research assistant to ensure an uncritical participation. Subsequently, participants changed their clothes and were instructed how to apply the chest belt. The heart rate monitor watch was applied to the left wrist. Before the experiment started, participants were inducted into to the ergometer and the rating scales. After the research assistant calculated the individual maximal pulse, the task started with the low exertion phase as a warm-up. To define the two varying physical loads in the experimental phases, the maximal pulse was calculated by using a formula developed by Spanaus (2002). The maximal pulse equals: 214 – (0.5 × [age of participant in years] – 0.11 × body weight [in kilograms]). Hence, low exertion was represented by 60–65% of the maximal pulse, whereas and high exertion was represented by 80–85% of the maximal pulse. In the low exertion phase, no music was played. In the high exertion phase, the physical load was increased to the target pulse range. If the participants exceeded this range, the resistance in the ergometer was adjusted accordingly. The high exertion phase consisted of four different loudness conditions. In condition 1, music was still at 0 dB; in condition 2, a first song was played at an average sound pressure level of 65 dB. In conditions 3 and 4, the sound intensity was increased in 10-dB-steps to 75 and 85 dB intensity, respectively. Each intensity level was marked also with a new song. The order of the four conditions, as well as the order of the songs, was randomized for all participants. The two phases alternated four times. Each phase lasted 5 min. Thirty seconds before the phases ended the experimenter asked the participant to rate their current perception of fatigue and to evaluate the acoustic environment. After the last low exertion phase, participants filled the questionnaire about their subjective perception during the experiment. Every participant was provided with a cash incentive of 8.00 €. Each session lasted about 65 min in total. **Figure 1** depicts the time line of the experimental session.

This study was carried out in accordance with the recommendations of the Carl von Ossietzky University's Ethics Committee. This committee approved the protocol of the current study. All subjects gave written informed consent in accordance with the Declaration of Helsinki.

# Data Preparation and Analysis

Physical performance, force on the pedal, and pedaling frequency data were exported from the ergometer as CSV-data files. Mean values for every condition were calculated. When heart rates fell out of the target range of 80 to 85% during the high exertion condition for more than a third of the training session, physical measures were excluded from further analysis. First, a one-way ANOVA with repeated measures was conducted across all dependent variables to detect effects of order. Dependent variables were analyzed by a 4 × 2 repeated measures ANOVA with group (low and high trained) as a between-subjects and condition (0 dB/65 dB/75 dB/85 dB) as within-subject factor. Preconditions for conducting ANOVAs were assessed (normality Box's M test of equality of covariance matrices and Mauchly's test of sphericity). Accordingly, degrees of freedom were estimated in the F-statistics using Greenhouse–Geisser corrections where appropriate. Bonferroni's test was used for post hoc comparisons of means. In all statistical tests, p-values were set to 0.05. In addition,

partial eta-square was calculated as a measure of the effect size.

G <sup>∗</sup>Power (Faul et al., 2009) was used to conduct an a priori power analysis using the F-tests function and the algorithm for ANOVA (repeated measures, within-between interactions). According to this program, a total samples size of 36 participants was needed to obtain an effect size of f = 0.25 [α-level: 0.05, Power (1 – β): 0.95, correlations among repeated measures: 0.5]. Due to differences from targeted heart rates one participant had to be excluded from the 65 dB condition and two participants from the 75 dB condition.

#### RESULTS

**Table 1** summarizes the descriptive statistics for physical and psychological measures across low and high trained groups during the four exertion phases. No effects of sound intensity on physical performance and force on the pedal were found, all Fs < 0.68, all ps > 0.54. However, there was a trend for the main effect in terms of pedaling frequency, F(2.45,86.14) = 2.65, p = 0.07, η 2 <sup>p</sup> = 0.07. Bonferroni-adjusted post hoc tests indicated significantly lower frequencies during the 0 dB condition [CI 95% (66.18, 75.33)] in comparison to the 75 dB condition [CI 95% (69.29, 78.90)].

#### Order and Time Effects

There were significant effects of order regarding physical performance, F(1.86,66.77) = 60.05, p < 0.001, η 2 <sup>p</sup> = 0.63, force on the pedal, F(1.40,50.55) = 32.50, p < 0.001, η 2 <sup>p</sup> = 0.47, and pedaling frequency, F(1.68,60.47) = 8.70, p < 0.001, η 2 <sup>p</sup> = 0.20. While physical performance and force on the pedal significantly decreased over time, an increase of pedaling frequency was observed. In addition, perceived fatigue increased over time, F(2.47,96.26) = 28.99, p < 0.001, η 2 <sup>p</sup> = 0.43.

#### Training Level and Music Stimulation

Two-factorial ANOVA including training level and presence or absence of music during ergometer exercise were calculated for each of the dependent measures. To these ends, the three music conditions (65, 75, and 85 dB) were averaged and mean values entered into the analyses. There were significant main effects for physical performance, F(1,76) = 35.41, p < 0.001, and force on the pedal, F(1,76) = 12.40, p < 0.001. No further main or interaction effects were observed for the remaining dependent variables.

#### Perception of the Acoustic Environment

There was a significant main effect for perceived appropriateness of the acoustic environment, F(2.15,81.70) = 8.68, p < 0.001, η 2 <sup>p</sup> = 0.19. Bonferroni-adjusted post hoc tests revealed significant differences between the 0 dB condition [CI 95% (3.66, 5.04)] and all other conditions, namely 65 dB [CI 95% (5.10, 6.25)], 75 dB [CI 95% (5.61, 6.89)], and 85 dB [CI 95% (5.41, 6.84)]. Results showed lowest ratings in the 0 dB conditions and highest ratings in the 75 dB condition (see **Table 1** for details).

#### Music Evaluations

Majorities of participants rated the music as not too loud (65%) and unfamiliar (97.50%). The music was rated as quite motivating (M = 3.85, SD = 1.00) and appropriate for this sports exercise (M = 4.00, SD = 0.99). The tempo of 128 bpm was perceived

TABLE 1 | Means (and standard deviations) of physical and psychological measures across low and high trained groups during different conditions.


Physical performance are measured in Watt (W), the force on the pedal in Newton meter (Nm), and the pedaling frequency as revolutions of the crank per minute (rpm).

as appropriate (M = 3.25, SD = 0.59). **Table 2** summarizes the descriptive statistics for subjective responses. No differences between groups occurred, all ts < 0.01 and ps > 0.20, except for the perception of tempo, t(36.96) = −2.26, p < 0.05, d = 0.72. High trained participants (M = 3.45, SD = 0.61) perceived the music significantly faster as low trained participants (M = 3.05, SD = 0.51). However, ratings of both groups are in a positive range.

#### DISCUSSION

We asked whether music listening facilitated the performance and experience of strenuous ergometer exercise in trained and untrained healthy males. We assumed that music listening induced positive effects with respect to physical and subjective measures in the sense that loud music enhances performance and reduces perceived stress or effort. It was ensured that the music selection for this trial was appropriate and acceptable to the participants. And we took measures that the exercise was strenuous thus reflecting a typical workout protocol. Despite these efforts to construct a laboratory trial with high ecological validity, we failed to find unequivocal patterns of positive effects of music listening during the trial and across participant groups.

The observation that presenting music at 0 or 85 dB did not lead to any significant differences in the dependent measures of this study has important theoretical and practical implications that may warrant both further investigation and reconsideration of the use of music during fitness exercise. Theoretically, music listening may still have positive effects, but the mechanisms causing such effects are yet unclear. Practically, policies of the use of music in fitness studios particularly with respect to their intensity levels and potential risks for exercisers should be reconsidered. We will discuss these points in turn.

First of all, it is of note that our findings are in conflict with previous work which suggests more beneficial effects of loud music on performance during sports exercise (e.g., Edworthy and Waring, 2006). The rationale of such observations and interpretations is that loudness enhances the arousal potential of music stimulation and facilitates to distract attention from bodily perceptions to external cues (e.g., Murgia and Galmonte, 2015). However, previous evidence suggesting that loud music might reduce perceived exertion, or enhance physical aspects

TABLE 2 | Means (and standard deviations) of subjective ratings across low and high trained groups.


Scales range from 1 to 5. Higher values of motivation and genre appropriateness representing greater impact of the respective measure. Concerning tempo, a response in the middle of the scale indicates an ideal rating.

of performance, appears rather limited. To our knowledge, the current study is one of the first to systematically address this issue. Our results suggest that the hypothesis of performance enhancing effects of loud music during strenuous ergometer exercise must be rejected.

Fritz et al. (2013) have argued that music listening cannot be understood as a mere distraction, but instead can evoke brain mechanisms that lead to releases of hormones to reduce the perception of strain and enhance the experience of positive emotions. This interpretation is grounded on a body of research which has shown that music that is perceived as highly pleasurable can evoke brain systems associated with reward and emotion (e.g., Blood and Zatorre, 2001; Zatorre and Salimpoor, 2013). These observations resonate with potentially painreducing effects of more active music behaviors such as singing (Weinstein et al., 2016) or dancing (Tarr et al., 2015). These studies provide converging evidence by showing that performing synchronous musical activities in groups can modulate tolerance for individual pressure pain afflicted to the upper arm using a manchette for blood pressure measurement. However, the current observations are not necessarily in conflict with those previous findings. Fritz et al. (2013), for example, speculate that synchronicity between exercise movements and musical sound could be one key factor that contributed to the superior exercise performance and experience as compared to music listening to recorded music. Therefore, similar mechanisms that are believed to contribute to elevated pain-thresholds during singing and dancing in the above-cited studies may extend to workout exercise. Moreover, the findings that music listening can stimulate pleasure centers in the brain from PET-studies require participants to lay down silently and with minimal bodily movements in a scanner. At present, it seems difficult to measure and ascertain emotional brain responses to music listening during strenuous exercise.

Loud music has been identified as a potential source of hearing problems in both work and leisure environments. The size of the risk and the implications for needs of further regulation is a matter of continued and controversial debate (Morata, 2007; Zhao et al., 2010; Beach et al., 2013; Gilles et al., 2014). However, the choice of high loudness levels per se rests on the basic assumption that music listening induces positive effects on performance and perceived effort or strain. It is likely that this assumption must be specified in order to be of any practical use. For example, listening to loud music is generally assumed to contribute positively to fitness culture, although the research conducted to confirm this assumption is scarce and restricted to very few well-defined scenarios that do not entail the range of activities and contexts in which fitness sports happens.

Previous research on music and sports points toward a positive role of choice of tempo, which also suggests the importance of a certain coordination between auditory or audiovisual stimulation and bodily movement. Therefore, it may well be that temporal aspects such as synchronicity, tempo, and rhythm rather than sound intensity and respective loudness play a far greater role in supporting the music-aids-workout-hypothesis (Terry and Karageorghis, 2006; Fritz et al., 2013). The fitness industry already

responded to such an idea by producing electronic music in specific formats to entail well-defined tempo ranges. But again, the empirical support attributing a crucial role in temporal aspects is as yet insufficient.

#### Limitations

In this study, participants from a student population were invited to take part in a laboratory experiment. As is the case for a large number of psychological studies, this selection restricts the representativeness of findings to a significant degree. There are other methodological aspects that can be seen as limiting the interpretation of findings. For example, the ergometer per se emanates a certain type of background noise that could interfere with the music. However, increasing loudness levels also enhanced masking of the ergometer noise, but without inducing more positive effects during trials. Therefore, it seems unlikely that background noise influenced on the current findings in any systematic way. Moreover, the individual testing of the participants does not preclude a potential influence of group workout as opposed to individual workout. The presence of two experimenters and students during sessions, however, at least suggests that the presence of others per se might not alter the results. Finally, the music was not at an excessive loudness level and participants were exposed to the highest level (85 dB) only for few minutes according to the study protocol. Therefore, it may be that prolonged exposure to sound pressure levels above 85 dB could induce higher levels of positive affect and, consequently, a still more positive experience of the workout. However, it is obvious that the potential hearing risk outweigh the to-be-expected gains, if those exist at all.

#### REFERENCES


#### CONCLUSION

We tested the hypothesis that loud music positively influences workout at physical and subjective levels. The hypothesis was disconfirmed. Moreover, individual training status had no systematic influence on these findings. Nevertheless, there are important implications of the study. First, theories attributing a motivating role of music listening beyond distraction and entertainment during sports exercise must be revisited. Second, public policies regulating the use of music in fitness and workout contexts are advised to recommend lower levels as effective as higher levels of volume.

#### AUTHOR CONTRIBUTIONS

GK and JS conceived of and designed the research. DS and JN collected the data. DS provided a first draft. GK, JS, and AB analyzed the data and wrote the paper.

# FUNDING

This research was conducted by the authors without external funding.

#### ACKNOWLEDGMENTS

We would like to thank the staff and students from the sports science lab at the Carl von Ossietzky University of Oldenburg for their assistance.



**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Kreutz, Schorer, Sojke, Neugebauer and Bullack. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# How the Experimental Setting Influences Representativeness: A Review of Gaze Behavior in Football Penalty Takers

#### Johannes Kurz\* and Jörn Munzert

Neuromotor Behavior Laboratory, Department of Psychology and Sports Sciences, Justus-Liebig-University Giessen, Giessen, Germany

This article reviews research on the gaze behavior of penalty takers in football. It focuses on how artificial versus representative experimental conditions affect gaze behavior in this far-aiming task. Findings reveal that—irrespective of the representativeness of the experimental conditions—different instructions regarding the aiming strategy and different threat conditions lead to different gaze patterns. Results also reveal that the goal size and the distance to the goal did not affect the gaze behavior. Moreover, it is particularly run-up conditions that lead to differences. These can be either artificial or more natural. During a natural run-up, penalty takers direct their gaze mainly toward the ball. When there is no run-up, they do not direct their gaze toward the ball. Hence, in order to deliver generalizable results with which to interpret gaze strategies, it seems important to use a run-up with a minimum length that is comparable to that in a real-life situation.

#### Edited by:

Mauro Murgia, University of Trieste, Italy

#### Reviewed by:

Derek Panchuk, Australian Institute of Sport, Australia Ralf Kredel, Universität Bern, Switzerland

\*Correspondence: Johannes Kurz Johannes.kurz@sport.uni-giessen.de

#### Specialty section:

This article was submitted to Movement Science and Sport Psychology, a section of the journal Frontiers in Psychology

Received: 08 February 2018 Accepted: 19 April 2018 Published: 08 May 2018

#### Citation:

Kurz J and Munzert J (2018) How the Experimental Setting Influences Representativeness: A Review of Gaze Behavior in Football Penalty Takers. Front. Psychol. 9:682. doi: 10.3389/fpsyg.2018.00682 Keywords: football, penalty, far-aiming task, gaze behavior, experimental conditions

# INTRODUCTION

Perception–action coupling is a promising field of study offering new insights into sensorimotor control. This article reviews studies focusing on gaze behavior in football penalty takers and how the representativeness of the experimental setting influences gaze behavior in this faraiming task. The idea of creating representative task designs has been an important issue in experimental psychology for many years (Brunswick, 1956) and has been introduced to the field of perception-action coupling (e.g., Dicks et al., 2009, 2010). It has been suggested that task designs should represent the organism's natural environment (see Araújo et al., 2007, for details) and that task designs should comprise representative stimuli and allow participants to respond with unrestricted movements. Both aspects play an essential role when studying perception and action. Dicks et al. (2009) suggested that representative experimental conditions are mandatory to gain generalizable conclusions. Several studies have supported this suggestion by showing how representativeness affects performance and gaze behavior. For example, Mann et al. (2010) have shown that cricket batsmen's performance was better in representative experimental conditions compared to artificial experimental conditions. Another study by Dicks et al. (2010) found that football goalkeepers' performance and gaze behavior differed between artificial and representative experimental conditions: In representative experimental conditions, performance was better and gaze was directed toward the ball earlier and for a longer duration. Studies on walking in real

life using a mobile eye tracker compared to watching videos of walking also found significantly different patterns of gaze behavior ('t Hart et al., 2009; Foulsham et al., 2011): In real life, gaze was more centralized due to participants making head movements instead of large saccades, and gaze was directed more toward near objects and toward the path they were walking on. Additionally, results have revealed that the visual angle is smaller when watching videos compared to real-life conditions. This leads to restrictions of head movements. These restrictions lead, in turn, to limitations in gathering further information (e.g., vestibular and other crossmodal information). Furthermore, in cycling Zeuwts et al. (2016) found that gaze is directed more toward the path in real life than in the laboratory. Such findings reveal the need to create representative experimental conditions to investigate gaze behavior in its natural environment (see Land and McLeod, 2000; Hayhoe and Ballard, 2005). They emphasize that gaze behavior is task-specific (Yarbus, 1967), and, moreover, that it is based on a just-in-time mechanism when examined under natural interactive conditions (Ballard et al., 1995; 't Hart et al., 2009).

The present review focuses on gaze behavior in football penalties. This far-aiming task is of paramount importance in football. The specificity of this task is that it comprises two different task-related goals: the ball (proximal goal) that has to be hit with high precision and the corner (distal goal) where the ball has to be placed successfully (Kurz et al., 2018). A third area of interest for gaze behavior is defined by the goalkeeper who tries to prevent the penalty taker from scoring a goal. There have been discussions regarding whether either observing or ignoring the goalkeeper's reaction is the more successful strategy (van der Kamp, 2006). Although research shows that instructions for both strategies can indeed influence gaze behavior, the keeperindependent strategy has proven to be the more successful one (Noël and van der Kamp, 2012). However, up to now, no study has investigated gaze behavior in open-play situations, but only in artificial or representative conditions (McGuckian et al., 2018). Therefore, several aspects that might effect performance and gaze behavior have not been studied so far (e.g., minute of play, current score, presence of spectators, or goalkeeper characteristics). As a result, we did not consider these aspects in the present review. We review research on the gaze behavior of penalty takers focusing on how gaze behavior in this far-aiming task is affected by artificial versus representative experimental conditions. The aim of the present review is to deliver support for the need to reinterpret data on gaze behavior in artificial experimental conditions and to emphasize the need for research in visual science to be carried out under representative experimental conditions.

## LITERATURE SEARCH

We searched for literature in the following electronic databases (**Figure 1**): Web of Science, PubMed Central, and SPORTDiscus. Within each database, we used the keyword penalty combined with one of the following four keywords: eye tracking, gaze

behavior, eye movement, or visual search. We also examined the references in relevant articles. Studies were included when (1) the task was to shoot a penalty in football, (2) gaze behavior was recorded, and (3) the article was written in English. We manually excluded studies addressing the socalled "quiet-eye" phenomenon (e.g., Vickers, 2007) because these focus mainly on the last fixation and do not take gaze behavior during the complete run-up into account. **Table 1** reports relevant descriptive information on the included studies.

We are well aware that a dichotomy between artificial and representative experimental conditions does not exist (e.g., Hadlow et al., 2018), but we prefer to treat this problem as a continuum with experimental conditions closer to one end of an artificial–representative dimension. To define studies using artifical and studies using representative experimental conditions, we performed a two-step categorization: In the first step, we defined conditions that include penalty shots toward a goal with a real goalkeeper who tries to prevent the penalty taker from scoring. In a second step, we defined conditions in which a minimum length of run-up should be given. The second step was defined to study the impact of the proximal and the distal goal. We categorized the situation as representative only when a run-up involved more than one step. Our argument was that only then is the ball relevant as the proximal goal. These two categories correspond with two

TABLE 1 | Overview of studies on gaze behavior of penalty takers in football.


categories defined by McGuckian et al. (2018): (1) laboratory in situ where participants were allowed to move freely, but non-live stimuli were presented and (2) controlled in-situ where participants were allowed to move freely and live stimuli were presented.

#### GAZE BEHAVIOR IN FOOTBALL PENALTY TAKERS

Based on this two-step categorization (**Table 1**), we defined studies as either artificial or representative. We then arranged the studies in ascending order according to goal size. As **Table 1** shows, the goal size and the distance between the goal and the penalty spot are interdependent. This is because most studies chose a visual angle between the penalty spot and both goalposts that is similar to the visual angle (∼36◦ ) in a real-life situation (goal size: 7.32 × 2.44; distance: 11 m). Furthermore, studies using artificial experimental conditions applied a mean goal size of 2.40 × 0.85 m and a mean distance of 3.4 m (SD = 1.1 m). In contrast, studies using representative experimental conditions applied a mean goal size of 6.12 x 2.23 m and a mean distance of 9.6 m (SD = 2.2 m).

The first study (Bakker et al., 2006) examining the gaze behavior in football penalties used artificial experimental conditions. Penalty takers had to shoot toward a screen onto which a goal and a goalkeeper were projected. Penalty takers did not perform a run-up and they were asked to follow three different instructions: (1) to shoot as well as possible, (2) to shoot as well as possible and make sure to attend to the goalkeeper, and (3) to shoot as well as possible and make sure to hit the open space. When penalty takers shot according to Instruction 1, they directed their gaze for about 38% of the trials toward the goalkeeper and for about 59% toward the open space within the goal. Under Instruction 2, they directed their gaze for about 77% of the trials toward the goalkeeper and for about 22% toward the open space. In contrast, under Instruction 3, they directed their gaze for about 20% of the trials toward the goalkeeper and for about 79% toward the open space. In two further studies by Binsch et al. (2008, 2010) using artificial experimental conditions, penalty takers were asked to shoot toward a goal projected onto a screen. There was no run-up and penalty takers were asked (1) to shoot as accurately as possible, (2) to shoot as accurately as possible and not to shoot within the reach of the goalkeeper, and (3) to shoot as accurately as possible and to shoot into the open space within the goal. Results revealed that, irrespective of instructions, penalty takers first directed their gaze toward the goalkeeper and afterward toward the open space within the goal until they hit the ball. Some participants directed their gaze toward the goalkeeper again shortly before they hit the ball. Wilson et al. (2009) asked penalty takers to shoot toward a goal with a real goalkeeper using one step as run-up. Penalty takers were instructed to shoot toward the areas of the goal where they expected the best chance of scoring under a low-threat and high-threat condition. Results showed no significant differences for total number of fixations between the locations goalkeeper and open space within the goal irrespective of threat conditions. However, gaze was directed significantly longer toward the goalkeeper (M = 3.9 s) compared to the open space within the goal (M = 1.9 s) irrespective of threat conditions. In another study by Wood and Wilson (2010b) using artificial experimental conditions, penalty takers had to shoot toward a goal with a real goalkeeper. However, they were allowed to perform only one step as run-up. In this study, penalty takers were instructed to score as many goals as possible. Three typical gaze strategies were instructed: (1) to ignore the goalkeeper's reaction, (2) to ignore the goalkeeper's reaction and to direct their gaze toward the opposite corner from that to which they intended to aim the ball, and (3) to observe the goalkeeper's reaction. Results showed no significant differences between conditions, though the duration of last fixation was shorter in Instruction 1 (M = 223 ms) compared to Instruction 2 (M = 317 ms) and Instruction 3 (M = 329 ms). No results were provided on which locations penalty takers directed their gaze toward. In a further study by van der Kamp (2011), penalty takers had to shoot toward a screen on which a goal and a goalkeeper were projected. Penalty takers were asked to score a goal and to shoot the ball toward the opposite side to that toward which the goalkeeper dived. In contrast to the aforementioned studies in which penalty takers performed no

or only one step as run-up, here, penalty takers were required to take exactly 2 s and the start of the run-up was 2 m behind the ball. In the first section of the run-up, gaze was directed mainly toward the goalkeeper's upper and lower body. However, during the last section, gaze was directed mainly toward the open space within the goal and toward the floor (including the ball).

In a study by Timmis et al. (2014) using representative experimental conditions, penalty takers had to shoot a ball toward a goal with a real goalkeeper and run up individually. Penalty takers were asked to score as many goals as possible, to avoid attempting to deceive the goalkeeper, and to take either a placement or a power penalty. Irrespective of taking a placement or a power penalty, penalty takers directed their gaze mainly toward the ball (M = 63%) and less toward the goalkeeper (M = 6%) and less toward the open space within the goal (M = 13%). Noël and van der Kamp (2012) asked penalty takers to shoot toward a goal with a real goalkeeper, and the run-up was a matter of individual choice. At the beginning of the task, penalty takers directed their gaze mainly toward the goalkeeper, toward the open space within the goal, and toward the ball (which seems to be necessary for a spatial calibration for the runup; see Kurz et al., 2018). Closer to foot–ball contact, penalty takers then directed their gaze almost exclusively toward the ball. This gaze pattern appears to be independent from the penalty takers' strategy of either ignoring or observing the goalkeeper's reaction. When penalty takers tried to ignore the goalkeeper's reaction, the time gaze was directed toward the ball increased from about 21% at the beginning of the task to about 90% just before foot–ball contact. When penalty takers tried to observe the goalkeeper's reaction, the time gaze was directed toward the ball increased from about 5% at the beginning of the task to about 46% just before foot–ball contact. In another study by Wood and Wilson (2010a), penalty takers were asked to shoot a ball toward a goal with a real goalkeeper and run up individually. Penalty takers were also asked to do their best in a low- and a highthreat condition. Results showed that during the aiming phase, penalty takers distributed their gaze between the goalkeeper and the open space within the goal. Additionally, in the high-threat condition (M = 462 ms), gaze was directed longer toward the open space within the goal than in the low-threat condition (M = 347 ms). During the run-up, gaze was directed exclusively toward the ball (M = 430 ms) and not toward the goalkeeper (M = 0 ms) or the open space within the goal (M = 0 ms). Similar results were found by Kurz et al. (2018) when penalty takers had to shoot toward a goal with a real goalkeeper and run up individually while ignoring the goalkeeper's reaction. At the beginning of the task, penalty takers distributed their gaze mainly between the goalkeeper, the open space within the goal, and the ball. Closer to foot–ball contact, they directed their gaze almost exclusively toward the ball. When the goalkeeper tried to save the ball, penalty takers directed their gaze toward the ball for about 45% of the time at the beginning of the task and for about 70% during the last three steps. Gaze was hardly ever directed toward the open space within the goal (M = 4%) during the last three steps. Using representative experimental conditions, Hüttermann et al. (2014) asked penalty takers to shoot toward the side opposite to the one the goalkeeper dived toward and to score as many goals as possible. Penalty takers had to shoot toward a goal with a real goalkeeper and they were required to start their run-up at least 3.5 m behind the ball. Penalty takers received two different instructions concerning their gaze behavior: (1) a condition in which they received no further instruction and (2) a condition in which they were instructed to direct their gaze toward a 1 × 1 m area between the ball and the goalkeeper. In compliance with Instruction 2, penalty takers directed their gaze toward the 1 × 1 m area. Under Instruction 1, penalty takers mainly distributed their gaze between the ball, the goalkeeper, and the open space within the goal on most trials (77%).

# SUMMARY AND CONCLUSION

In recent years, research on the gaze behavior of penalty takers in football has become an interesting topic. A series of studies has been carried out to gain a better understanding of penalty takers' gaze behavior. The aim of this article was to review research on the gaze behavior of penalty takers in football and focus on research on how artificial versus representative experimental conditions affect gaze behavior in this far-aiming task. Furthermore, we aimed to deliver support for a reinterpretation of data on gaze behavior in artificial compared to representative experimental conditions.

The first and foremost question is whether participants performed the same task in the aforementioned studies. Most studies applied different experimental settings, such as different goal sizes, distances, balls, and lengths of run-up. Therefore, one could argue that participants performed different tasks. However, irrespective of these differences, all studies applied a similar visual angle (M = 36.9◦ , SD = 2.5◦ ) ensured by adjusting the distance to the goal size. Thus, a smaller goal size resulted in a smaller distance and vice versa. Furthermore, in each study, participants were asked to shoot a ball toward a target within a goal; and in some studies, a real goalkeeper tried to prevent the participants from scoring a goal. Therefore, we argue that participants had to perform a similar task, and this justifies comparing the studies.

As outlined above, results showed differences in the gaze behavior of penalty takers depending on whether studies used artificial or representative experimental conditions. In studies with artificial experimental conditions, penalty takers directed their gaze mainly toward the goalkeeper and the open space within the goal. In studies with representative experimental conditions, penalty takers distributed their gaze between the goalkeeper, the open space within the goal, and the ball during the preparation phase. During the last three steps gaze was directed mainly toward ball. Gaze was even directed toward the ball when penalty takers were instructed explicitly to observe the goalkeeper's reaction (Noël and van der Kamp, 2012). Thus, we suggest that this gaze pattern shown in studies using representative experimental conditions can be considered to be generalizable. Furthermore, we suggest that findings from studies using artificial experimental conditions cannot be compared to the preparation or the execution phase from studies using representative experimental conditions. It can be argued that

the differences in gaze behavior depend on whether or not penalty takers perform a run-up irrespective of the presence of a real goalkeeper (van der Kamp, 2011; Noël and van der Kamp, 2012). When penalty takers did not perform a run-up, their position in relation to the ball was constant and, as a consequence, they did not have to refresh the relative position of the ball. This is one possible explanation why they did not direct their gaze toward the ball. In contrast, when penalty takers performed a run-up, their relative position to the ball changed, and they therefore had to refresh the relative position of the ball continuously. This seems to be necessary in order to obtain optimal foot–ball contact (Kurz et al., 2018). We suggest that this is the reason why gaze behavior changed during the run-up and why penalty takers directed their gaze almost exclusively toward the ball the closer they came to foot–ball contact.

Dicks et al. (2010) and Mann et al. (2010) have already shown that representative experimental conditions are mandatory to gain generalizable conclusions on performance environments. They demonstrated that findings from artificial and highly controlled experimental conditions are unlikely to be comparable with findings from more natural and less controlled experimental conditions. Thus, we suggest that future studies should consider this point and try to create representative experimental conditions (Dicks et al., 2009). These should include a runup of a minimum length. Furthermore, we suggest that results on gaze behavior gained from studies using artificial experimental conditions without a run-up should be interpreted with caution, because these studies overestimate the number and the duration of fixations focused on the goalkeeper and the open space within the goal. Recently, Cañal-Bruland and Mann (2015) extended this approach by arguing that future studies should also consider situational and contextual (non-kinematic) information. However, it seems to be a real challenge to consider these important aspects in controlled experimental conditions.

In addition to differences in gaze behavior, we also identified similarities between studies using artificial and studies using representative experimental conditions. As shown repeatedly, different instructions on the same task result in different gaze patterns (Yarbus, 1967). This has also been shown for the gaze behavior of penalty takers in football irrespective of experimental conditions. For example, Bakker et al. (2006) used artificial experimental experimental conditions and Noël and van der Kamp (2012) used representative experimental conditions to show that penalty takers' gaze was directed more toward the goalkeeper when they were asked to observe the goalkeeper's reaction. In contrast, when penalty takers were asked to ignore the goalkeeper's reaction, they directed their gaze less toward the goalkeeper. However, all other studies applied a huge number of different instructions to manipulate the gaze behavior. These studies reveal that gaze behavior of football penalty takers can be influenced by instructions. In particular, findings from studies using representative experimental conditions showed that gaze behavior of football penalty takers is taskspecific and that it is based on a just-in-time mechanism. Another similarity is that irrespective of the experimental conditions, penalty takers made more fixations and directed their gaze longer toward task-relevant locations in highthreat conditions compared to low-threat conditions (Wilson et al., 2009; Wood and Wilson, 2010a). This has been found irrespective of whether or not a real goalkeeper was present and whether or not penalty takers performed a run-up.

Furthermore, findings from studies using representative experimental conditions showed that differences in goal size and distance do not affect gaze behavior. For example, Timmis et al. (2014) applied a goal size of 3.66 x 1.83 m and a distance of 6 m; Noël and van der Kamp (2012), a goal size of 5.0 x 2.0 m and a distance of 9 m; and Kurz et al. (2018), a goal size of 7.32 x 2.44 m and a distance of 11 m resulting in a mean visual angle of 33.9◦ (33.9◦ , 31.0◦ , and 36.8◦ , respectively). Additionally, these studies also used different instructions. However, results revealed that prior to the beginning of the run-up, gaze was distributed between the goalkeeper, the goal, and the ball; and during the run-up, gaze was directed mainly toward the ball. Based on these findings it remains unclear whether differences in goal size and distance affect shooting performance.

Finally, some other aspects were not considered due to the limited number of studies. For example, we did not review whether the expertise level of the participants resulted in different gaze behavior because 10 out of 11 studies had recruited university or intermediate football players with a mean experience of playing football on a competitive level for 12.9 years (SD = 2.2). Only one study (van der Kamp, 2011) compared different expertise levels in the participants. Furthermore, we did not review other aspects such as anxiety, environmental conditions, and knowledge of the opponent, because such aspects have not been studied so far.

In general, this review provides further insight into how the artificial versus representative distinction—and particularly whether participants had to perform a run-up—impacts on the interpretation of gaze strategies in studies on football penalties. We identified the length of the run-up as key feature which influences gaze behavior even if we have to consider that the length of the run-up is correlated with other features such as goal size or distance. In general, the review shows that gaze behavior in studies using artificial or representative experimental conditions differs. Even if the basic task, i.e., shooting a ball toward a target within the goal, seems to be the same, we would still argue that the task is modified substantially when reducing the run-up to a minimum. The essential task characteristic that changes in most artificial conditions is the reduced difficulty to obtain an optimal foot–ball contact. Thus, we suggest that results from studies using artificial experimental conditions are hardly comparable with studies using representative experimental conditions. Furthermore, we suggest that future studies should apply a minimum length of a run-up (more than one step).

## AUTHOR CONTRIBUTIONS

JK made substantial contributions to the literature search, conception, writing, and interpretation of data. JM made substantial contributions to the conception, writing, and

interpretation of data. JK and JM participated in drafting the article and revising it critically for important intellectual content; and gave final approval of the version to be submitted and agreed

#### REFERENCES


to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Kurz and Munzert. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Movement Sonification in Stroke Rehabilitation

Gerd Schmitz <sup>1</sup> \* † , Jeannine Bergmann2,3†, Alfred O. Effenberg<sup>1</sup> , Carmen Krewer 2,4 , Tong-Hun Hwang1,5 and Friedemann Müller 2,3

1 Institute of Sports Science, Leibniz University Hannover, Hannover, Germany, <sup>2</sup> Schön Klinik Bad Aibling, Bad Aibling, Germany, <sup>3</sup> German Center for Vertigo and Balance Disorders, Ludwig-Maximilians University of Munich, Munich, Germany, <sup>4</sup> Department of Sport and Health Sciences, Technical University Munich, Human Movement Science, Munich, Germany, 5 Institute of Microelectronic Systems, Leibniz University Hannover, Hannover, Germany

Stroke often affects arm functions and thus impairs patients' daily activities. Recently, several studies have shown that additional movement acoustics can enhance motor perception and motor control. Therefore, a new method has been developed that allows providing auditory feedback about arm movement trajectories in real-time for motor rehabilitation after stroke. The present article describes the study protocol for a randomized, controlled, examiner, and patient blinded superiority trial (German Clinical Trials Register, www.drks.de, DRKS00011419), in which the method will be applied to 13 subacute stroke patients with hemiparesis during 12 sessions of 30 min each as additional feedback during the regular movement therapy. As primary outcome, a significant pre-post-change in the Box and Block Test is expected that exceeds the performance increase of 13 patients who will be provided with sham-acoustics. Possible limitations of the method as well as the study design are discussed.

Keywords: movement sonification, motor rehabilitation, stroke rehabilitation, arm movements, acoustic feedback

# INTRODUCTION

#### Background

Stroke is the second most common cause of death among the neurological disorders. The great majority of patients who survive a stroke have to rely on health care support afterwards (1). Sensory and motor impairments can lead to dramatic limitations of everyday motor skills and temporary or permanent disability. Most often arm functions are impaired and hamper patients during activities of daily living (2). Hemiparesis, for example, affects spatial and temporal arm motor control and results in disturbed movement trajectories, lower movement amplitudes and enhanced movement times (3). Therefore, one important goal of motor rehabilitation is the improvement of arm functions. Some therapies like the Arm Ability Training (4) or the Constraint Induced Movement Therapy (5) predominantly focus on the improvement of the motor components of the arm movement system. However, Bastian points out that efficacy of stroke rehabilitation might be improved by methods that combine perceptual- and motor oriented approaches (6). A recent study with healthy participants showed a higher efficacy of a sensorimotor compared to a purely motor orientated approach, accordingly, although both approaches address the same adaptation mechanisms (7). An example for a perception-oriented approach for stroke rehabilitation is Ramachandran's mirror visual feedback method. It seems to reestablish congruency between motor commands and visual feedback in patients that watch a mirror image of the unimpaired arm during bilateral movements. Some of these patients report not only to see the impaired arm, but also to feel

#### Edited by:

Mauro Murgia, University of Trieste, Italy

#### Reviewed by:

Marta Bienkiewicz, Medical School, University of Exeter, United Kingdom Eckart Altenmüller, Hanover University of Music Drama and Media, Germany

#### \*Correspondence:

Gerd Schmitz gerd.schmitz@ sportwiss.uni-hannover.de

†These authors have contributed equally to this work.

#### Specialty section:

This article was submitted to Movement Disorders, a section of the journal Frontiers in Neurology

Received: 28 February 2018 Accepted: 14 May 2018 Published: 01 June 2018

#### Citation:

Schmitz G, Bergmann J, Effenberg AO, Krewer C, Hwang T-H and Müller F (2018) Movement Sonification in Stroke Rehabilitation. Front. Neurol. 9:389. doi: 10.3389/fneur.2018.00389

**48**

it moving. A probable explanation is that mirror visual feedback revives temporarily inactive motor neurons and/or ipsilateral corticospinal pathways (8).

As alternative to vision-oriented approaches, a specific feature of recently developed methods is the implementation of auditory signals and sounds to generate additional perceptual information about movement quantities and qualities (9, 10). In particular, music has been shown to be an efficient add-on in stroke therapy: Schneider et al. (11) showed that a music based arm therapy can outperform highly established approaches like the constraint induced movement therapy. Chen et al. (12) reported from a proof of concept case study on five stroke patients that rhythmic auditory cueing enhanced movement speed. Furthermore, twostate continuous musical feedback increased elbow extension as well as shoulder flexion and reduced compensatory trunk movements. Growing evidence suggests that music-supported therapy is superior to conventional physiotherapy without music, probably because it acts on multiple levels and addresses motor, cognitive, and emotional mechanisms (13).

Furthermore, some studies indicate beneficial effects of continuous auditory feedback for movement rehabilitation after stroke. For example, Maulucci and Eckhouse (14) reported that stroke patients relearned functional movement paths faster when they were provided with auditory feedback about spatial deviations from reach paths performed by healthy persons. Secoli et al. (15) found that auditory feedback improved performance in a movement tracking task performed during robot-assisted arm training in patients with chronic left hemiparesis. However, other results were equivocal: According to Robertson et al. (16), feedback about hand orientation during reaching seems to be beneficial for patients with right hemisphere lesions, but detrimental for patients with left hemisphere lesions. Based on a systematic literature review, Molier and colleagues see a possible benefit of performance feedback and augmented auditory feedback, although the determinants for their efficacy remain largely unknown (17).

Since stroke often impairs somatosensation (18, 19), recovery of arm functions might benefit from methods that support proprioception, particularly. Hereto, Sihvonen et al. (13) argue that music-supported therapy might be effective, again, because patients generate an internal expectation about when the next note is going to be heard and thereby improve their movement timing. However, by considering proprioception as integrated percept of multiple sensory streams from multiple receptors which is experienced as motion and position sense, further methods might address specific proprioceptive mechanisms and thereby support the relearning of functional movement patterns after stroke. The method of movement sonification might have this potential. Movement sonification represents a concept for mapping movement parameters to sound in order to create novel perceptual streams congruent to the time course of kinematic or dynamic movement parameters (20). This method differs conceptually from providing feedback on performance errors, because it allows to design artificial perceptual streams structurally equivalent to perceptual streams from other modalities. It has been shown that the amendment of visual motion information by movement acoustics amplifies the activity of multimodal integration areas in the brains of observers and furthermore, activates the basal-ganglia-frontocortical motor loop (21, 22). Accordingly, movement sonification has been shown to support learning (23, 24) and adaptation (25) of fine motor skills, (re)learning of arm joint coordination patterns (26) and acquisition of gross motor skills (27, 28) in healthy persons. In deafferented patients, it can substitute proprioception (29). Studies on immediate effects of movement sonification on movement pattern recognition, movement synchronization and own-other discrimination (30–33) indicate that movement sonification unfolds its potentials on perception and action by linking to internal movement representations.

#### Theoretical Approach

The present approach is based on a further development of a method presented in Vinken et al. (31) and Schmitz et al. (34). It differs from other approaches as it focuses on sensorimotor representations of hand and arm movements as suggested by Bastian (6) for arm training in stroke rehabilitation. Studies indicate that hand and arm movements are represented in body-centered reference frames and that arm trajectories are realized on the basis of muscle synergies (35–38). Findings from Overduin and colleagues indicate that muscle synergies are represented in the brain in a time-invariant spatial as well as a time-varying spatiotemporal manner (39). d'Avella et al. (40) showed that different muscle synergies are active during movements to different directions, but a few synergies can sufficiently explain coordinated muscular activity during movements with different amplitudes, loads, forearm postures, as well as movement sequences. Such results indicate that synergies serve the implementation of a few global movement features like movement direction and amplitude which are coded by independent neuronal populations in the brain (35). Accordingly, it seems to be reasonable to design feedback related to movement direction and amplitude in an egocentric reference frame to address arm movement control and muscle synergies. Since stroke seems to disrupt muscle synergy patterns of the impaired arm (41), significant effects of such feedback might be expected for the rehabilitation of arm functions. Moreover, muscle synergy patterns are highly correlated between arms and the reorganization of muscle synergy patterns is part of the recovery process after stroke (41). By providing homogenous feedback on movements of each arm, the unimpaired arm can serve as individualized movement model as well as auditory mirror image and might support the reorganization process.

The goal of the intended clinical trial is to prove the impact of a novel method for arm movement sonification in motor rehabilitation after stroke. The method provides real-time feedback about three-dimensional wrist movements in relation to the trunk. Auditory feedback informs about the angular direction of movements in the horizontal and vertical plane, the radial amplitude as well as the absolute velocity of the wrist. Accordingly, each movement produces an unequivocal sound which represents additional sensory feedback. We hypothesize that patients benefit from the real time feedback during movements with the impaired arm due to the structural equivalence of the sound to movement information from other modalities which can amplify activity of multisensory integration areas in the brain, support motor control and substitute partially lost proprioception as indicated by Scheef et al. (21), Schmitz et al. (22), Effenberg et al. (28) and Danna and Velay (29). A pilot study with a precursor version of this method provided encouraging results as four stroke patients showed improved performance in the Box and Block test after five training sessions (34). Furthermore, a related method has recently been applied in a randomized controlled clinical trial by Scholz et al. (42). After an exploration-phase, stroke patients learned to play simple melodies by moving their impaired arm in 3D-Cartesian space. Ten training sessions of 20 min each with this musical sonification reduced pain according to the pain-score of the Fugl-Meyer test, enhanced hand functions as assessed by the Stroke Impact Scale and increased smoothness of reaching as shown by kinematic analyses. Effect sizes were moderate. The present approach differs from the method presented in Scholz et al. (42) mainly by the way arm movements are sonified and by capturing the whole upper body. This allows providing intuitive feedback for both arms while controlling for upper body movements. Thus, the present method prospects even larger effects.

# Objectives

The primary objective of this trial is to investigate the effectiveness of the above mentioned approach for real-time movement sonification on motor abilities of the paretic upper limb in subacute stroke patients. The hypotheses are that, compared to patients provided with an auditory control stimulus, patients provided with real-time movement sonification (1) improve gross motor dexterity of the paretic upper limb assessed with the Box and Block test, and (2) motor function of the paretic arm and hand measured with the Action Research Arm Test and the Stroke Upper Limbs Capacity Scale.

# Trial Design

The present trial is designed as a randomized, controlled, assessor and patient blinded superiority trial with two parallel groups. Randomization is performed as block randomization with 1:1 allocation.

# METHODS

## Study Setting and Eligibility Criteria

All subjects included in the study are inpatients at a rehabilitation hospital in Germany. They meet the following inclusion criteria: hemiparesis of the upper extremity (SULCS score ≥3) after a unilateral ischemic or hemorrhagic stroke (4 weeks to 6 months after stroke onset), the functional ability to pick up a wooden cube (2.5 cm in size) with the paretic hand, and age between 18 and 80 years. Patients with unstable fracture, the inability to sit for 30 min, or severe aphasia or cognitive impairment, which compromises the implementation of the assessments or the therapy are excluded from the trial.

#### Interventions

Subjects enrolled in this study are randomized in equal proportions between sonification and sham-acoustics, receiving either training for the upper extremities with real-time movement sonification (intervention group) or training with sham-acoustics (control group). During the intervention phase, subjects of both groups receive movement therapy for the upper extremities at 4 days per week for 3 weeks, i.e., a total of 12 sessions. A therapy session takes 30 min. Within each study therapy session, gross motor arm movements are performed focusing on (a) reaching, (b) grasping, (c) bimanual activities, and (d) velocity. Exercises belonging to those categories are performed in blocks of 5 min. An exercise catalog can be used by the therapists containing ideas for arm movements of each category. Content and repetitions are recorded for later analysis. A break and a short calibration of the XSens system is scheduled between the 5-min blocks. Subjects of both groups wear the sonification system (straps, sensors, head phones, onbody controller) during the therapy.

Once a patient is enrolled in the study, the study site makes every reasonable effort to follow the patient for the entire study period. If study sessions are canceled due to indisposition of the patient or the therapist, or due to technical issues, one additional session per week can be scheduled. The intervention period should not exceed 3 weeks. Adherence to therapy is monitored by documenting therapy failures, therapy durations, and drop-outs.

The following individual criteria were defined for discontinuing the allocated intervention: incidence of a new disease or complication of the underlying disease, which makes continuation of the study impossible, and at the patients' request or at request of the legal representative. Patients who discontinue the intervention are considered off intervention and follow the same schedule of measurements as patients who finished the intervention. Discontinuation of the intervention is not a reason for withdrawal from the study. Patients are free to withdraw from the study for any reason at any time. The investigator also may withdraw patients from the study to protect their safety or if they are unwilling and unable to comply with the study procedures.

All patients included in the study are inpatients at the rehabilitation hospital and receive the normal therapy setting during the study period. The only intervention which is prohibited during the intervention period is robot-assisted training for the upper extremities.

# Arm Motion Tracking and Sonification

Arm movements are tracked with a mobile motion capture system (MTx miniature 3DOF inertial orientation tracker; Xsens Technologies BV, Enschede, The Netherlands). It contains seven inertial sensor units, which are composed of three accelerometers, three gyroscopes, and magnetic sensors allowing measuring three dimensional orientation. All sensors are connected by cable with an on-body controller (XBus Master) worn at a belt which transmits synchronized sensor data (50 Hz) wirelessly to a laptop (Bluetooth protocol 2,400–2,500 MHz). Sensors are fixated by velcro straps and aligned to seven body parts representing a kinematic chain (sternum, shoulders, upper and lower arms). By comparing orientation of two interconnected sensors and considering predefined segments' lengths it becomes possible to determine joint angles and calculate relative joint positions based on forward kinematics (43), here the relative wrist position in relation to the intersection of shoulder axis and spine of a biomechanical upper body model.

Wrist position is calculated in spherical coordinates, i.e., each posture is determined by the azimuth angle, the elevation angle as well as the radial distance between wrist and origin of the spherical coordinate system (**Figure 1**). These data are submitted to the open source software applications PureData and CSound for sonification. The sonification concept is inspired by ecological relationships between sound and energy like it is given for the sound amplitude, which is usually determined by the amount of energy being transformed by the soundemitting event: The harder a tennis player hits the ball with the racket, the louder the impact boom will sound (20). Such ecological relations are well established within the hearing system and the perceptual generation of such kind of auditory information does not need conscious attention. The sonification technique is based on frequency modulation of a synthesized sound with a sawtooth wave form. The carrier frequency, which is the basic frequency, is set to 200 Hz for the left arm and to 300 Hz for the right arm when the arms are hold in a neutral position besides the body (elevation angle 0 ◦ ). Arm elevation increases sound frequency by a maximum of 200 Hz, which is achieved when both arms are stretched above the head (elevation angle 180◦ ). The azimuth angle determines the panning (equal power panning) and thus the interaural intensity difference. Radial amplitude modifies the perceived brightness of the sound by a logarithmic change of the frequency regulation index between 0 and 0.15. Finally, the absolute velocity of the wrist defines the sound amplitude and thus the loudness. Higher velocities result in higher sound amplitudes. Thus, right and left arm movements produce and modify one sound each which are provided to the patients of the experimental group wirelessly via headphones. Notably, no sound can be heard when the arms are at rest. The control group is provided with sham-acoustics. Arm movements produce the sound of ocean waves, which are not altered by the movement trajectory.

The volume of the movement sonification and the shamacoustics is adapted according to the patients' preference (maximal 65 decibel).

#### Outcomes

Primary outcome measure is the Box and Block Test (44). The test is a measure of unilateral gross manual dexterity. It counts the number of wooden blocks that can be transported from one compartment of a box to another compartment within 1 min. The Box and Block test shows high test-retest and interrater reliabilities in elderly subjects and subjects with neurological disorders (45, 46). The construct validity of the test is high when compared with the ARAT and the Fugl-Meyer test (45, 46). The Box and Block test is suitable to detect changes over time in stroke patients (47).

Secondary outcome measures are the Action Research Arm Test (ARAT) and the Stroke Upper Limbs Capacity Scale (SULCS). The ARAT assesses mainly the ability to handle smaller and larger objects with a variety of qualitatively rated items. It

includes four subtests: grasp (6 items), grip (4 items), pinch (6 items), and gross movement (3 items). The scores for each item range from 0 to 3. We use the standardized protocol published by Yozbatiran et al. (48) to assess the ARAT. The test shows high intrarater reliability and interrater reliability. Validity is high when compared to the Fugl-Meyer test and it is sensitive to detect changes (47).

The SULCS assesses the capacity of the paretic upper limb in stroke patients. It consists of 10 items which represent tasks that are related to daily activities (49). The items assess proximal arm capacity without need for active wrist or finger movement (3 items), arm capacity combined with basic hand capacity (grasp tasks without manipulating, 4 items), and advanced hand capacity (manipulating tasks, 3 items). The scale has good interrater reliability and a high construct validity when compared with the ARAT and the Rivermead Motor Assessment (RMA) (50).

All these outcome measures are assessed at baseline before the start of the intervention, at post-test after the last intervention, and at follow-up test 2 weeks after the end of the intervention. The measures are assessed by an assessor blind to treatment allocation. The assessor is experienced and trained in performing the clinical assessments.

Differences between baseline and post measurement, and between baseline and follow-up measurement will be calculated for all outcome measures to determine short-term and long-term changes. These changes will be compared between groups. The


primary and secondary outcome measures will be presented as means and standard deviations or as medians and 25th and 75th percentiles for each group.

# Participant Timeline

The study timeline shown in **Table 1** presents an overview of the time schedule of enrolment, interventions, and assessments of the outcome measures.

#### Sample Size

The sample size calculation for this trial is based on the pilot study by Schmitz et al. (34) investigating the feasibility of movement sonification in stroke patients. The study found a small, but statistically significant effect and a high correlation between the number of blocks which were transported in the Box and Block test before and after five 20-min sessions with movement sonification. As the intervention period is much longer in this trial, we assume a medium effect. For an effect size of f = 0.2, a correlation among repeated measures of 0.7, a power of 80%, and a significance level of α = 0.05 a total sample size of 26 subjects is required. The sample size calculation was performed using G∗Power.

A dropout rate of 20% was anticipated, consequently a minimum number of 32 subjects has to be enrolled in the study.

#### Recruitment

In the pre-screening, a scientific staff member determines on a daily basis all stroke patients admitted to the hospital. These patients are screened for eligibility by the study coordinator. Patients who fulfill all inclusion and exclusion criteria are approached with the study information. If the patient is interested in the study and agrees to participate, written informed consent is obtained. If the patient has a legal representative, the study information is also provided to the legal representative and he gives written informed consent. Patients who are not yet, but potentially may become eligible, are followed by the study coordinator until they meet all the eligibility criteria.

## Allocation

Patients included in the study are randomly assigned to either the control or the experimental group with a 1:1 allocation as per a computer generated randomization schedule stratified by age (<60 and ≥60 years) and lesion side (left and right sided) using blocks of random sizes. The block sizes will not be disclosed to ensure concealment. The randomization schedule will be concealed until the primary endpoint will be analyzed. The allocation is done by a scientific staff member not directly involved in the project. The staff member sends a form with the allocated intervention to the therapist who is not involved in assessing the outcome measures.

#### Blinding

The information about treatment allocation is not given to the patient in order to ensure blinding as long as possible. However, due to the nature of the intervention, blinding of the patient during the intervention may be difficult. Blinding of the therapist is not possible.

#### Data Management

Data is collected by a blinded assessor using data based case report forms. All data are entered into an electronic database by a scientific staff member at the study site who is not involved in data collection. Original data forms will be kept on file at the study site in locked cabinets. Access to the study data will be restricted to authorized staff members. Incremental back-ups of the electronic database will be performed on a daily basis. The database is protected by a password.

After termination of the study and the data verification, all files will be archived for a period of 10 years.

# Statistical Methods

Descriptive statistics including means and standard deviations or medians and 25th and 75th percentiles for continuous data, and frequencies for categorical data will be determined. The appropriateness of the randomization will be examined by testing for between group differences in demographical and clinical variables (e.g., age, time since stroke, SULCS score).

Among the cases available for analyses, intention-to-treat analyses will be performed. For all outcome measures, the within-subject differences between the baseline and post-test, and the baseline and follow-up test are of central interest in the intervention group compared to the control group. For the primary outcome measure, a repeated measures analysis of variance will be used. Pairwise comparisons will be generated using Tukey's method. A subgroup analysis will be performed to investigate the influence of lesion side. For the secondary ordinal outcome measures, non-parametric statistics will be used. Between-group comparisons (intervention vs. control group) will be performed to compare the short-term changes (baseline post-test) and long-term changes (baseline—follow-up) between groups using Mann-Whitney U-tests. In addition within-group comparisons will be performed using Friedman tests. If the Friedman test showed significant differences, Wilcoxon matchedpairs tests will be used to compare baseline and post, and baseline and follow-up measures. Effect sizes (r) of changes between groups and within-groups will be calculated.

In addition, a per protocol analysis will be done, excluding patients who deviated from the protocol. Missing data will be replaced by the last value carried forward method.

## Data Monitoring

No external monitoring of the trial procedures or data collection processes will occur and no auditing is planned for this trial.

No interim statistical analyses are planned. The study will be stopped if risks emerge which were not known before.

# Harms

In this trial, an adverse event is defined as any untoward medical occurrence in a subject without regard to the possibility of causal relationship. Adverse events will be collected and reported after the patient or his/her legal representative has provided written informed consent and the patients is enrolled in the study until follow-up test. All adverse events are evaluated with regard to the anticipation and severity of the adverse event, and the causal relation to the study intervention or study procedure.

An adverse event which occurs after enrolment but before the intervention is started, will be reported as not related to the study intervention.

An adverse event that meets the criteria for a serious adverse event between study enrolment and follow-up test will be reported to the German Federal Institute for Drugs and Medical Devices (BfArM).

## Research Ethics Approval

This trial is performed according to the World Medical Association Declaration of Helsinki as well as the guidelines for good scientific practice of the German Research Foundation and of the University of Hannover. It has been approved by the Ethics Committee of the Bavarian State Chamber of Physicians.

## Protocol Amendments

Any modifications in the study protocol will be reported to the relevant Ethics Committee and the registration in the German Clinical Trials Register will be amended.

# Consent

The study coordinator introduces the trial to patients who fulfill all the eligibility criteria of the study. The patients also receive an information sheet about the study (informed consent form) and the study coordinator discusses the trial with the patients in light of the information provided. Patients are then able to have an informed discussion with the principle investigator and ask questions. At least 24 h after the informed discussion, the principle investigator obtains written consent from patients willing to participate in the trial. The informed consent involves a confirmation that the patient understands the research and an assurance that their agreement to participate is voluntary. If a patient has a legal representative the informed consent form will also be provided to the legal representative. The legal representative will also have a informed discussion with the principle investigator and gives written informed consent if he agrees with study participation.

# Confidentiality

All administrative and data collection forms are identified by a coded ID number only to maintain patient confidentiality. All records that contain names or other personal identifiers, such as informed consent forms, will be stored separately from study records identified by code number. All study-related information will be stored securely at the study site. Access is limited to the staff involved in quality control and data analysis. The electronic database is password-protected. Data which will be transmitted to co-investigators of the University Hannover for analysis do not include personal identifiers.

## Access to Data

Authorized research staff at the Schön Klinik Bad Aibling will have direct access to the data sets. Project team members at the University Hannover will have access by request. To ensure confidentiality, data dispersed to project team members will be blinded of any identifying participant information.

# Ancillary and Post-trial Care

All participants are inpatients at the rehabilitation hospital. After completion of the study, all patients receive rehabilitation treatments and therapy according to their functional level. No specific post-trial care is planned.

The study site has an insurance to cover for harms associated with the trial. This includes cover for additional health care, compensation, or damages.

# Dissemination Policy

Results of this trial will be disseminated through presentations at scientific conferences and peer-reviewed publications.

# DISCUSSION

Stroke patients often show an impaired spatial and temporal arm control which results in disturbed movement trajectories. Movement sonification is a novel approach to map movement trajectories to sound and provide the patient with real-time auditory feedback. We hypothesize that these method might support the relearning of functional movement patterns after stroke. The goal of this clinical trial is to scrutinize the efficacy of a recently developed method for real-time movement sonification on motor abilities of the paretic upper limb in subacute stroke patients. In addition, it assesses adherence to therapy and adverse events.

The combination of perceptual and motor oriented approaches seems effective to improve motor rehabilitation after stroke (6). While visual feedback training is widelyused, acoustic feedback methods are much less prevalent and insufficiently investigated. The method of movement sonification which is applied in this trial provides the patient with additional auditory feedback about three-dimensional wrist movements in relation to the trunk during regular movement therapy. This perception-oriented approach links to internal movement representations and might address specific proprioceptive mechanisms and support relearning of functional movement patterns. Despite its potential benefits, the method has several possible limitations which have to be discussed as they might influence the study outcome.

The mobile sonification system allows to sonify up to 16 from several hundred movement parameters, concurrently. Therefore, it is highly adaptable to different movement categories. Unfortunately, applicability in arm motor rehabilitation is limited to gross-motor functions, because the motion capture is based on inertial sensor units that do not allow capturing finger or grasping movements. A system for the sonification of handand finger movements to support grasping actions and fine motor skills could be developed in future on the basis of data gloves with an adapted kinematic-auditory framework as the one used in the intended study.

A second limitation results from the calibration procedure in which orientations of sensor units are aligned to orientations of body limbs. Repeated recalibration is necessary, because inertial sensor data tend to drift. Thereto, patients have to take up a pre-defined pose, in which both arms are stretched. Although the pose is standardized and patients are supported by the therapist, inaccuracies have to be expected that induce noise in the kinematic-acoustic mapping. Such noise might reduce the impact of the auditory movement information during the multisensory fusion process with information from other sensory modalities (51). Although a higher accuracy might be achieved with optical motion capture systems, it was decided to base motion capture on inertial sensor units to maintain mobility as well as time efficient motion data processing to minimize latency of auditory feedback.

A third possible limitation concerns the necessity to standardize the kinematic-acoustic mapping inter-individually in the clinical study. It might be argued that a higher efficiency of the method will be achieved by adapting the kinematic-acoustic mapping to each patient individually, since impairments vary inter-individually. But by sonifying spherical coordinates (angles

#### REFERENCES


and standardized amplitudes), inter-individual differences seem to play a minor role and the need to adapt the kinematicacoustic mapping diminishes. However, during the 3 weeks of movement therapy, many different arm movement have to be practiced (uni- as well as bilateral movements, different velocities, cyclic/acyclic etc.), and different movement types might require feedback on different movement parameters to achieve highest efficiency. Thereto, an adapted mapping-strategy might be beneficial.

One major study limitation of the study protocol is the blinding of the patients. Due to ethical reasons, patients are informed in the information sheet and the informed discussion that they will be randomly allocated to one of two treatment groups. They are told that the control group is provided with sham-acoustics which is not related to arm movements. After enrolment in the study, the information about treatment allocation is not given to the patient. However, some patients might notice whether they train with movement sonification or sham-acoustics. As knowledge of group allocation might influence the study outcome, it will be documented if a patient mentions awareness of his treatment group.

The study investigates the effectiveness of movement sonification in patients in the subacute phase after stroke. Further work should determine its effects in acute or chronic stroke patients.

#### AUTHOR CONTRIBUTIONS

GS and JB drafted the manuscript. AOE, CK, T-HH, and FM revised it critically for important intellectual content. AOE developed the sonification. GS and AOE developed the framework for the arm sonification. T-HH contributed to the software-application and sound-synthesis. CK conceived, and JB designed the study. CK, JB, AOE, GS, and FM participated in the development of the intervention. All authors read and approved the version of the submitted manuscript.

#### ACKNOWLEDGMENTS

The publication of this article was funded by the Open Access fund of Leibniz Universität Hannover.

a single-blind, randomized, controlled trial. Arch Phys Med Rehabil. (2001) **82**:961–68. doi: 10.1053/apmr.2001.23982


homogeneity model, unidimensionality, fit of the double monotonicity model, differential item functioning, internal consistency, and feasibility of the stroke upper limb capacity scale, SULCS. Arch Phys Med Rehabil. (2011) **92**:214–27. doi: 10.1016/j.apmr.2010.10.034


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Schmitz, Bergmann, Effenberg, Krewer, Hwang and Müller. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Profiles of Motor Laterality in Young Athletes' Performance of Complex Movements: Merging the MOTORLAT and PATHoops Tools

Marta Castañer 1,2 \*, Juan Andueza<sup>1</sup> , Raúl Hileno<sup>1</sup> , Silvia Puigarnau<sup>1</sup> , Queralt Prat <sup>1</sup> and Oleguer Camerino1,2

<sup>1</sup> National Institute of Physical Education of Catalonia (INEFC), University of Lleida, Lleida, Spain, <sup>2</sup> Lleida Institute for Biomedical Research Dr. Pifarré Foundation (IRBLLEIDA), University of Lleida, Lleida, Spain

Laterality is a key aspect of the analysis of basic and specific motor skills. It is relevant to sports because it involves motor laterality profiles beyond left-right preference and spatial orientation of the body. The aim of this study was to obtain the laterality profiles of young athletes, taking into account the synergies between the support and precision functions of limbs and body parts in the performance of complex motor skills. We applied two instruments: (a) MOTORLAT, a motor laterality inventory comprising 30 items of basic, specific, and combined motor skills, and (b) the Precision and Agility Tapping over Hoops (PATHoops) task, in which participants had to perform a path by stepping in each of 14 hoops arranged on the floor, allowing the observation of their feet, left-right preference and spatial orientation. A total of 96 young athletes performed the PATHoops task and the 30 MOTORLAT items, allowing us to obtain data about limb dominance and spatial orientation of the body in the performance of complex motor skills. Laterality profiles were obtained by means of a cluster analysis and a correlational analysis and a contingency analysis were applied between the motor skills and spatial orientation actions performed. The results obtained using MOTORLAT show that the combined motor skills criterion (for example, turning while jumping) differentiates athletes' uses of laterality, showing a clear tendency toward mixed laterality profiles in the performance of complex movements. In the PATHoops task, the best spatial orientation strategy was "same way" (same foot and spatial wing) followed by "opposite way" (opposite foot and spatial wing), in keeping with the research assumption that actions unfolding in a horizontal direction in front of an observer's eyes are common in a variety of sports.

Keywords: laterality profiles, PATHoops (spatial orientation), MOTORLAT (motor laterality inventory), contralateral synergy, complex movements

## INTRODUCTION

Our bodies are able to move among all kinds of surroundings thanks to hemispheric dominance, which, when linked to orientation in spatial contexts, shapes our usage of laterality with regard to our limbs. Thus, as explained in **Figure 1** below, the human body is anatomically symmetric (bilateral) but functionally asymmetric (contralateral), depending on its movement needs and contextual circumstances (for a review, see Brancucci et al., 2009). These circumstances give rise to different contralateral usages of the two sides of the body, performed mainly by the limbs during

#### Edited by:

Mauro Murgia, University of Trieste, Italy

#### Reviewed by:

Pamela Bryden, Wilfrid Laurier University, Canada Gudberg K. Jonsson, University of Iceland, Iceland

> \*Correspondence: Marta Castañer mcastaner@inefc.es

#### Specialty section:

This article was submitted to Movement Science and Sport Psychology, a section of the journal Frontiers in Psychology

Received: 23 February 2018 Accepted: 18 May 2018 Published: 07 June 2018

#### Citation:

Castañer M, Andueza J, Hileno R, Puigarnau S, Prat Q and Camerino O (2018) Profiles of Motor Laterality in Young Athletes' Performance of Complex Movements: Merging the MOTORLAT and PATHoops Tools. Front. Psychol. 9:916. doi: 10.3389/fpsyg.2018.00916

**57**

the execution of motor actions that define our personal motor laterality profiles. In recent years, a growing number of studies have mentioned laterality in relation to technical, behavioral, physical, and tactical factors in sports (Carling et al., 2005; Hodges et al., 2006), but without delving deeper into this subject or offering specific tools.

In this study, we present two tools suited to the study of laterality profiles and spatial orientation that fit with the assumptions of spatial stimulus-response compatibility and ideomotor action in the framework integrating perception and action addressed within the Theory of Event Coding (Hommel et al., 2001). The existing tools for assessing laterality do not take into account this framework of perception-action integration or the polymorphism of laterality usages and therefore are used primarily for studying the functions of the upper limbs (mainly the hands).

# Richness of Motor Skills and Lateralization Uses

The laterality of the body underpins all motor skills that allow for the richness of movements in everyday situations as well as in specific contexts such as sports (for a review, see Brancucci et al., 2009; Tran and Voracek, 2016). Indeed, laterality must not be reduced to right- or left-handedness, as it is clear that our bodies perform specific and personal uses of lateralization, thereby defining a varied tapestry of motor laterality profiles. Greater research on laterality terms (Gabbard, 1997; Hart and Gabbard, 1998; Westmoreland, 2016) could help to enhance motor performance in all types of movements involving basic, specific, and specialized motor skills, including the mechanical aspects of a technique—the way in which the skill is performed in terms of the kinetic and kinematic details of the movement involved (O'Donoghue, 2010). The roots of fundamental motor skills—locomotor, stability, and manipulation (Gallahue and Cleland-Donnelly, 2003; Castañer et al., 2009, 2015, 2016a) lie in the phylogenetic contribution (Anderson et al., 2001) and their singular characteristics depend on ontogeny (Assaiante and Amblard, 1995; Salesse et al., 2005), with each individual being optimally geared to adapt to multifaceted environments (Johnson, 2007) such as the complex and dynamic context of sports.

Given the dynamism and complex nature of sports, motor laterality profiles detected using specific tools are of interest for the purposes of optimizing athletes' performance of complex movements (Loffing et al., 2015), which are built on complex intentional actions (Murgia et al., 2014; Schaefer, 2014). Laterality refers not only to left-right preference (Hagemann, 2009; Teixeira et al., 2011; Bishop et al., 2013) but also to how an athlete orients his or her body spatially (Bishop et al., 2013; Loffing et al., 2015). In this regard, previous research related to football (Castañer et al., 2016a) demonstrated that Lionel Messi—a leftfooted player—is a good example of laterality, given that he has achieved some of his best results while playing on the right wing. This study showed that Messi "tends to occupy the right midfield and right wing more often than the other parts of the pitch as he moves toward the goal, as this would logically afford him a better angle from which to shoot with his left foot" (Castañer et al., 2016a, p. 8). Although the richness and diversity of sports are due to the high complexity of the athletes' body movements and the contexts in which they perform, general research has exhibited certain flaws: (a) a lack of specific practical tools for observing and detecting a broader range of motor laterality profiles; (b) the simplification of the broad range of right-left and ambidexterity profiles; and (c) a failure to take into account fundamental factors such as spatial orientation and the complementary functions of postural support and gestural precision of the limbs.

# Mastering Contralateral Body Synergy: Merging Gestural Precision and Postural Support

Specific studies from the late 1980s and 1990s (Peters, 1988a; Previc, 1991; Coren, 1993; Hart and Gabbard, 1998) noted that laterality is described in a bilateral context in which the role of one limb is to execute an action while the role of the other limb is to establish postural stabilization. In terms of more detailed motor conceptualization, we refer to these two roles as gestural precision and postural support, respectively. We argue that it is essential to distinguish between these two functions performed by limbs working harmoniously together in contralateral synergy and underpinning a particular motor laterality profile.

Despite the large amount of scientific literature related to laterality, there is scant discussion of the conceptual basis of the constituent elements of motor actions (e.g., motor skills, perceptual and conditioning capabilities, technique, and tactics) that underpin laterality. Therefore, gestural precision and postural support functions are based on the diverse—and, at the same time, bilateral—structure of our corporeity, which enables us to simultaneously generate bodily gestures (dynamism) and postures (staticity) (Castañer et al., 2010a, 2016b). In fact, postural sequences are nested in each body gesture (Out et al., 1998; Gabbard and Hart, 2002).

Optimizing motor actions involves mastering physical activity practices and, consequently, making effective use of laterality, with the aforementioned contralateral body synergy playing a key role. This is clear in elite athletes such as Rafa Nadal, who trained to reverse his innate manual preference in order to obtain an advantage over his opponents, and Lionel Messi, who, despite being left-footed, signs his contracts with his right hand. In previous research (Castañer et al., 2016a, 2017a), we detected the role played by laterality in Messi's extraordinary goal-scoring achievements. We reported, for example, that his right turn with his back to the rival goal line was directly related to the use of his left leg. While remaining steady on his right leg, he turns his body, thereby allowing his left leg to perform precise actions. In that study, which compared the motor skills performed by Messi and Cristiano Ronaldo, the term mastering lateral synergy was used to refer to an athlete's ability to combine the precision of his/her dominant limb with the stability offered by the other, non-dominant limb.

## Limb Dominance and Spatial Orientation

With regard to limb dominance, the scientific literature (Büsch et al., 2010; Edlin et al., 2015) has shown that the inventories used to assess laterality have certain weaknesses, including (a) insufficient differentiation between dominant precision actions (i.e., the foot that kicks a ball) and support actions (i.e., standing on a foot) (Elias et al., 1998; Peters, 1998b; Gabbard and Hart, 2002) and (b) excessive focus on the handedness of human beings (Westmoreland, 2016), to the detriment of other body parts (i.e., Oldfield, 1971; Kelley, 2012; Papadatou-Pastou et al., 2013; Hardie and Wright, 2014). Hence the need for a useful exhaustive inventory designed to assess the laterality of the body as a whole and the versatility and complexity of its motor actions.

Versatility of complex movements in both individual and team sports requires the integration of multiple skills (Bishop et al., 2013) directly linked to motor anticipation (Murgia et al., 2014) and the linkage of behaviors to outcomes for teams and individual athletes (Glazier and Robins, 2013). Complementarily, lateral asymmetry in sports performance is due to greater use of the dominant limb, particularly for complex motor actions such as shooting, and is largely determined by use, habit and technique acquisition (Teixeira et al., 2011; Edlin et al., 2015).

Nevertheless, an exhaustive assessment of laterality must take into account that all kinds of locomotor, stability, and manipulation motor skills are rooted in the acquisition of spatial concepts (Pitchford et al., 2016) such as spatial structuring, organization, and orientation, which are directly related to uses of hemispheric dominance. In any case, this is a complex reality (Edlin and Lyle, 2013) that also fits in with the "moving while perceiving and thinking" line of analysis (Schaefer, 2014). The complex merging of hemispheric dominance and spatial orientation reinforces the framework integrating perception and action that was first addressed within the Theory of Event Coding (Hommel et al., 2001), which aims to improve our knowledge about how complex movements are performed. Furthermore, temporo-spatial information plays a fundamental role in the multifaceted surroundings where complex movements are performed. In this sense, as Murgia et al. (2017) point out, "the combination of temporal information processing and biological movement perception has rarely been addressed by researchers, nevertheless, it represents an interesting research challenge which might reveal how athletes, dancers, and musicians process temporal information related to complex human movements." We found in previous studies that spatial information also has a strong influence on perceptive-cognitive processes in the performance of motor actions in a range of different groups, including children and adolescents (Castañer et al., 2016c) and adults and the elderly (Alves Franco et al., 2013; Saüch and Castañer, 2014; Castañer et al., 2015, 2017b; Puigarnau et al., 2016). As shown in **Figure 1**, laterality is not merely a question of handedness or footedness, but a process that develops in conjunction with the way in which our body uses and orients itself in space (Salesse et al., 2005; Castañer et al., 2012a) and emerges as a factor of perceptual-motor experience.

# Laterality as a Factor of Perceptual-Motor Experience

Performance in sports depends on specific perceptual or anticipatory skills (Williams et al., 1999; Hagemann et al., 2006) that are directly related to managing spatial circumstances. Expert athletes can predict, for example, the direction of an opponent's action earlier and more precisely than novices (Hagemann, 2009) and are more skilled at anticipating actions (Chi, 2006; Hodges et al., 2006). Recent studies on perceptualmotor experience in the mastery of various sports (Murgia et al., 2014, 2016; Woods et al., 2014; Pizzera and Hohmann, 2015; Castañer et al., 2016a, 2017a; Camponogara et al., 2017; Sors et al., 2017) have shown that the observer's perceptual-motor experience is a crucial factor for accurate perception of biological movements (Calvo-Merino et al., 2006; Schütz-Bosbach and Prinz, 2007; Murgia et al., 2016). Research in this line using point-light displays has demonstrated how the observer's perception system fits with the kinematic parameters in specific contemporary dance actions (Castañer et al., 2012a; Torrents et al., 2013) and shown how accuracy can be recognized in spatial representation (Fumarola et al., 2016). Likewise, in a recent study (Castañer et al., 2017b), we found, through a mixed methods research analysis (Anguera et al., 2012, 2017), unexpected interpersonal heart rate synchrony between participants during motor-cognitive tasks, which could be related to the cue factors of the Theory of Event Coding: codes (cognitive structures) and sensorimotor synchronization. Perceptual-motor experience implies the enhancement of cognition (Kenny et al., 2016), but traditional approaches tend to consider cognitive and motor skills in isolation, thus preventing the adoption of an integrative approach. In fact, the ability to efficiently and effectively execute skilled movement patterns—which requires the application of cognitive and motor skills to rapidly changing situations—is the most important aspect of an athlete's performance (Ali, 2011).

## The Present Study

On the basis of the theoretical underpinnings set out above, in the present study we determined how young athletes approach a novel perceptual-motor situation by studying their contralateral uses of the limbs and spatial orientation during the performance of tapping locomotion skills. In parallel, we determined the athletes' laterality profiles by asking them to perform 30 motor skills of increasing complexity that underpin all sorts of complex movements (Camerino et al., 2012). The items were correlated in order to guarantee a perception-action way of detecting these profiles.

With this procedure, we went beyond the traditional procedures for detecting laterality (for a review, see Edlin et al., 2015), which were established on the basis of the terms left-handedness and right-handedness as they are used in sport sciences (for a review, see Tran and Voracek, 2016). Instead, we used the term motor laterality profile—right, left, or mixed—which encompasses the whole body, taking into account the lateral synergy that merges postural support and gestural precision (Castañer et al., 2017a).

In sum, we believe that the determination of laterality profiles should include a more detailed study of laterality in relation to the performance of the fundamental and specific motor skills that make up complex movements. Thus, the overall objective of this study was to obtain a broad view of motor laterality profiles by applying two complementary instruments, one which analyzes the contralateral distribution of postural support and gestural precision in a broad spectrum of motor skills (from simple to complex), and another which allowed us to detect spatial orientation by presenting participants with a novel motor situation that activated an ideomotor action as an empirical domain of the perception-action integration framework.

# MATERIALS AND METHODS

## Participants

A total of 95 young athletes (73 males, 22 females) ranging in age from 17 to 26 years (Mage = 19.7 years; SD = 2.01) provided informed consent and participated in the study, which was approved by the ethics committee at the University of Lleida, Spain (code CEIC-1665). Participants were taking part in a program to improve their physical capabilities and motor skills. As part of this program, they signed up for the two tasks included in this study. Participants were required to have practiced their sport—by training or competing—for at least the previous 6 months. Those who were injured at the time of data collection or in the previous month were excluded.

# Materials

#### MOTORLAT: An ad-hoc Motor Laterality Inventory

To detect laterality profiles from motor skills performance, we designed a motor laterality inventory called MOTORLAT (**Table 1**) as an optimized extension of previous research (Castañer et al., 2012a) and we applied measures of inter-rater agreement. MOTORLAT comprised four criteria based on the motor skills-related criteria from the Motor Skills Observation System (OSMOS) instrument (Castañer et al., 2009, 2012a). These four criteria were as follows: (1) locomotion skills, referring to actions that require the body to travel from one point to another across space; (2) stability skills, referring to actions that do not require the body to travel from one point to another across space (i.e., jumping, balancing, and turning); (3) manipulation skills, referring to actions that require the manipulation of objects or other people with the limbs of the body; and (4) combined skills, referring to actions that combine one or more of the aforementioned criteria. Each criterion was expanded to build an exhaustive and mutually exclusive total of 30 items of fundamental and combined motor skills (12 related to the lower limbs, 9 related to the upper limbs, and 9 related to the direction taken to execute an action). Moreover, next to each item, there was a clear description of the aspect to evaluate and the boxes for left and right were arranged intuitively for the observer.

#### PATHoops: The Precision and Agility Tapping Over Hoops Task

Precision and Agility Tapping over Hoops (PATHoops) consisted of the following task. Participants, standing on both feet, were asked to perform a path by stepping in each of 14 hoops arranged in a triangular shape on the floor. In addition, participants were asked to perform the PATHoops task from both sides (**Figure 2**). Performing the task from both sides is in keeping with the assumption of Loffing et al. (2016) that actions unfolding in a horizontal direction in front of an observer's eyes are common to a variety of sports. To measure PATHoops, the researchers recorded the strategies used by the participants: (a) Same way: The athlete goes to the same wing as the foot used in the first step (e.g., right-right); (b) Opposite way: The athlete goes to the opposite wing as the foot used in the first step (e.g., left-right); or (c) Other: The athlete performs some other type of spatial orientation strategy. Given that novelty in motor situations involving fundamental acquired skills guarantees a spontaneous stimulus-response, thus preventing the use of automatic or rehearsed responses (Hommel et al., 2001; Castañer et al., 2010b, 2011, 2012b, 2016c, 2017b; Stöckel and Weigelt, 2012; Torrents et al., 2013), we designed PATHoops to be a novel situation involving locomotor skills. We decided to focus on the locomotor skill of walking quickly—i.e., feet-tapping—because this is a fundamental and automatic motor skill and because it involves multisensory information such as vestibular, visual, and

#### TABLE 1 | MOTORLAT motor laterality inventory.


(Continued)

#### TABLE 1 | Continued


kinesthetic information (Hart and Gabbard, 1997; Gabbard and Hart, 2002; Santoro et al., 2017).

#### Procedure

Two researchers—experts in physical activity and sports administered the instruments to all participants one by one in a sports facility divided by a curtain into zone A and zone B. In order to guarantee that none of the participants had worked out immediately before data collection, all participants attended a non-practical session in a classroom at the facility. Participants were called, one by one, into zone A, where a researcher administered the PATHoops task. Once finished, participants proceeded to zone B, where they completed the MOTORLAT inventory under the supervision of a second researcher.

The materials required for the application of MOTORLAT are as follows: a chair, a step or stairs, and a foam ball. Participants indicated their age, gender, and sports specialty on a data entry form. For each participant, the observer administered the 30 items one at a time, in the indicated order, and checked the box corresponding to the limb (left or right) that the participant used to execute the aspect being evaluated. The observers stated the wording of the items loudly and clearly, and materials were provided to the participant as required for each inventory item.

The materials required for the application of PATHoops are as follows: 14 hoops, each measuring no more than 50 cm in diameter, arranged on the floor in the shape of a triangle (**Figure 2**). Participants had to perform the task twice, first from the narrow side and then from the wide side, allowing the researchers to observe which foot the participant used to start the task, as well as the participant's left-right preference and spatial orientation. The spatial orientation strategies used by the athletes after the first step to complete the PATHoops task are included in the results section.

The researchers did not perform any examples or models of the PATHoops task or demonstrate any of the motor skills included in the MOTORLAT items.

## Data Analysis

Firstly, measures of inter-rater agreement with standard errors and confidence intervals were used to validate the MOTORLAT instrument. This validation was carried out by 35 international experts on physical activity and sports. The 30 items were validated (Wongpakaran et al., 2013; Gwet, 2014) using a Likert scale of 1–3 with the following criteria: unambiguity, appropriateness, and relevance (**Table 2**). Laterality profiles were obtained by means of cluster analysis. As an internal assessment of these clusters, a correlational analysis was carried out for each cluster between the motor skills of the MOTORLAT items. A contingency analysis was used to cross the limb dominance criteria from the MOTORLAT inventory and their relationships with the spatial orientation criteria from the PATHoops task.

## RESULTS

In accordance with the declared objectives of the study, our results were as follows: (a) motor laterality profiles were obtained by analyzing contralateral distribution of postural support and gestural precision for a broad spectrum of motor skills (from simple to complex); (b) spatial orientation was detected from a novel motor situation in which participants were asked to activate an ideomotor action. We present our results in three sections: (a) Motor laterality profiles obtained; (b) Related motor skills in lateralization uses; (c) Spatial orientation and laterality profile.

## Motor Laterality Profiles Obtained

Laterality profiles were obtained by means of cluster analysis and subsequent correlational analyses were carried out. Cluster analysis showed that the criteria of locomotion, stability and manipulation reveal clear motor laterality profiles (**Table 3**): (1) ambidexterity (1%), (2) left laterality (6%), (3) right laterality (74%), and (4) mixed laterality (19%) (meaning that the athletes perform locomotion and stability motor skills with the left lower

#### TABLE 2 | Measures of inter-rater agreement with standard errors and confidence intervals.


Number of raters = 35; number of items = 30; number of rating categories = 3 (unambiguity, appropriateness, relevance).

#### TABLE 3 | Athlete profiles by dimension of laterality.


Cluster analysis showed that the criteria of locomotion, stability and manipulation reveal clear motor laterality profiles: (1) ambidexterity (1% of participants), (2) left laterality (6%), (3) right laterality (74%), and (4) mixed laterality (19%).

limb and in a leftward direction but perform manipulation skills with the right upper and lower limbs). This mixed laterality is made clear by an in-depth analysis of complex movements—for example, those which include the action of jumping. A cluster analysis of jumping skills reveals four profiles (**Table 4**). Most athletes usually used their right hand to touch an elevated object, orienting their body to the left side. The first profile (cluster 1) corresponds to athletes who raised their right hand and foot during the jump. The second and the third profiles (clusters 2 and 3) indicate an inverse relationship between the right hand and left foot, although the direction of the turn varies. Finally, the fourth profile (cluster 4) corresponds to athletes who raised their left hand and right foot.

#### Related Motor Skills in Lateralization Uses

Correlational analysis showed that the locomotion categories involving the first step taken to perform an action were significantly correlated (**Table 5**), the stability categories

#### TABLE 4 | Laterality profiles and jumping skills.


The first profile (cluster 1) corresponds to athletes who raised their right hand and foot during the jump. The second and the third profiles (clusters 2 and 3) indicate an inverse relationship between the right hand and left foot, although the direction of the turn varies. Finally, the fourth profile (cluster 4) corresponds to athletes who raised their left hand and right foot.

involving turn direction were significantly correlated (**Table 6**), and the manipulation skills with upper and lower limbs were significantly correlated (**Table 7**).

#### TABLE 5 | Significant correlations between specific motor skills involving locomotion skills.


\*\*The correlation is significant at p < 0.01 (two-tailed).

TABLE 6 | Significant correlations between specific motor skills involving stability skills.


\*\*The correlation is significant at p < 0.01 (two-tailed).

In relation to the fourth criterion—combined skills— **Table 8** shows a significant correlation among the skills that required a change of spatial direction. In contrast, this correlation is not as evident for activities that required jumping. Additionally, the fourth MOTORLAT criterion differentiates athletes' uses of laterality by type of sport. The motor skills that best explain uses of laterality by sport are turn direction and jumps. The frequencies shown in **Table 9** suggest that athletes in various sports—with the exception of gymnastics—tend to prefer performing turns to the left.

#### Spatial Orientation and Laterality Profile

The spatial orientation strategies used by athletes after the first step to complete the PATHoops task were classified as follows:


the hoops anarchically, which results in mistakes such as failing to step in some hoops or stepping in some hoops more than once.

**Table 10** shows that the most common spatial orientation strategy used by the athletes after the first step was "same way," followed by "opposite way."

## DISCUSSION

The aim of this study was to use the complementary tools of MOTORLAT and PATHoops to perform an objective analysis of young athletes' use of laterality in an increasingly complex range of motor skills and spatial orientation tasks involving a novel motor situation. To obtain a broad interpretation from the two instruments, we have conducted a contingency analysis to cross the limb dominance criteria from the MOTORLAT inventory and their relationships with the spatial orientation criteria from the PATHoops task. The discussion section is structured in the following sections: (a) Athletes' laterality profiles; (b) Laterality profiles and sport specialization; and (c) Spatial orientation and laterality profile (based on the findings from PATHoops crossed with the profiles obtained from

#### TABLE 7 | Significant correlations between specific motor skills involving manipulation skills.


\*\*The correlation is significant at p < 0.01 (two-tailed).

\*The correlation is significant at p < 0.05 (two-tailed).

TABLE 8 | Significant correlations between specific motor skills involving a change of direction.


\*\*The correlation is significant at p < 0.01 (two-tailed).

\*The correlation is significant at p < 0.05 (two-tailed).

MOTORLAT). Each section ends with clues about how coaches, educators and athletes can better understand how the laterality of the whole body and limbs underpins the diversity of motor skills used in sports and improve the performance of complex movements.

#### Athletes' Laterality Profiles

Despite the anatomical symmetry of the body, humans exhibit a broad range of asymmetric usage of the limbs in the execution of motor actions (Palmer, 2004; for a review, see Brancucci et al., 2009). This evidence supports an integral perspective on the whole body that takes into account the contralateral synergy that combines postural-support and gestural-precision functions (Castañer et al., 2017a). We have therefore gone beyond the terms handedness and footedness as used in sports science (for a review, see Tran and Voracek, 2016), focusing instead on the concept of motor laterality profile. To move forward with this concept and offer MOTORLAT as a suitable inventory for assessing laterality, we based the tool on fundamental motor skills—locomotor, stability, and manipulation (Gallahue and Cleland-Donnelly, 2003; Castañer et al., 2009, 2015) which have their roots in the phylogenetic contribution, display personalized ontogeny (Anderson et al., 2001; Salesse et al., 2005) and are geared toward adapting to multifaceted surroundings (Johnson, 2007) not only in sports but also in everyday life.

The MOTORLAT items used in this study allowed us to observe these contralateral uses, since the cluster analysis involving the three first criteria—locomotion, stability, and manipulation motor skills—reveal four clear motor laterality

#### TABLE 9 | Frequencies involving turn direction skills by sport.


TABLE 10 | Contingency table of spatial orientation strategies used after the first step.


profiles: ambidexterity (1.1%), left laterality (2.6%), right laterality (3.72%), and mixed laterality (18%). Mixed laterality corresponds to athletes who perform locomotion and stability motor skills with the left lower limb and in a leftward direction but perform manipulation skills with the right upper and lower limbs.

As mentioned above, hand preference is a long-studied topic, including in the field of sports science. It has been suggested that left-handers may have an advantage over righthanders in various interactive sports, as demonstrated, for example, in our study of Lionel Messi's motor skill expertise in goal-scoring (Castañer et al., 2016a). However, as noted above, the left-footed Lionel Messi signs his contracts with his right hand. Even without knowing the full motor laterality profile of the best sportsman exhibiting mixed laterality, our results, obtained using the MOTORLAT combined motor skills criterion, differentiate athletes' uses of laterality by type of sport, showing a clear tendency toward mixed laterality profiles.

#### Laterality Profiles and Sport Specialization

The fourth MOTORLAT criterion—combined skills—is the best criterion for explaining the use of laterality in sports because the most complex movements tend to include the actions of turn direction and jumping. The results show that athletes in various sports perform turn direction mainly to the left. The results of a cluster analysis on jumping skills show that most athletes usually use their right hand to touch an elevated object, orienting their body to the left side. These results also fit with the cluster mentioned in the laterality profiles section, in which athletes with a mixed laterality profile (18%) perform locomotion and stability motor skills with the left lower limb and in a leftward direction but perform manipulation skills with the right upper and lower limbs.

These results clearly offer more evidence to support the argument that "postural support enables stasis and blocks movement, which allows the zone involved in gestural precision to execute the dynamics of the corresponding motor action" (Castañer et al., 2012a, p. 133). In another study by Castañer et al. (2017a), this quality was observed in Messi, who, with his back to the goal, would turn on his right leg, leaving his left leg to execute the goal-scoring action with greater precision. However, since laterality does not refer only to left-right preference (Velotta et al., 2011; McGrath et al., 2015) but also to how players orient their bodies spatially (Castañer et al., 2012a; Loffing et al., 2015), our study supports the notion that mixed laterality profiles enhance the performance of complex movements in athletes, adding value advantages in sport sciences (Tran and Voracek, 2016).

We therefore advise athletes, coaches and teachers that the successful use of certain patterns of mixed laterality promotes versatility of movement and could be used to enhance expertise in the performance of complex technical movements (Murgia et al., 2014; Schaefer, 2014). For example, stability skills are a versatile aspect because jumps, turns, balancing actions, and swinging actions serve to redistribute body weight, to play with gravity, or to prepare for or initiate the next move. Therefore, if an athlete uses his or her left leg for postural support to allow the right leg perform precise actions—such as those mentioned above—this contralateral use is the best option.

#### Spatial Orientation and Laterality Profile

The acquisition of spatial concepts (Pitchford et al., 2016) is a process directly related to uses of hemispheric dominance. The complex merging between hemispheric dominance and spatial orientation reinforces the framework integrating perception and action that was first addressed within the Theory of Event Coding (Hommel et al., 2001), which is related to assumptions such as spatial stimulus-response compatibility, sensorimotor synchronization, and ideomotor action. If hemispheric dominance is directly related to how one's body performs motor skills, having to manage the spatial orientation of the body is an allocentric point of view: "Spatial updating allows people to keep track of the self-toobject relations during movement" (Santoro et al., 2017). In PATHoops, participants are asked to perform a path by stepping in each of 14 hoops arranged on the floor, allowing researchers to observe their feet, their left-right preference and their spatial orientation. This task allowed us to achieve our objective of detecting spatial orientation from a novel motor situation that required participants to activate an ideomotor action as an empirical domain of the perception-action integration framework. It also fit with the assumption that the acquisition of locomotor skill is linked to developmental changes in an infant's ability to regulate posture on the basis of information available in patterns of optic flow (Anderson et al., 2001).

Our results show that the best strategies for the PATHoops task, after the first step, were "same way" (the athlete goes to the same wing as the foot used in the first step, e.g., rightright) followed by "opposite way" (the athlete goes to the opposite wing as the foot used in the first step, e.g., leftright). The most commonly used spatial orientation strategy was "same way," followed by "opposite way," in keeping with the assumption of Loffing et al. (2016) that actions unfolding in a horizontal direction in front of an observer's eyes are common in a variety of sports. These findings are consistent with the findings of previous research (Castañer et al., 2016a, 2017a).

We encourage athletes, coaches, and teachers to use tasks like PATHoops, which participants—not only athletes but also people of various ages and motor capabilities must perform from both sides using locomotor skills. This guarantees a spontaneous stimulus-response, thereby avoiding previous automatic or rehearsed responses (Hommel et al., 2001; Castañer et al., 2010b, 2011, 2012b; Stöckel and Weigelt, 2012; Torrents et al., 2013). Despite involving the use of a fundamental and automatic locomotor skill—walking quickly—the novelty of the PATHoops task requires the use of multisensory information such as vestibular, visual, and kinesthetic information (Santoro et al., 2017). This could be suitable in decision-making, for example, when athlete must execute a feint or react to an object and decide which direction to take and, therefore, which foot to use to support his or her bodyweight and which foot to use to begin the movement.

## CONCLUSIONS AND FUTURE LINES OF STUDY

The objective of this study was to further our understanding of body laterality, taking into account the two main functions combined by the upper and lower limbs of the body—precision and support—as well as the spatial direction and orientation of the body. To achieve this objective, we used a combination of two instruments—the MOTORLAT inventory and the PATHoops task—to describe the "tapestry" of motor skills and contextual aspects that make up the singular style of each participant. In particular, spatial orientation and turning and jumping which demand more complexity of movement—are described in this study. The 30 MOTORLAT items cover a range of movements from simple to complex motor skills, allowing experts to choose which ones might be of interest. The PATHoops task is a good complement for observing the spatial orientation strategies employed by participants. We consider that both instruments are a good fit for motor laterality studies because there is a need for deeper study of the motor skills underpinning the complex movements (Murgia et al., 2014; Schaefer, 2014) that athletes use in sports. As for the practical implications of this study, we would like to highlight the following:


With this study, we hope to have contributed to extending the research on motor laterality and spatial orientation. We agree with Loffing et al. (2016) that mixed laterality in sports has become a recent focus of research and requires more extensive study in order to explore contralateral functions in motor skill acquisition, technique learning, and dealing with complex movements in order to optimize performance in sports (Murgia et al., 2014; Schaefer, 2014).

From a methodological point of view, the use of two instruments that combine bilateral limb usages and spatial orientation for a broader assessment of laterality is in keeping with the mixed methods approach, which combines techniques to offer a better way of achieving objectives (e.g., Creswell, 2015). Given that we generally conduct our research using mixed method designs that combine qualitative and quantitative data and analytical techniques such as triangulation, embedded, or explanatory designs—in a parallel or sequential way (Anguera et al., 2014, 2017), we encourage researchers to move beyond the use of a single instrument and embrace the combination of multiple instruments to find a better way of achieving research objectives.

#### AUTHOR CONTRIBUTIONS

MC developed the project, supervised the design of the study, the method section, and the drafting of the manuscript. SP was responsible for the review of the literature. OC was responsible for the critical revision of the content. RH performed the validation of the laterality inventory and the drafting of the manuscript. JA was responsible for the drafting of the manuscript and collected and codified the data. QP supervised the drafting of the manuscript. All authors approved the final, submitted version of the manuscript.

# REFERENCES


# FUNDING

We gratefully acknowledge the support of INEFC (National Institute of Physical Education of Catalonia), and the IRBLLEIDA (Lleida Institute for Biomedical Research Dr. Pifarré Foundation), University of Lleida, Lleida, Spain and the support of two Spanish government projects (Ministerio de Economía y Competitividad): (1) La Actividad Física y el Deporte Como Potenciadores del Estilo de Vida Saludable: Evaluación del Comportamiento Deportivo Desde Metodologías no Intrusivas (Grant number DEP2015-66069-P); (2) Avances Metodológicos y Tecnológicos en el Estudio Observacional del Comportamiento Deportivo (PSI2015-71947-REDP); and the support of the Generalitat de Catalunya Research Group, Grup de Recerca i Innovació en Dissenys (GRID). Tecnología i Aplicació Multimedia i Digital als Dissenys Observacionals (Grant number 2017 SGR 1405).


Sport, Physical Education and Dance, eds O. Camerino, M. Castañer, and M. T. Anguera (Abingdon: Routledge), 146–176.


functional performance - a systematic review. J. Sport Sci. 34, 289–302. doi: 10.1080/02640414.2015.1050601.


reliability coefficients: a study conducted with personality disorder samples. BMC Med. Res. Methodol. 13:61. doi: 10.1186/1471-2288-13-61

Woods, E. A., Hernandez, A. E., Wagner, V. E., and Beilock, S. L. (2014). Expert athletes activate somatosensory and motor planning regions of the brain when passively listening to familiar sports sounds. Brain Cogn. 87, 122–133. doi: 10.1016/j.bandc.2014.03.007

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Castañer, Andueza, Hileno, Puigarnau, Prat and Camerino. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Musical Training in Congenital Hearing Impairment. Effects on Cognitive and Motor Skill in Three Children Using Hearing Aids: Pilot Test Data

Sara Ghiselli <sup>1</sup> \*, Elena Ciciriello<sup>1</sup> , Giovanni Maniago<sup>2</sup> , Enrico Muzzi <sup>1</sup> , Sandra Pellizzoni <sup>3</sup> and Eva Orzan<sup>1</sup>

*<sup>1</sup> Department of ENT and Audiology, Institute for Maternal and Child Health IRCCS Burlo Garofolo, Trieste, Italy, <sup>2</sup> Prematuri Si Nasce Association, Pordenone, Italy, <sup>3</sup> Department of Life Science, University of Trieste, Trieste, Italy*

Keywords: congenital hearing impairment, hearing aid, early intervention, cognitive ability, musical training

#### INTRODUCTION

#### Edited by:

*Penny McCullagh, California State University, East Bay, United States*

#### Reviewed by:

*Frank A. Russo, Ryerson University, Canada Laura Traverso, University of Genoa, Italy*

\*Correspondence: *Sara Ghiselli sara.ghiselli@burlo.trieste.it*

#### Specialty section:

*This article was submitted to Movement Science and Sport Psychology, a section of the journal Frontiers in Psychology*

Received: *27 February 2018* Accepted: *04 July 2018* Published: *24 July 2018*

#### Citation:

*Ghiselli S, Ciciriello E, Maniago G, Muzzi E, Pellizzoni S and Orzan E (2018) Musical Training in Congenital Hearing Impairment. Effects on Cognitive and Motor Skill in Three Children Using Hearing Aids: Pilot Test Data. Front. Psychol. 9:1283. doi: 10.3389/fpsyg.2018.01283* Congenital hearing impairment (CHI) is one of the most common sensory deficits (Davis and Davis, 2011) with significant repercussions on cortical brain development and behavior (e.g., Pisoni et al., 2008; Kral, 2013). Studies have shown that cortical auditory pathways develop largely on the basis of sensory experience (Kral et al., 2000; Sharma et al., 2002; Kral and Eggermont, 2007). Late treatment and rehabilitative intervention may therefore critically affect neuropsychological and language development, preventing full access to educational opportunities and hindering participation and self-reliance (Kral, 2013).

Restoration of auditory functions with current hearing device technology (i.e., digital hearing aids and cochlear implants) together with auditory training therapy are crucial to help children learn to understand how to interpret auditory signals, form meaningful sound representations, and develop their own listening strategies. Nevertheless, children with CHI still appear to lag behind their normal-hearing peers in terms of linguistic, cognitive, and motor skills (Kral and O'Donoghue, 2010).

Recent findings indicate that brain regions involved in the processing of music and language at cortical (Tillmann et al., 2006; Koelsch et al., 2009) and subcortical (Strait and Kraus, 2014) level tend to overlap, thus showing a significant degree of affinity between music and language (Patel, 2008). Music training has therefore been increasingly offered to both children with typical development (Besson et al., 2011; Kühnis et al., 2013) and children with various types of atypical development, including individuals with auditory impairments (Yucel et al., 2009; Torppa et al., 2014). Research in this field yielded the following results: (1) children with implants or hearing aids display poorer music perception skills compared to their peers (Scorpecci et al., 2012) but this seem to be related to differences in acoustical hearing prior to cochlear implantation (Hopyan et al., 2012); (2) however, when offered musical training their perception of musical acoustic features improves (e.g., Chen et al., 2010; Stabej et al., 2012; Dastgheib et al., 2013; Rochette et al., 2014; Good et al., 2017). This transfer effect of musical training seems to affect not only language (Yucel et al., 2009; Torppa et al., 2014), but also other cognitive functions (Moreno et al., 2011; Rochette et al., 2014; Benz et al., 2016). Studies mentioned so far present the following limits; (1) their main focus is on primary school children and (2) they investigate the effect of musical training on phonetic awareness, language, and memory (auditory or digit span task), disregarding all other cognitive abilities potentially influenced by music. "Even if the effect of music training on non-auditory

**71**

findings are preliminary and in need of further corroboration, musical program seem to enhance working memory (Bergman Nutley et al., 2014; Zuk et al., 2014), motor and visual-spatial skills (Benz et al., 2016), and visual attention (Roden et al., 2014; Rodrigues et al., 2014)." This case report focuses on non-instrumental musical training offered to three pre-school children with CHI with the aim of investigating its effects on complex cognitive functions governing language development, such as memory, attention, and motor skills.

# BACKGROUND

The study was carried out at the Audiology Unit of the Institute for Maternal and Child Health—IRCCS "Burlo Garofolo" of Trieste, Italy. Inclusion criteria for patients were set as follows: (1) CHI with mean auditory threshold between 50 and 70 dB HL, corrected with bilateral digital hearing aids; (2) age between 2 and 4 years; (3) absence of cognitive impairment; (4) permanent abode in the area of the Friuli-Venezia-Giulia region. The selection process resulted in the identification of seven eligible candidates for the study, although only three of them took part in the actual activities. The other four lived too far away from the rehabilitation center to be able to attend the training sessions. The Ethical committee of the Institute approved the project (protocol number 614/2016) and informed consent was signed by both parents of each participating child. The three participants were two girls (L.D. and S.V.I.) and one boy (N.C.). At the beginning of their musical training all three of them were aged between 35 and 44 months, had attended nursery school, and had been admitted to their first year of kindergarten. The three participants were also receiving speech-language therapy, which was not interrupted during the study.

#### Measures

Individual neuropsychological evaluation was carried out before, immediately after and 6 months after the conclusion of the musical training program (MTP). Testing took place in a quite room of the hospital during 60-min sessions. In the following paragraphs each tested task is illustrated, together with the respective evaluation tool.

## Attention

Attention was measured using the Leiter-R Attention Sustained task (Roid and Miller, 1997), which ensures good internal consistency (Cronbach's α = 0.83) and good test-retest reliability (r = 0.85).

# Memory

Memory abilities were measured using the NEPSY tool, a neuropsychological assessment tool designed for children of 3– 12 years of age. NEPSY psychometric properties have proved satisfactory (Korkman et al., 2011). The following NEPSY tasks were selected for our study:

(1) Memory for Designs: this subtest is designed to assess spatial memory by presenting new visual material. It uses a 1– 20 scale (M = 10, SD = 3), with good psychometric properties as regards validity and reliability (see Korkman et al., 2011).

(2) Narrative Memory: This subtest is designed to assess recognition and (free or cued) recollection of organized verbal material. It uses a 1–20 scale (M = 10, SD = 3), with good psychometric properties as regards validity and reliability (see Korkman et al., 2011).

# Motor Skills

The NEPSY task used was the Manual Motor Sequences subtest, which is designed to assess the ability to imitate a series of rhythmic movement sequences using one or both hands. It uses a 1–20 scale (M = 10, SD = 3), with good psychometric properties as regards validity and reliability (see Korkman et al., 2011).

Evaluation was integrated with two musical ability surveys, administered to the children's parents and music teacher, respectively. The two surveys are described in the following paragraphs.

#### Musical Ability Survey Administered to the Children's Parents

The survey is a 26-item questionnaire; responses were rated on a Likert-type scale ranging from 1 (never) to 5 points (always). Questions concern five areas of musical ability: (a) general reaction and awareness of sounds; (b) music exposure; (c) melody and dynamic variations; (d) rhythm variations; (e) emotional response (Yucel et al., 2009). The total score was calculated as the sum of questionnaire responses over the full range of the scale, with the highest possible score set at 155 points.

#### Musical Ability Survey Administered to the Children's Music Teacher

The survey is a 15-item questionnaire designed by the music teacher. Responses were rated on a Likert-type scale ranging from 1 (never) to 5 points (always). Questions concern various areas of musical training (posture, synchronization, distinction between binary and ternary forms, identification of duration, reading rhythmic notation). The total score was calculated as the sum of questionnaire responses over the full range of the scale, with the highest possible score set at 75 points (Piatti, 1993; Gordon, 2003).

#### Musical Training

Musical training was organized in two separate sessions with a 2-month break in-between; each session consisted of 10 individual encounters (25 min) and 10 group encounters (45 min). Encounters took place twice per week, for a total of 20 individual encounters and 20 group encounters. This training relied primarily on listening activities and did not involve the acquisition of any specific instrumental program. Activities provided a combination of motor, perceptual, and cognitive tasks**,** and included training in movement, rhythm, melody, sound-language production, and basic musical concepts.

In the following paragraphs each clinical case is presented individually.

# Clinical Cases

#### N.C. Clinical Case

N.C. was diagnosed with bilateral sensorineural CHI of moderate-to-severe degree. He had been fitted with behindthe-ear digital hearing aids since he was 3 months old. N.C. started MTP when he was 44 months old and attended 36 encounters out of 40. Before the MTP, N.C. had scored 107 in the Short Leiter QI index (Roid and Miller, 1997). Results of the neuropsychological tests administered at the beginning of the MTP (pre-MTP), immediately after (post-MTP), and during the follow-up visit (follow-up) are illustrated in **Figure 1** and described in the following paragraphs.

Pre-MTP: N.C. performed very poorly in the narrative memory neuropsychological test of recognition and recall. Results obtained in the memory for drawings and motor sequences test were within the mid-lower average score. Conversely, N.C. performed well in the visual attention test, with results within the higher average bracket. Musical ability surveys returned a score of 78 points out 155 and 5 out of 75 when filled out by the parents and the music teacher, respectively.

Post-MTP: all weighted scores displayed an increase of at least two points. N.C. still performed poorly in the narrative memory test, although his post-MTP results were within the lower average score. Results obtained in the memory for drawings and motor sequences test were significantly higher in the post-MTP phase, ranking within the higher average score. N.C.'s performance in the visual attention test was above the average. Musical ability surveys returned a score of 97 points out 155 and 57 out of 75 when filled out by the parents and the music teacher, respectively, bearing further evidence of the child's improvement.

Follow up: all scores displayed a decrease, with the sole exception of results obtained in the motor coordination test. More specifically, N.C.'s performance in the narrative memory test of recognition and visual attention regressed to their pre-MTP values. Weighted scores obtained in the narrative memory test of recollection and memory for drawings test registered a slight decrease. On the contrary, N.C. obtained 133 points out of 155 in the musical ability survey administered to his parents, thus showing a consistent learning trajectory by indirect evaluation.

#### L.D. Clinical Case

L.D. was diagnosed with bilateral sensorineural CHI of severe degree. She had been wearing hearing aids since she was 3 months old. L.D. started her MTP when she was 35 months old and attended only 26 encounters out of 40. Before the MTP, L.D. had scored 105 in the Short Leiter QI index (Roid and Miller, 1997). Results of the pre- and post-MTP neuropsychological tests are illustrated in **Figure 2** and described in the following paragraphs.

Pre-MTP: L.D.'s initial deficit was so severe that the narrative memory test of recognition and recollection could not be carried out. Results obtained in the memory for drawings and motor sequences test were within the average, whereas L.D.'s scores in the visual attention test could only reach the lower average score. Musical ability surveys returned a parental score of 105 points out 155 while only 3 out of 75 points in the case of the music teacher scoring.

Post-MTP: all weighted scores displayed an increase of at least five points, with the sole exception of L.D.'s results in the memory for motor sequences test. L.D. performed poorly in the narrative memory test, with post-MTP recognition scores within the lower average score. However, her post-MTP recollection results were within the average. Results obtained in the memory for motor sequences test are consistent with scores obtained in the pre-MTP phase. L.D.'s performance in the visual attention test and memory for drawings test improved, with scores ranking within the higher average score. Musical ability surveys returned a score of 150 points out 155 and 32 out of 75 when filled out by his parents and the musical teacher, respectively, bearing further evidence of the child's improvement.

FIGURE 2 | Cognitive and motor abilities performed by L.D. before the training (pre-MTP), just after the training (post-MTP) and 6 months after the training (follow-up). (*M* = 10; *SD* = 3).

FIGURE 3 | Cognitive and motor abilities performed by S.V.I. before the training (pre-MTP), just after the training (post-MTP) and 6 months after the training (follow-up). (*M* = 10; *SD* = 3).

Follow up: all weighted scores display a decrease, although registered values remain within the average. L.D. obtained 133 points out of 155 in the musical ability survey administered to her parents, thus showing a consistent learning trajectory.

#### S.V.I. Clinical Case

S.V.I. was diagnosed with bilateral sensorineural CHI of moderate degree. She had been wearing hearing aids since she was 3 months old. S.V.I. started the MTP when she was 38 months old and attended 40 encounters out of 40. Before the MTP, S.V.I. had scored 123 in the Short Leiter QI index (Roid and Miller, 1997). Results of the pre- and post-MTP neuropsychological tests are illustrated in **Figure 3** and described in the following paragraphs.

Pre MTP: results obtained in the memory for drawings and motor sequences test were within the mid-lower average bracket. S.V.I. obtained average scores in all the other tests. Musical ability surveys returned a parental score of 155 points out 155 and 19 out of 75 points in the case of the music teacher scoring.

Post-MTP: all weighted scores were in the mid-higher average score. S.V.I.'s performance in the memory for drawings and motor sequences test, which had obtained the lowest scores in the pre-MTP phase, registered the most significant improvement, with an increase of six points. Musical ability surveys confirmed a score of 155 points out 155 in the case of S.V.I.'s parents, while the music teacher reported an improvement with a scoring of 54 out of 75.

Follow up: S.V.I.'s weighted scores obtained in the memory for drawings and motor sequence test registered a slight increase compared to her post-M.T. values. Results obtained in the visual attention test and narrative memory test of recognition registered an improvement consistent with the scores obtained in the post-M.T. phase. S.V.I.'s performance in the narrative memory test of recollection indicates a stability in her learning process, with scores lower than those obtained in the pre-M.T. phase, although still within the higher average score. Parent again confirmed a scoring of 155 points out of 155 in the musical ability survey.

## DISCUSSION

Our case report is an initial attempt at quantifying the beneficial effects of non-instrumental musical training on pre-school children with CHI. Tests aimed at investigating various cognitive functions, including memory, attention, and motor skills. Musical training was specifically designed to enhance children's attentional and auditory skills, focusing on the recognition of sound-related aspects, such as timbre, intensity, duration, and pitch, as well as the reproduction of rhythmic models and the combination of rhythm and melody. Investigated learning patterns also included neural-motor skills and sound-gesture coordination skills. Musical training proved beneficial in all the analyzed cognitive areas. More specifically, narrative memory, whose deficit was most significant in all three patients due to CHI repercussions on language production, registered significant improvements, with post-MTP scores ranking within the average and/or its lower scores. Previous studies indicate that patients with CHI greatly rely on their visual skills; our study confirms these results, as all three children registered a significant post-MTP improvement in the visual attention test and memory for drawings test. Such results corroborate the hypothesis according to which visual skills may serve as an effective starting point for future learning. Post-MTP results concerning memory for motor sequences were either preserved or improved, thus bearing witness of MTP's beneficial potential in this area, under-studied in the literature. Moreover, our results provide corroborating evidence of the need for promoting the development of specific motor skills in hearing-impaired children, which may result in

# REFERENCES


severe motor and balance deficits (Gheysen et al., 2008) and difficulties in motor sequencing (Conway et al., 2011), when left untreated.

The report limits lay in the reduced dimension of the analyzed sample of participants, as well as in the fluctuating nature of their performance in time. We strongly believe that these preliminary data need to be further confirmed by new studies with a larger sample and with specific control groups. At the same time, however, these data seem to highlight the need for a recurrent administration of the proposed activities to ensure retention and consistent improvement (Moreno et al., 2015).

# CONCLUSIVE REMARKS

This case report is a preliminary observation that seems to show the efficacy of a multimodal training involving cognitive and motor skills as an effective clinical and rehabilitative tool offered to very young children with CHI using a hearing device. A combined approach may in fact enhance the child's overall skills. Further research with better methodological approach is needed to confirm the extent of benefit from specific musical training activities.

# AUTHOR CONTRIBUTIONS

SG designed the scientific work, drafted the first version of the manuscript and gave the final approval of the manuscript. EC followed the testing procedure of the children and drafted the first version of the manuscript. GM programmed and developed the musical training. EM designed and programmed the musical training. SP critically interpreted the data and gave the final approval of the manuscript. EO supervised the scientific work, provided a critical revision of the work and gave the final approval of the version to be published.

# FUNDING

This paper was supported by MHO, under the internal project RC n◦ 17/2017.

# ACKNOWLEDGMENTS

The authors wish to thank Association Prematuri si Nasce and the families who participated in the study.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Ghiselli, Ciciriello, Maniago, Muzzi, Pellizzoni and Orzan. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# The Study of Visual-Auditory Interactions on Lower Limb Motor Imagery

#### Zhongliang Yu<sup>1</sup> , Lili Li <sup>2</sup> \*, Jinchun Song<sup>1</sup> and Hangyuan Lv <sup>1</sup>

<sup>1</sup> School of Mechanical Engineering and Automation, Northeastern University, Shenyang, China, <sup>2</sup> School of Physics, Liaoning University, Shenyang, China

In order to improve the activation of the mirror neuron system and the ability of the visual-cued motor imagery further, the multi-stimuli-cued unilateral lower limb motor imagery is studied in this paper. The visual-auditory evoked pathway is proposed and the sensory process is studied. To analyze the visual-auditory interactions, the kinesthetic motor imagery with the visual-auditory stimulus, visual stimulus and no stimulus are involved. The motor-related rhythm suppression is applied on quantitative evaluation. To explore the statistical sensory process, the causal relationships among the functional areas and the event-related potentials are investigated. The results have demonstrated the outstanding performances of the visual-auditory evoked motor imagery on the improvement of the mirror neuron system activation and the motor imagery ability. Besides, the abundant information interactions among functional areas and the positive impacts of the auditory stimulus in the motor and the visual areas have been revealed. The possibility that the sensory processes evoked by the visual-auditory interactions differ from the one elicited by kinesthetic motor imagery, has also been indicated. This study will promisingly offer an efficient way to motor rehabilitation, thus favorable for hemiparesis and partial paralysis patients.

#### Edited by:

Mauro Murgia, University of Trieste, Italy

#### Reviewed by:

Ambra Bisio, Università di Genova, Italy Valter Prpic, De Montfort University, United Kingdom

> \*Correspondence: Lili Li lilili\_mail@163.com

#### Specialty section:

This article was submitted to Perception Science, a section of the journal Frontiers in Neuroscience

Received: 03 May 2018 Accepted: 05 July 2018 Published: 24 July 2018

#### Citation:

Yu Z, Li L, Song J and Lv H (2018) The Study of Visual-Auditory Interactions on Lower Limb Motor Imagery. Front. Neurosci. 12:509. doi: 10.3389/fnins.2018.00509 Keywords: visual-auditory interactions, motor imagery, mirror neurons, brain-computer interface, event-related desynchronization

# INTRODUCTION

Motor imagery (MI) is a type of mental simulation of motor behavior and however, without any actual execution (Qiu et al., 2017). The premotor area, the primary motor area, the somatosensory motor area, and the cerebellum have been reported to be activated in MI as motor execution (Taube et al., 2015). MI based brain computer interface (BCI) has been widely studied in motor rehabilitation (Xu et al., 2014) and physical disability assistance (Choi and Kang, 2014). For MI depends on the participant's imagination ability which is individual difference, the motor ability acquirement has been limited (Cirstea et al., 2003). Mirror neuron system (MNS) is a series of visuomotor neurons, and it is first discovered in F5 area of macaque (Cattaneo and Rizzolatti, 2009). MNS is considered as the physiology basics of the prediction of the action's effects (Knoblich and Flach, 2010). It also plays an important role in motor skills acquirement (Garrison et al., 2010). MI is mediated by the MNS (Babiloni et al., 2003; Eaves et al., 2014). MI ability is believed to represent the ability to arrange movement and to utilize internal forward model

for the prediction of the motor outcome before the available sensorimotor feedback (Reynolds et al., 2015). MI ability has been identified to benefit from the MNS activation by researches on hand, mouth, and foot in human (Dickstein and Deutsch, 2007). MI is reported to improve motor performance by the promotion of MNS activation (Gatti et al., 2013) and to generate changes in structure and function of high-order motor cortical areas, (Slagter et al., 2011). Hence, it has been considered to be an effective way of the motor performance improvement. In order to improve the MNS involvement and MI ability, many researches concerning on stimulus evoked MI have been carried out. The effectiveness of the improvement on the MI performance by visual stimulus has been revealed (Hanakawa et al., 2003). An abundant guidance information provided by the auditory stimulus is demonstrated (Schreuder et al., 2010). A promising way of MI ability improvement has been reported by a video-cued unilateral lower limb MI (Boord et al., 2010). The MNS involvement has been improved by the object-oriented visual stimulus (Li et al., 2015). The MNS has been proved as a crucial role during the ecological stimuli (Murgia et al., 2015, 2016, 2018; Sors et al., 2015; Pizzera et al., 2017), and furthermore, the ecological visual and auditory stimuli can effectively affect complex movements (Kennel et al., 2014; Camponogara et al., 2016; Murgia et al., 2017). The rhythmic auditory stimulation is also indicated to facilitate gait rehabilitation (Thaut et al., 1993, 1996; Pau et al., 2016; Dalla Bella et al., 2017; Bailey et al., 2018). The possibility of achieving better performances on brain wave response and information transfer rate (ITR) by rich multi-sensory synergism is indicated (Moonjeong et al., 2012). Nevertheless, within our knowledge, the ability improvement approach and the sensory process of the multi-stimuli-cued unilateral lower limb MI are still not clear.

The mu frequency oscillation within the range of 8–12 Hz is relevant to the MNS activity (Pfurtscheller et al., 2008; Lapenta and Boggio, 2014). The beta frequency within the range of 13–30 Hz may be also related to the motor-related neuron activity (Li et al., 2015). During MI, electroencephalogram (EEG) desynchronization resulting from thalamocortical stimulus is a reliable correlate of the activated cortex, while EEG synchronization is a correlate of the deactivated cortex (Wriessnegger et al., 2013). The event-related desynchronization (ERD) and synchronization (ERS) on mu and beta frequencies are the indexes of the MNS involvement and the MI ability (Perry et al., 2011).

Event-related potential (ERP) is the neurophysiological activity that responds to the sensory stimulation in the background EEG. ERP can be divided into the endogenous and exogenous components. The endogenous component provides a sensitive measurement to assess information processing. The most studied endogenous component P300 is elicited by infrequent novel stimulus (P3a) and/or infrequent target stimulus (P3b). It reflects high-order information processing associated with the contextual evaluation of attended stimuli. The latency of the P300 indicates the time taken for the activation (Wang et al., 2003). The exogenous component is related to the attention and the sensory processing. One of the most studied exogenous components is N100. This is activated by several intra-cranial generators and is regarded as the reflection of the general and nonspecific cerebral excitability (Cortoos et al., 2014). In the neurophysiological research, the neural mechanism underlying the cognitive process can be reflected by the precise timing of ERP.

To improve the activation of MNS and the MI ability further, the multi-stimuli-cued unilateral lower limb MI is studied in this paper. In view of the positive performance with the visual stimulus, the effects of visual-auditory interactions on lower limb MI and the sensory process are investigated. The suppressions of mu and beta EEG oscillations and the ERP are applied for quantitative evaluation and analysis. This work devotes to explore the underlying neural mechanism of multi-stimuli-cued lower limb MI, and hopefully to provide an efficient path for motor rehabilitation especially lower limb rehabilitation, thus favorable for hemiparesis and partial paralysis patients.

#### MATERIALS AND METHODS

#### Participants

In this study, 10 participants composed of 8 males and 2 females with the mean age of 22.4 ± 1.43 years old are involved. They are able-bodied and free from medication and any disorders of or injuries to the central nervous system. The study is approved by the ethics review board of Northeastern University and is conducted according to the declaration of Helsinki. Studies are implemented after signing written consent forms by participants.

#### Recordings

EEG signals are recorded with 32 Ag-AgCl active electrodes including the motor area, the visual area and the auditory area, with the g.HIamp (g.tec Inc., Austria) system according to the 10- 5 electrode location system (Jurcak et al., 2007). The motor area consists of the premotor and supplementary motor cortex (A6), the primary motor cortex (A4), and the primary somatosensory and somatosensory association cortex (A1-2-3-5), etc. The visual area is composed of the primary visual cortex (A17), the secondary visual cortex (A18) and the associative visual cortex (A19). The distribution of electrodes is illustrated in **Figure 1**. EEG signals are referenced to a unilateral earlobe and grounded at frontal position (Fpz) with a sampling rate of 1,200 Hz. To suppress artifacts and power line interference, online band-pass filter between 2 and 100 Hz and notch filter between 48 and 52 Hz are applied on the recorded EEG. All impedances of active electrodes are kept below 30 k during experiments. To avoid the influences of electromyographic (EMG), the differential voltages between EMG electrode pairs on rectus femoris and biceps femoris of each leg are also recorded using the g.HIamp system with a sampling rate of 1,200 Hz. The EEG trials with any actual leg movement are discarded from further analysis to avoid the EMG disturbance.

#### Experimental Procedures

To evaluate the effect of visual-auditory interactions on lower limb MI, three kinaesthetic MI tasks, "visual-auditory context," "visual context," and "imagery context," are conducted. The visual stimulus that is applied on the "visual-auditory context"

and the "visual context," presents the extension and restoration movements of the unilateral leg using the 1.7 s color video frames. The binaurally auditory stimulus, extending leg and restoring it, is introduced using the 1.7 s recordings of native language and it is applied to the "visual-auditory context." During the "visual-auditory context," participants are instructed to imagine the unilateral leg extension and restoration movements with the paralleled visual and auditory stimuli of the same direction. During the "visual context," participants are instructed to imagine the same movements accompanied by visual stimulus. During the "imagery context," participants are instructed to imagine the same movements without any stimulus. The experiments are conducted in a dark and electrically shielded room. Participants are seated in an armchair comfortably with the distance of 95 centimeters between nose and computer screen approximately. There is a half an hour training session for each participant to be familiar with the trial design by motor execution before experiments. The three tasks are presented in pseudorandom for participants. The trials of the experiments are shown in **Figure 2**. The trials start with a crosshair that lasts for 2 s at the center of the screen. Participants are required to focus on the crosshair to reduce ocular movement. Subsequently, the arrowhead randomly pointing to left or right at the center of the screen lasts for 1 s as a reminder. When the arrowhead disappears, the stimulus or blank appears. Meanwhile, participants are instructed to perform the kinaesthetic MI. The imagery processes of the "visual-auditory context" and "visual context" last for 1.7 s. With regard to the third task, the imagery process lasts for 3 s in view of the initiative MI. There is a random break of 2–4 s at the end of each trial and a 1-min break after every ten trials for rest. Each run is comprised of five trials for the left and five trials for the right leg. During every task, 75 trials for each leg of the participants are implemented. Presentation of the visual, auditory, blank and their reversals are controlled by the psychophysics toolbox 3.0 (Brainard, 1997).

#### Analysis

To reduce the ocular artifacts, the EMD-regression algorithm (Li et al., 2013) is employed. All trials are visually inspected based on EMG during MI process, and nearly 9% contaminated trials are discarded from further analysis. To reduce the influences of the volume conduction and the reference electrode selection (Li et al., 2015), as well as to improve the spatial resolution and the signalto-noise ratio of EEG, the surface Laplacian is applied (Boord et al., 2010). Subsequently, all trials are extracted from data flow. To observe the brain activity, the average suppression index (ASI) illustrated by Equation (1), is applied. It is the average power ratio of the imagery process and the baseline (Pfurtscheller and da Silva, 1999). In this study, the EEG signals between −2.5 and −1.5 s of each trial are used as the baseline. As there is no relative information of kinesthetic MI during the crosshair and reminder process, only the imagery process of the three tasks is analyzed in this study.

$$\begin{split} \text{ASI}(f) &= \frac{1}{p+1} \sum\_{t\_a=t\_{sa}}^{t\_{ta}+p} \langle \frac{1}{n} \sum\_{i=1}^{n} I^2(f, t\_a) \\ &- \frac{1}{k+1} \sum\_{t\_b=t\_o}^{t\_o+k} \frac{1}{n} \sum\_{i=1}^{n} R^2(f, t\_b) ) / \frac{1}{k+1} \sum\_{t\_b=t\_o}^{t\_o+k} \frac{1}{n} \sum\_{i=1}^{n} R^2(f, t\_b) \end{split} \tag{1}$$

where, I(f, t) and R(f, t) denote the imagery process and baseline on the concerned frequencies f ; n is the trial number; k and p are in connection with the point number of baseline and imagery process.

The relationships of the visual, auditory and motor areas during different tasks are studied using the Granger causality analysis to explore the neural meditation mechanism and the sensory process evoked by the visual-auditory interactions. The Granger causality analysis is a statistical measurement based on the time sequence forecast. If the past information from one time sequence is benefit to a better prediction accuracy of another sequence, the first sequence has a causal influence on the second one. Due to the mutual interactions elicited by the volume conduction and the multi-electrodes, the multiple vector autoregressive (MVAR) model of the Granger causal analysis (Seth, 2010) is applied in this study. The ratio of the Akaike information criterion to the Bayesian information criterion is used to calculate the order of the MVAR model.

#### Statistics

To evaluate the differences of the MI abilities and the MNS activation during the three tasks, the analysis of variance (ANOVA) is applied to analyze the ASI of the imagery process. ANOVA is employed on mu and beta frequency oscillations to evaluate the differences of the three tasks in each functional area and to analyze the functional differences. The factors are within-subjects factors, "condition" ("visual-auditory context"


vs. "visual context" vs. "imagery context"), "rhythm" (mu and beta) and "area" (A6, A4, A1-2-3-5, A17, A18, A19, and auditory area). To study the differences of the three tasks in the contralateral hemisphere and in the ipsilateral hemisphere, the mu and beta ASIs are analyzed by ANOVA. The factors are "condition" ("visual-auditory context" vs. "visual context" vs. "imagery context"), "area" (motor area, visual area and auditory area), "rhythm" (mu and beta) and "hemisphere" (contralateral vs. ipsilateral). To evaluate MI abilities by the differences between the contralateral and ipsilateral hemispheres, ANOVA is applied on the mu and beta ASIs. The factors are "condition" ("visualauditory context" vs. "visual context" vs. "imagery context"), "area" (motor area, visual area and auditory area), "hemisphere" (contralateral vs. ipsilateral) and "rhythm" (mu and beta). Moreover, the ERP differences of the three tasks are also analyzed by ANOVA. The peaks of ERPs are adopted. The factors are "condition" ("visual-auditory context" vs. "visual context" vs. "imagery context"), "ERP" (P2, N1, N2, and P3) and "electrode" (Fz, Cz, Oz, T7, and T8). All the analysis and calculation are performed using MATLAB.

# RESULTS

The suppressions of mu and beta frequencies are applied to analyze the cortical excitement. The topographical views of the average ERD/ERS on the mu and beta frequencies during the three tasks are illustrated in **Figure 3**. Under the "visual-auditory context" and the "visual context," the unilateral leg MI provides the mu and beta suppressions in the contralateral hemisphere. In the "imagery context," the mu suppression can be found at the central area. The statistical results of the three tasks in the functional areas present significant mu differences in the A6 area [F(2, 98) = 14.62 P < 0.05], the primary motor area [F(2, 98) = 8.72 P < 0.01], the primary visual cortex [F(2, 58) = 14.47 P < 0.01], and the auditory area [F(2, 118) = 5.84 P < 0.01]. Besides, there are significant beta differences of the three tasks in the A6 area [F(2, 98) = 29.18 P < 0.01] and the primary motor area [F(2, 98) = 26.05 P < 0.01]. ANOVA results of the three tasks in the contralateral hemisphere and in the ipsilateral hemisphere indicate that both of the mu and beta suppressions are significantly different in the contralateral motor area {mu: [F(2, 238) = 18.72 P < 0.001], beta: [F(2, 238) = 6.56 P < 0.01]} and in the ipsilateral visual area {mu: [F(2, 158) = 3.13 P < 0.05], beta: [F(2, 158) = 4.02 P < 0.05]}. The statistical results between the contralateral and ipsilateral hemispheres demonstrate the significant mu differences between the contralateral and ipsilateral hemispheres in the motor area [F(1, 119) = 5.67 P < 0.05] and the auditory area [F(1, 59) = 8.23 P < 0.01] under the "visual-auditory context." The "visual context" presents a significant beta difference in the visual area [F(1, 79) = 6.92 P < 0.05]. Other comparisons by ANOVA which are not listed, are not significant difference (P > 0.05). The statistical results are shown in **Figure 4**.

The relationships of the visual, auditory and motor areas under the "visual-auditory context" and the "visual context" are studied by the Granger causality analysis to explore the underlying neural mechanism evoked by stimulus. The average analysis results of the imagery process of the participants are illustrated in **Figure 5**. The connectivity between electrodes presents a significant connection (P < 0.01). The analysis results demonstrate the causal influences from the auditory area of the right hemisphere to the visual and motor areas, and from the visual area to the motor area under the "visual-auditory context." Besides, the causal connectivity from the visual area to the motor area under the "visual context" is also revealed.

ERPs on Fz (frontal), Cz (central), Oz (posterior), T7 (left) and T8 (right) are studied to evaluate the potential neural mediation during the imagery process. The average ERP waveforms of the three tasks on these electrodes are illustrated in **Figure 6**. In the figure, the red line, blue line and black line represent the "visual-auditory context," "visual context," and "imagery context" respectively. In the "visual-auditory context," there are N100 (N1) on Fz and Oz, P200 (P2) on Fz and Cz, P300 (P3) on Oz, T7 and T8. In the "visual context," P3 and N1 are found on Oz. Meanwhile, P3 can be also discovered on Fz and Cz. In the "imagery context," N200 (N2) is found on Cz and Fz. The statistical results of the three tasks demonstrate the significant differences of the three tasks on P2 [F(2, 98) = 119.97 P < 0.05], N1 [F(2, 98) = 22.51 P < 0.05], P3 [F(2, 98) = 78.3 P < 0.05], and N2 [F(2, 98) = 24.49 P < 0.05]. There is a significant difference among the four kinds of ERPs [F(3, 447) = 521.42 P < 0.05]. In addition, there is a significant interaction of "condition" × "ERP" [F(6, 249) = 20.76 P < 0.05] that indicates the significant ERP difference among conditions.

## DISCUSSION

The mu rhythms which are originated at parietal lobe, are attenuated during attending motor behavior (Gastaut, 1952), such as motor imagery (Pfurtscheller et al., 2006). Many studies

have suggested that the desynchronization and attenuation in mu rhythm activity reflect MNS modulation (Muthukumaraswamy et al., 2004; Oberman et al., 2005). Therefore, the mu rhythm has been treated as a physiological indicator of MNS (Honaga et al., 2010). The suppressions of beta rhythms which originate from the precentral cortex, have also been regarded as indicators of MNS (Honaga et al., 2010) and motor behavior (Bai et al., 2008). Based on the above, the suppressions of mu and beta rhythms are related to the mirror neurons activation and the MI ability. In this study, the mu and beta suppressions have been discovered in the MI process under the "visual-auditory context" and the "visual context." There are significantly different

mu and beta suppressions among the three tasks in the A6 and the primary motor area. These suggest the differences of the motor neuron mediation among these tasks, and a greater MNS activation evoked by stimulus. Both of the mu and beta contralateral suppressions have presented significant differences in the motor area among the three tasks. In addition, the mu rhythm has exhibited a significant difference between the ipsilateral and the contralateral motor areas under the "visualauditory context." The above results reveal the greater motorrelated rhythm suppression under the "visual-auditory context" than under other tasks. Namely, the visual-auditory interactions can promote the MNS activation and the MI ability. The MNS plays a crucial role during MI evoked by the visual-auditory interactions. The MNS theory provides a path to study motor behavior. This mirror-like system has been convinced that it contributes to the social behavior by many researches (Wicker et al., 2003; Leslie et al., 2004). MNS is suggested to be involved in complex functions except for motor interpretation. It is constrained by motor mode and is differently encoded (Cattaneo and Rizzolatti, 2009). Hence, the results of this study indicate that the visual-auditory interactions may activate more perceive activities than other tasks by the promotion of MNS activation.

The activation of the auditory cortex is closely related with memory-scanning task (Krause et al., 1995). In the study of mu and beta frequency oscillations, significant mu difference among the three tasks in the auditory cortex has been revealed. Only

under the "visual-auditory context," a significant mu difference between the ipsilateral and the contralateral auditory cortices has been verified. These results reveal that the mu rhythm fluctuation may have a relationship with the activity of the auditory cortex, and a greater ipsilateral-auditory mu suppression has been presented by the binaural stimulation. During MI process evoked by the visual-auditory interactions, the activation of the auditory cortex that may be involved in the memory recall activity of the brain is asymmetrical. Besides, the significant differences of the mu suppression in the primary visual cortex and of the mu and beta suppressions in the ipsilateral visual area have been indicated among the three tasks. That is, the activation of the visual area varies with the tasks. Both of the mu (alpha) and beta rhythms may be affected by the visual stimulus. Among the three tasks, a significant beta difference between the ipsilateral and the contralateral visual areas has been discovered only under the "visual context." Namely, the introduction of the auditory cue may suppress the beta difference between the ipsilateral and the contralateral visual areas under the "visual-auditory context." As a result, the auditory area activation may have an effect in the visual area.

With the aim to explore the relationship of the motor, auditory and visual areas during the MI process evoked by stimulus, the Granger causality analysis has been employed. The results concerning the "visual-auditory context" have indicated that both of the motor and visual areas are affected by the auditory area of the right hemisphere. Besides, the motor area is also affected by the visual area. Under the "visual context," there is a causal effect from the visual area to the motor area. The study about visual-auditory interactions (Molholm et al., 2002) indicates the possibility of the impact of the auditory stimulus in the auditory and visual areas and of the impact from the primary auditory or the auditory cortex to the visual cortex during the button-press response task under the auditory and visual instructions. The relative anatomy research indicates that some axons of the visual cortex pass by the thalamus, and end in mesencephalon (Benevento, 1975). The mesencephalon is relevant to the reflection of the visual and auditory stimuli. This may be the anatomical basis of the causal connection from the auditory cortex to the visual cortex under the "visual-auditory context." The auditory area in the right hemisphere plays a predominant role in the attention control (Heilman and Van Den Abell, 1980) and the listening task without any specific strategies (Tervaniemi and Hugdahl, 2003). As a result, it has dominated the causal connections from the auditory to the visual and the motor areas under the "visual-auditory context." Based on the outstanding MI ability, the causal connections indicate a positive effect of the auditory cortex in the motor and visual cortices under "visual-auditory context." This results demonstrate that the auditory stimulus may activate the similar cognitive process by memory recall with the one by visual stimulus and kinesthetic MI, as the auditory cortex activation is closely related with memory recall (Krause et al., 1995). The dorsal pathway of the visual cortex is not a strict serial hierarchy. While, in general, A17 accepts the nervous discharge from the lateral geniculate nucleus. Then the projections extend to A18 and A19, and finally reflected in the somatosensory area by A18 and A19, etc. (Van den Stock et al., 2011). Accordingly, the significant causal connection from the visual area to the motor area under the "visual-auditory context" and the "visual context" may indicate the information transmission process of the dorsal pathway evoked by the visual stimulus.

ERPs with a high temporal resolution offer a sensitive path to monitor brain electrical activity and to observe cognitive process (Delle-Vigne et al., 2014). In this study, significantly different EPRs are presented among the three tasks. Furthermore, there is a significant interaction of "condition" × "ERP." Namely, the brain activity and the cognitive process vary with tasks. Under the "visual-auditory context" and the "visual context," N100 and P300 emerge on Oz, while these ERPs can not be observed under the "imagery context." N1 has been proved as a type of the visual evoked potentials, which can be elicited by Yu et al. Visual-Auditory Interactions

visual stimulus. It is significantly affected by the early phase of perception and attention processing (Bar-Haim et al., 2005). As a result, the ERPs of the N100 and P300 above may be the response of visual stimulus and the reflection of the visual area's activity. Besides, N1 is also thought to be evoked by the auditory stimulus (Annic et al., 2014). This auditory N1 is a measurement of the initial registration, the affiliation selection and the process of the auditory stimulus (Woldorff et al., 1987). Therefore, in view of the causal influence from the auditory area to the motor and visual areas, N100 on Fz and Oz may be also the reflection of cognitive process evoked by the visualauditory interactions under the "visual-auditory context." The P200 elicited by auditory stimulus presents over the vertex (Cz) prominently, with a typical peak latency of 150–250 ms approximately (Ferreira-Santos et al., 2012). It reflects the later stage of the stimulus processing, regarded as an index of some aspects of the stimulus classification process (Annic et al., 2014). The P200 only observed under the "visual-auditory context" may be the response of the recognition process evoked by auditory stimulus. During the auditory stimulus processing, the frontal lobe and the parietal lobe may be involved. N2 as a type of cognitive potential can be only observed under the "imagery context." The sensory process of the "imagery context" are different with the one of the other two tasks. The auditory and visual stimuli may convert the cognitive process of kinesthetic MI.

#### CONCLUSION

With the aim to explore the effect of the visual-auditory interactions on lower limb MI and the sensory process, three kinds of kinesthetic MI have been involved in this study. The study results have demonstrated the noteworthy performances

#### REFERENCES


of the visual-auditory evoked MI on the improvement of the mirror neurons activation and the MI ability. The visual-auditory evoked MI has presented the abundant information interactions among the functional areas and the positive impacts of the auditory stimulus on the motor and visual areas. Besides, the study results also reveal that the cognitive process of kinesthetic MI may be converted by the visual-auditory and visual stimuli.

The hemiparesis and partial paralysis are the common sequelae after stroke, affecting the daily life quality of patients directly. To recover the patients' somatic and sensory motor abilities, MI assisted therapy is a promising path of motor rehabilitations. This study about the visual-auditory interactions on lower limb MI will be favorable for motor learning and rehabilitation. Other imaging technology of brain will be explored to study the effect of visual-auditory interactions in further work, in order to overcome the low spatial resolution of EEG.

#### AUTHOR CONTRIBUTIONS

The data was analyzed by ZY and LL, the paper written by ZY, the materials and analysis tools supplied by ZY and LL, the language corrected by JS and HL.

#### FUNDING

This work is supported by the Young Scientists Fund of the National Natural Science Foundation of China (Grant No. 51605085), the Fundamental Research Funds for the Central Universities (Grant No. N170304021) and the Postdoctoral Science Foundation of China (Grant No. 2016M5 90229).


attentional event-related potentials. Front. Hum. Neurosci. 8:106. doi: 10.3389/fnhum.2014.00106


disease: new frontiers in assessment and intervention protocols. Open Psychol. J. 8, 220–229. doi: 10.2174/1874350101508010220


in hemiparetic gait of stroke patients. Neurorehabil. Neural Repair 7, 9–16. doi: 10.1177/136140969300700103


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Yu, Li, Song and Lv. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Information-Based Social Coordination Between Players of Different Skill in Doubles Pong

A. A. M. (Daphne) van Opstal1,2, Niek H. Benerink<sup>2</sup> , Frank T. J. M. Zaal<sup>1</sup> , Remy Casanova<sup>2</sup> and Reinoud J. Bootsma<sup>2</sup> \*

<sup>1</sup> Center for Human Movement Sciences, University Medical Center Groningen, University of Groningen, Groningen, Netherlands, <sup>2</sup> Institut des Sciences du Mouvement, Aix-Marseille Université, CNRS, Marseille, France

We studied how teams of two players of different skill level intercepted approaching balls in the doubles-pong task. In this task, the two players moved their on-screen paddles along a shared interception axis, so that the approaching ball was intercepted by one of the paddles and that the paddles did not collide. Earlier work revealed the presence of a fuzzy division of interception space, with a boundary between interception domains located in the space between the two initial paddle positions. In the present study, using the performance of the players in their individual training sessions, we formed teams of players of varying skill level. We considered two accounts of how this boundary should be understood. In a first account, the players have shared knowledge of this boundary. Based on the side of the boundary at which the approaching ball will cross the interception axis, the players would decide whose paddle is to make the interception. Under this account, we expected that a better-skilled player would take responsibility for a larger interception domain, leading to a boundary closer to the lesser-skilled player. However, our analyses did not reveal any systematic effect of skill difference on the location (or degree of fuzziness) of the boundary: location of boundaries and overlap of interception domains varied over teams but were not systematically related to skill differences between team members. We did find effects of ball speed and approach angle. In a second account, the boundary emerges from (information-driven) player– player–ball interactions. An action-based model consistent with this account was able to capture all the patterns in boundary positions and overlaps that we observed. We conclude that the interception patterns that players demonstrate in the doubles-pong task are best understood as emerging from the unfolding of the dynamics of the system of the two players and the ball, coupled through information.

Keywords: social coordination, visual information, interception, team performance, skill level, emergent behavior

# INTRODUCTION

Team work implies coordination. Teams are made of individuals, and individuals differ. How do these differences play out in the coordination among the team members and their environment? Consider, for instance, the situation in which a group of friends helps to move furniture to a new apartment. To carry a sofa up the stairs, at least two people are needed. Best practice learns that the stronger person of the two best carries more weight standing under the sofa and

#### Edited by:

Mauro Murgia, University of Trieste, Italy

#### Reviewed by:

Patrick Nalepka, Macquarie University, Australia Mario Di Bernardo, University of Bristol, United Kingdom

> \*Correspondence: Reinoud J. Bootsma reinoud.bootsma@univ-amu.fr

#### Specialty section:

This article was submitted to Movement Science and Sport Psychology, a section of the journal Frontiers in Psychology

Received: 02 March 2018 Accepted: 27 August 2018 Published: 19 September 2018

#### Citation:

van Opstal AAM, Benerink NH, Zaal FTJM, Casanova R and Bootsma RJ (2018) Information-Based Social Coordination Between Players of Different Skill in Doubles Pong. Front. Psychol. 9:1731. doi: 10.3389/fpsyg.2018.01731

**87**

the other person carries less weight but also guides the sofa's movement through the stairwell. Such a division of labor can be planned and communication throughout the operation facilitates the coordination (e.g., Vesper et al., 2017). In professional or sports situations, teams are often composed of specialists who work together on a shared goal. For instance, Gray et al. (2017) studied how teams of baseball infielders coordinated their actions in response to balls hit to the infield. Being at their own position (i.e., the position that they were used to play at) with teammates (i.e., players whose action capabilities they knew best) led to the most successful team decisions. In this situation, team decisions had to be made quickly and some predictability from experience with teammates was beneficial (cf. Glover and Dixon, 2017). Even clearer differences in expertise can be found in teams flying drones for reconnaissance purposes. Cooke and colleagues studied teams composed of a pilot, a navigator, and a photographer, who collaborated in flying simulated drone missions in order to take photographs of reconnaissance targets (for an overview, see Cooke et al., 2013). These studies demonstrated that to understand how successful team decisions come about, a good understanding of the interactions among the team members is indispensable. For instance, when comparing different types of training, Gorman et al. (2010) demonstrated that teams that had received a training focused on interactions among the team members were better able to adapt to novel situations that asked for performance under increased workload. Thus, interaction among team members seems key to success. Team members often have different roles, each contributing to the shared goal. But what if team members have the same role but different abilities?

The current study builds on previous work on joint interception, with teams of two individuals performing a doublespong task (Benerink et al., 2016, 2018). An innovative aspect of these studies was that team members were not assigned specific roles as to who was supposed to intercept balls where. In the doubles-pong task, each team member controlled their own paddle that could be moved along an interception axis at the bottom of a large, shared computer screen. Starting from different positions, balls moved along rectilinear paths from the top of the screen downward, under different angles with the vertical. With overt communication being banned, the task of the team on each trial was simply to intercept the ball. Importantly, the task constraints dictated that successful interception could in fact only be accomplished with a single paddle, as contact between the paddles led both to immediately disintegrate rendering future interception impossible. Inspection of how teams dealt with this joint-interception task revealed that they systematically showed a division of interception space. There was a distinct boundary between the interception domains of both players, together with a fair amount of overlap. When considering the teams in the Benerink et al. (2016) study, this boundary was generally located roughly halfway between the two paddles' initial positions, although some inter-team variability in its location was present. One notable exception in this study was a team with a boundary between interception domains clearly located away from the middle. Particularly interesting for the present purposes was that this specific team was characterized by a considerable difference in the individual skill levels of its two members and that the boundary was shifted toward the lesser-skilled player's initial paddle position. In other words, it seemed that the better-skilled player had taken responsibility of a larger interception domain. Moreover, a pilot experiment, in which we had teams perform the doubles-pong task while both players operated paddles of a different size, accidentally included skill differences between team members. Here too, these skill differences seemed to affect the location of the boundary, such that the boundary was closer to the lesser-skilled player at a distance from the mid-screen vertical that seemed linearly related to the skill difference (Benerink et al., 2015). The current study was inspired by these findings and set out to explore the question how joint interception plays out when team members differ in skill level on the same task.

The boundaries observed in the Benerink et al. (2016, 2018) studies bring to mind the boundaries in so-called Voronoi diagrams (e.g., Rein et al., 2017) or dominant regions (Taki and Hasegawa, 2000), as applied in a number of team-sport situations. When considering soccer, for instance, the pitch can be tessellated into areas such that each area is comprised of all positions on the field closest to the player occupying that area. Boundaries between these areas are lines halfway between adjacent player positions. Such spatial tessellations (i.e., Voronoi diagrams) have been applied in soccer (Taki and Hasegawa, 2000; Kim, 2004; Rein et al., 2017), futsal (Fonseca et al., 2012), volleyball (Paulo et al., 2018), and handball (Taki and Hasegawa, 2000), for example, in relation with passing opportunities (Gudmundsson and Wolle, 2014). Interestingly, one of the earlier studies that sought to apply the Voronoi diagrams took the tessellation in a direction that is directly relevant for the current study. Taki and Hasegawa (2000) suggested that the determination of the boundaries between Voronoi cells should not be limited to purely geometrical considerations but should also include the speed that players can adopt in all different directions. Indeed, in the same time, a player can reach a larger distance running in the forward direction than running in the backward direction. Taking into account the players' orientations, asymmetries can thus appear when drawing the Voronoi diagrams. Analogously, when looking at boundaries between two players who differ in the maximum speed they are able to reach, an asymmetry better captures the situation. In other words, the boundaries would be drawn based on the action capabilities of the players rather than simply on the geometry of the distribution of the players across the pitch.

While, evidently, boundaries can be drawn between the interception domains in the doubles-pong task and other team-based sport examples (using Voronoi diagrams or other methods), the status of such boundaries, however, remains unclear. In one account, the boundaries form the a priori basis for team coordination; alternatively, the boundaries are but the a posteriori result (by-product) of team coordination dynamics (cf. Benerink et al., 2016, 2018). Returning to the doubles-pong task, successful coordination implies a successful interception by one, and only one, of the two players. In the first-mentioned account, the decision that the players make – about intercepting an approaching ball or leaving that to the teammate – would be based on whether or not the ball will pass the interception axis either on their side of the boundary or on the teammate's side.

Since in our doubles-pong task overt communication between players was banned, such an account would assume that the two players tacitly agreed on the location of the boundary. Furthermore, such an account would (have to) assume that, at some point, players are able to accurately know where a ball will pass the interception axis. In other words, the decision of each player to either intercept or forfeit would then be understood as resulting from their shared understanding of a separation of interception domains (see Vesper et al., 2017 for a review on the role of shared knowledge in joint action) combined with their sufficiently accurate individual predictions of the future ball arrival position. Yet, current models of the control of interception cast doubt on proficiency of performing such predictions. Rather than relying on predictive control (i.e., predict the interception location from early target kinematics and move to this location), interception has been found to be controlled prospectively (i.e., through continuous guidance of the hand to the interception location on the basis of prospective information, e.g., Peper et al., 1994; Montagne et al., 1999; Dessing et al., 2002; Michaels et al., 2006; Ledouit et al., 2013).

An alternative to the account of reliance on an a priori boundary that delineates interception domains is one in which the boundary emerges from the unfolding of the dynamics of team coordination (see Richardson et al., 2015; Nalepka et al., 2017). Both players not only see the ball, but also their own paddle and the paddle of their teammate. Benerink et al. (2016)suggested that the division of labor between the two players emerges from the informational couplings within this tripartite system. This account thus focuses on the interactions rather than on the individuals (cf. Cooke et al., 2013). The relevant information is captured by the rates of change of the base angles β (see **Figure 1**) of the triangle formed by the ball (apex) and the two paddles that can move along the horizontal interception axis (base). When either the ball and/or one (or both) paddle(s) move, the relevant angles change. However, in the situation that paddle and ball movement are coordinated in such a way that the corresponding base angle β remains constant (i.e., dβ/dt = 0), ball-paddle contact is forthcoming (e.g., Fajen and Warren, 2007; Bootsma et al., 2016). Benerink et al. (2016, 2018) showed that, for balls heading for positions located between the players' initial paddle positions, both players often started to move and that attributing interception to the first player whose paddle moved such that its dβ/dt reached (or, in fact, exceeded) zero captured the division of interception space very well. In other words, when interception was afforded to one player, the teammate abandoned his or her movement, to avoid collision and thus allow successful team performance. The latter account assumes that players not only are able to see the affordance of interceptability for themselves (Postma et al., 2018) but also for the other (e.g., Stoffregen et al., 1999; Ramenzoni et al., 2008; Fajen et al., 2009; Weast et al., 2011). Note that in this account the boundaries between interception domains (a posteriori) describe the patterns resulting from the unfolding dynamics but do not form the (a priori) basis for these patterns.

With the exception of the one team mentioned before, Benerink et al. (2016, 2018) only considered teams with players of similar skill level. In both studies, each player started the experiment with an individual session, not only serving as training on the interception task and apparatus, but also allowing individual skill levels to be determined. Teams were then composed for subsequent doubles sessions by combining players of comparable skill levels (i.e., having similar performance scores) in their individual sessions. The one-team exception herein in the Benerink et al. (2016) study resulted from having to combine a limited pool of 12 players into six teams. In order to investigate effects of skill-level differences between the two team members, in the present study, we deliberately composed teams of players having demonstrated different levels of performance in the preliminary individual sessions. If teams were to rely on a (tacitly) shared understanding of a boundary separating interception domains, balls moving toward the left side of the boundary would be for the left player to intercept and vice versa for the right player. In order to optimize team performance, under this logic, the boundary could then be expected to be shifted toward the lesser-skilled player, with the better-skilled player thereby taking responsibility for a larger interception domain. An account of emergent boundaries, on the other hand, does not necessarily lead to specific predictions concerning the location of the boundary as a function of the skill differences, since it is based on the way players move during an interception attempt, rather than on final outcome. Of course, skill differences might be accompanied by differences in movement kinematics and the interactions between players would then play out into one player intercepting balls at certain locations on the interception axis rather than the other. However this may be the account of emergent division of labor would under all circumstances predict that observed patterns in boundary and overlap can be captured by the model that attributes the interception to the player whose paddle moves such that its associated dβ/dt first reaches zero.

# MATERIALS AND METHODS

#### Participants

In the framework of the present study, participants took part in three separate sessions: one individual session and two doubles sessions. A group of 28 right-handed (post)graduate students from the Aix-Marseille University (17 men and 11 women, with an average age of 24.7 ± 2.2 years, M ± SD) volunteered for participation in the first (individual) session. From this group of 28 participants, 12 (eight men and four women, with an average age of 24.8 ± 1.2 years) were retained for the present purposes; the other 16 participated in a separate study (cf. Benerink et al., 2018). The selection of participants for the present study was based on their levels of performance in the individual session, allowing teams (i.e., dyads) with different individual performance levels to be composed (details follow later).

All participants provided written consent before participating in the study that was approved by the local institutional review board of the Institute of Movement Sciences (Comité Ethique de l'Institut des Sciences du Mouvement d'Aix-Marseille Université) and conducted according to University regulations and the Declaration of Helsinki.

# Experimental Setup

The experimental setup used for the present experiment was the same as that of Benerink et al. (2016). The experiments were all performed in a darkened room equipped with a large table with two adjacent seats on one side and a large television screen (Samsung 55" LED ED55C, operating at a frame rate of 100 Hz with a 1920 × 1080 pixel resolution) on the other side. Seated participants faced the middle of the screen at eye-height from a 2-m distance. Participants were separated by a curtain, hanging down from the ceiling, that prevented them from seeing (any part of) the other during the doubles sessions. With verbal communication between participants being banned, headphones (3M Peltor Optime2) and earplugs furthermore prevented them from picking up (auditory) information about their partner's behavior.

Participants individually controlled the position of their on-screen paddle by moving a hand-held knob laterally over an in-house constructed linear-positioning device placed on the table in front of them (for further details, see Benerink et al., 2016). The on-screen paddle moved along the (invisible) horizontal interception axis, located just above the bottom of the screen that extended horizontally (X-axis) from −60.5 to +60.5 cm and vertically (Y-axis) from −2 to +66 cm (see **Figure 2**). A proportional gain ensured that participants could cover the full (121-cm) range of the on-screen interception axis with their paddle without reaching the extremities of the (75-cm long) linear-positioning device. Unless specified otherwise, positions and distances reported from here on correspond to distances on the screen, with the origin corresponding to the center of the horizontal interception axis.

Positions of the participants' paddles and the ball were sampled at a frequency of 100 Hz and stored on an external disk. Prior to analysis, the kinematic data were filtered with a recursive

low-pass second-order Butterworth filter with a cut-off frequency of 5 Hz (cf. Benerink et al., 2016, 2018).

#### TABLE 1 | Individual characteristics for session 1 (S1) of the 12 participants.

#### Task and Procedure

fpsyg-09-01731 September 17, 2018 Time: 10:22 # 5

The participants' task was to intercept virtual balls (2-cm diameter white circles depicted against a black background), moving downward across the screen at various angles and speeds, by making these bounce back upward after contact with a white (3-cm wide and 0.8-cm high) paddle.

The first session (S1), in which participants performed the interception task individually, consisted of five blocks of 50 trials, for a total of 250 trials per participant. In this session, half of the participants performed the task while seated on the left side of the table (left position) and the other half performed the task while seated on the right side of the table (right position). As reported in Benerink et al. (2016), performance generally increased over the first two blocks before leveling off on the last three blocks. The skill level demonstrated in S1 by each participant was therefore operationally defined by score S, calculated as S = (B3 + B4 + B5 + Max/2)/3.5, where B3, B4, and B5 correspond to the percentage of balls intercepted in blocks 3, 4, and 5 and Max corresponds to the highest percentage of balls intercepted in any of the five blocks.

Individual S-scores (see **Table 1**) were used to form teams composed of individuals with different skill levels for the doubles sessions. Since pilot work indicated that participants that took part in an individual session in either the left or the right position subsequently performed equally in both positions, team composition for sessions 2 (S2) and 3 (S3) did not take into account individual participant positions in S1. However, all participants performing in the left position in S2 performed in the right position in S3 (hereafter referred to as P1 participants). Likewise, all participants performing in the right position in S2 performed in the left position in S3 (hereafter referred to as P2 participants). In order to test the basic hypothesis of a shift in boundary location (toward the less-skilled player) in the presence of within-team skill-level differences, in S2, the 12 participants were combined into six teams with relatively homogeneous differences in S-scores within teams, ranging from 6.3 to 9.4% (M ± SD = 7.7 ± 1.2%). In order to test the hypothesis that the shift in boundary location varied as a function of the degree of within-team skill-level differences, for S3, six new teams were formed, with differences in S-scores within teams now varying from 2.3 to 13.4% (M ± SD = 6.4 ± 5.3%). While perhaps seemingly moderate, these skill-level differences between team members (see **Table 2** for details) are to be appreciated in the light of the 1.8 ± 1.5% (range 0.4–4.8%) and 1.7 ± 0.8% (range 1.0–2.5%) within-team differences in individual performance for, respectively, all eight teams of the Benerink et al. (2018) study and five of the six teams of the Benerink et al. (2016) study<sup>1</sup> . Over the two doubles sessions of


Pos S1, S1 in left (L) or right (R) position; Perf, performance expressed as the mean percentage balls intercepted over all five blocks of S1; S, skill-level score used for composing teams.

TABLE 2 | Team characteristics and results.


P2–P1, within-team S-score difference between P2 and P1 participants; TP, team performance (% intercepted); B-Loc, boundary location between interception domains; overlap, overlap between interception domains.

the present experiment, within-team differences in S-scores were on the average 7.1 ± 3.7%.

In both doubles sessions, participants were instructed that the task they had to perform was to intercept as many balls as possible as a team by moving the on-screen paddles laterally along the invisible horizontal interception axis. Importantly, participants were warned that they should avoid contact between their on-screen paddles, as this led both paddles to immediately disintegrate, thereby rendering future interception impossible. Participants were explicitly instructed that the number of individual interceptions did not matter and that the team performance was the only thing that counted.

For a trial to start, participants had to move their paddle to the designated start position (30 cm to the left or to the

<sup>1</sup>The sixth team of Benerink et al. (2016), with an individual performance level difference of 10.0%, revealed the idiosyncratic team behavior mentioned in the introduction and interpreted as potentially due to differences in skill level between its members.

right of the center of the screen in S2 and S3; see **Figure 2**) marked by a 3-cm wide translucent red rectangle. If the center of the participant's paddle arrived within 0.3 cm of the center of the rectangle, the rectangle turned green indicating that the paddle was located at the right place. After participants had remained in place for 2 s, the green rectangles disappeared and after another second the ball appeared. Balls moved downward with vertical speeds of 0.40 [slow ball speed (BS)] or 0.64 m/s (fast BS), corresponding to movement durations for the ball to arrive at the interception axis of 1.6 and 1.0 s, respectively. Successful interception required that one of the participants' paddles touched the ball before it crossed the interception axis. If so, both paddles turned green and the ball moved back up again. In trials in which neither of the two participants reached the arrival position of the ball in time (i.e., unsuccessful trials), the paddles turned red and the ball continued moving downward. As mentioned before, if the participants' paddles touched each other before the ball reached the interception axis, both paddles disintegrated, resulting in a failure to intercept the ball (i.e., unsuccessful interception). The occasional trials in which such a collision occurred after ball interception were considered successful as the common goal of intercepting the ball was achieved. Two seconds after ball arrival at the interception axis (regardless of a successful or unsuccessful interception), the paddles turned to their original white color and the translucent red rectangles would appear again for the team to start a new trial.

Balls moved downward following rectilinear trajectories and approached the interception axis under different angles. Similar to our previous studies (Benerink et al., 2016, 2018), the design included five standard ball departure positions (Y = +64 cm) and five standard arrival positions (Y = 0 cm), both at X = −42, −21, 0, +21, and +42 cm. Combining the five departure positions with the five arrival positions gave rise to a total of 25 standard trajectories. On each trial, a random distance between −10.5 cm and +10.5 cm was added to both the standard departure and arrival positions of the selected trajectory, shifting the entire trajectory to the left or right, while keeping trajectory incidence angle [or, equivalently, lateral ball movement (LBM) between the X-coordinates of ball departure and arrival positions] the same. This way, balls could appear and arrive anywhere between X = −52.5 cm and X = +52.5 cm (see **Figure 2**). In each block, all 25 trajectories appeared with two different vertical ball velocities resulting in a total of 50 fully randomized trials per block.

Both experimental doubles sessions (S2 and S3) started off with ten familiarization trials. Besides intercepting a number of balls, participants were asked to purposely miss one as well and to make contact with the other participant's paddle, so as to experience all action possibilities, constraints and their outcome during these familiarization trials. In each doubles session, all teams completed four blocks consisting of 50 trials that were presented in random order. This resulted in a total of 200 trials for each team per doubles session, which took a team about an hour to complete.

# RESULTS

## Interception Performance

Team compositions (in terms of differences in individual S-scores) as well as their performances (in terms of percentage intercepted balls over all blocks) are presented in **Table 2**. Team performance varied between 76.0 and 94.0%, for an overall mean of 87.3 ± 4.3% (corresponding to a total of 2095 successful interceptions). Collisions leading to unsuccessful interception were rare, occurring in 1.3% (i.e., 31) of all 2400 trials. **Figure 3** provides a graphical summary of the interception results as a function of the ball's arrival position on the interception axis for all 200 trials of each team separately. To this end, interceptions accomplished by the P1 (dark blue circles) and by the P2 (light blue circles) players were plotted on separate axes (corresponding to the probability of interception by the P1), allowing visual discrimination of who intercepted the balls where. Trials in which both participants failed to intercept the ball (red circles) and trials resulting in a collision between the participants' paddles (purple circles) are also presented. As also observed in Benerink et al. (2016, 2018), collisions mainly occurred around the center of the interception axis, while misses were widely distributed over the interception axis.

# Boundary Location and Overlap Between Interception Domains

Largely corroborating the general observations of Benerink et al. (2016, 2018), inspection of **Figure 3** revealed a clearly visible but nevertheless somewhat fuzzy separation of interception domains for all teams. In a first step to assess the effect of within-team skilllevel differences on the separation of interception domains, we followed the procedure adopted in Benerink et al. (2016, 2018) for determining the location of the boundary and the amount of overlap between interception domains. To this end, simple logistic probability curves for interception by P1 (p = 1) and P2 (p = 0) were derived for each team<sup>2</sup> , using ball arrival position (BAP) along the interception axis as a predictor (green lines in **Figure 3**). From the logistic regression equations, the location of the boundary between interception domains was calculated for each team as the location of the symmetry (p = 0.5) point and the amount overlap as the distance along the interception axis between the p = 0.05 and p = 0.95 points (see Cox and Snell, 1989). As can be seen from **Table 2**, boundary locations varied over teams between −2.8 and +3.9 cm, for an overall mean of 1.8 ± 2.2 cm. Interception domains of individual team members revealed overlaps varying between 10.2 and 24.8 cm for an overall mean of 14.9 ± 4.9 cm.

We tested whether the better-skilled player took responsibility for a larger interception domain, resulting in a boundary location shifted from the center of the interception axis in the direction of lesser-skilled player. Contrary to this hypothesis, however, such a shift was not systematically observed in our data. Plotting the boundary location as a function the within-team skill-level differences (**Figure 4A**) did not reveal the expected association,

<sup>2</sup>All statistical analyses were performed in R version 3.4.3 (https://www.r-project. org/).

and purple dots (collisions). The green curves depict the logistic curves representing the probability that P1 (p = 1) or P2 (p = 0) will intercept the ball as a function of ball arrival position. The horizontal dashed gray lines at ball arrival position 0 cm indicate the center of the interception axis. For each team, S-scores from the individual sessions, as well number of misses (M) for each individual player in the BAP ranges outside ± 15 cm observed during the team sessions are indicated.

with positive P2–P1 difference leading to a shift in boundary location to the left (points in the fourth quadrant of **Figure 4A**) and negative P2–P1 difference leading to a shift to the right (points in the second quadrant of **Figure 4A**). From the 12 teams studied, only six (three out of six in S2 and three out of six in S3) revealed boundary locations in the predicted quadrants. For the relatively homogeneous within-team skilllevel differences in S2, the chance of finding a smaller (larger) interception domain for lesser-skilled (better-skilled) player was thus as large as finding a shift in the opposite direction. Even for the two teams with the largest skill-level differences (S3 teams 16 and 17; see **Table 2**) one did not reveal the expected behavior: Team 16 had a larger interception domain for the lesser-skilled player. Likewise, plotting the amount of overlap between interception domains as a function of the within-team skill-level differences (**Figure 4B**) did not reveal any systematic relation.

## GLMER Analysis

While within-team skill-level differences did not reveal systematic effects on the location of the boundary and the amount of overlap between interception domains, we noted that these global analyses were, for each team, based on the full set of ball trajectories presented. Yet, balls could not only arrive at different positions on the interception axis but could also arrive there with different speeds and different angles of approach (i.e., different amplitudes of LBM resulting from the combination of BAPs with different ball departure positions). In order to test whether within-team skill-level differences might indeed be observed for specific BSs and/or specific amplitudes of LBM, we extended the analysis to a generalized linear mixed effects regression (GLMER), using the glmer function from the lme4 package (Bates et al., 2015). In addition to within-team skill-level difference (P2–P1; see **Table 2**), potential predictors of the binary outcome (interception by P1 = 1 and by P2 = 0) were BAP, LBM,

FIGURE 4 | (A) Boundary location as a function the within-team skill-level differences. (B) Overlap between interception domains as a function the within-team skill-level differences.

TABLE 3 | Generalized linear mixed model fit by maximum likelihood (Laplace approximation) for final model<sup>a</sup> .

#### Model variables


Significance codes: ∗∗∗<0.001; ∗∗<0.01; <sup>∗</sup><0.05. CIs were calculated using parametric bootstrapping (n = 1000). <sup>a</sup>Model formula in R-notation: result ∼ BAP + speed + LBM + BAP × speed + (1 | team).

BS, and session (fixed effects). Team was included in the overall analysis as a random-effect variable.

We started out with a null model, which included only the effect of BAP (thereby comparable to the set of simple logistic regressions described above). Predictors were then added to the model in a stepwise forward manner, starting with the main predictors, followed by their two-way interactions. In order to avoid possible multicollinearity, predictors were not included in the model simultaneously if they showed high correlation (ρ > 0.7). Predictors were retained in the model if they turned out to be significant (α = 0.05) and simultaneously led to a decrease of the Akaike information criterion (AIC) of more than 2 (cf. Burnham and Anderson, 2004). This procedure was followed until no further improvement of the model could be achieved. Collinearity of the model was then reassessed on the basis of the variance inflation factor (VIF). If the VIF was above a threshold value of 3 (as suggested by Zuur et al., 2010), removal of the predictor from the model was considered.

Importantly, whereas adding LBM and BS to the original null model with BAP as predictor variable improved outcome prediction, this was not the case for the within-team skill-level difference and session<sup>3</sup> variables; inclusion of within-team skilllevel difference, either as a continuous or as a binary variable, did not significantly improve the prediction (either through a main effect or through an interaction with other variables) nor did it lead to the criterion reduction in AIC.<sup>4</sup> We therefore conclude

<sup>3</sup>For the session variable, we note that adding BAP × session and LBM × session interaction effects did lead to a significant improvement of the model performance, but at the cost of increased VIF values. For this reason (i.e., to avoid collinearity), these two interaction effects were not included in the final model.

<sup>4</sup>To explore the option that players when acting as team members showed different skill levels than when acting alone (i.e., in S1), we determined for each participant in both S2 and S3 (i.e., the team sessions) the number of misses in an area that was clearly to be covered by this specific participant. **Figure 3** gives the number of misses in the areas to the left or to the right of −15 and +15 cm for the P1 and P2, respectively. Although not directly comparable with the skill-level scores for S1 (because these scores were computed for players covering the full length of the interception axis), the number of misses provided an impression of individual skill levels during the team sessions, correlating significantly with skill level scores,

that, even for specific BSs and for specific ball trajectories, withinteam skill-level differences did not systematically affect which player intercepted the ball where. Considering that, on the other hand, systematic effects of BAP, LPM, and BS were observed, our overall pattern of findings thus provides quite compelling evidence against a systematic role of within-team skill-level differences in the location of the division of interception space.

The final model included the fixed effects of BAP, LBM, BS, and the BAP x BS interaction effect, as well as a random effect of Team (see **Table 3**). While the (strong) effect of BAP was, of course, to be expected from observation of **Figure 3**, the others were not. First of all, the analysis demonstrated that the effect of BAP was moderated by BS. For balls moving at the lower speed, the probability curve was somewhat shallower than for balls moving at the higher speed, implying a larger degree of overlap between interception domains of the players when they had more time at their disposal. Second, the effect of LBM indicated that angle of approach to the BAP influenced which player intercepted the ball. This finding most likely reflects the so-called angle-of-approach effect observed in individual lateral interception tasks: balls arriving at the same position after the same motion duration give rise to kinematic interception patterns that vary systematically as a function of the incidence angle of the ball's trajectory (Peper et al., 1994; Montagne et al., 1999; Michaels et al., 2006; Arzamarski et al., 2007; Ledouit et al., 2013).

Evaluation of the statistical pertinence of the GLMER model by a trial-by-trial examination of its predictions revealed that it correctly predicted interception by P1 or P2 in 98.4% of all successfully intercepted trials. In other words, of all 2095 trials resulting in interception, the GLMER-based model provided an incorrect prediction of who intercepted the ball in (only) 33 cases. As can be seen from **Figure 5**, the GLMER prediction errors (red circles) generally concerned balls arriving close to center, with a mean BAP of 3.4 ± 6.3 cm. **Figure 6** allows appreciating the supplementary effect of LBM, with direction (positive or negative) and magnitude of LBM revealing a relation with BAP of incorrectly predicted interceptions [correlation between LBM and BAP for prediction errors: r(31) = 0.75, p < 0.001].

Analysis of the GLMER model and its predictions of who intercepts which ball indicated that, while overall the correct prediction rate was very high, it required inclusion of the LBM variable. We will discuss the consequences hereof further on, but first move on to evaluate the action-based model of continuous interaction proposed by Benerink et al. (2016) to explain who intercepts which ball.

## Action-Based Model of Continuous Interaction

While the results of the present study did not reveal systematic effects of within-team skill-level differences, a separation of interception domains with a more or less fuzzy boundary was observed in all 12 teams. Benerink et al. (2016) suggested that, rather than being somehow predefined, such a separation in fact emerged from a continuous information-based interaction between the team members. More precisely, they suggested that this interaction was captured by the rates of change of angles β (see **Figure 1**). With both team members potentially engaging in interception for each ball, the one that first reaches a positive dβ/dt (indicating that the ongoing interceptive movement is expedient) will be the one that intercepts the ball (see Benerink et al., 2016, 2018 for further details). When applied to the present data set, in its simplest form, the continuous interaction model correctly predicted the results in 97.8% of all successfully intercepted trials (i.e., for 2049 of the 2095 trials concerned, with 46 erroneous predictions). Analogously to **Figures 5**, **7** presents the predictions of the continuous interaction model, and their correctness compared to the measured outcome of the trials, for each team separately. In visualizing these results, it is important to bear in mind that, contrary to the GLMER analysis, the continuous interaction model was not fitted to the observed results: the resulting (fuzzy) separation of interception domains shown in **Figure 7** is a consequence of the betweenplayer interaction prior to interception.

Incorrect prediction by the action-based model of who will intercept which ball occurs when the non-intercepting player is the one who reaches positive dβ/dt first. Trials in which both players reached positive dβ/dt occurred in 144 of the 2095 successfully intercepted trials, of which 46 resulted in incorrect prediction of who intercepted the ball. As can be seen from **Figure 8**, in almost all these trials, both team members reached positive dβ/dt at approximately the same moment (i.e., within 200 ms from each other), implying that they hardly had time to adjust their behavior to that of their team mate. Moreover, as can be seen from **Figure 9**, in these trials, one of the team members often maintained the state of positive dβ/dt for only a short (<200 ms) duration; that is, they did in fact not pursue their interceptive movement in an expedient way. Enriching the criterion for attributing the interception to a given team member by selecting the team member that first reached positive dβ/dt and maintained it for at least 200 ms gave rise to correct predictions of who intercepted the ball in 99.1% of the 2095 successfully intercepted trials, leaving a mere 19 trials with incorrect predictions. We emphasize that our goal in developing the action-based model of continuous interaction to a certain extent here (by including a supplementary criterion) is not necessarily intended to be taken as a proposal for durably refining it (as it may lose its attractive parsimony when additional criteria are added), but to demonstrate that it is capable of explaining, solely on the basis of the informational dynamics of the two-paddle-and-ball system, which team member will pursue the interception attempt and which will abandon it.

# DISCUSSION

The current study was designed to investigate the effects of skilllevel differences between team members on how the doublespong task is performed. Replicating earlier findings (Benerink et al., 2016, 2018), all teams showed a distinct but fuzzy boundary between interception domains. From an account of

r(22) = −0.72, p < 0.001. Next, we checked if these numbers of misses had any predictive value in the GLMER model. This turned out not to be the case.

shared understanding of a tacitly agreed-upon boundary as a basis for assigning the interception to either player (i.e., the left and right player take responsibility for balls that will arrive left or right of the boundary, respectively), we expected that the boundary would be closer to the lesser-skilled player. As observed for the specific team in the Benerink et al. (2016) study that was characterized by large differences in individual skill levels, the hypothesis was that the better-skilled player would cover a larger interception domain. No matter how we analyzed the data (performing logistic-regression analyses separately for each team – that is, applying the methods that we used before in the Benerink et al., 2016, 2018 studies – or using linear mixed-effects logistic regression, controlling for potential unanticipated effects of other variables that were part of the design), we did not find any systematic effect of within-team skill-level differences on the location of the boundary. Rather, the GLMER analyses indicated that other factors, such as BS and the lateral movement of the ball, affected the division of interception space between the two team members. The GLMER model was able to correctly predict the player who intercepted the ball in about 98% of all successful trials. We also considered the action-based model of continuous interaction introduced by Benerink et al. (2016), in which the prediction of the division of labor between the players is based on the first player to be in a situation that affords interception, as specified by a zero-rate of change in participant-related base angle β. This model also predicted about 98% of who of the players made the successful interception.

Although of similar predictive power, the two models represent two diagonally different accounts. The GLMER model is a statistical model that was fit to the data a posteriori (i.e., which player intercepted the ball was used as an input variable to derive the model), optimizing a fair amount of degrees of freedom. The underlying logic of this model fits with an account in which the players base their decisions on who of the two will make an interception on an a priori boundary. In contrast, the action-based model goes with an account in which the boundaries can be identified a posteriori (i.e., the boundary and

the center of the interception axis.

its characteristics emerge from the dynamics of the ball–player– player triad) but observed coordination patterns are predicted from a priori principles (cf. Benerink et al., 2016), without

recourse to any form of data fitting. As mentioned before, one account of how the two players each intercept their specific subset of balls is that they choose who will take which ball using a tacitly agreed boundary dividing the interception space. The agreement must be tacit because in the present experiments, players were not allowed to communicate other than through moving their paddle on the screen. In this account, presumably then, they arrive at such shared knowledge (e.g., Vesper et al., 2017) from interactions early on in their team session. For each approaching ball, the players have to determine on which side of the boundary it will pass and base their shared decision on this information. The statistical GLMER model that we built to account for the present data indicated that not only the BAP but also the way how the ball arrived at this position – the LBM and BS – affected the division of labor. Particularly, the factor of LBM is interesting, because it is indicative of an angle-of-approach effect in joint interception. When we translate these results to an account of joint decisions based on which side of the boundary a ball will pass, the predictions involved in these decisions will have to take many factors (including BS and angle of approach ball) into account. Interestingly, in individual lateral interception, the angle-of-approach effect has been reported repeatedly and has been taken to imply that an interceptive movement is not controlled toward a predicted future arrival position (e.g., Peper et al., 1994; Montagne et al., 1999; Dessing et al., 2002; Michaels et al., 2006; Ledouit et al., 2013). Thus, invoking an explanation relying on the prediction of a future BAP in joint interception seems problematic.

The alternative to predictive control in lateral interception is the use of continuous prospective information (e.g., Bootsma et al., 2016). A zero rate of change of angle β in the pong task qualifies as prospective information because upcoming interception is specified for current ball and paddle movement. The action-based model proposed by Benerink et al. (2016)

the center of the interception axis.

FIGURE 6 | Effect of lateral ball movement (LBM) and ball arrival positon (BAP) combinations on GLMER model predictions for the total of 2095 intercepted trials. Red dots represent trials with incorrect predictions of which player intercepted the ball. Black dots represent trials with correct predictions of which player intercepted the ball.

and tested in the present study (see also Benerink et al., 2018) capitalizes on the use of this informational variable. Saying that the rate of change of angle β has reached zero boils down to saying that paddle movement is such that successful interception is forthcoming (if current conditions persist). In other words, a zero rate of change of angle β specifies expediency of current movement (cf. Benerink et al., 2016). When one of the two players is moving in such a way that s/he has reached a zero rate of change of β, the other player can (and should) stop moving and leave interception to the teammate. As mentioned before, this model accounted for about 98% correct predictions

of the intercepting player. When we inspected the trials with incorrect predictions, we noted that in many of these cases both players reached a zero rate of change of β at about the same time and/or that the rate of change of β remained above zero for only a fraction of a second. Some straightforward fine-tuning the model to deal with these spurious results led to an almost perfect prediction of the intercepting player. This is not to suggest that elaborating the model toward better prediction should be the goal, but more to show how an action-based account seems to capture the phenomena very well without losing the elegance of its simplicity.

The doubles-pong task is an instance of the many ways in which two persons work together to attain a shared goal. As demonstrated in the current study, as well as in previous doublespong studies (Benerink et al., 2016, 2018), the two players in this task appear to have divided up interception space (with a fuzzy boundary), each taking care of a subset of the approaching balls. We argue that explaining this division of labor as emerging from the dynamics of the player–player–ball system leads to a more parsimonious account than one in which the players explicitly use a tacitly agreed-on boundary for deciding the player to take a specific ball. Previous studies also showed how different roles that members of dyads might take up emerge from the dynamics of the task. For instance, Davis et al. (2017) had two players coordinate two circular avatars (of different size), controlled by hand movements and presented on a shared screen. The stability of the balance of the players was manipulated by having them either stand with a normal base of support or in a heel-to-toe tandem stance. The balance manipulations led to one player taking on the role of leader and the other that of follower, without any instruction to do so. Another example comes from Richardson et al.'s (2015) study on an interpersonal collisionavoidance task. Here, two members of a dyad were to cyclically move a pointer between two targets on a shared screen. The

targets were positioned on the corners of an invisible square, and each player was to move along one of the two diagonals. The instruction was to have the pointers not collide. Meeting task requirements, in theory, could have been realized in many ways, but the dyads all turned out to show a solution in which one dyad member moved along a straight path and the other along an elliptical path, while synchronizing target contacts. A final example involves dyads that have to perform a reciprocal aiming task (a Fitts' task), either unimanually between two targets (as studied most often), bimanually with one hand moving the pointer and the other moving the set of targets, or in a dyad with one member of the dyad moving the pointer and the other moving the set of targets (Mottet et al., 2001). Interestingly, when allowed, people did move the set of targets, and when considering relative movements (i.e., pointer with respect to targets), the movement patterns essentially were the same across these three conditions. What is common in all these examples is that roles were not prescribed to the participants, but rather emerged from the dynamics given the task constraints (see also Riley et al., 2011). What sets apart the doubles-pong task, though, is that successful performance requires interception by only one of the two dyad members while the other member's movement has to be such that no collision occurs. That is to say, whereas in the other tasks, both members of the dyad continuously interact in attempting to meet the common task goals, the doubles-pong task more resembles the emergence of discrete decisions.

#### CONCLUSION

All in all, the aim of the present study was to investigate the effects of skill difference between the two participants in

#### REFERENCES


the doubles-pong task on the division of interception space. The results of our analysis did not suggest the presence of any straightforward effect of skill difference. Since we did not question our participants on this, we cannot say whether skilllevel differences were consciously perceived by the players. Still, the boundaries between interception domains varied over teams. Whereas these boundaries were mostly halfway in-between the initial positions of the players' paddles in the Benerink et al. (2016) study, there was considerably more variability among teams in the present study (see also Benerink et al., 2018). Perhaps, the asymmetries (see Lagarde, 2013) introduced by the skill differences in the present study did have an effect on the emerging patterns. However these asymmetries played out, the action-based account was able to capture the observed patterns. These patterns emerged from the informational couplings among the players and their environment (e.g., Mottet et al., 2001; Richardson et al., 2015; Davis et al., 2017; Nalepka et al., 2017), coordinating under the constraint of not colliding (see also Richardson et al., 2015). Although we cannot totally rule out an account based on shared knowledge of a boundary and the ability to correctly predict BAP, this action-based account seems the most parsimonious, and, thus, most promising to us.

#### AUTHOR CONTRIBUTIONS

NB, RC, FZ, and RB conceived and designed the study. NB ran the study. AvO, NB, FZ, and RB analyzed and interpreted the data. All authors participated in drafting the work and/or revising it critically for important intellectual content. The final version submitted was approved by all authors.


Kim, S. (2004). Voronoi analysis of a soccer game. Nonlinear Anal. 9, 233–240.

Lagarde, J. (2013). Challenges for the understanding of the dynamics of social coordination. Front. Neurorobot. 7:e18. doi: 10.3389/fnbot.2013.00018


behavioral dynamics of an interpersonal collision-avoidance task. J. Exp. Psychol. Hum. Percept. Perform. 41, 665–679. doi: 10.1037/xhp0000041


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 van Opstal, Benerink, Zaal, Casanova and Bootsma. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Responding to Other People's Posture: Visually Induced Motion Sickness From Naturally Generated Optic Flow

Henry E. Cook IV, Justin A. Hassebrock and L. James Smart Jr.\*

Department of Psychology, Miami University, Oxford, OH, United States

Understanding the relationship between our actions and the perceptual information that is used to support them is becoming increasingly necessary as we utilize more digital and virtual technologies in our lives. Smart et al. (2014) found that altering the relationship between perception and action can have adverse effects, particularly if the perceptual information cannot be used to guide behavior. They also found that motion characteristics varied between people who remained well and those that became motion sick. The purpose of this study was to determine the influence of naturally produced virtual motion on postural regulation and examine how people respond to different types of optical flow (produced by other people). Participants were either exposed to optic flow produced by the postural motion of a person who did not become motion sick, or a person who did exhibit motion sickness from Smart et al. (2014). It was discovered that participants exhibited both stronger coupling and more incidents of motion sickness in response to optic flow generated by a non-sick participant. This suggests that participants recognized the potentially usable nature of the well-produced optic flow- but the open loop nature of the stimuli made this perception disruptive rather than facilitative.

Keywords: motion sickness, posture, optic flow, perception and action, virtual reality, head mounted displays

# INTRODUCTION

IMAX (large screen format) theaters, high definition television, immersive virtual and gaming environments, as well as commercial grade head-mounted displays (HMDs) are becoming increasingly commonplace technologies that are expanding the realm of possibilities for novel experiences and interactions, while at the same time facing some enduring challenges for widespread successful engagement. One of the most common challenges is the potential for motion sickness and similarly documented ailments (e.g., cyber sickness, simulator sickness); particularly when depicting some form of self-motion. Further complicating this issue is that "simply" improving the technology does not mitigate this issue and may in many instances make it more prevalent (Biocca, 1992; Palmisano et al., 2017). Thus solutions to preventing motion sickness may reside within changing how virtual technology is implemented rather than how it is designed. Successfully changing implementation necessitates understanding the relationship between our actions and the perceptual information that is used to support them.

#### Edited by:

Mauro Murgia, University of Trieste, Italy

#### Reviewed by:

Rahul Goel, Baylor College of Medicine, United States Zhixian Cheng, Yale University, United States

#### \*Correspondence:

L. James Smart Jr. SmartLJ@miamioh.edu

#### Specialty section:

This article was submitted to Movement Science and Sport Psychology, a section of the journal Frontiers in Psychology

Received: 21 February 2018 Accepted: 18 September 2018 Published: 08 October 2018

#### Citation:

Cook HE IV, Hassebrock JA and Smart LJ Jr (2018) Responding to Other People's Posture: Visually Induced Motion Sickness From Naturally Generated Optic Flow. Front. Psychol. 9:1901. doi: 10.3389/fpsyg.2018.01901

**101**

Exploring the link between perception and action in the context of motion sickness can be traced back to Riccio and Stoffregen (1991) Postural Instability theory, which states that poor outcomes such as motion sickness should be characterized as perception-action problems rather than perceptual-processing issues (such as sensory conflict theory; Reason and Brand, 1975; Oman, 1990). Simply put, Riccio and Stoffregen (1991) assert that motion sickness and other negative outcomes emerge from degraded postural control strategies (i.e., instability) that develop over time, rather than a cognitive inability to resolve sensory conflicts. Since proposing postural instability as a causal mechanism, researchers have provided support by demonstrating that the manipulation of visual stimulation (i.e., optic flow) can perturb postural stability and in turn produce an increase in subsequent reports of motion sickness (Stoffregen and Smart, 1998; Stoffregen et al., 2000; Smart et al., 2002, 2007; Villard et al., 2008). These researchers were able to produce disruptions in participants' actual postural motion by exposing participants to computer-generated motion; in these studies, a sum of 10 sine waves that simulated the optic flow that is typically produced by postural sway. Importantly, these studies were able to show that postural disruptions occurred prior to reports of motion sickness symptomology.

In an extension of this paradigm, Smart et al. (2014) sought to determine how changes in the complexity of optic flow and changes in the manner of behavioral coupling (i.e., the relation between available optical information and participants' physical motion) influence differences in postural regulation between Motion Sick individuals and Well participants. The researchers manipulated the complexity of optic flow by presenting simple sinusoidal motion or naturally generated (from the participants' own movements), complex motion. Coupling was manipulated by either playing back previously recorded sway, or generating flow in real time based on the participants' movements. In the real time conditions, the relationship between sway and optic flow was either anti-phase (what we normally experience) or in-phase (where moving forward produces contraction rather than expansion of the stimuli). It was found that incidences of motion sickness increased with more complex motion and when the behavioral coupling was altered (in-phase) or when the researchers presented participants with their own motion, decoupled (i.e., not real-time). Interestingly what this study revealed were differing patterns of postural motion/structure for Well participants and Motion Sick participants. What was discovered is that postural motion preceding reports of motion sickness tends to increase in magnitude and spatial complexity over time while remaining temporally rigid (motion pattern persists over time, once the disruption occurs the participants do not recover), while postural motion for those that remain well tend to exhibit the opposite trend. Importantly in this study, each participant's motion was reflected in the structure of optic flow presented in the virtual environment (VE). This raises the question of whether these patterns of optic flow are inherently facilitative or disruptive, that is, do they carry the "information" for successful (or unsuccessful) behavioral regulation?

The current study was designed to examine the behavioral characteristics exhibited by participants who are exposed to other people's postural (OPP) motion. In particular, our goal was to determine how participants respond to OPP; specifically can they utilize OPP to successfully regulate their own postural sway? We addressed these questions by exposing participants to two types of optical motion in a VE by way of a HMD. Participants were exposed to optical flow created from (1) postural motion previously recorded from an individual in a previous study (Smart et al., 2014) who successfully completed her/his postural regulation trials without reporting motion sickness (Well-flow) or, (2) postural motion previously recorded from an individual who completed his/her postural regulation trials but reported motion sickness (Sick-flow). The specific trials employed in this study were chosen because their motion parameters (PL, EA, PLN, SEn) closely matched the overall means obtained by Smart et al. (2014) for Well and Motion Sick participants.

While we expected based on the results of Smart et al. (2014) that both conditions should produce motion sickness, it was not theoretically clear which condition (Well-flow or Sick-flow) should have produced higher incidences of motion sickness. Following the wave interference hypothesis (Stoffregen and Smart, 1998; Smart et al., 2002) that suggests that the interaction of similar waveforms will result in greater instability, we expected that the Well-flow condition would produce higher rates of motion sickness because its structure would likely be close to that that could be produced by the current participants (prior to incidences of instability/sickness, which develop over time). However, Smart et al. (2014) found that the greatest incidence of motion sickness occurred in the condition that had the least informative stimuli suggesting that the Sick-flow condition would be likely to produce higher rates of sickness given the likelihood that the information (structure) provided in this stimuli would not be supportive of successful postural regulation. Fortunately, this is an empirical question that can be addressed by the current study.

Whatever the incidence rate between conditions, we expected to find a similar divergence in the postural sway dynamics between motion sick participants and those who remain well as was found by Smart et al. (2014). The same set of postural sway measures employed by Smart et al. (2014) [Path Length (PL), Elliptical Area (EA), Normalized Path Length (PLN), and Sample Entropy (SEn)] were utilized in the current study.

Finally, to address the utility of the "information" provided by OPP we assessed the degree to which participants coupled or become entrained with the stimuli. We hypothesized that Wellflow, which represents a person who successfully regulated his or her sway, would potentially allow for easier regulation of sway in the VE. In contrast, Sick-flow, which depicts motion from a person who became unstable (and subsequently motion sick), would provide insufficient information for regulation. Thus we expected to see less coupling in the Sick-flow condition.

To determine whether coupling differs between optic flow conditions, a set of non-linear synchronicity analyses were conducted on the postural sway data. The analyses used to determine synchronized behavior include; (1) Average Mutual Information (AMI), which examines the amount of information shared (dependency) between two time series (Thomas et al., 2014); (2) Cross-Correlation (CC), which determines how

linearly correlated two time series are while accounting for time lags between stimuli and response (Strang et al., 2014); (3) Coherence (CoH), which examines frequency coupling (similarity) across the two time series; and (4) Cross-Fuzzy Entropy (CFEn), which determines temporal stability of the coupling between two time series (Strang et al., 2014). We expected that higher AMI, CC, and CoH as well as lower CFEn values would indicate stronger coupling with the optic flow. Given this we predicted that coupling should be higher in the Well-flow condition and with participants who remain well.

## MATERIALS AND METHODS

#### Participants

Forty participants (19 male, 20 female, and one participant who did not specify gender) drawn from the psychology department participant pool were randomly assigned to one of two conditions: Well-flow (10 male, 10 female), and Sick-flow (9 male, 10 female, 1 undisclosed). None of the current participants were involved in the studies reported in Smart et al. (2014). Male participants had a mean (SE) height of 1.81 (0.02) m and weight of 78.23 (2.67) kg, while female participants had a mean (SE) height of 1.69 (0.01) m and weight of 62.43 (1.56) kg. Participants reported being in their normal state of health, and had normal or corrected to normal vision. No participants reported any history of falls, dizziness, or vestibular dysfunction and all participants were able to stand on 1 ft for 30 s with their eyes closed. Participants were instructed not to eat 2 h prior to their experimental session and compliance with this request was verified at the beginning of the sessions. Participants received course credit for their time and were aware that they could cease participation at any time and for any (or no) reason without loss of benefits. As part of the informed consent process, participants were made aware that the experiment could have produced mild motion sickness, but were unaware of the specific hypotheses of the study. The study protocols were approved by the Miami University Institutional Review Board (#00116r). All participants gave written consent in accordance with the Declaration of Helsinki.

# Materials

Materials used in this study were the same as employed by Smart et al. (2014) and described below. The single deviation from the original study involves the baseline stimulus which is discussed in the procedure.

#### Questionnaires

Two different questionnaires were used in this study. The first asked for basic demographic information, motion sickness history and perceived susceptibility to motion sickness (10 point scale with one being not susceptible and 10 being very susceptible). The second questionnaire was the widely used and accepted simulator sickness questionnaire (SSQ; Kennedy et al., 1993), which determines the level of common motion sickness symptoms prior to exposure and the extent to which immersion in a VE subsequently produces and/or elevates those symptoms (determination of sick/well was by verbal report of the participants, not their score on the SSQ).

#### Postural Sway Measurement

A magnetic tracking system was used to record the postural sway of participants (Flock of Birds; Ascension, Inc., Burlington, VT, United States) in the anterior-posterior (AP) and medial-lateral (ML) planes. The system consisted of an emitter that created a low-level magnetic field extending 1 m in radius. A sensor was placed on the top of the participant's head and held in place with athletic prewrap. The AP and ML motion of the sensor disturbed the magnetic field, and these disturbances were then recorded by the computer at a sampling rate of 50 Hz.

#### Head Mounted Display (HMD)

One pair of virtual i-glasses SVGA 3D ASO1317 (I-O display systems, CA) personal displays were used to present the VE. The displays simulated a 1.78 m screen (diagonally) that is 3.96 m away from the viewer's eyes resulting in a field of view of 24◦ (diagonally). The HMDs were only partially immersive (participants could see the lab below and peripherally). Thus, during exposure, the laboratory lights were turned off.

#### Virtual Environment

The VE consisted of a spherical "star field" consisting of a pattern of randomly placed white dots on a black background in the shape of a sphere. The sphere was positioned such that participants were "standing" in the center of the sphere with "stars" located at a starting distance of about 3.3 m away. The stars in the field were made to translate in the AP plane for all conditions and trials. In addition, stars would change from white to red for a period of 3 s at quasi-random intervals during experimental trials (14 shifts in each trial) and were used in the manipulation check to ensure that participants were engaged in the task. Following Smart et al. (2014) AP motion of the star field was amplified (15x) relative to the motion of the participant so that visual change was both observable and smooth. The motion path for the star field was generated from the data of two participants' last experimental trial from Smart et al. (2014); one who did not become motion sick, and one who did report motion sickness. The data was chosen because the sway properties of these two participants most closely matched the overall pattern of results discovered by Smart et al. (2014). The Well participant's data showed decreases over time in PL, PLN, and EA coupled with relatively higher SEn. The Sick participant's data exhibited the opposite pattern (the general finding of Smart et al., 2014; see **Figure 1**).

#### Hardware/Software

One computer (Dell Optiplex GX270) was used in this study to display the stimulus through the HMD and simultaneously record the postural sway of participants. Participants' sway was recorded using the same software package that was used to create the star field stimuli and display it to participants (Vizard; version 2.53; World Viz, Santa Barbara, CA, United States).

#### Procedure

Upon entering the lab, participants were presented with a consent form that explained the purpose of the experiment and their

rights. After signing the consent form, participants were asked to complete the two questionnaires described above (i.e., SSQ and sickness history). Also, participants were asked to keep the symptoms described in the SSQ in mind during the experiment, and in the event of an increase or emergence of these symptoms, to inform the researchers immediately so that the experiment could be halted.

For safety, and to ensure that participants had comparable balance capabilities prior to exposure, participants were asked to complete two balance checks. The first involved walking a line in heel-toe fashion (standard field sobriety test) and the second had participants stand on their preferred leg with their eyes closed for 30 s. If a participant was unable to complete either of these checks, she or he was excused from the study. No participants were excluded from the study as a result of the balance checks.

The experiment consisted of up to three trials (depending on whether the participant became motion sick), each with a duration of 10 min. The first trial was used to assess baseline (with static computer-generated stimulus) postural sway, during which the participants stood bipedally in the lab. The stimulus was the star field sphere zoomed out in the HMD so that it appeared to be a flat circle of white dots in an otherwise black background about 0.5 m in diameter. The star field was static during this baseline trial, however, it did still shift occasionally from white to red.

Following the baseline trial, up to two experimental trials (20 min of exposure) were conducted, depending on whether the participants became motion sick via self-report. This a reduction in trials from Smart et al. (2014) as they noted that the majority of motion sickness reports occurred by the end of the second trial. In the Well-flow condition participants were exposed to optical flow generated from the postural motion of a participant who did not become motion sick in Smart et al. (2014) study. In the Sick-flow condition participants were exposed to optical flow generated from the postural motion of a participant who became motion sick in Smart et al. (2014) study. Both conditions were open-loop presentations in that the participants' current movements did not impact the optical motion generated in the HMD. The well and motion sick data were chosen by the experimenters and reflected the general motion profiles for well and motion sick participants obtained by Smart et al. (2014) (see **Figure 1**). In the experimental trials, the participants were asked to remember how many times the stars in the field changed from white to red (color shift task; a manipulation employed to ensure that they were paying attention to the stimuli). At the conclusion of each trial, the participants were asked (1) how many times the stimuli changed from red to white and (2) how the participants felt (if they experienced any symptoms of motion sickness).

In the event that the participants indicated symptoms of motion sickness, the experiment was stopped (even if it was in the middle of a trial). The participants once again filled out the SSQ indicating the new level of their symptoms. They were allowed to rest and asked to stay in the laboratory for observation for 15 min. After this time, if the participants felt better, they were allowed to leave after successfully repeating the two balance

checks. If the participants had no symptoms of motion sickness at any time during the trials, they were asked to complete the SSQ after completing the last trial. As before, the participants were only allowed to leave after successful completion of the balance checks. In either case, the participants were given a third copy of the SSQ. In the event that the participants exhibited symptoms at some time (up to 24 h) after leaving the laboratory, they were asked to fill out the questionnaire at that time and return it. If the participants had no symptoms, they were asked to complete and return the questionnaire approximately 24 h after completing the experiment.

# RESULTS

As mentioned in the introduction the purpose of this analysis is threefold; (1) to determine if the optic flow generated by previous Motion Sick and Well participants produce different rates of motion sickness, (2) to determine if similar divergences in postural sway characteristics emerge between current Motion Sick and Well participants as found in Smart et al. (2014), and (3) to determine if the optic flow generated by previous Motion Sick and Well participants differentially influence the postural regulation of participants in the current study. To address these questions, we recorded motion sickness incidence rates and symptomology, and analyzed both structural and temporal properties of participants' postural motion individually as well as in relation to the stimuli (i.e., coupling). As in Smart et al. (2014), postural motion was analyzed in 2 min windows (such that a person completing both experimental trials would have 10 values for each measure). As in the previous research a linear slope was derived from the trend line created by the measures at each time window (i.e., the value obtained from each 2 min window, five values per 10 min trial) as an additional index of how regulation evolved over time.

# Color Shift Performance (Manipulation Check)

For the baseline trial all participants regardless of condition correctly identified the number of color shifts (14). For the experimental trials, a 2 (Condition) × 2 (Health) betweengroups ANOVA revealed a significant main effect for Health, F(1,36) = 8.62, p < 0.05, η<sup>p</sup> <sup>2</sup> = 0.19. Well participants [M(SE) = 14 (0.3), 100% accurate] were more accurate in detecting red shifts than Motion Sick participants [M(SE) = 12.34 (0.5), 88% accurate]. Flow condition did not significantly influence accuracy although participants in the Well-flow condition [M(SE) = 13.5 (0.4), 96% accurate] were slightly more accurate than those in the Sick-flow condition [M(SE) = 12.9 (0.4), 92% accurate].

#### Motion Sickness History and Incidence

Participants were asked to rate their susceptibility to motion sickness on a 10 point scale (with 10 being very susceptible) prior to exposure to the stimuli. Participants who became motion sick in the current study reported a mean (SE) susceptibility of 3.08 (0.58) out of 10; participants who remained well reported a mean (SE) susceptibility of 3.64 (0.39) out of 10. This difference was not significant. In addition, reported susceptibility did not differ significantly between males [3.32 (0.48)] and females [3.55 (0.46)] nor between Well-flow [3.60 (0.51)] and Sick-Flow [3.35 (0.40)].

Overall there were 12 (6 male and 6 female) explicit reports of motion sickness (30%). Five (42%) of these participants reported past motion sickness. Notably, the majority of sickness reports [8 participants (67%)] occurred during exposure to the Well-flow stimulus (5 male and 3 female, 40%). In the Sick-flow condition 1 male and 3 females (20%) reported motion sickness. A chisquared analysis of the incidence rates revealed that they were not significantly different from the average incidence rate of 42% for visually induced motion sickness studies (Playback and Normal Coupling conditions – Stoffregen and Smart, 1998; Smart et al., 2002, 2014; Villard et al., 2008), nor were they significantly different from each other.

#### SSQ

Simulator Sickness Questionnaire data for the two flow conditions were analyzed together. Pre-Post (Wilcoxon Signed Rank test), and Sick-Well (Mann–Whitney U test) comparisons were performed. We also ran a comparison analysis across flow conditions (Mann–Whitney U test) averaging over health of the participant. While Kennedy et al. (1993) developed a method for normalizing SSQ scores, since the original data is at best ordinal level measurement, we felt that non-parametric statistics were more appropriate to run in this case. The analyses revealed that pretest scores did not differ significantly between Motion Sick and Well participants, or between flow conditions, for any of the subscales or total SSQ scores. However, posttest scores for each subscale as well as the total score differed significantly for Motion Sick and Well participants (p < 0.05). The magnitude of reported symptom severity by the participants who self-identified as motion sick across conditions (See **Figure 2**) was comparable to those typically reported in VEs (typical range is 19–55; Kennedy et al., 2003). There was a significant difference (p < 0.05) in Oculomotor post subscales scores between those exposed to Well-flow (30.18) and those exposed to Sick-flow (19.71). There were no differences between Motion Sick participants in the two conditions nor were there differences between the Well participants in the two conditions. It should be noted that in general post test scores (both total and subscale) were significantly higher (p < 0.05) than pretest scores (regardless of health) suggesting that while only 30% of the participants explicitly reported motion sickness, nearly all participants reported increases in symptomology. This highlights the caution that should be taken with relying on SSQ responses as the main tool used to determine motion sickness.

#### Postural Response

As in Smart et al. (2014), we examined two measures of magnitude (PL and EA) and two measures of structure (PL<sup>N</sup> and SEn) to determine if there were characteristic sway differences between sick and well participants. As in Smart et al. (2014) we calculated these measure for each 10 min trial in 2 min windows (corresponding to 6000 data point time-series) such that if participants completed both experimental trials they would

have 10 values for each measure. Replicating the analysis of Smart et al. (2014) we analyzed both raw values and derived slopes over the trials/windows to look for trends that may emerge over time. Three way mixed ANOVAs [Condition (2) × Health (2) × Window (5)] were performed on the raw data and Two way between ANOVAs [Condition (2) × Health (2)] were performed on the slope measures.

#### Baseline Trial (No Stimulus Movement)

Analysis of the raw values revealed significant effects of time window for all four postural response measures. PL F(4,140) = 7.56, p < 0.05, η<sup>p</sup> <sup>2</sup> = 0.18, and PL<sup>N</sup> F(4,140) = 12.76, p < 0.05, η<sup>p</sup> <sup>2</sup> = 0.27 both exhibited u-shaped patterns with higher values during the first 2 min (0–2 min) window and last 2 min (8– 10 min) window. **EA** F(4,140) = 7.46, p < 0.05, η<sup>p</sup> <sup>2</sup> = 0.07 and **SEn** F(4,140) = 6.97, p < 0.05, η<sup>p</sup> <sup>2</sup> = 0.17 both showed linear increases over time windows. There were no significant differences for Flow condition (Well-flow, Sick-flow), Participant Health (Well, Motion Sick), nor the interaction between Condition and Health.

#### Path Length (PL)

The analysis of the raw values revealed a significant interaction between condition and health, F(1,70) = 3.74, p < 0.05, ηp <sup>2</sup> = 0.05. In the Well-flow condition Motion Sick participants [M(SE) = 2.4 (0.38) m] moved to a greater extent than the Well participants [M(SE) = 1.48 (0.25) m]. In the Sick-flow condition the opposite pattern emerged with Well participants [M(SE) = 2.02 (0.22) m] exhibiting more sway than Motion Sick participants [M(SE) = 1.55 (0.52) m]. See **Figure 3** for a depiction of these results. There were no other significant effects for the raw values, nor were there any significant effects revealed by the analysis of the slope data.

#### Elliptical Area (EA)

The analysis of the slope data revealed a significant effect of health, F(1,72) = 3.77, p < 0.05, η<sup>p</sup> <sup>2</sup> = 0.05. Motion Sick

participants [M(SE) = 15.01 (8.18) cm<sup>2</sup> /2 min] exhibited a higher rate of change in magnitude of motion over time while Well participants [M(SE) = −3.45 (4.83) cm<sup>2</sup> /2 min] exhibited a lower rate of change in magnitude of motion over time. This indicates that Motion Sick participants generated larger movement overall and at relatively faster rate than Well participants. The Well participants tended to decrease their movement overall, doing so at a slower rate. See **Figure 4** for a depiction of these results. There were no other significant effects, nor were there any significant effects revealed by the analysis of the raw values.

#### Normalized Path Length (PLN)

The analysis of the raw values revealed a significant effect of window (time), F(4,280) = 10.74, p < 0.05, η<sup>p</sup> <sup>2</sup> = 0.13. This effect was produced by the complexity of motion being significantly lower in the second 2 m window [minutes 2–4; M(SE) = 81.28 (4.66) a.u.] than the other time windows which did not differ significantly [minutes 0–2 and 5–10; M(SE) = 92.56 (4.94) a.u.] regardless of health or condition. No other significant effects were

FIGURE 4 | Mean (SE) Elliptical Area Slope as a function of Condition and Health (N = 40).

revealed, nor were there significant effects revealed by the analysis of the slope data.

#### Sample Entropy (SEn)

The analysis of the raw values revealed a significant effect of window (time), F(4,284) = 8.13, p < 0.05, η<sup>p</sup> <sup>2</sup> = 0.1. SEn values increased over windows regardless of flow condition or health status of the participants. The increase occurred between the first 2 min window [minutes 0–2; M (SE) = 0.17 (0.01)] and third 2 min window [minutes 4–6; M (SE) = 0.2 (0.0)]. This indicates that participants' movement strategies became more variable during the early stages of the trials. No other significant effects emerged, nor were there any significant findings from the analysis of the slope data.

# Postural Coupling

#### Average Mutual Information (AMI)

The analysis of the raw values revealed a significant effect of condition, F(1,72) = 12.87, p < 0.05, η<sup>p</sup> <sup>2</sup> = 0.15. Regardless of health status, participants' motion exhibited higher magnitudes of coupling with the stimulus during the Well-flow [M(SE) = 0.36 (0.01)] condition than during the Sick-flow [M(SE) = 0.3 (0.01)] condition. Participants were influenced to a greater degree by the Well-flow stimulus than the Sick-flow stimulus. See **Figure 5** for a depiction of these results.

#### Cross-Correlation

As we were interested in the magnitude of coupling rather than the type of coupling per se, the analysis of raw and slope values were performed on absolute values of the correlations. The analysis of the raw values revealed a significant effect of condition, F(1,70) = 8.36, p < 0.05, η<sup>p</sup> <sup>2</sup> = 0.11. Regardless of health status, participants' motion exhibited higher magnitudes of coupling with the stimulus during the Well-flow [M(SE) = 0.198 (0.01)] condition than during the Sick-flow [M(SE) = 0.148 (0.01)] condition. As shown in the AMI analysis Participants were influenced to a greater degree by the Well-flow stimulus (although the correlations were in the weak range). See **Figure 6** for a depiction of these results.

#### Cross Fuzzy Entropy and Coherence

Analysis of these measures failed to reveal any significant effects.

#### DISCUSSION

In this study, we exposed participants to naturally generated optical flow produced from OPP motion. Importantly, the motion profiles represented a person who reported motion sickness and a person who remained healthy; more generally, these stimuli represented the divergent motion patterns observed in healthy and motion sick participants (Smart et al., 2014). We observed that participants' sway patterns were differentially influenced by the two flow types and importantly different rates of reported motion sickness occurred with twice as many participants becoming motion sick in the Well-flow condition (however, the difference in rates while noteworthy, was not significant).

Similar to the findings of Smart et al. (2014), increases in the magnitude of motion (PL, EA slope) were observed in Motion Sick participants. However, a key difference from the previous study was revealed by examining sway PL, in that the pattern of increase was specific to the Well-flow condition. In the Sick-flow condition, we observed that Well participants exhibited higher magnitudes of motion (PL). In this case the increased motion may have been adaptive as the frequency of motion sickness was less than we observed in the Well-flow condition. Overall, we observed a similar divergence in EA slope (rate as with the magnitude of the participants' motion changes) that was reported

by Smart et al. (2014); with the Motion Sick participants showing a more rapid rise in magnitude, while the Well participants were slower to change and tended to have lower magnitudes of sway. It is also important to note again that the major difference between the current study and that Smart et al. (2014) is that in the previous study the open loop (playback) condition presented participants with their own motion rather than another person's movements.

It may seem surprising that optical flow from a non-sick participant would seemingly produce more motion sickness in current participants, however, the incident rate (40%) is consistent with what was observed in response to baseline recordings (prior to any sickness) of participants' own motion (playback condition of Smart et al., 2014) and in fact was statistically equivalent to the rate (20%) observed in the Sickflow condition. In general the incidences of motion sickness in these conditions are consistent previous research utilizing open loop presentations of optic flow (Stoffregen and Smart, 1998; Villard et al., 2008). The data also lend support to the wave interference hypothesis posited by Stoffregen and Smart (1998) and Smart et al. (2002). This hypothesis states that like in other physical systems, when two waveforms interact, the closer in nature (amplitude, frequency) the two waveforms are the more catastrophic the interaction will be. Additionally, the differential postural response to the two types of optic flow, suggest that participants were sensitive to structural differences in the flow.

Functionally what this may indicate is that the Wellflow condition presented participants with structure that they perceived as "useful" or "usable" (i.e., sufficient to guide behavior) as evidenced by the stronger coupling (increased synchrony: AMI, CC) to the Well-flow stimulus (and to slightly higher extent for Well participants). This suggests that what may be occurring is that the participants are attempting to dynamically synchronize with the stimuli (evidenced by the stronger coupling exhibited in the Well-flow condition) but at times failing to do so appropriately, hence the increase in reports of motion sickness. Despite the perceived "usability" of the Well-flow, the open-loop nature of the stimulus prevents a true perception-action coupling and renders it disruptive rather than facilitative. This is supported in part by the significantly higher post immersion reports of oculomotor discomfort in the Well-flow condition. In the case of the decreased coupling observed in the Sick-flow condition, this may represent participants' ability to discriminate abnormal or non-usable structure and their attempts to adjust their sway to compensate for the lack of "appropriate" structure. The analysis suggests that this may be the case as we observed some increases in PL, SEn, and PL<sup>N</sup> for Well participants (although not significant).

The divergent patterns of sway characteristics between Well and Motion Sick participants observed in this study not only lend support to the assertion that postural motion can be used as a reliable means to assess potential motion sickness, but also supports the idea that behavior requires perceivable causal mechanisms to enact (successful) actions in support of an intended goal. The Well-flow condition seemingly provides information that participants are not only able to detect, but specify how to support an ongoing action (stable posture). It would appear, however, that providing information without any means of actualizing their function can lead to clear disruptions in behavior.

These findings also have design implications for virtual technologies as there is a resurgence in attempts to make head-mounted, first-person displays commercially viable. Motion Sickness continues to be a significant issue with the technologies that cannot be alleviated with general design improvements alone. Instead, the solutions sought should examine how one can support the emergence of "natural" perception-action relations in these virtual contexts. Doing so requires the examination of both what information/structure is available to the person as well as what actions are supported. If you are going to provide information that suggests that a given behavior or regulatory strategy is possible, the system needs to allow for that behavior/strategy to be implemented. This is important as the data from this study reveal that "open loop" presentations of information that are perceived as consequential can lead to disruptions in behavior and ill-effects. For example in many first-person perspective games, "bob and sway" are often coded into the stimulus to represent body movement. The addition of this non-controllable sway information is analogous to our experimental manipulation, and has been indicated as a factor in the emergence of motion sickness (Dong et al., 2011; Sharp, 2013). In this study, some participants were unable to modulate their behavior successfully at least in part due to the absence of consequential feedback which is characteristic of open-loop presentations. The disruptions observed in these open-looped systems illustrate the consequences of natural perception-action suppression commonly seen in VE and simulations, especially when potentially exploitable information can be acquired, but not fully utilized by the user. In short, the mere presentation of swaylike optical flow may not be sufficient for successful regulation of behavior in virtual environments, particularly without the ability to engage in real–time interaction with this optical information.

# AUTHOR CONTRIBUTIONS

This project was a graduate project of HC and JH. LS supervised the study. HC, JH, and LS designed, conducted, and analyzed the project. Each author contributed significantly to the writing of the manuscript.

# FUNDING

This research was supported by a National Research Foundation (United States) Graduate Research Fellowship awarded to HC.

# ACKNOWLEDGMENTS

Portions of this research were presented at the 2014 International Society for Ecological Psychology North American Meeting, Oxford, OH, United States and at the 2016 International Society for Ecological Psychology North American Meeting, Clemson, SC, United States.

# REFERENCES

fpsyg-09-01901 October 5, 2018 Time: 17:47 # 9


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Cook, Hassebrock and Smart. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# My Action, My Self: Recognition of Self-Created but Visually Unfamiliar Dance-Like Actions From Point-Light Displays

Bettina E. Bläsing1,2 \* and Odile Sauzet3,4

<sup>1</sup> Neurocognition and Action – Biomechanics Research Group, Faculty of Psychology and Sport Science, Bielefeld University, Bielefeld, Germany, <sup>2</sup> Center of Excellence Cognitive Interaction Technology (CITEC), Bielefeld University, Bielefeld, Germany, <sup>3</sup> Bielefeld School of Public Health/AG 3 Epidemiology & International Public Health, Bielefeld University, Bielefeld, Germany, <sup>4</sup> StatBeCe, Center for Statistics, Bielefeld University, Bielefeld, Germany

Edited by:

Mauro Murgia, University of Trieste, Italy

#### Reviewed by:

Selenia Di Fronso, Università degli Studi G. d'Annunzio Chieti e Pescara, Italy Bernadette Ann Murphy, University of Ontario Institute of Technology, Canada

\*Correspondence: Bettina E. Bläsing bettina.blaesing@uni-bielefeld.de

#### Specialty section:

This article was submitted to Movement Science and Sport Psychology, a section of the journal Frontiers in Psychology

Received: 25 May 2018 Accepted: 18 September 2018 Published: 16 October 2018

#### Citation:

Bläsing BE and Sauzet O (2018) My Action, My Self: Recognition of Self-Created but Visually Unfamiliar Dance-Like Actions From Point-Light Displays. Front. Psychol. 9:1909. doi: 10.3389/fpsyg.2018.01909 Previous research has shown that motor experience of an action can facilitate the visual recognition of that action, even in the absence of visual experience. We conducted an experiment in which participants were presented point-light displays of dance-like actions that had been recorded with the same group of participants during a previous session. The stimuli had been produced with the participant in such a way that each participant experienced a subset of phrases only as observer, learnt two phrases from observation, and created one phrase while blindfolded. The clips presented in the recognition task showed movements that were either unfamiliar, only visually familiar, familiar from observational learning and execution, or self-created while blind-folded (and hence not visually familiar). Participants assigned all types of movements correctly to the respective categories, showing that all three ways of experiencing the movement (observed, learnt through observation and practice, and created blindfolded) resulted in an encoding that was adequate for recognition. Observed movements showed the lowest level of recognition accuracy, whereas the accuracy of assigning blindfolded selfcreated movements was on the same level as for unfamiliar and learnt movements. Self-recognition was modulated by action recognition, as participants were more likely to identify themselves as the actor in clips they had assigned to the category "created" than in clips they had assigned to the category "learnt," supporting the idea of an influence of agency on self-recognition.

Keywords: action recognition, self-recognition, motor learning, point-light walker, dance-like actions

# INTRODUCTION

Human body motion has been studied by many authors using point-light displays in which only white dots on a black background indicate relevant parts (joints) of a moving body (Johansson, 1973). Such displays are used as stimuli that contain only movement information without any additional information about the person (see Thornton, 2006; Blake and Shiffrar, 2007). Studies using point-light walkers have shown that the moving dots representing a body in motion reliably convey information about the person's familiarity (Cutting and Kozlowski, 1977; Troje et al., 2005),

gender (Kozlowski and Cutting, 1977; Pollick et al., 2005), and emotional state (Dittrich et al., 1996; Atkinson et al., 2004). Fewer studies focused directly on identifying the type of action performed (Dittrich, 1993), however, studies in which point light displays representing different types of action were compared and showed that actions differ with regards to the information they reveal about the actor's identity (Loula et al., 2005; Sevdalis and Keller, 2009). Recent literature suggests that body motion (e.g., gait) in general contributes significantly to person recognition in real-world scenarios, in particular from a distance or in uncertain viewing conditions, whereas from close-up, the face is the primary cue for recognition (Rice et al., 2013; Hahn et al., 2015). Yovel and O'Toole (2016) provide a framework explaining person recognition in the real world, suggesting that dynamic information, in the form of dynamic identity signatures, plays the central role in binding together information from face, body, and voice into a multi-modal dynamic representation of a person, and that this binding function is the main contribution of the superior temporal sulcus to social cognition. Taken together, these studies corroborate that body motion plays a crucial role in person recognition and that the type of action as well as the action context interacts with this process.

Comparing different types of actions represented as pointlight displays, Sevdalis and Keller (2009) found that free dancing resulted in better self-recognition from point-light displays compared to walking and clapping, and suggested that this finding was based on the more pronounced "kinematic fingerprint" of the improvised dance movement compared to other actions. Dance movements often do not involve interactions with objects or persons and have no obvious external goal that can be referred to with respect to its anticipated outcome (e.g., Prinz, 1997; Hommel et al., 2001). Instead, dance movements typically possess internal goals that are related to the movement itself, its trajectory, dynamics, and expression. Schachner and Carey (2013) refer to actions that do not have obvious external goals as "dance-like," even if these actions are not performed in a dance context. The authors showed that observers tended to interpret actions as intentionally movementrelated (and thereby "dance-like") if they were not able to infer external goals from observing the actions, or if the actions seemed inefficient or inappropriate with respect to any potential external goal. In dance training, movement learning is most commonly practiced by observation of a human model, and observational learning has proved to be the most successful learning mode (Schmidt, 1975, 2003; Blandin and Proteau, 2000; Hodges et al., 2007). Performing movement with closed eyes, however, is considered a meaningful practice in modern and contemporary dance training, as it provides an unusual experience with enhanced perception of kinesthetic, proprioceptive, haptic, and acoustic information. In this study, our aim was to apply the movement-based approach to action and actor recognition provided by the use of point-light displays to dance-like actions that had been acquired in the absence of visual feedback.

Casile and Giese (2006) showed that motor experience of an action (walking) can facilitate the visual recognition of that action, even in the absence of observational learning or visual experience. In their study, they applied a learning paradigm based on verbal and haptic feedback to dissociate visual and motor learning of unusual gait patterns. The results showed that visual recognition of the non-visually learnt material was improved compared to similar but untrained movement material and that recognition performance correlated with the ability to perform the learnt movement. The authors concluded that non-visual motor learning has a direct influence on visual action recognition.

Our aim was to extend the findings by Casile and Giese (2006) to non-cyclic full-body movements other than gait patterns. Additionally, in order to make sure that movement representations were based solely on proprioceptive and kinaesthetic feedback, we used motor actions that were not learnt through haptic guidance or verbal instruction, but created by the participants themselves in the absence of vision. In our study, we investigated to what extent participants were able to recognize movements they had created and performed while they were blindfolded from visual observation of point-light displays. Visual recognition performance of these blindfolded self-created movements was compared to that of learnt (via observation and movement practice) and only observed (without physically moving along) movements, and to unfamiliar movements as control. We expected that the "blind-created" movements could be recognized successfully from visual observation equally well as movements that had been learnt from observation and imitation, based on the multimodal nature of the action representation built up during the creation and execution of the movement. Furthermore, based on previous studies (e.g., Loula et al., 2005; Sevdalis and Keller, 2009), we expected that participants would be able to distinguish between themselves and others as performers of the action equally well for actions they had learnt from observation and actions they had created while being blindfolded. The ability to differentiate between oneself and others on the basis of visual and acoustic action information has been investigated by many authors [see Knoblich and Flach (2003) for review], and rich evidence exists that dynamic visual cues (such as those provided by point-light displays) are particularly well suited for self-recognition (e.g., Troje et al., 2005). Self-other discrimination from sound information representing complex motor actions has also been studied in the absence of visual information for sports (e.g., Murgia et al., 2012; Kennel et al., 2014) and musical contexts (see Sevdalis and Keller, 2014). These findings support the view that action representations stored in memory are based on motor execution and practice, and thereby include individualized information deriving from the performer's motor system that can be accessed through different modalities (Flach et al., 2004; Repp and Knoblich, 2004; Repp and Keller, 2010).

# MATERIALS AND METHODS

#### Participants

Nineteen sports students (22–26 years of age, mean 23.4 years; all right handed, four males) participated in this study. Seven out of the 19 students practiced dance or gymnastics regularly, 9 trained ball games (mostly soccer and volleyball), others most practiced sports included tennis, running, and fitness training. All students took part in the same seminar,

participation in the experiment was recommended for their own experience but was not necessary for course credit. The students were not informed about the purpose of the movement recording session before taking part in the following point-light experiment. This study was carried out in accordance with the recommendations of the ethics committee of Bielefeld University. A prospective ethics approval was not required in agreement with the institutional institution's guidelines and national regulations. All subjects gave written informed consent in accordance with the Declaration of Helsinki.

# Procedure

The procedure consisted of two sessions, a movement recording session and a recognition session. During the first session, movement phrases were recorded with groups of students in the biomechanics lab using Vicon motion capture to produce point-light displays as stimuli for the following action recognition experiment. In the second session, each student participated individually in the action recognition experiment. The first session took place on the same day for all participants; the second session was conducted 14–21 days after the first session.

# Action Recording Session

For the first session, students were randomly assigned to six groups of three (in one case four). Two groups entered the biomechanics lab together for a recording session of approximately 1 h, resulting in three recording sessions for all 19 participants. The recording session with two groups took place in the following way: After entering the biomechanics lab, one group was defined by the experimenter as "observers" and instructed to sit on the side of the lab watching the other group attentively and quietly, without moving themselves. The other group was defined as "active group," and each member of this group were equipped with 15 retro-reflective markers positioned the body (one on each foot, knee, hip, hand, elbow, shoulder; two on the forehead; and one on the sternum). Subsequently, one member of the active group was blindfolded with a sleeping mask, and was lead to the middle of the recording space (approximately 2 × 2 m). The two other members of the active group were standing outside the recording space, with sufficient space around them to move freely and to watch their blindfolded group member. The blindfolded participant was then instructed to start moving and create a short movement phrase that s/he considered novel and unusual, and to repeat it until s/he felt confident (creating movement blindfolded had been tried out in the seminar once before, so this practice was not entirely new to the students, but they had not been informed that this would be done during the movement recording session). The two other students were instructed to watch the "movement designer" and to learn the movement by imitating or marking. After the two partners indicated that they felt confident performing the movement, each member of the group was recorded performing the movement individually three to five times using the Vicon motion capture system. The "movement designer" remained blindfolded throughout until the

recording of "his/her" movement with all partners was finished, whereas the "learners" were performing with their eyes open. After the recording of this particular movement, the blindfold was removed from the "movement designer's" eyes and one of the partners ("learners") was assigned the new "movement designer." The procedure was repeated for each member of the "active" group so that everyone took the role of the blindfolded movement designer once, and each member of the active group was recorded performing his/her own and every other member's movement. Subsequently, the active group and the group of observers swapped roles, the observers were seated on the side of the lab, and the whole procedure was repeated with the new active group.

## Recognition Task

The Vicon recordings were transformed into 2D video clips, with all movements being shown from the same distance and perspective (designated front view). Short clips each containing one full performance of each movement were cut from the footage to produce the stimulus material for the movement recognition task. For stimulus presentation, Presentation <sup>R</sup> software (14.8) was used. During the experiment, 36 movements were shown once in randomized order. Each movement clip was preceded by a screen with the text "Movement no. x" (with x being the number of the presented clip, counting from 1 to 36) for 2000 ms, a black screen (500 ms) and a fixation cross (500 ms), and followed by a black response screen during which the stimulus presentation was paused. To continue the experiment (with presentation of the next stimulus), the participant had to press the space bar. Each participant was presented 36 video clips showing 12 different movements, each performed by three different persons (one "movement designer" and two "learners"). Six of the movements (18 clips) were familiar (i.e., recorded during the session the participant had taken part in) and six were unfamiliar (recorded during sessions with other groups). For half of the familiar movements (nine clips), the participant had been observer, whereas the other half had been recorded with him/her being "active" (i.e., the participant had performed these movements him/herself, two sighted as learner and one blindfolded).

Each participant performed the movement recognition task individually in a quiet laboratory. The student was instructed to sit in front of the computer screen and watch the displayed pointlight clips, and to mark the responses in a paper questionnaire with a pen. After each movement clip, when the stimulus presentation was paused, the participant had to answer two questions by ticking the appropriate boxes, and subsequently to press the space bar to activate the presentation of the next stimulus. The questions that had to be answered for each pointlight clip during the experiment (originally in German) were the following:

Question 1: I have ..


Question 2: The person in the video clip ..


#### Statistical Analysis

fpsyg-09-01909 October 13, 2018 Time: 11:59 # 4

The results are presented with descriptive table and comparisons between rates of answers between categories unadjusted for the dependence of answers made by the same participants are obtained with chi-squared tests. Analyses accounting for multiple responses from the same participants were done using multilevel logistic regression. All analyses were performed using Stata (StataCorp., 2015, Stata Statistical Software: Release 14, College Station, TX, United States: StataCorp LP.).

#### RESULTS

#### Action Recognition

Answers to Question 1 (action recognition) were categorized into correct answers (i.e., clips correctly assigned to one of the four categories: unfamiliar, observed, learnt, or self-created) and incorrect answers (clips incorrectly assigned). Numbers of answers given for Q1 are displayed in **Table 1**. The distribution of true and false positives per action category answered and of correct answers per action category are provided unadjusted (**Tables 2**, **3**) as well as adjusted for multiple responses (**Table 4**).

The unadjusted comparison of the distribution of true and false positive per answer categories (**Table 2**) showed a significant difference. The adjusted comparison (multilevel logistic regression) showed that the odds of a true positive when answering "observed" is 77% lower [OR: 0.23; 95% CI: (0.07, 0.81); p = 0.02] than the odds of a true positive when answering "created." The odds of true positives for the answer categories "unknown" and "learnt" did not significantly differ from the answer category "created" (**Table 4**). With respect to our hypothesis, that means that participants indeed recognized and categorized the actions they had created while blindfolded equally well as actions they had learnt from observation and unfamiliar actions they had neither watched nor performed, whereas they were less successful in recognizing actions they had only observed but not performed themselves.

The distribution of correct answers per scenario categories depends significantly on the category (p-value for chi-squared test: 0.01 for **Table 3**). However, the adjusted comparison (multilevel logistic regression) showed no significant difference in the odds giving a correct answer for any type of action category compared to the "created" action category situation (see **Table 4**, Model 2).

#### Actor Recognition

Answers to Question 2 (actor recognition) were categorized into correct answers (i.e., clips correctly identified as showing oneself or not showing oneself) and incorrect answers (clips incorrectly identified). Numbers of answers given for Q2 are displayed in **Table 5**.

Identification of the actor as oneself or not oneself in this task was only meaningful for actions that the participant had performed him- or herself. The next important step therefore is



TABLE 2 | True and false positives per action category answered.


TABLE 3 | Correct answer per action category.


#### TABLE 4 | Results of the multilevel logistic regression.


TABLE 5 | Numbers of participants' answers given for identification of the actor as self or non-self.


to analyze the results for Question 2 with regards to those for Question 1. This is especially important as participants' answers to the two questions were not independent of each other, but were given successively for each clip, both in the same trial. When relating self-recognition to action recognition, a difference has to be made between self-recognition with respect to actions the participant had (correctly or incorrectly) assigned to the categories "learnt" or "created" (**Table 6**), and self-recognition with respect to actions that indeed belonged to these categories (**Table 7**). Therefore, in the following, we will differentiate between these two scenarios.

Given that the answer to the first question was "learnt" or "created," the odds of self-recognition was 75% [OR: 0.24, 95% CI (0.11,0.52)] lower for those answering "learnt" compared to those answering "created." Reducing the comparison to the questions for which "self " was true (50 answers for 19 participants) the odds of self-recognition was 98% [OR: 0.02, 95% CI (0.00, 0.56)] lower for those answering "learnt" compared to those answering "created." Reducing the comparison to the questions for which "self " was not true (107 answers for 19 participants) the odds of self-recognition was 65% [OR: 0.34, 95% CI (0.12, 0.98)] lower for those answering "learnt" compared to those answering "created." This means that participants were more likely to recognize themselves as actors (correctly or not) in clips they had assigned to the category "created" than those they had assigned to the category "learnt." This effect was much stronger in the situation were self was actually true.

# DISCUSSION

In a study with participants who had only basic dance experience, we were interested in the participants' ability to recognize

TABLE 6 | Numbers of participants' answers given for identification of the actor as self or non-self for action categories categorized as "created" or "learnt."


TABLE 7 | Numbers of participants' answers given for identification of the actor as self or non-self for action categories answered "created" or "learnt" separately for self true and not true.


movement phrases they had experienced through learning from observation and practice, from pure observation, or from improvisation without vision while being blindfolded. We presented our participants with 36 video-clips showing pointlight displays of dance-like actions that had been recorded with the same participants during a previous session. The clips showed dance-like movements of four categories: unfamiliar, observed (but not performed), learnt (observed and performed), and self-created while blindfolded (performed, but not observed). Based on previous studies (e.g., Casile and Giese, 2006), we expected that participants would be able to assign the presented movement phrases to the correct categories, independent of the modality of their specific previous experience of that action (visual, kinaesthetic, both, or none); in particular, we expected the recognition accuracy for the blindfolded self-created movement phrases to be on the same level as for the other categories. Results showed that participants assigned movements of all four categories correctly to the respective categories, showing that all three ways of experiencing the movement (observed, learnt, created blindfolded) resulted in an encoding of the movement in long-term memory that was sufficient for recognition (Schmidt, 1975, 2003). Observed movements showed the lowest level of recognition accuracy, whereas the accuracy of assigning blindfolded self-created movements was on the same level as for unfamiliar and learnt movements.

As main finding of this study, the recognition of point-light displays from movements that the participants had created and performed, but never visually experienced, was equally high as for the movements they had learnt through observation and practice, and higher than for the movements they had only observed. This finding corroborates results of a previous study in which participants learnt gait patterns without visual feedback, based only on haptic and verbal cues (Casile and Giese, 2006). The performance of the participants in assigning the visually displayed movements correctly points toward a perceptual equivalence of movements learnt from observation and those created blind-folded during the recording session, as both could be accessed via visual observation of the pointlight display equally well. These findings support the idea of an intermodal mapping of kinesthetic and proprioceptive movement representations to the visual domain (Schütz-Bosbach and Prinz, 2007).

According to recent approaches, motor learning and execution is based on the integration of visual, auditory, verbal, proprioceptive, and kinaesthetic information into a holistic multimodal mental representation of the learnt action in long-term memory (Zacks et al., 2007; Barsalou, 2008). Such representations are supposed to comprise declarative and non-declarative memory content that is integrated and updated with every new access and are therefore thought to underlie the physical execution as well as the mental imagery of a motor action, with their internal structure depending on the quality of performance (Land et al., 2013). Nomikou et al. (2016) argue in favor of rich multimodal representations continuously developed through and for action and interaction, suggesting that such representations are built early during development by acting and interacting in the physical and social world. Such representations

have to be dynamic in nature to capture temporal progression and allow for prediction; they need to express temporal relations allowing for synchronization and co-occurrence as prerequisites for social behaviors. Evidence exists, in particular for audio– visual information, that multimodal action representations are transferable between sensory modalities and can even be accessed through senses that were not actively involved in the process of action acquisition (Rosenblum et al., 2017). Rosenblum et al. (2017) propose that the architecture of the brain implies perceptual parity between the senses, and that cross-sensory integration occurs completely and early in the perceptual stream. The authors argue in favor of task rather than sensory modality as primary organizing principle, and suggest that perceptual learning might involve extracting amodal primitives that are not specifically tied to sensory modalities, therefore perceptual learning within the same task context should be transferable between senses. This argument provides explanatory ground with regards to the results of the present study in which participants showed that they were able to exploit movement information gained through physical execution without visual feedback for a visual recognition task. In real-world motor learning tasks in sports and dance, information from other sensory modalities such as action-related sound contributes significantly to motor learning (e.g., Camponogara et al., 2017; Sors et al., 2017). Camponogara et al. (2017) showed that expert basketball players were able to infer opponents' movement intentions from action-based sound more accurately than novices, by picking up action-specific movement information and using it to anticipate the opponent's future position. The authors suggest that the experts pick up relevant kinematic features such as velocity, trajectory, and position of deceptive movements through structural and transformational invariants of the movement sounds by directly mapping sound characteristics onto action intentions. These findings are supported by fMRI results showing that sports experts display specific activation in brain areas involved in action planning when passively listening to task-relevant sounds from their own area of expertise, but not in response to irrelevant sounds (Woods et al., 2014).

With regards to the creating movement task applied in the current study, it cannot be ruled out that the participants, while being blindfolded, created mental images of the performed action using visual imagery. As this is not unlikely, it might have been interesting to investigate the influence of cognitive strategies on action recognition, for example by means of a post hoc questionnaire or interview. In dance, mental imagery is applied for different purposes including the rehearsal, creation, and interpretation of movement and the preparation or recreation of the body (Hanrahan and Vergeer, 2001; Nordin and Cumming, 2007), and dance training has been found to increase the efficiency of imagery techniques (Golomer et al., 2008; Fink et al., 2009). Even participants without dance training experience might have used visual imagery during the experimental task to compensate for the lack of visual feedback.

The finding that recognition accuracy for observed movements was below that of movements learnt through observation and practice supports the notion that action execution is generally more beneficial for learning than observation alone (e.g., Badets et al., 2006). In sports and dance training, the learning of dance-like actions (see Schachner and Carey, 2013) is most commonly practiced in the form of observational learning from a visual model, typically augmented by verbal comments as teacher feedback (Wulf and Prinz, 2001) and supported by simultaneous movement execution or marking (Kirsh, 2011; Warburton et al., 2013). The often-observed superiority of combined motor and visual learning, compared to visual learning alone, can be explained with reference to the integration of multisensory information during action acquisition (Land et al., 2013), by stating that the participation of more sensory modalities in the learning process might result in a richer representation that involves more complementary information and therefore leads to a better learning outcome. Even though the majority of studies supports the view that physical execution results in better learning than mere observation, evidence against such an enactment effect has also been found, in particular for complex "real-world" type tasks involving longer action sequences (von Stülpnagel et al., 2016a,b). Findings by Allerdissen et al. (2017) suggest that it might not be the mere redundancy of information that enhances learning success in multimodal conditions, but rather the contribution of different modalities providing slightly different information that is then integrated in a meaningful way, and that the ability to integrate relevant information into a consistent action representation and omit irrelevant or contradictory information can be considered a feature of domain-specific expertise. Plenty of evidence exists that auditory information is more accurate than vision with regards to temporal action features and that action control therefore relies more strongly on sound if timing, speed, or rhythm is crucial (e.g., Repp and Penel, 2002), which is of particular relevance in sports (Camponogara et al., 2017; Sors et al., 2017). Studies using audio-based interventions in sports support the view that auditory information is more pertinent than visual information with regards to rhythmic movement features and precisely timed actions (Sors et al., 2015). A study with tap dancers showed that temporal properties of rhythmic dance movement can be better perceived through auditory than visual stimuli (Murgia et al., 2017). In this study, experts' accuracy in recognizing dance steps was higher than non-dancers' in the auditory domain, and in the auditory than in the visual domain.

As a second point of interest, we investigated self-recognition from the point-light displays presented in the recognition task by asking the participants to identify the actor as self or nonself. Previous studies had shown that people can easily distinguish between themselves, familiar persons, and strangers from pointlight displays of different types of actions (e.g., Loula et al., 2005).

Discrimination between one's own compared to another's motor actions on the basis of action-based auditory information has been proved successful for different sports (Murgia et al., 2012; Kennel et al., 2014), and EEG evidence has supported these findings by reporting activation of an evaluation network for agent identification through action-related sound stimuli (Justen et al., 2014). Even though the rhythmic structure has been identified as relevant factor, self recognition from action-based sound does not depend on rhythmic features exclusively, but on a more complex auditory "gestalt"

(Kennel et al., 2014). Sevdalis and Keller (2009) found that free dancing resulted in better self-recognition from pointlight displays compared to walking and clapping, and argued in favor of a more pronounced "kinematic fingerprint" of the improvised full-body motor action. Loula et al. (2005) also observed that participants identified themselves and familiar persons successfully from point-light displays of dancing, boxing, and playing ping-pong, but failed to reach chance level for displays of walking and running. Mitchell and Curry (2016), in contrast, found that participants identified themselves above chance level from walking point-light walkers presented from different perspectives. In our study, participants did not identify themselves correctly above chance level if only the video clips of self-created and learnt movements are taken into account (self-identification in clips showing unfamiliar or only observed movements would not make much sense in the given context). Only 22 out of 57 clips (38.6%) showing the participant himor herself performing a self-created of learnt movement were identified as "self," and 20 clips were erroneously identified as "self."

Furthermore, we found an interesting interaction between action and actor recognition: participants were more likely to identify the actor as "self " in clips they had assigned to the category "created" than in clips they had assigned to the category "learnt." This effect was much stronger in the situation if the "self " judgment was actually true. These results show that selfrecognition and action recognition influenced each other and that categorization of a movement as "learnt" or "self-created" had a biasing effect on actor identification, which points toward a significant role for agency for self-recognition (see Knoblich and Flach, 2003; Jeannerod and Pacherie, 2004).

Previous studies had shown that actor recognition and action recognition are not independent from each other in different conditions, for example that knowing an actor's identity and intention can influence action perception (e.g., Knoblich and Sebanz, 2006; Sebanz et al., 2006). Ferstl et al. (2017) suggest that neural mechanisms might exist that link actor information to action information by encoding actor identity on the basis of specific cues (facial features, clothing, posture) in service of action prediction. They claim that action recognition should be sensitive to actor identity for reasons of ecological validity, as information about the actor is fundamental for understanding observed actions. Schütz-Bosbach et al. (2006) showed that observing others' actions facilitated the motor system, whereas observing one's own actions rather suppressed motor activation. Based on their results, the authors argued strongly against agent-neutral action representations, suggesting that neural mechanisms underlying action observation are intrinsically social. These studies support the view that action recognition is influenced by actor recognition, however, in the present study, it could be claimed that we found a reverse effect, namely that actor recognition is influenced by action recognition. In this regard, the order in which participants were asked to identify action and actor the recognition task might be relevant. For each presented video clip, the participant had to answer two questions within the same trial, before the next clip was shown; in each trial, action recognition (or action categorization) came before actor recognition (or action identification). This way, participants were judging the presented movement first on its own merit, however, it cannot be excluded that the action recognition thereby had a priming effect on actor recognition, which might have caused or enhanced the observed interaction bias. First identifying a presented movement as self-created might have influenced the "self or other" decision by shifting it toward "self," which is reflected by the results. It would be interesting to know if the same interaction had been found if the questions had been asked in the opposite order (self-identification before action categorization), or if the two questions had been asked separately in different blocks.

Another limitation of the presented experiment might be seen in the choice of participants. This study was conducted with 19 sports students whose dance experience differed (seven practiced dance or gymnastics regularly, whereas the others practiced other types of sports). Even though none of the participants reached professional level in dance, their different experience might have affected their individual approach to learning and creating movement (however, none of the movement phrases created in the first session was particularly complex or too difficult to be picked up easily by dance novices). Repeating the experiment with more homogeneous expertise-based groups (professional dancers vs. non-dancers) could provide relevant novel insights regarding these aspects.

# CONCLUSION

The results of the presented study support findings of a direct influence of motor experience on visual action perception and recognition for actions that have been learnt without visual feedback. They extend previous results (Casile and Giese, 2006), to dance-like actions that have been acquired exclusively through movement exploration and practice, in the absence of vision and without haptic or verbal feedback. The "blind" execution and creation of full-body actions (as it is typically applied in contemporary dance training) obviously results in a multimodal representation that can be accessed via visual cues, despite the lack of visual experience. Furthermore, the results corroborate that agency plays a significant role for self-identification, which adds new aspects to perspectives taken in social cognition contexts (Ferstl et al., 2017).

# AUTHOR CONTRIBUTIONS

BB planned and conducted the study and pre-analyzed the data. OS analyzed and interpreted the data. Both authors contributed equally to the manuscript.

# FUNDING

This research/work was supported by the Cluster of Excellence Cognitive Interaction Technology "CITEC" (EXC 277) at Bielefeld University, which is funded by the German Research Foundation (DFG).

# REFERENCES

fpsyg-09-01909 October 13, 2018 Time: 11:59 # 8



**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Bläsing and Sauzet. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Stepovers and Signal Detection: Response Sensitivity and Bias in the Differentiation of Genuine and Deceptive Football Actions

Robin C. Jackson<sup>1</sup> \*, Hayley Barton<sup>2</sup> , Kelly J. Ashford<sup>3</sup> and Bruce Abernethy<sup>4</sup>

<sup>1</sup> School of Sport, Exercise and Health Sciences, Loughborough University, Loughborough, United Kingdom, <sup>2</sup> Centre for Sports Medicine and Human Performance, Brunel University London, Uxbridge, United Kingdom, <sup>3</sup> Cardiff School of Sport, Cardiff Metropolitan University, Cardiff, United Kingdom, <sup>4</sup> Faculty of Health and Behavioural Sciences, The University of Queensland, Brisbane, QLD, Australia

Edited by:

Mauro Murgia, University of Trieste, Italy

#### Reviewed by:

Itay Basevitch, Anglia Ruskin University, United Kingdom Bettina E. Bläsing, Bielefeld University, Germany

> \*Correspondence: Robin C. Jackson r.c.jackson@lboro.ac.uk

#### Specialty section:

This article was submitted to Movement Science and Sport Psychology, a section of the journal Frontiers in Psychology

Received: 30 July 2018 Accepted: 04 October 2018 Published: 29 October 2018

#### Citation:

Jackson RC, Barton H, Ashford KJ and Abernethy B (2018) Stepovers and Signal Detection: Response Sensitivity and Bias in the Differentiation of Genuine and Deceptive Football Actions. Front. Psychol. 9:2043. doi: 10.3389/fpsyg.2018.02043 The ability to differentiate genuine and deceptive actions was examined using a combination of spatial and temporal occlusion to examine sensitivity to lower body, upper body, and full body sources of visual information. High-skilled and low-skilled association football players judged whether a player genuinely intended to take the ball to the participant's left or right or intended to step over the ball then take it in the other direction. Signal detection analysis was used to calculate measures of sensitivity (d 0 ) in differentiating genuine and deceptive actions and bias (c) toward judging an action to be genuine or deceptive. Analysis revealed that high-skilled players had higher sensitivity than low-skilled players and this was consistent across all spatial occlusion conditions. Low-skilled players were more biased toward judging actions to be genuine. Receiver Operating Characteristic (ROC) curves revealed that accuracy on deceptive trials in the lower body and full body conditions most accurately classified participants as highskilled or low-skilled. The results highlight the value of using signal detection analysis in studies of deceptive actions. They suggest that information from the lower body or upper body was sufficient for differentiating genuine and deceptive actions and that global information concurrently derived from these sources was not necessary to support the expert advantage.

Keywords: anticipation, deception, signal detection, perception, bias

# INTRODUCTION

The ability to judge the intentions of an opponent using advance visual information confers an advantage in many competitive sport encounters (Mann et al., 2007). A potential disadvantage of being highly attuned to early visual information is that it leaves performers vulnerable to deception, resulting in misreading the intentions of an opponent characterized by incorrect or inefficient responses (Jackson et al., 2006). In early research on deception, researchers showed that expert players in the French martial art savate (a form of kick boxing) made more 'false alarm' responses to fake attacks ('feints') than intermediate and novice players (Ripoll et al., 1995). In light of the very different consequences of failing to respond to a genuine attack and responding to a feint, this

may well have reflected a strategic or perceptual bias on the part of experts rather than indicating their greater susceptibility to deception. The weight of evidence now supports a clear advantage for high-skilled over low-skilled performers in using kinematic information to judge deceptive intent. This has been shown in studies of deceptive 'sidestep' actions (Jackson et al., 2006; Brault et al., 2012; Mori and Shimada, 2013), football penalty kicks (Smeeton and Williams, 2012), football 'stepovers' (Bishop et al., 2013; Wright et al., 2013; Wright and Jackson, 2014), and discriminations between genuine and deceptive actions in volleyball (Güldenpenning et al., 2013), handball (Cañal-Bruland and Schmidt, 2009), and basketball (Sebanz and Shiffrar, 2009).

An important question in the perception of deceptive intent concerns the information sources used by skilled and lessskilled performers to discriminate between genuine and deceptive actions. Researchers have shown that experts use information from distributed sources to anticipate action outcomes (Ward et al., 2002; Huys et al., 2008, 2009; Cañal-Bruland et al., 2011; Diaz et al., 2012; Loffing and Hagemann, 2014). For example, expert badminton players became increasingly accurate at judging the depth of a shot as markers for the racket arm, head plus non-racket arm, and lower body were progressively added to those depicting the racket and shuttle. In contrast, recreational players relied more on the arm and racket and did not improve when lower body information was added (Abernethy et al., 2008). Similarly, Williams et al. (2009) showed that tennis players were less able to differentiate between cross-court and 'inside-out' forehand tennis shots when local motion from the two shots was selectively interchanged. For skilled players, judgments were impaired when the manipulation was applied to a number of local sources, namely motion of the arm and racket, shoulders, hips, and legs. By contrast, judgments of less-skilled players were only impaired when motion of the arm/racket region was manipulated. From this evidence some researchers have inferred that experts use 'global' processing whereas low-skilled or novice performers are more reliant on 'local' processing of specific sources of information (Huys et al., 2008; Williams et al., 2009). Greater sensitivity to distributed sources of information need not imply global processing as different sources may be processed sequentially. For example, consistent with the constraints attunement hypothesis (Vicente and Wang, 1998) applied to dynamic anticipation tasks, Abernethy et al. showed that expert badminton players process information in a proximalto-distal manner. For trials occluded early in the striking action, expert players predicted shot depth more accurately when only the player's lower body or head plus non-racket arm was visible than when only the arm (holding the racket) or racket was visible. By contrast, in later-occluded actions predictions were more accurate when viewing the racket arm or racket than when viewing the player's lower body or head plus non-racket arm. While this shows that skilled performers make better use of early proximal information they are also more attuned to sources close to the end effector that undergo the greatest displacement (Abernethy and Russell, 1987; Ward et al., 2002; Jackson and Mogan, 2007; Abernethy et al., 2008; Huys et al., 2009). In a series of studies of how cricket batters anticipate bowler actions, Müller and colleagues concluded that the expert advantage is primarily driven by pick-up of advance information from upper body sources, notably the bowling hand and arm over the time period in which the underlying kinematics undergo the greatest change (Müller et al., 2006, 2010).

Instructions for executing common deceptive actions such as the football stepover and rugby sidestep refer to movements of the lower and upper body. To perform sidesteps, expert coaches instruct players to "step wide with the outside leg at the same time leaning your body weight directly over the top of that foot. . . drive off the outside leg back inside" (Biscombe and Drewitt, 1997, p. 36). Similarly, to execute a football stepover players are instructed to "Go across the ball with the outside of the right or left foot, feint with the upper part of the body and cut inside" (Simpson and Hesse, 2013, p. 36). In sidestep actions, Brault et al. (2010) found that differences in lower body movement (outer foot displacement and lower trunk yaw), upper body movement (head yaw; upper trunk yaw; and upper trunk roll), and centre of mass (COM) displacement differentiated genuine from deceptive actions and characterized more and less effective sidesteps. Brault et al. (2012) showed that expert players were more attuned to the 'honest' COM displacement signal whereas non-players were more attuned to deceptive signals. To determine COM at any given moment requires knowledge of both the lower and upper body so implies that skilled judgments of deceptive intent rely on global processing rather than enhanced local processing of specific sources. If this is the case then the ability to differentiate genuine and deceptive actions should be attenuated when this information is unavailable, for example when only the lower body or upper body is visible.

The aim of the present study is to test whether concurrent use of lower and upper body sources of information is necessary for judging deceptive intent in a common deceptive action: the football stepover. To address this question, high-skilled and recreational football players judged the direction an opponent would take the ball under three levels of spatial occlusion in which (1) the whole player, (2) only the player's upper body, and (3) only their lower body, were visible. To ensure results could be attributed to player motion, full-video and point-light tests were constructed. Point-light displays present key joint centers against a dark background and were developed by Johansson (1973) as a means of isolating information in biological motion from cues relating to form. They have been successfully applied to studies of anticipation in sport as a simple means of isolating kinematic information as the performer interacts with an object.

A limitation of previous research on deceptive actions is that judgment accuracy has been assessed separately for genuine and deceptive actions. This yields important information regarding response accuracy for each type of trial; however, it is limited in at least two ways. First, it does not directly measure a fundamental goal of the task, which is to determine whether the intent conveyed by an action (e.g., a football player showing intent to take the ball to the right) is genuine (she takes the ball to the right) or deceptive (she steps over the ball then takes the ball to the left). This ability is captured by a measure of sensitivity that is derived from both the proportion of correct responses for genuine trials and the proportion of 'correct rejections' in

deceptive trials. A second limitation of analyzing genuine and deceptive trials separately is that differences in accuracy might reflect different biases toward judging an action to be genuine or deceptive. For example, higher-skilled performers might obtain higher accuracy scores than lesser-skilled performers on deceptive actions because they are more biased toward judging actions to be deceptive, perhaps born of greater exposure to deceptive actions in competitive play. Analysis originating in signal detection theory (Green and Swets, 1966) enables us to examine these issues but has very rarely been employed in studies of deceptive actions in sport.

To the best of our knowledge, the only study to date to employ signal detection analysis in judgments of deceptive actions in sport was conducted by Cañal-Bruland and Schmidt (2009), who asked skilled handball goalkeepers, outfield players and novices to judge whether penalty throws were genuine or deceptive. While goalkeepers and outfield players showed the same level of sensitivity in differentiating genuine and deceptive actions, only the goalkeepers were biased in favor of judging penalty throws to be fake (i.e., judging the shooter would not release the ball). The authors suggested this might reflect knowledge of situational probabilities of the respective actions or an assessment that there are greater costs associated with missing a deceptive action. The source of bias can also be perceptual and this was neatly illustrated by Witt et al. (2015) in their model of the effect of tail orientation on judgments of line length using the Müller-Lyer illusion. Likewise, perceptual bias applies to deceptive actions such as the football stepover, in which the goal of the actor is to 'fool' an observer into judging an action to be genuine when it is in fact deceptive. In these tasks the extent to which participant responses are biased toward judging the action to be genuine are an additional measure of the effectiveness of deception and can be assessed at different time points as the action unfolds.

Another feature of signal detection analysis is that one can quantify the degree to which test results differentiate participants on a binary classifier such as membership of a high-skilled and low-skilled group. To do this, Receiver Operating Characteristic (ROC) curves are plotted that depict the rate of true positive identifications (e.g., membership of the high-skilled group) against the rate of false positives (e.g., membership of the lowskilled group) as one progresses through the list of ranked test scores. The area under the curve (AUC) measures the degree to which the test 'diagnoses' group membership. This and associated ROC analyses that compare the rates of true positives and false positives for different decision criteria have been extensively applied in a diverse range of fields including medical diagnosis and eye witness identification (Zweig and Campbell, 1993; Swets, 2014; Wixted and Mickes, 2014).

In the present study, we used response accuracy scores to calculate measures of (perceptual) sensitivity (d 0 ) and response bias (c). 'Hits' were defined as correct responses on genuine trials and 'false alarms' were defined as incorrect responses on deceptive trials. In the analysis that follows, a d 0 value of 0 indicates an inability to distinguish between genuine and deceptive actions, which can result from any proportion of 'hits' on genuine trials as long as it is matched by the same proportion of 'false alarms' on deceptive trials. When the proportion of 'hits' is greater than the proportion of 'false alarms' this will yield positive values of d 0 ; conversely, fewer 'hits' on genuine trials than 'false alarms' on deceptive trials will result in negative d 0 values. In regard to bias, negative values of c reflect a bias toward judging actions to be genuine and positive values of c reflect a bias toward judging an action to be deceptive. Last, we conducted ROC analysis to examine which elements of the test best differentiated high-skilled and low-skilled participants.

In regard to the measure of sensitivity (d 0 ), we hypothesize that (1) sensitivity will be greater for high-skilled players than low-skilled players, reflecting their greater ability to distinguish between genuine and deceptive actions. Consistent with the global processing hypothesis we further hypothesize that (2) sensitivity, and (3) the difference in sensitivity between highskilled and low-skilled players, will be greater when the whole body is visible than when the upper and lower body are seen in isolation. In regard to the measure of response bias (c), we hypothesize that (4) low-skilled players will have a stronger bias toward judging actions to be genuine than high-skilled players. We further hypothesize that (5) bias will be stronger, and (6) the difference in bias between high-skilled and lowskilled players will be greater, in the whole body condition than in the lower body and upper body conditions. In regard to the ROC analysis, we hypothesize that (7) group membership will be better 'diagnosed' by judgment accuracy on deceptive trials than genuine trials, and that (8) the AUC will be greatest for the deceptive trials in the full body condition.

#### MATERIALS AND METHODS

#### Participants

Forty-eight female football players (24 high-skilled, Mage = 21.9 years, SD = 4.3; 24 low-skilled, Mage = 21.6 years, SD = 1.4) participated in the experiment. High-skilled participants were competing in the Football Association Women's Super League at the time of the experiment and had a mean of 12.3 years (SD = 3.8) of competitive football experience. Low-skilled participants had a mean of 5.1 years (SD = 3.5) of recreational football experience. High-skilled and low-skilled participants were randomly allocated to the 'full video' and 'point-light' test formats. Power analysis was conducted in G <sup>∗</sup>Power (version 3.1, see Faul et al., 2007). For a medium effect size (f = 0.25), alpha set at 0.05, and power set at 0.80, the mixed-factor ANOVA calculation yielded a recommended total sample size of 40 for the interaction between group (four levels) and spatial occlusion (three levels), and of 36 for the interaction between group and time of occlusion (four levels).

## Experiment Design and Test Stimuli

The task was designed to simulate a one-on-one football scenario in which one player runs toward an opposing player before attempting to evade the other player by taking the ball to the left or right, with or without a deceptive action. Two skilled female football players with a mean of 13.5 years of competitive National level playing experience were used to create the test stimuli. The video sequences were filmed using a Canon HD digital video

camera (Canon HV40, Toyko, Japan) mounted on a tripod at a height of 1.4 m recording at 25 frames per second. For each video clip, the player ran from a starting position located 11.5 m from the video camera and was instructed to change direction in the region of a marker placed 5.3 m from the point directly beneath the video camera lens. At this point, the player moved toward one of two training cones placed at an angle of 45 degrees to the left and right of the initial approach. In the non-deceptive condition, the player was instructed to change direction to the left or right of the camera, while in the deceptive condition the player was instructed to perform a 'stepover' by moving their lead foot in front of and across the ball before taking the ball in the opposite direction. The task for participants was to judge whether the approaching player intended to take the ball to their left or right, which required them to judge whether the initial intention conveyed by a movement to the left or right was genuine or deceptive. Participants were told that there would be an equal number of action outcomes to the left and right and an equal number of genuine and deceptive actions.

To select the highest quality actions for the test, three UEFA 'B' License football coaches rated each video clip for speed, straightness of approach, and technical execution. The two highest-rated clips for each player changing direction to the left and right with and without a stepover were included in the final test. This generated 16 unique clips, which were then digitally edited using Pinnacle Studio and Jasc Paint Shop Pro software to create three levels of spatial occlusion and four times of occlusion. For simplicity, the three levels of spatial occlusion refer to the information sources that were visible: (A) full body: original video with no areas removed, (B) lower body: each player's head, arms, hands and torso above the hips were removed, and (C) upper body: each player's legs, feet and torso from the hips down were removed.

#### Full-Video Stimuli

To create the spatial occlusion conditions, each frame of the 16 video sequences was edited by cloning a background image of the experiment set up to 'paint over' the relevant region of the player. The edited images from consecutive frames were then recombined to create a new video clip. The resulting 48 video stimuli were cropped at four time points relative to the frame before the foot made contact with or passed in front of the ball: t1 (−240 ms), t2 (−120 ms), t3 (0 ms), and t4 (+120 ms) (see **Figure 1**).

#### Point-Light Video Stimuli

Each frame of the 16 unique video sequences was edited to produce sparse binary (black/white) point-light representations consisting of 19 small disk markers corresponding to principal body joints and extremities (forehead; chin; heads of the left and right humerus; left and right elbow; left and right wrist; navel; left and right iliac spines of the pelvis; left and right patella; left and right heel; mid-points of the lateral and medial malleoli of the left and right ankle; and distal phalanx of the second toe of the left and right foot). In addition, the ball was represented in each frame by a white disk of the same circumference such that the looming effect was retained as the player approached the camera. The 10 markers corresponding to the hips, knees, ankles, and feet of the player were retained to create the 'lower body' stimuli. The remaining nine markers were retained to create the 'upper body' stimuli (see **Figure 1**).

The full video and point-light tests each comprised 192 test trials, presented in four blocks of 48 trials. The tests were identical except for display format and were presented on a 15.6" widescreen monitor viewed from a distance of approximately 0.5 m, such that the vertical visual angle subtended by the player at the point of direction change was approximately 10 degrees. In the first two blocks of trials, participants were shown video clips from one of the two players and in the second two blocks were shown video clips from the other player. Player order was counterbalanced across participants to control for possible order effects. The order of trials associated with each player was randomized with respect to levels of deception, spatial occlusion and temporal occlusion. The duration of each trial was approximately 2.0 s and we employed a 5.0 s inter-trial interval.

#### Procedure

Institutional ethical approval was granted and all participants gave written consent prior to participating in the study. After completing the participant information and consent forms participants were told that their task was to judge whether the player in the video would take the ball to the left or right of the screen from the participant's viewing perspective. They were informed that the clips would vary in terms of when they were occluded, that the player would take the ball to the left and right an equal number of times, and that on half of the trials the players would try to deceive them by feigning to take the ball in one direction before moving in the other. Participants were also told the clips would vary in terms of how much of the performer would be visible, namely their whole body, just their upper body, or just their lower body. Participants who viewed the point-light test were informed that the two players would be represented by a group of white dots set against a black background so that sometimes they would see all the dots, sometimes only the dots from the player's upper body, and sometimes only the dots from the player's lower body.

Participants were instructed to indicate the direction they thought the player would go by making a verbal response ('left' or 'right'). To familiarize participants with the test format and response requirements, they were shown 16 practice trials in their designated display format (full video or point-light) comprised of eight video clips from each player. These contained examples of each level of deception, spatial occlusion, and temporal occlusion and were generated from different clips to those used in the test.

#### Statistical Analysis

The primary dependent variables were sensitivity (d 0 ) and bias (c), which were calculated for each group in each combination of spatial and temporal occlusion. To calculate d 0 , the proportions of correct responses on genuine trials ('hits') and incorrect responses on deceptive trials ('false alarms') were converted to z-scores. The values for deceptive trials were then subtracted from the values for genuine trials. To calculate c, the z-scores for deceptive trials were added to those for genuine trials and

FIGURE 1 | A schematic representation of single frames from the point-light and full video test sequences, showing the three levels of spatial occlusion and four times of occlusion as one of the players performs a stepover. Written informed consent was obtained from the depicted individual for the publication of these images.

multiplied by −0.5. To account for the possibility of infinite z-scores, values of 0 and 1 were replaced with 1/2n and (n−0.5) ÷ n, respectively, where n is the number of trials in the relevant condition (Stanislaw and Todorov, 1999).

Using these measures and setting up the analysis in this way addresses the key judgment to be made when viewing a step-over action, namely whether the outcome intention initially conveyed by the actor is genuine or deceptive, regardless of whether the intention conveyed is to take the ball to the left or the right. Conceptually, it is important to note that while the participant makes a directional judgment (left or right) rather than one of deceptive intent (genuine or deceptive) the latter is implicit in the former so is subject to analysis for sensitivity and bias. Specifically, a correct response to a genuine action (whether to the left or right) implies a correct judgment that the action was genuine (a 'hit'). An incorrect response to a deceptive action (whether the initial intention conveyed was to the left or right) implies an incorrect judgment that the action was genuine when it was in fact deceptive (a 'false alarm'). Conversely, an incorrect response to a genuine action (whether to the left or right) implies an incorrect judgment that the action was deceptive (a 'miss'). Last, a correct response to a deceptive action (whether the initial intention conveyed was to the left or right) implies a correct judgment that the action was deceptive (a 'correct rejection').

A 2 (Expertise: high-skilled, low-skilled) × 2 (Test Display: full video, point-light) × 3 (Spatial Occlusion: full body, upper body, lower body) × 4 (Time of Occlusion: t1, t2, t3, and t4) mixedfactor analysis of variance (ANOVA) was conducted for the d 0 and c variables, with expertise and test display entered as betweenparticipant factors, and spatial and temporal occlusion serving as within-participant factors. Alpha was set at 0.05 for all analyses and partial eta squared (η<sup>p</sup> 2 ) was used to indicate effect size. The Greenhouse-Geisser adjustment to the degrees of freedom was applied when Mauchly's test of sphericity was violated. For the ROC analysis, group membership (high-skilled or low-skilled) served as the binary classifier and the AUC was calculated for each combination of spatial and temporal occlusion. In this analysis, classification at chance level produces a diagonal line for the rates of true positives (correct classifications) and false positives (incorrect classifications) so significance is tested against an AUC value of 0.5.

## RESULTS

#### Descriptive Data

The combined accuracy data across the two tests for nondeceptive and deceptive trials in the three spatial occlusion

conditions are displayed in **Figure 2**. In all three spatial occlusion conditions, high-skilled players were slightly more accurate than low-skilled players when judging genuine actions and were considerably more accurate than low-skilled players in judgments of deceptive actions. Although not the primary analysis of interest, this replicates previously reported findings (Brault et al., 2012; Wright and Jackson, 2014) and resulted in a significant Expertise × Deception interaction, F(1, 44) = 24.26, p < 0.001, ηp <sup>2</sup> = 0.36. Overall, there was no significant difference between response accuracy in the full video (M = 0.68, SE = 0.01) and point-light (M = 0.66, SE = 0.01) tests, F(1, 44) = 3.27, p = 0.08, ηp <sup>2</sup> = 0.07.

#### Signal Detection Analysis

Overall discriminability was slightly higher for the full-video test (d <sup>0</sup> = 1.11, SE = 0.04) than for the point-light test (d <sup>0</sup> = 0.98, SE = 0.04); however, the difference was nonsignificant, F(1, 44) = 3.97, p = 0.053, η<sup>p</sup> <sup>2</sup> = 0.08, as was the Test Display × Expertise interaction, F(1, 44) = 0.30, p = 0.59. Consistent with Hypothesis 1, analysis of sensitivity (d 0 ) revealed that the ability to distinguish genuine and deceptive actions was substantially greater in high-skilled players (d <sup>0</sup> = 1.46, SE = 0.04) than low-skilled players (d <sup>0</sup> = 0.63, SE = 0.04), F(1, 44) = 175.73, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.80. As expected given the nature of the test, the ability to distinguish between genuine and deceptive actions increased as more of the action was revealed, resulting in a significant effect of time of occlusion, F(1, 44) = 296.38, p < 0.001, ηp <sup>2</sup> = 0.87. The difference between high-skilled and low-skilled players was stable across t1, t2, and t3 then decreased after the foot contacted or passed in front of the ball (t4), reflected in a significant Expertise × Time of Occlusion interaction, F(2.4, 105.6) = 7.28, p = 0.001, η<sup>p</sup> <sup>2</sup> = 0.14 (see **Figure 3**).

Consistent with Hypothesis 2, sensitivity was higher in the full body condition (d <sup>0</sup> = 1.20, SE = 0.04) than the upper body condition (d <sup>0</sup> = 0.97, SE = 0.06), F(1, 44) = 16.26, p < 0.001, ηp <sup>2</sup> = 0.27, and lower body condition (d <sup>0</sup> = 0.98, SE = 0.04), F(1, 44) = 21.91, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.33. However, a significant Spatial Occlusion × Time of Occlusion interaction, F(4.7, 205.7) = 11.79, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.21, reflected that sensitivity at t3 was higher in the full body (d <sup>0</sup> = 1.35, SE = 0.11) and upper body (d <sup>0</sup> = 1.33, SE = 0.09) conditions than in the lower body condition (d <sup>0</sup> = 0.87, SE = 0.06), and that sensitivity increased more in the full body and lower body conditions than the upper body condition after the foot had been seen contacting or passing in front of the ball (t3– t4; see **Figure 4**). Hypothesis 3 was not supported as sensitivity for high-skilled players was greater than for low-skilled players in all spatial occlusion conditions, resulting in a non-significant Spatial Occlusion × Expertise interaction, F(2, 88) = 1.10, p = 0.34, ηp <sup>2</sup> = 0.02 (see **Figure 5**).

Consistent with Hypothesis 4, analysis of response bias revealed that the low-skilled players (c = −0.79, SE = 0.04) had a stronger bias toward judging actions to be genuine than the highskilled players (c = −0.54, SE = 0.04), F(1, 44) = 22.86, p < 0.001, ηp <sup>2</sup> = 0.34. As can be seen in **Figure 6**, bias in low-skilled players was already strong at t1, increased further at t2, then stabilized at t3 before decreasing markedly at t4 after the foot had taken or passed in front of the ball. In the high-skilled players, bias was

strongest at t1 and t2, decreased markedly at t3, and was almost eliminated at t4. This resulted in a significant Expertise × Time of Occlusion interaction, F(2.3, 103.2) = 7.87, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.15.

The hypotheses that response bias would be strongest and the expertise effect greatest in the full body condition were not supported. As can be seen in **Figure 7**, before veridical information became available (i.e., from t1 to t3) bias toward judging actions to be genuine was strongest in the lower body condition then full body condition, and was weakest in the upper body condition. The effect of expertise was consistent across the three conditions of spatial occlusion, resulting in a non-significant Expertise × Spatial Occlusion interaction,

F(1.5, 66.0) = 1.35, p = 0.26, η<sup>p</sup> <sup>2</sup> = 0.03. A significant Spatial Occlusion × Time of Occlusion interaction, F(4.3, 188.8) = 17.12, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.28, reflected relatively stable bias across time

FIGURE 6 | Response bias (c) for the high-skilled and low-skilled participants at each time of occlusion. Negative values indicate a bias toward judging the action to be genuine.

of occlusion in the upper body condition in contrast to bias in the full body and lower body conditions, which strengthened from t1 to t2 then weakened at t3 and t4.

#### ROC Analysis

action to be genuine.

To determine the elements of the test that best categorized high-skilled and low-skilled performers, we conducted ROC analyses on the accuracy scores for genuine and deceptive trials in each spatial occlusion condition at each of the four times of occlusion. The AUC values are displayed in **Table 1**, in which three themes can be identified. First, most of the values are higher for trials occluded at t1, t2, and t3 than for those occluded at t4, when veridical information was available (the foot taking or passing in front of the ball). Second, the highest AUC values


TABLE 1 | Area under the curve (AUC) values from the receiver operating characteristic analysis.

The values indicate how well different elements of the test differentiate between the binary classifier (high-skilled and low-skilled). P-values indicate AUC values that are significantly greater than 0.5, which represents classification at chance level. 95% confidence intervals (CI) associated with the AUC values are also stated. Values in bold are the highest associated with the genuine and deceptive trials and ROC curves for these are presented in Figure 8.

are found in the deceptive trials, which is consistent with the hypothesis that judgment accuracy on deceptive trials would be more "diagnostic" of expertise. Third, the highest values for genuine trials are found at t1, showing that the ability to make accurate early judgments of genuine actions distinguishes highskilled and less-skilled performers. In contrast, the highest values for deceptive trials are found at t3, just before the foot passes in front of the ball. These findings are illustrated in **Figure 8** and are characterized by the apices of ROC curves for the deceptive trials (occluded at t3; Panel B) bowing further from the diagonal line than those for genuine trials (occluded at t1; Panel A). In contrast to the hypothesis that the highest AUC values would be in the full body deceptive trials, judgment accuracy for deceptive trials in the lower body condition (AUC = 0.94) and full body condition (AUC = 0.90) distinguished expertise extremely well and slightly better than deceptive trials in the upper body condition (AUC = 0.83; **Figure 8B**).

#### DISCUSSION

Expertise in perceiving deceptive intent has been linked to an ability to attend to 'honest' signals, such as center of mass, while ignoring deceptive signals (Brault et al., 2012). This fits with the narrative that high-skilled performers use 'global' information from distributed sources whereas less-skilled performers are more reliant on local sources of (potentially deceptive) information (Ward et al., 2002). If true, skilled perception of deceptive intent may involve processing the same sources of visual information in a different, more holistic manner, rather than enhanced sensitivity to the critical sources that convey deceptive intent. This would have significant implications for perceptual training protocols, for example, in regard to the degree to which performers should be made aware of information linked to local sources as opposed to more global, relational information. To test this experimentally we manipulated the sources of information available to participants as they attempted to judge the direction a football player would take the ball by determining whether the initial intention conveyed by the player was genuine or fake. Overall, we found clear differences between the performance of high-skilled and low-skilled performers that were consistent across the full-video and point-light tests, highlighting the importance of kinematic information in anticipation and judgment of deceptive intent (Abernethy et al., 2001, 2008).

The results of the signal detection analysis revealed that highskilled participants were better at differentiating genuine and deceptive actions and were most sensitive on trials occluded before the foot contacted or passed in front of the ball (t3). Averaged across all spatial occlusion conditions they made proportionately more correct responses to genuine actions ('hits') than they made incorrect responses to deceptive actions ('false alarms'), which yielded positive values of d 0 . This contrasts with the overall performance of low-skilled participants, who made fewer correct responses to genuine actions than incorrect responses to deceptive actions on trials occluded at t1 and t2, resulting in negative values of d 0 . Negative sensitivity values are uncommon in most of the tasks in which signal detection analysis is used; however, they can be accounted for by use of exaggerated movements to convey a false intention (Brault et al., 2010). Exaggeration has been shown to make some actions more recognizable (Pollick et al., 2001) so when exaggeration is associated with deceptive actions it is logical that the proportion of false alarms can exceed the proportion of hits. Negative sensitivity scores (higher proportions of false alarms than hits) were also found in groups of police investigators and trained students who judged the innocence or guilt of individuals in mock crime interviews (Meissner and Kassin, 2002). Collectively, these scores reveal a key attribute of skilled judgments of deceptive intent, namely the ability to differentiate genuine and deceptive actions earlier in the action sequence. Analysis of the ROC curves confirms it is the ability to judge deceptive actions that best differentiates the two groups (Jackson and Cañal-Bruland, in press; Wright and Jackson, 2014).

The analysis also revealed a clear pattern of results with respect to response bias. There was support for the hypothesis that lowskilled performers would show a stronger bias toward judging actions to be genuine than would high-skilled performers. Moreover, the strength of this bias in the low-skilled group increased from t1 to t2, and in both groups weakened considerably after the player's foot contacted or passed in front of the ball. Combined with the instructions participants received regarding the equal number of genuine and deceptive trials, this implies that the main source of bias was perceptual, which reflects the goal of the actor in conveying a false intention. Low-skilled

to classify participants into either group and would yield an area under the curve (AUC) of 0.50. Better classification of individuals as high-skilled and low-skilled is reflected by curves above and left of the horizontal line and yields AUC values greater than 0.50. participants were fooled more frequently so made more 'false

alarm' responses, peaking at trials occluded at t2 and decreasing considerably after veridical information became available. Highskilled players were fooled less frequently and bias peaked earlier in the action sequence (t1), which supports the interpretation of earlier differentiation of genuine and deceptive actions.

In regard to the sources of information that support accurate judgments of deceptive intent, high-skilled players had the same advantage in sensitivity over low-skilled participants in the full body, lower body, and upper body occlusion conditions (see **Figure 5**). This indicates that information from the lower body or upper body was sufficient to support the expertise effect but that global information, or other relational information concurrently derived from both sources, was not necessary. Instead, the data suggest that high-skilled players were more sensitive than lowskilled players to kinematic information from both the lower and upper body. The picture is a little more nuanced in that upper body and lower body information appear to have been processed sequentially or weighted differently across times of occlusion (**Figure 4**). Specifically, in all three spatial occlusion conditions sensitivity was very low at t1 and improved very little from t1 to t2. Sensitivity then increased more from t2 to t3 when the upper body was visible but improved more from t3 to t4 when the lower body was visible. Information from the upper body was therefore more useful for early differentiation of genuine and deceptive actions while veridical information provided by the lower body became dominant later in the action. Consistent with this interpretation, the effect of spatial occlusion on response bias was stronger for the full body and lower body conditions than for the upper body condition in early-occluded trials, which implies that the lower body was the primary source for conveying deception. This was supported by attenuation of bias in the full body and lower body conditions after the foot contacted or passed in front of the ball (**Figure 7**).

Relating our findings to those of Brault et al. (2012), it is important to note that while tau of COM displacement (the ratio between current motion-gap size and its rate of closure) accounted for most of the variance (74%) in expert responses to rugby sidesteps the deceptive signals accounted for more than 50% of the variance. Some signals (e.g., head yaw) have minimal impact upon COM displacement, which suggests that expert sensitivity extended beyond a globally derived source to assessing the veracity of more local deceptive sources. Williams et al. (2009) argued that by using distributed sources of information high-skilled players might be harder to deceive because they would be more resistant to local perturbations. Our results are consistent with this insofar as there was no advantage for globally derived information over information gleaned from local sources, at least in the coarse distinction between lower body and upper body sources. The results are also consistent with research showing that kinematic differences between nondeceptive and deceptive actions span multiple markers across the upper and lower body (Smeeton and Williams, 2012). At the same time, while our results suggest sequential processing of local information from the upper and lower body they do not preclude holistic processing of information within each source. Indeed, Lopes et al. (2014) found that the variables that best differentiated genuine and deceptive football penalty kicks were stronger predictors of kick direction when expressed as a compound variable. By implication, deceptive actions may be most effective when the player attends to specific isolated cues within a broader source. In the present task attending to the lead foot as it gathered or passed in front of the ball ultimately provided veridical information about the player's intentions but was also the primary source for conveying a false intention. Sequential attention to different sources of information in discrete tasks was shown in a study of expert futsal goalkeepers while they faced penalty kicks (Navia et al., 2017). The goalkeepers focused predominantly on the penalty taker's head during the early phase of the run up then on the ball in the final stride of the player's approach.

A similar analysis of the spatiotemporal characteristics of visual gaze in judgments of stepover actions may provide corroborative evidence for sequential processing of lower and upper body information.

#### CONCLUSION

It is becoming increasingly clear that high-skilled performers have a sizeable advantage over less-skilled performers in their ability to judge deceptive intent. The present study shows how signal detection analyses can be used to capture the essence of these tasks, which is to discriminate between a genuine and deceptive action. This analysis revealed that the advantage of high-skilled football players resides in their ability to use information from both the lower and upper body, yet also showed that they are not dependent on global information concurrently derived from these sources. Moreover, expertise was reflected in different levels of (perceptual) bias toward judging actions to be genuine. Last, ROC analysis revealed that, within the context of a task that contains both genuine and deceptive actions, judgment accuracy for deceptive actions strongly differentiates high-skilled and low-skilled performers. How information from different sources is used to resolve genuine and deceptive actions, and the extent to which the present results relate to in situ physical responses, warrants further investigation. Some researchers have shown no discrepancy between verbal and physical responses (Jackson and Mogan, 2007) while others have found that expertise effects are greater when

#### REFERENCES


participants make coupled physical responses (Mann et al., 2010). In addition, other sources of bias warrant further investigation. In sport, performers commonly have knowledge of situational probabilities regarding player preferences. We expect that this will bias performer responses and there are early indications that this is the case (Jackson and Barton, 2018). How such information and other sources of bias affect response sensitivity is critical for developing a full understanding of how anticipation skill relates to judgments of deceptive intent.

## ETHICS STATEMENT

This study was carried out in accordance with the recommendations of Brunel University London Institutional Ethics Committee with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the Brunel University London Institutional Ethics Committee.

#### AUTHOR CONTRIBUTIONS

RJ led design of the study, analysis, and writing of the paper. HB contributed to the design of the study and drafting of the paper. KA and BA contributed to the design of the study and writing of the paper.



**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Jackson, Barton, Ashford and Abernethy. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Collision Avoidance With Multiple Walkers: Sequential or Simultaneous Interactions?

Laurentius Antonius Meerhoff <sup>1</sup> \*, Julien Pettré<sup>1</sup> , Sean Dean Lynch<sup>2</sup> , Armel Crétual <sup>2</sup> and Anne-Hélène Olivier <sup>2</sup>

1 Inria, Univ Rennes, CNRS, IRISA - UMR 6074, Rennes, France, <sup>2</sup> Univ Rennes, Inria, M2S - EA 7470, Rennes, France

Collision avoidance between multiple walkers, such as pedestrians in a crowd, is based on a reciprocal coupling between the walkers with a continuous loop between perception and action. Such interpersonal coordination has previously been studied in the case of dyadic locomotor interactions. However, when walking through a crowd of people, collision avoidance is not restricted to dyadic interactions. We examined how dyadic avoidance (1 vs. 1) compared to triadic avoidance (1 vs. 2). Additionally, we examined how the dynamics of a passable gap between two walkers affected locomotor interactions. To this end, we manipulated the starting formation of two walkers that formed a potentially pass-able gap for the other walker. We analyzed the interactions in terms of the evolution over time of the Minimal Predicted Distance and the Dynamics of the Gap, which both provide information about what action is afforded (i.e., passing in front/behind and the pass-ability of the gap). Results showed that some triadic interactions invited for sequential interactions, resulting in avoidance strategies comparable with dyadic interactions. However, some formations resulted in simultaneous interactions where the dynamics of the pass-ability of the gap revealed that the coordination strategy emerged over time through the bi-directional interactions between all walkers. Future work should address which circumstances invite for simultaneous and which for sequential interactions between multiple walkers. This study contributed toward understanding how collision is avoided between multiple walkers at the level of the local interactions.

Keywords: locomotion, multiple interactions, collision avoidance, dynamic gap, interpersonal coordination, affordance, perception-action, pass-ability

#### INTRODUCTION

In a crowd, interactions between people at the micro-level construe how the crowd moves at the macro-level (Vicsek and Zafeiris, 2012). From a movement science perspective, these interactions are a form of interpersonal coordination: the coordination of one's movements with one (or more) other(s) (Schmidt and Richardson, 2008). An important aspect of interpersonal coordination in a crowd is regulating one's distance with others. Distance regulation requires a continuous coupling between perception and action of all persons involved, as each persons' actions affect - to some extent - the actions of others. Numerous studies have addressed distance regulation between two

Edited by:

Mauro Murgia, University of Trieste, Italy

#### Reviewed by:

James L. Croft, Edith Cowan University, Australia Michael J. Richardson, Macquarie University, Australia

> \*Correspondence: Laurentius Antonius Meerhoff rensmeerhoff@gmail.com

#### Specialty section:

This article was submitted to Movement Science and Sport Psychology, a section of the journal Frontiers in Psychology

Received: 28 February 2018 Accepted: 09 November 2018 Published: 30 November 2018

#### Citation:

Meerhoff LA, Pettré J, Lynch SD, Crétual A and Olivier A-H (2018) Collision Avoidance With Multiple Walkers: Sequential or Simultaneous Interactions? Front. Psychol. 9:2354. doi: 10.3389/fpsyg.2018.02354

**130**

persons (i.e., dyads) with regards to interception (e.g., Passos et al., 2008; Zhao and Warren, 2017), and following/tracking (Meerhoff and de Poel, 2014; Meerhoff et al., 2014, 2017; Rio et al., 2014). Moreover, using an orthogonal avoidance task, multiple studies have addressed how dyadic interactions unfold through bidirectional interactions (Olivier et al., 2012, 2013; Basili et al., 2013; Huber et al., 2014; Lynch et al., 2017). When extrapolating these findings to distance regulation with more than two people (e.g., walking through a crowd of people), the multiple interactions make it more complex to study how human behavior emerges (Davids et al., 2014). An extensive literature on pedestrian crowds exists. Some studies adopt a macro-level approach disregarding the micro-level interactions (e.g., Degond et al., 2013). These macro-level approaches are highly informative to predict how crowds behave, however, the perception-action loops that underlie these macro-level patterns cannot be specifically studied. In contrast, microscopic approaches focus on local interactions (Paris et al., 2007; Van den Berg et al., 2008); however, in simulations arbitrary rules are set to combine multiple interactions, as the way humans deal with this is unexplored. Therefore, we examined how dyadic (between two persons) and triadic (between three persons) interactions compare in an orthogonal avoidance task.

The coordination of pedestrians at the level of local interactions can be described using an affordance-based approach (e.g., Fajen, 2007). An affordance is an opportunity for action that is furnished by the environment to an agent (i.e., an entity with decision-making ability such as a pedestrian). Gibson (1979) emphasized that affordances simultaneously depend on the agent's action boundaries and the configuration of the environment. For example, it has been shown that humans can perceive the pass-ability of a gap as a ratio of the width of the gap and their shoulders (Warren and Whang, 1987; Wilmut and Barnett, 2010; Franchak et al., 2012; Hackney et al., 2015). In other words, these affordances cannot be attributed to either the agent or the environment but must be considered in the agentenvironment system (Warren, 2006). A pedestrian in a crowd usually does not come close to its action boundaries, however, the environment - cluttered with other pedestrians - may indeed present a strong limitation on what behavior is afforded. For a walker to interact with this environment, the interactions with the other pedestrians determine for the most part what behavior is afforded. By describing the relation between two pedestrians, Olivier et al. (2012, 2013) developed a measure that describes this agent-environment system. They quantified the time-evolution of the Minimal Predicted Distance (MPD), which is the linearly extrapolated predicted minimal interpersonal distance (i.e., the future distance of closest approach assuming a constant speed and heading). Although MPD is not a perceptual variable, it is an apt descriptor of the action afforded to either walker. A high enough MPD affords passage without collision. By comparing MPD at the end of an interaction with the start, Olivier et al. (2012) showed that walkers consistently adapted their trajectories when the risk of collision was high enough (i.e., initial MPD < ∼1 m). Hence, an MPD below this threshold did not afford a collision-free passage and thus required some form of adaptation. Moreover, the temporal evolution of MPD indicates that collision is avoided proactively, with distinct observation, reaction and regulation phases (Olivier et al., 2013). Such proactive control has been put forward as one of the characteristics of an affordance (Fajen et al., 2009). Therefore, we adopt similar metrics that describe the agent-environment system with a strong emphasis on the other agent(s) in the environment.

Locomotor trajectories toward a target have been described as a stereotyped behavior in terms of path geometry and velocity profile (Hicheur et al., 2007), indicating that some generic principles may govern trajectory generation. However, when an obstacle is in motion, collision avoidance may need to be controlled on-line, as the behavior afforded in relation to moving objects may change over time (Cutting et al., 1995; Plumert and Kearney, 2014). Cinelli et al. (2008) showed that when passing through a moving aperture, participants used a perceptual feedback mechanism to assess whether collision could be avoided by solely changing speed or that shoulder rotations were required as well. Moreover, interpersonal coordination has a strong social component (Schmidt et al., 2011); for example, humans also regulate distance to preserve personal space (Bailenson et al., 2003; Gérin-Lajoie et al., 2005). Additionally, humans reciprocally influence each other, but not necessarily symmetrically (Meerhoff and de Poel, 2014). Therefore, it is important to study collision avoidance behavior in the context of human-to-human interactions. Nevertheless, dyadic (i.e., pairwise) pedestrian interactions show robust regularities in terms of adaptation thresholds (Olivier et al., 2012). Furthermore, these dyadic interactions often take place without inversion of crossing order, that is, the walker that was predicted (based on a linear extrapolation) to cross first at the start was most likely to indeed cross first at the end of the interaction (Olivier et al., 2013; Knorr et al., 2016). It can thus be surmised that although avoiding collision with other people requires a more adaptive strategy compared to avoiding static obstacles, these reciprocal interactions follow some clear regularities.

Some of the characteristics of dyadic interactions may be extrapolated to situations where many pedestrians interact (e.g., a crowd). However, when multiple persons coordinate their movements, the complexity rapidly increases, as each person can potentially interact with each other person (and vice versa). This has previously been described in interactive sports (e.g., Davids et al., 2014; Passos et al., 2016), and specific joint-action tasks (e.g., Richardson et al., 2015). It raises the question whether collision avoidance between many pedestrians can be described as a sequence of many dyadic interactions, or as one simultaneous interaction. One of the few studies (Dicks et al., 2016) that compared pedestrian interactions between two and three walkers, examined the potential for social interaction during a pedestrian crossing. In their study, the potential for social interaction was manipulated by having the oncoming walkers cross with or without looking at a mobile phone. Results revealed that the potential for interaction decreased the velocity, perhaps because the predictability is increased when somebody is looking at their phone. Additionally, the authors noted that participants took longer to complete a crossing with two compared to only one oncoming pedestrian. However, it was beyond the scope of their study to tease apart how these interactions differed. In this paper, we therefore aim to contrast the principles that govern dyadic and triadic interactions in a collision avoidance task.

Using an affordance-based approach (Fajen, 2007), the interactions between many pedestrians can be considered as a collection of gaps that may afford either passing through, or going around (Fajen et al., 2009). In traffic, such gaps have been studied extensively (e.g., Chihak et al., 2010; Louveton et al., 2012; Plumert and Kearney, 2014). For example, Louveton et al. (2012) suggested that drivers interact with the gap that exists between two cars when crossing a busy interaction. The action that is afforded can be described as the "pass-ability" of the dynamic gap that exists between these cars (Plumert and Kearney, 2014). Chihak et al. (2010) found evidence for a multistage interception strategy when passing through a moving gap on a bicycle. Rather than changing and maintaining heading and speed to shift the point of constant bearing to the desired point of intersection, participants consistently accelerated between 4 and 6 s before the passing through the gap. That is, initially participants decelerate (more than strictly necessary to cross behind the first object) and subsequently accelerate to safely pass through the gap. It could be argued that the invitation to act upon the affordance of passing through the gap only becomes apparent as the interaction unfolds (Withagen et al., 2017). Although pedestrian interactions are different from interactions in traffic (because of the imposed traffic rules and the different velocities -and thus "costs" of collisions), the notion of online control and emerging affordances is highly relevant for pedestrian interactions. This was for example highlighted by Cinelli et al. (2008), who put forward that walkers pass through a moving door (cf., dynamic gap) by controlling their trajectories on-line to constantly adjust to changing affordances. By describing the interactions in terms of their affordances, it becomes apparent that monotonic control laws may not entirely explain how trajectories emerge. In this paper, we thus also set out to quantify which behavior is afforded in relation to the gap that may exist between two persons.

Human movement follows regularities at various levels (cf., law-like principles, Turvey, 1990). Schmidt et al. (1990), for example, argued that patterns from within-person coordination (Kelso, 1984), may also apply to between-person coordination (e.g., Harrison and Richardson, 2009; Riley et al., 2011; Meerhoff et al., 2014). Following the same reasoning, we examine whether the principles from dyadic interactions (Olivier et al., 2012, 2013; Basili et al., 2013) may also apply to triadic interactions, as a step toward understanding the micro-level interactions in crowds of pedestrians. Therefore, the aim of this study was to examine how dyadic and triadic interactions compare. First, we examined whether similar initial parameters in terms of MPD yielded similar changes in MPD over time in triadic compared to dyadic interactions. We hypothesized that triadic interactions evoke a simultaneous interaction, therefore yielding different changes to similar initial parameters. Second, we examined whether the hypothesized simultaneous adaptation affected how often the crossing order inverted. We hypothesized that in the triadic interactions these role inversions are more frequent compared to dyadic interactions, as the strategy simultaneously depends on multiple persons. Additionally, we explored whether we can describe triadic interactions in a measure that quantifies the action-opportunities that are afforded to the walkers. To this end, we manipulated the relative starting formation of the group, which formed a potentially pass-able gap for the other walker (see **Figure 1**). With two pedestrians crossing the trajectory of another, there are three actions afforded to the single pedestrian: (1) in front of the other two, (2) through the gap between the other two, or (3) behind the other two. As an extension of MPD, we adopted the measure Dynamic Gap (DG) that described the pass-ability of the gap at each point in time. We hypothesized that the initial parameters of the triadic interaction in terms of DG better predict the outcome compared to the MPD.

In short, we addressed three research questions: (1) Are triadic interactions similar to two subsequent dyadic interactions (i.e., sequential treatment)? (2) Does a triadic interaction yield a rigorous adaptation subverting the starting parameters of each of the two dyadic interactions in a triadic interaction? (3) Can the avoidance strategy in a triadic interaction be explained in terms of a dynamic gap? We considered interactions to be treated sequentially when MPD was similar in the dyadic and triadic trials both at the start and at the end of the interaction. Subsequently, we assessed whether the simultaneous interaction strategy can be captured using DG. We hypothesized that the same consistencies in dyadic interactions do not transfer to triadic interactions, as multiple persons may be interacted with simultaneously. As an alternative we provided a triadic variable that provides additional insight as to how multiple persons are interacted with.

# METHODS

# Participants and Apparatus

Twelve participants (8 males and 4 females, aged 30 ± 7 years) volunteered to take part in this experiment. The participants were recruited through general advertisement within the research institute and local notice boards. The participants were randomly allocated to four groups of three participants. Two groups consisted of males only (aged 34 ± 11 and 32 ± 8 years), one group consisted of females only (aged 30 ± 7.5 years) and one group was mixed with 2 males and 1 female (aged 29 ± 5 years). Participants typically did not know each other before the experiment; however, this was not recorded. All participants had normal or corrected-to-normal vision and no known motor impairments that affected their walking ability. This study was carried out in accordance with the recommendations of the research institute. All participants gave their written informed consent in accordance with the declaration of Helsinki. The experiment took place in a motion capture laboratory with an 18 camera VICON motion capture system (Oxford Metrics Group Ltd., Oxford, UK) covering an interaction area of 13.5 × 17.5 m. Participants' trajectories were recorded with a sampling rate of 120 Hz using a retro-reflexive marker on each shoulder, the center of which was used to represent the participants' displacement. Additionally, we used a set of markers on a helmet to identify each participant.

6.25 m. In the triadic trials, the center of W2 and Walker 3 (W3) was set at 6.25 m. The formation of W2 and W3 was angled at −45◦ , 0◦ , 45◦ , or 90◦ , with a diameter of 2 or 4 m. The participants provided written informed consent for the publication of these images.

# Procedure

**Figure 1** provides an overview of the experiment showing the dyadic (**Figures 1A,B**) and triadic trials (**Figures 1C,D**). For each of the four experimental sessions, we recruited three participants to fulfill the roles of Walker 1 (W1), Walker 2 (W2), and Walker 3 (W3). W2 and W3 formed a group and crossed perpendicular to W1. We instructed all walkers to "reach the target line on the opposite side of the interaction area." We did not provide any additional information as to how they were to do so. In the dyadic interactions (see **Figures 1A,B**), both W1 and W2 started at 6.25 m from the center of the interaction area. In the triadic interactions (see **Figures 1C,D**), the starting positions of W2 and W3 were varied in relative angle (−45 ◦ , 0 ◦ , 45 ◦ , or 90 ◦ ) and radius (2 or 4 m), yielding eight different formations providing a range of starting parameters that would change the characteristics of the gap between W2 and W3. The relative position of W2 and W3 was symmetrical as such that the center between W2 and W3 was fixed at 6.25 m. Theoretically, this implied that without any adaptation of any of the walkers (i.e., constant speed and perfectly straight trajectories), W1 is precisely in the center of the gap between W2 and W3 (both equally far, but in opposite direction) after 6.25 m.

Each of the three participants of a group of participants performed each role (W1, W2, or W3) in every possible configuration, yielding 6 role configurations. For reasons of time, we performed all trials for one role configuration consecutively. The order of the configurations was randomized, and for each configuration all trials were presented in 3 randomized blocks. In the first block of 8 trials, participants performed each of 8 triadic formations crossing from a random side (i.e., to the left or to the right of W1). In the second block, W1 performed two dyadic trials with both remaining walkers in random order. In the third block, each of the 8 triadic formations was repeated, crossing from the opposite side as the first block. Once all 18 trials of a role configuration were completed, the participants were assigned new roles and the process was repeated. In total, we recorded 432 trials: 96 triadic trials (8 formations, 6 role configurations and 2 sides) and 12 dyadic trials (3 role configurations, 2 sides and 2 repetitions) for each of the four experimental sessions. No data points for any included trials needed to be interpolated. One trial (formation [90◦ , 4 m]) was excluded due to an unexpected technical malfunction, leaving 431 trials for further analysis. For each triadic formation and the dyadic formation, we analyzed 48 trials, except for the one missing trial in formation [90◦ , 4 m].

# Timing

We normalized the time-series from tstart until tend. Although of course a walker can already make adjustments during the acceleration phase, we are only interested in the adaptation made once a walker has reached a stable velocity. Therefore, we identified tstart as the first instant that any of the walkers had reached 90% of its maximum speed during that trial (see **Figure 2A**), which coincides with the highly variable instants at the start of a trial (as illustrated by the rate of change in heading in **Figure 2B**). Note that for each trial tstart was the same for all walkers during that trial. Next, we determined tend of each interaction as the instant the minimal interpersonal distance between W1 and the other walkers occurred (tMD12 and tMD13, see **Figure 2C**). The trial duration was consistent (mean ± SD = 4.50 ± 0.06 s), but to allow for a direct comparison between trials we normalized time from tstart (0%) until tend (100%).

#### Kinematic Analysis

We first post-processed the raw kinematic data. The medio-lateral sway movements that occur during gait were removed with a 3rd order low-pass Butterworth filter (cut-off frequency = 0.5 Hz). All kinematic analyses were performed in MATLAB R2015b (The MathWorks Inc., Natick, MA, 2015)<sup>1</sup> . We expressed specific interactions based on the walkers involved; we labeled the interaction between W1 and W2 as I12 and the interaction between W1 and W3 as I13. Additionally, any outcome variable that has the subscript "12" or "13" specifically refers to I12 or I13, respectively. We addressed the research questions using three different timeseries variables. Each outcome measure is explained below in the context of the specific research question.

#### Initial Parameters

To assess whether similar initial parameters yielded similar avoidance strategies in triadic compared to dyadic interactions, we computed the Minimal Predicted Distance (MPD in meters) to their respective interactions (MPD<sup>12</sup> and MPD13). This future distance of closest approach is a linear extrapolation of two walkers' current speed and heading to determine at which interpersonal distance they are predicted to cross assuming constant heading and speed (for more details, see Olivier et al., 2012, 2013). We tested the difference between the dyadic and triadic formations with Statistical Parametric Mapping (SPM; Pataky, 2010), which makes a two-tailed paired comparison at every time step<sup>2</sup> . We separately compared each of the eight triadic formations with the dyadic trials. To account for these multiple comparisons, we applied a Bonferroni correction to the critical p-value (p < 0.05). The corrected critical p-value was therefore set at p < 0.00625. When a significant effect was found during a trial, we reported the t-statistic corresponding to the maximum difference during that trial. Whenever a triadic formation had a similar MPD(tstart) to the dyadic trials, a direct comparison was meaningful as the starting parameters were similar. If MPD(tstart) was similar, but MPD(tend) different, this was considered evidence that the triadic interactions were not simply a summation of sequential dyadic interactions.

#### Crossing Order Inversions

To further test whether triadic interactions are engaged simultaneously, we assessed whether the crossing order (i.e., W1 crossing first or second) changed more often in the triadic compared to the dyadic trials. Based on the same assumptions that we make to compute MPD (i.e., constant speed and heading), the crossing order can be computed by estimating who will first reach the point where the two trajectories are predicted to cross. The crossing order can be easily represented with MPD by assigning a positive sign to MPD(tend), and whenever during a trial the crossing order was predicted to be different compared to the crossing order at tend, the sign of MPD was negative. In the exemplar trial in **Figure 3A**, the negative MPD values of I13 (red dashed line) indicate that the predicted crossing order of W1 in relation to W3 was the opposite of the final crossing order from start (0%) until when the inversion occurred (62%). The predicted crossing order in I12 (blue solid line) on the other hand was the same throughout the whole trial as indicated by the consistently positive values. We reported the number of trials during which an inversion of crossing order occurred. To quantify the effect of formation on crossing in front, through or behind W2 and W3, we used a χ 2 test (p < 0.05). If inversions occurred relatively more often in the triadic compared to the dyadic trials, it was considered evidence that W1 avoided collision with W2 and W3 simultaneously.

#### Gap Pass-Ability

As an alternative to treating triadic interactions as a sequential summation of dyadic interactions, we explored whether we can describe triadic interactions with a measure that quantifies the action-opportunities that are afforded to W1 by simultaneously avoiding W2 and W3. To this end, we computed the Dynamic Gap (DG, see **Figure 3C**). DG is a combination of the Interaction Distances of I12 and I13 (ID<sup>12</sup> and ID13, see **Figure 3B**), a derivative of MPD<sup>12</sup> and MPD13. This derivative has the same magnitude as the MPD, however, the sign was based on the predicted crossing order at every time point in relation to W1: if W1 was predicted to cross first, ID was positive; if W1 was predicted to cross second ID was negative. Together, the signs of ID<sup>12</sup> and ID<sup>13</sup> could then be used to determine the predicted state (i.e., open or closed) of the gap that may exist for W1 between W2 and W3. If both IDs have the same sign, W1 is predicted to go around the gap (IDs > 0 represent W1 in front, see **Figure 4A**;

<sup>1</sup>The code for the kinematic analyses can be found in the supplementary material and on https://github.com/Rens88/PW\_to\_Multiple\_Public. The 'exampleRun.m' can be used to run the code with some mock-data. Data can be made available upon request.

<sup>2</sup> SPM (Statistical Parametric Mapping) is a commonly used open source statistical tool in neuroimaging (available on http://www.spm1d.org), specifically, through the application of Random Field Theory (RFT) that makes inferences of topological brain image features that are continuous functions of space or time (Adler, 1981; Pataky, 2010, 2016). The methods are based on a General Linear Model (GLM) of analysis; in its simplest expression, SPM runs an analysis per time point, grouped as a continuous statistical process and tested for probabilistic behavior through RFT using univariate probability and spatial covariance. The probabilities are computed from expected Gaussian random fields (Adler and Hasofer, 1976), which in turn leads to probabilistic descriptions of distribution that in turn allow for common parametric analyses such as GLM and ANOVA. See for a more detailed explanation Pataky (2010) section 2; and for full technical details Pataky (2016). Of the software package, we specifically used the paired

comparison in spm1d (spm1d\_stats\_ttest2.m) and our inferences were based on a GLM (spm1d\_stats\_glm.m).

FIGURE 2 | An exemplar trial showing the speed (A), change in heading (B) of each walker, and interpersonal distances (C) between W1 & W2 and between W1 & W3. The speed (A) during a trial was used to identify tstart, which corresponds to the instant the variable initiation phase was finished (B). The interpersonal distances (C) were used to derive tMD.

Dynamic Gap (DG). The solid and dashed line refer to the interactions between Walker 1 (W1) & Walker 2 (W2) (i.e., interaction between W1 and W2, I12), and between W1 and W3 (i.e., interaction between W1 and W3, I13), respectively. The vertical lines in (B) indicate which interaction had the smallest absolute ID and upon which DG was based. The dotted horizontal line extending I12 in (A, B) denotes that the specific values were no longer updated as the time of minimal distance (tMD12) had already passed. See also Video 1 in the Supplementary Material for an animated display.

IDs < 0 represent W1 behind, see **Figure 4C**) and the gap is thus closed. On the other hand, if the IDs have the opposite sign, it means that W1 is predicted to cross in front of one (i.e., positive ID) and behind the other (i.e., negative ID) walker and the gap is thus open (see **Figure 4B**). The interaction with the smallest absolute ID is then the interaction that constrains the state of the gap, as it represents the minimum adaptation required to change the state of the gap. This margin, in turn, can specify whether the gap affords passing through. Therefore, we determined DG as the smallest absolute ID and signed it based on whether W1 was predicted to go through (DG > 0 m) or around (DG < 0 m) the gap between W2 and W3. Note that the size of the gap between W2 and W3 as at least twice the magnitude of DG, as DG represents the smallest distance to one side of the gap.

In the exemplar trial in **Figure 3B**, given that ID<sup>12</sup> was positive and ID<sup>13</sup> negative at 100%, it can be deduced that W1 passed in front of W2 and behind W3. In **Figure 3C**, the negative DG indicates that until 62% the gap was predicted to be closed. However, from that point onward DG was positive, meaning that W1 eventually crossed between W2 and W3. To assess

each \* and ◦ corresponds with the ID, which is projected on the right in each panel as a portal that Walker 1 (W1) is predicted to pass through (B) or not (A, C). DG is

whether our manipulation of the formation yielded a broad range of behaviors, we examined whether formation affected the gap crossing behavior using a χ 2 test (p < 0.05). Then, we compared the relative occurrence of inversions of DG in trials where W1 went through the gap with trials where W1 went around the gap using a χ 2 test (p < 0.05). Using SPM, we performed six pairwise comparisons of DG, making each possible comparison between Open and Closed trials, with and without inversion. Again, we applied a Bonferroni correction to the critical p-value (p < 0.05) to account for the multiple comparisons. The corrected critical p-value was therefore set at p < 0.0083. The relative frequency of inversions in combination with when the difference between Open and Closed trials provides a description of how well this triadic measure (DG) describes the avoidance strategy.

the distance to the closest side of the projected portal.

# RESULTS

# Initial Parameters MPD

We examined the difference between the dyadic and triadic trials using SPM, comparing the normalized time-series of MPDDyadic with MPD<sup>12</sup> and MPD<sup>13</sup> in each formation separately (see **Figure 5**). All significant differences with a corrected critical pvalue of 0.00625 are indicated in **Figure 5** with a horizontal bar above the plot. Whenever the difference did not start at tstart, short vertical bars were added to highlight the first instant a significant difference occurred. The SPM analysis revealed that in formation [90◦ , 2 m], MPD<sup>12</sup> and in formation [0◦ , 4 m], MPD<sup>13</sup> was significantly greater than the dyadic trials at tend, but not at tstart [t(94) = 3.687, p = 0.002; t(94) = 6.251, p = 0.001, respectively]. Additionally, this difference was already apparent from 7% onward in formation [90◦ , 2 m], and from 3% onwards in formation [0◦ , 4 m]. In formation [45◦ , 2 m], we also found a significant deviation of MPD<sup>12</sup> from MPDDyadic during the trial. Between 49 and 68% MPD<sup>12</sup> was significantly larger compared to MPDDyadic [t(94) = 3.186, p = 0.003]. Furthermore, the SPM analysis revealed that MPD<sup>12</sup> was significantly different from the dyadic trials from tstart until tend in formation [0◦ , 4 m] [t(94) <sup>=</sup> 8.043, <sup>p</sup> <sup>&</sup>lt; 0.001], [45◦ , 4 m] [t(94) = 13.632, p < 0.001], [90◦ , 4 m] [t(94) = 6.875, p < 0.001]. Similarly, MPD<sup>13</sup> was significantly different from the dyadic trials from tstart until tend in formation [45◦ , 2 m] [t(94) <sup>=</sup> 4.438, <sup>p</sup> <sup>&</sup>lt; 0.001], [45◦ , 4 m] [t(94) <sup>=</sup> 11.283, <sup>p</sup> <sup>&</sup>lt; 0.001], [90◦ , 4 m] [t(94) = 7.567, p < 0.001].

# Crossing Order Inversions

In the dyadic trials, 13% (6 out of 48) of the trials had an inversion of crossing order at some point during a trial. For the triadic interaction I12, inversions occurred in 12% (45 out of 383) of the trials. For I13, inversions occurred in 17% (65 out of 383) of the trials. The proportion of trials with inversions was not significantly different in the dyadic trials, I12 or I13: χ 2 (2, N = 814) = 4.401, p = 0.111.

#### Gap Pass-Ability

In **Figure 6**, the percentage of trials per formation with each gap crossing behavior (through or around) is shown. Formation significantly affected the gap crossing behavior, χ 2 (14, N = 383) = 201.305, p < 0.001. In some formations W1 almost never passed through the gap ([45◦ , 2 m] and [45◦ , 4 m]), other formations allowed W1 to cross through the gap in almost all trials ([−45 ◦ , 4 m], [0◦ , 4 m], and [90◦ , 4 m]). It stands out that particularly when the radius was 4 m, W1 often crossed through the gap, except in formation [45◦ , 4 m].

In **Figure 7**, ID<sup>12</sup> and ID<sup>13</sup> are visualized in relation to each other at tstart (**Figure 7A**) and tend (**Figure 7B**). **Figure 7A** provides a descriptive insight in the distribution of the starting parameters of the triadic trials in terms of ID and therefore DG. Each quadrant represents the (predicted) state (open or closed) of the gap between W2 and W3, which is highlighted

over time per formation. A horizontal line above the plots indicates when MPD12 (1) and MPD13 (\*) are significantly different from the pairwise MPDDyadic (solid line) after a Bonferroni correction was applied. When the difference did not start at tstart, small vertical bars were added to indicate first instant MPD was different from MPDDyadic.

with the different colors. Moreover, the open circles and filled dots indicate whether for that trial an inversion occurred or not, respectively. For trials where W1 went through the gap (n = 230), an inversion of DG occurred in 12% of the trials. On the other hand, for trials where W1 went around the gap (n = 153), an inversion of DG occurred in 41% of the trials. That is, 12% of the green data points (see **Figure 7B**; top left and bottom right quadrant) and 41% of the red data points (see **Figure 7B**; top right and bottom left quadrant) were at some point in a differently colored quadrant (see

**Supplementary Material**, **Video 2**). In the trials where W1 passed through the gap, DG inversions occurred significantly less often [χ 2 (2, N = 814) = 41.075, p < 0.001].

For trials where W1 went through and around, with and without inversions, we separately plotted the average (±SE) values of DG for every time-step (see **Figure 8**). Using SPM, we compared each gap crossing behavior against one another (i.e., through and around, with and without inversion) with a corrected critical p-value of 0.0083. For trials with inversion, the trials where W1 went through the gap were significantly different from the trials where W1 went around the gap from 18% until tend [t(151) = 46.560, p < 0.001], as highlighted in **Figure 8**. For the trials where W1 went around the gap, the trials with inversion were significantly different from the trials without inversion from 0% until 70% [t(151) = 13.582, p < 0.001] and later from 93% until tend [t(151) = 3.364, p = 0.004]. The remaining comparisons [around with inversion compared to through without inversion, t(262) = 51.266, p < 0.001; through with compared to without inversion, t(228) = 13.058, p < 0.001; through without inversion compared to around without inversion, t(291) = 60.700, p < 0.001; through with inversion compared to around without inversion, t(88) = 46.460, p < 0.001] were all significantly different from tstart until tend.

# DISCUSSION

In the current paper, we compared dyadic (1 vs. 1) and triadic (1 vs. 2) interactions. We aimed to examine whether the extra walker in the triadic interactions changed how the interactions emerged. We manipulated the starting formation in the triadic interactions to obtain a range of initial parameters. We first assessed whether similar initial parameters in terms of MPD resulted in similar changes in MPD over time. Secondly, we compared whether the potential simultaneous treatment of the multiple interactions in the triadic trials resulted in a higher number of inversions. Lastly, we explored whether the triadic interactions could be described at the group level in terms of the pass-ability of the dynamic gap that may be formed between the two grouped walkers (W2 & W3). We found evidence that the same initial parameters resulted in different adaptations in the triadic compared to the dyadic trials. However, this potentially simultaneous avoidance strategy was not corroborated by a higher number of inversions. Furthermore, we successfully described how the potential gap between walkers can be linked to its pass-ability. Moreover, the pass-ability of the gap appeared to unfold through the bi-directional interactions and stabilize over time. We discuss the influence of the extra walker on the avoidance strategy in relation to the triadic formations and discuss the interactions in terms of the affordance of passing through the gap. Finally, we highlight how in some cases multiple interactions were possibly treated simultaneously.

#### Dyadic vs. Triadic Interactions

We addressed our first research question - whether multiple interactions are treated sequentially - by examining the MPD. For some formations ([45◦ , 4 m] and [90◦ , 4 m]), the MPD was always significantly different from the dyadic trials. In these formations, both interactions could be negotiated without actively avoiding collision. When the risk of a collision is low, avoidance is not necessary (Olivier et al., 2012; Huang et al., 2017). It is thus uncertain whether the strategy in triadic interactions differed from the dyadic interactions. For the remaining formations, the same initial parameters (in terms of MPD) in the dyadic interactions did not always lead to the same avoidance behavior in the triadic interactions which we interpret in the context of simultaneous and sequential adaptations.

Out of the six comparable formations, three formations yielded a simultaneous collision avoidance strategy by W1. As evidenced by the difference in MPD at tstart in formation [90◦ , 2 m], the extra walker forced a quick and relatively large adaptation to secure a collision-free interaction. Once the collision risk of the first interaction was acceptable, the next interaction could be treated similarly to a dyadic interaction. In [0◦ , 4 m] on the other hand, the extra walker influenced the interaction, despite crossing at a large distance (i.e., with a low risk of collision). Moreover, in [45◦ , 2 m] the timeseries analysis (see **Figure 5**) revealed a significantly higher MPD halfway the interaction compared to the dyadic interactions. It

FIGURE 7 | Distribution of the Interaction Distance (ID) in the triadic trials to both Walker 2 (W2) and Walker 3 (W3) at tstart (A) and tend (B). The colors indicate the final crossing order, the open circles indicate trials with a Dynamic Gap (DG) inversion. The dashed horizontal and vertical lines indicate the border between crossing in front and behind the other walker. When crossing behind both W2 and W3 (IDs < 0 m) or in front of both W2 and W3 (IDs > 0 m), the gap between them is closed for Walker 1 (W1). See also Video 2 of the Supplementary Material for an animated display from tstart until tend.

can be argued that the extra walker resulted in these augmented responses, reflecting a degree of simultaneous treatment of the interactions. Similarly, Bruneau et al. (2015) showed that depending on social constraints such as group density and appearance, walkers might prefer to avoid having to interact with members of a group individually by going around a group as a whole. Such simultaneous treatment of the interactions could be a conservative strategy to simplify interacting with multiple walkers, which is a common strategy when uncertainty increases (Krell and Patla, 2002; Berard and Vallis, 2006; Lowrey et al., 2007).

Finally, some triadic trials showed no difference with the dyadic trials during the interaction ([−45 ◦ , 2 m], [−45 ◦ , 4 m], and [0◦ , 2 m]). In terms of the evolution of MPD, this indicates that the triadic interactions were negotiated as sequential dyadic interactions. It remains somewhat ambiguous whether these interactions were treated sequentially, as perhaps this was a result of the convenient positioning of the extra walker. The symmetrical set-up of the triadic formation allowed for W1 to go in front of one, and behind the other walker, with acceptable risks of collision. Indeed, formations [−45 ◦ , 2 m], [−45 ◦ , 4 m], and [0◦ , 2 m] had a high incidence of going through (see **Figure 6**). Overall, it can be argued that, when the extra walker is conveniently positioned, triadic interactions can be negotiated with sequential dyadic interactions. However, the extra walker can also interfere with the original strategy, hence resulting in a strategy where W2 and W3 are avoided simultaneously. These findings highlight that a good understanding of the micro-level interactions that construe crowd movements requires a close examination of how multiple interactions are engaged.

Previously, preservation of crossing order has been reported as one of the rigid characteristics of pairwise (i.e., dyadic) locomotor collision avoidance (Olivier et al., 2013; Knorr et al., 2016; Lynch et al., 2017). Therefore, we addressed our second research question in terms of crossing order inversion. Interestingly, we found a high rate of crossing order inversion compared to previous research, even in the dyadic trials (13%). In the triadic trials, inversions occurred in 12% and 17% in I12 and I13, respectively, but in contrast to our hypothesis inversions did not occur significantly more often in the triadic compared to the dyadic interactions. One difference with the previous studies by Olivier et al. (2012, 2013) is that in the current study participants could already see each other at the starting positions (similar to Basili et al., 2013; Huber et al., 2014). Early visual information could influence the interaction, particularly given that early adaptations have a bigger effect on the crossing distance. Inversions typically indicate that there was some asymmetry in the interaction, that is, one pedestrian contributed more to the change compared to the other. Vassallo et al. (2017) showed for example that in a human-robot interaction, humans prefer to let the robot cross first, even if that means inverting the crossing order. These asymmetries could be considered in the context of social motor coordination (cf., Schmidt et al., 2011). For example, Dicks et al. (2016) reported that the collision avoidance strategy depends on the potential for social interaction, which was manipulated by having oncoming walkers look at their mobile phones or not. In the absence of gaze, pedestrians adjust their strategy (Croft and Panchuk, 2017), arguably to anticipate that the other walker may not initiate any adaptation (cf., asymmetrical coupling in interpersonal coordination; Meerhoff and de Poel, 2014). This social component can also be modeled mathematically. For example, Colombi and Scianna (2017) made a first attempt to include the subjective perception of the attractor-state of multiple persons in their model. Although the model did not expand on an agent's action opportunities, it did report the that subjective perception influenced sequential (i.e., localized perception) and simultaneous (i.e., distributed perception) interactions.

## Pass-Ability

For our third research question, we assessed the pass-ability of the gap between W2 and W3 with respect to W1. Looking at the triadic interactions, the gap crossing behavior was clearly affected by formation (see **Figure 6**). In some formations ([45◦ , 2 m] and [45◦ , 4 m]), W1 almost never passed through the gap, other formations allowed W1 to cross through the gap in almost all trials ([−45 ◦ , 0 ◦ ] and [90◦ at 4 m]). It stands out that particularly with a 4 m radius, W1 often crossed through the gap, except at −45 ◦ . This indicates that participants were susceptible to the affordance of passing through a gap between others (Plumert and Kearney, 2014). Inversions were less frequent in trials where W1 ended up going through the gap (12%) compared to W1 going around the gap (41%). In other words, gaps that were initially predicted to be open ended up being closed more often than vice versa. This could indicate that going around was more inviting (Withagen et al., 2017) compared to going through. In other words, the attractor-state of going around was more stable (Schmidt and Richardson, 2008). In contrast to our hypothesis, the initial parameters in terms of DG did not result in less inversions compared to the initial parameters in terms of MPD, with inversions only occurring in 12–17% of the trials. This highlights how low the prediction accuracy of gap crossing behavior is at tstart. When subsequently examining the time-evolution of DG, it becomes clear that only after 18% of time the predicted crossing behavior on average matched the final gap crossing behavior. It may well be that the triadic interactions were inherently more dynamic because of the mutual (i.e., bi-directional) interactions, which has previously been observed even in a highly controlled setting (Meerhoff and de Poel, 2014). As can be seen in **Figure 8** and **Video 2** of the **Supplementary Material**, the predicted outcome in triadic trials can change until quite late in the interaction. Therefore, rather than looking at the initial parameters, the dynamics of a gap need to be considered to examine the reciprocal interactions between all walkers throughout an interaction using a descriptive variable such as DG. Our DG jointly captures the behavior of three walkers in terms of their speed and heading. which can shed light onto how the gap crossing behavior unfolds. However, one could argue that to classify a gap as pass-able, it is pertinent to consider the space necessary to pass through a gap. Previous work has highlighted how the pass-ability of a gap is tightly coupled to the shoulder to aperture ratio (e.g., Wilmut and Barnett, 2010; Franchak et al., 2012; Hackney et al., 2015). With DG, we have taken the first hurdle toward assessing multiple pedestrian interactions at the micro-level by quantifying the magnitude of a potentially pass-able gap. Future work could extend this measure by looking at how the magnitude of DG and the final gap crossing behavior relates to body-scaled characteristics such as the shoulder to aperture ratio. In addition to examining I12 and I13, future work could for example use gaze (Meerhoff et al., 2018) to also focus on I23 - the interaction between the grouped walkers - to tease apart whether pedestrians in a crowd form a coordinated strategy to let others pass between them. Depending on how the interactions between W2 & W3 (i.e., within a group) are perceived, the decision to go through or around are strongly affected (Bruneau et al., 2015). If W2 and W3 are perceived as a coordinated unit, it might be less inviting to go through a potential gap between two walkers.

The different gap crossing behaviors (through or around, and with or without DG inversions) may reveal an extent of sequential or simultaneous treatment of the interactions by W1. It could be argued that trials without DG inversion allowed for simultaneous treatment of the interactions, as the predicted crossing behavior corresponded with the final gap crossing behavior. Trials with DG inversions could be interpreted as sequential: First, W1 and W2 interacted, and then W1 and W3. This is perhaps similar to the MPD differences in formation [90◦ , 2 m], where I12 first reached an "acceptable" risk of collision to subsequently continue treating I13 as a dyadic interaction. However, similar to the MPD differences between dyadic and triadic trials (see above, [−45 ◦ , 2 m], [−45 ◦ , 4 m] and [0◦ , 2 m]), it remains ambiguous whether these interactions were treated simultaneously intentionally, or whether this was simply due to the convenient positioning of W2 and W3. It is difficult to say whether some formations simply did not afford sequential treatment, or whether the simultaneous treatment was in fact coincidental. More work is required to better understand to what extent interactions can be treated simultaneously. Current models of understanding how walkers combine multiple interactions are typically based on an arbitrary selection procedure, for example based on whether the other walkers are within view and within a critical distance (e.g., Helbing et al., 2001). However, human-to-human interactions seem to be governed by subtler behavioral laws. Despite the relatively few unique participants, our results showed that these interpersonal dynamics are highly adaptive and not exclusively restricted by a critical distance. The generalizability of this finding can be strengthened by adopting a random effects model (e.g., Barr et al., 2013), which more appropriately deals with the between-subject variability. Nevertheless, it can be surmised that the adaptive human-to-human behavior needs to be taken into account for understanding the micro-level interactions. However, more work is required to assess what makes the passability of a gap attractive enough to act upon it.

Another direction for future work could be to expand on how this behavior may be guided visually. As for example highlighted by Zhao and Warren (2017), on-line visual information is pertinent for locomotor interception of a moving target. Moreover, Dachner and Warren (2016) explain how depending on a person's location (in front or to the side), pedestrian following can be explained by a combination of a target's bearing and optical expansion. Both locomotor interception and pedestrian following are important components of pedestrians navigating through crowds of people. For example, it could be that the bearing angle provides pertinent information to avoid collision with multiple persons. However, Chihak et al. (2010) showed that when cyclists cross through gaps in traffic, humans do not simply adjust their action to the bearing angle. Although the initial strategy seems consistent with a bearing angle strategy, they observed a multi-stage strategy as a cyclist got closer to the actual gap. Nevertheless, it may be that some form of the bearing angle (e.g., a fractional order, see Bootsma et al., 2015) or another optical variable could indeed explain how collision with multiple walkers is guided visually. However, in the case of avoidance, the rate of change of bearing angle becomes infinite at the instant of smallest distance, which makes it difficult to apply it to this specific situation. As an alternative, we propose that navigation through a crowd of people can be considered as passing through a multitude of dynamic gaps. Watson et al. (2011) provide a good basis for determining how pass-ability of a gap can be optically specified. In their study, they assessed the perception of whether a gap between two rugby defenders affords passing a ball through. They showed that 82% of the variance could be accounted for based on tau-based information (Lee, 1976, 1998). Future work could examine which optical variables may specify the pass-ability of a dynamic gap between pedestrians.

# CONCLUSION

We showed that triadic (1 vs. 2) interactions are not always comparable with dyadic (1 vs. 1) interactions. Although it can be argued that a conveniently positioned extra walker allowed for similar triadic compared to dyadic interactions, the extra walker can also interfere with the original strategy, hence resulting in a different adaptation. Even more than dyadic interactions, triadic interactions strongly depend on how the reciprocal interactions between all walkers unfold. Moreover, the interpersonal dynamics are highly adaptive and not (only) restricted by a critical distance. We adopted a novel analysis to describe this dynamic character of the gap between two walkers. By describing the affordance of passing through a gap over time, the emergence of the final gap crossing behavior can be better understood. Furthermore, we propose that some interactions afforded simultaneous treatment and others required a more sequential treatment of the interactions. This study was a first attempt to understand the combination of multiple interactions at the micro level. However, future research should specifically address under which circumstances these different types of interactions occur. For example, using a virtual reality environment, the trajectories of the interfering walkers can be controlled to design a paradigm that contrasts simultaneous and sequential treatments of interactions. In sum, we revealed that in some cases, triadic and dyadic interactions yield different collision avoidance strategies and interestingly, the interactions between multiple persons unfold over time through bi-directional interactions.

# AUTHOR CONTRIBUTIONS

LM, JP, AC, and A-HO conceived and planned the experiments. LM, JP, AC, and A-HO carried out the experiments. LM took the lead in the data analysis and writing of the manuscript. LM, SL, JP, and A-HO provided critical feedback and helped shape the research, analysis and manuscript.

# FUNDING

This work was supported by the French National Research Agency ANR, project PERCOLATION number ANR-13-JS02- 0008, as well as the Brittany Region, project SAD 2015 - INTERACT (9297).

# SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg. 2018.02354/full#supplementary-material

# REFERENCES

Adler, R. M. (1981). The Geometry of Random Fields. Chichester: Wiley.


Helbing, D., Molnár, P., Farkas, I., and Bolay, K. (2001). Self-organizing pedestrian movement. Environ. Plan. B Plann. Design 28, 361–383. doi: 10.1068/b2697


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Meerhoff, Pettré, Lynch, Crétual and Olivier. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Watching or Listening: How Visual and Verbal Information Contribute to Learning a Complex Dance Phrase

Bettina E. Bläsing1,2 \*, Jenny Coogan<sup>3</sup> , José Biondi <sup>3</sup> and Thomas Schack <sup>1</sup>

<sup>1</sup> Neurocognition and Action Research Group & Center of Excellence Cognitive Interaction Technology, Bielefeld University, Bielefeld, Germany, <sup>2</sup> Department of Music and Movement in Rehabilitation and Therapy, Faculty of Rehabilitation Science, Technical University Dortmund, Dortmund, Germany, <sup>3</sup> Palucca Hochschule für Tanz Dresden, Dresden, Germany

While learning from observation is generally regarded as major learning mode for motor actions, evidence from dance practice suggests that learning dance movement through verbal instruction might provide a promising way to support dancers' individual interpretation of and identification with the movement material. In this multidisciplinary project, we conducted a study on the learning of dance movement through two modalities, observation of a human model in a video clip and listening to the audio-recording of a verbal movement instruction. Eighteen second year dance students learned two dance phrases, one from observation and one from verbal instruction, and were video-recorded performing the learned material. In a second learning step, they were presented the complementary information from the other modality, and their performance was recorded again. A third recording was carried out in a retention test 10 days after learning. Completeness scores representing the recall of the dance phrases, expert ratings addressing the performance quality and questionnaires reflecting the participants' personal impressions were used to evaluate and compare the performance at different stages of the learning process. Results show that learning from observation resulted in better learning outcomes in terms of both recall and approximation of the model phrase, whereas individual interpretation of the learned movement material was rated equally good after initially verbal and initially visual learning. According to the questionnaires, most participants preferred learning initially from observation and found it more familiar, which points toward an influence of learning habit caused by common training practice. The findings suggest that learning dance movement initially from observation is more beneficial than from verbal instruction, and add aspects with regards to multimodal movement learning with potential relevance for dance teaching and training.

Keywords: motor learning, observation, verbal instruction, recall, performance, dance

# INTRODUCTION

Interdisciplinary projects linking dance and neurocognitive research have recently come to increasing awareness in artistic and scientific communities (see Sevdalis and Keller, 2011; Bläsing et al., 2012). Recently, the claim that such research should be carried out with equal contribution of and benefit for the different communities, by multidisciplinary teams involved at all stages,

#### Edited by:

Mauro Murgia, University of Trieste, Italy

#### Reviewed by:

David Ian Anderson, San Francisco State University, United States Blandine Bril, School for Advanced Studies in the Social Sciences, France

\*Correspondence: Bettina E. Bläsing bettina.blaesing@tu-dortmund.de

#### Specialty section:

This article was submitted to Movement Science and Sport Psychology, a section of the journal Frontiers in Psychology

Received: 08 June 2018 Accepted: 12 November 2018 Published: 30 November 2018

#### Citation:

Bläsing BE, Coogan J, Biondi J and Schack T (2018) Watching or Listening: How Visual and Verbal Information Contribute to Learning a Complex Dance Phrase. Front. Psychol. 9:2371. doi: 10.3389/fpsyg.2018.02371

**144**

is been expressed with increasing emphasis (e.g., Jola, 2018). The project presented in this article represents an example of such research; it has been developed within an interdisciplinary network of scientists, scholars and artists (Dance engaging Science; The Forsythe Company|Motion Bank) and is motivated by dance-pedagogical questions on movement learning. The process of developing, planning and conducting this study has been monitored by the German society for dance medicine "tamed" (Tanzmedizin Deutschland e.V.), and the different stages of this process are presented and commented in a blog (www.blog.tanzmedizin.com), to provide a showcase for a multidisciplinary (German-speaking) audience. We expect that the outcomes contribute to our general understanding of movement learning in dance, and that they might yield potential implications for teaching and training in dance-related disciplines.

A decade after the discovery of the "mirror neurons" in the brains of macaque monkeys (Rizzolatti and Craighero, 2004) has given a new impetus for theoretical frameworks emphasizing the tight coupling of action and perception (Prinz, 1997; Hommel, 2015), scientists interested in related functions and systems in the human brain started to use video clips of dancers performing full-body movements as stimuli in brain imaging studies (Calvo-Merino et al., 2005, 2006; Cross et al., 2006). These influential studies showed that the activation of particular motor-related brain areas during the observation of human motor actions is modulated by the observer's own motor expertise (Calvo-Merino et al., 2005, 2006) and preference (Calvo-Merino et al., 2008). The interest in skilled motor action that resulted from such findings opened new roads for sport and exercise science to join forces with cognitive psychology and neuroscience in investigating action-perception coupling (Beilock, 2008; Moreau, 2015). An increasing number of studies addresses this and related topics in sports and dance contexts, as these are supposed to have a higher ecological relevance with regards to real-world scenarios (Jola et al., 2011). Beilock (2008) argues that the study of sportspecific scenarios has a high potential for advancing theories of cognitive neuroscience, in particular with regards to questions of motor control, motor learning, and expertise.

Motor actions from sports or dance are often referred to as naturally complex, in contrast to simple response actions (such as key presses) typically applied in experimental laboratory tasks. Despite the general use of the term "complex" with regards to motor actions or tasks, however, there seems to be no clear and reliable definition of that concept. Actions that are termed "complex," as opposed to "simple," often require specific training to be mastered even on a rather low level of performance (e.g., Meister et al., 2005; Cross et al., 2013), which makes them suitable for experimental learning tasks. Such actions typically involve the whole body (in sports or dance) or mainly the hands (in music or tool use), and consist of several independent elements that are either performed at the same time in a coordinated manner, or successively, as action sequences. In the case of action sequences, complexity often refers to the length of the sequence, the number of different components and the reproduceability of their order. The latter can be determined by a set of rules or an underlying "grammar," which also contributes to the over-all complexity (e.g., Opacic et al., 2009). Hossner et al. (2015) suggest that complex tasks in sports have a modular architecture acquired by the athlete on the level of motor control, and that this architecture can be revealed as sub-goal-related micro-structure via a functional task analysis. According to Schack (2004), who also regards motor control as being constructed in a hierarchical manner, complex motor actions are based on mental representations in long-term memory that mediate between volition, or intention, and effect representations, with the latter being deterministic for simple movements. The general idea that motor control, motor learning and the performance of skilled actions are based on cognitive representations has been expressed by many authors, going back to Lotze (1852) and James (1981), as well as Bernstein (1947), who applied the idea of cognitive representations to his model of the construction of movement. Bernstein, 1935/1967 pointed out that movements should be understood as goal-directed acts and assigned a decisive role to the model of the needed future as organizing principle in movement control (Bernstein, 1947, 1967, 1975). Furthermore, he regarded the sophisticated control of particular movements characteristic of human skills is a consequence of action development, rather than its basis. Crucially, according to Bernstein's concept of dexterity developed in the 1940s, actions are primary, and simple movements and postures are consequences of the organism's activities rather than building blocks of action (Bernstein, 1996). Following this line of argument, Reed and Bril (1996) point out that the ability to construct, coordinate and modulate movements independently, regardless of the functional context, and to fractionate actions into postures and movements in a controlled way (as is done in dance), can be regarded as one of the most sophisticated human achievements.

Bernstein (1947, 1967, 1975) also emphasizes the role of sensory (re-afferent) feedback in this context, arguing that motor control requires a continuous processing of sensory feedback, as well as comparison with the coded effect. According to this view, visual and auditive feedback play a substantial role in the control of complex motor actions for controlling the multitude of degrees of freedom present in the motor system. Coordination is thus conceptualized as transforming the degrees of freedom of the movement system into targeted movement effects (see Bernstein, 1971). Such a transformation requires specific means, including cognitive ones (e.g., representations), and it requires a functional mediation between the different building blocks of the movement system. Bernstein (1947) presented the most comprehensive compilation of descriptive and experimental data on the functional mediation of the building blocks of the movement system available at that time. His model of the interplay between movement goals, representations, and perceptual feedback is composed of hierarchically organized interdependent levels, including a superordinate symbolic or conceptual level for the organization for complex movements. More recently, authors from different fields have emphasized the role of mental representations in the control and learning of motor actions (e.g., Glenberg, 1997, 2010). According to Steels (2003), mental representations primarily co-evolve together with the corresponding actions and thereby become vehicles for higher mental functions, such as thinking and planning. Nomikou et al. (2016) argue in favor of a continuous development of rich representations through and for action and interaction, suggesting that children develop rich representations from the beginning on, and propose that representations are continuously shaped and enriched throughout development by acting and interacting in the physical and social world.

While mental representations can be regarded as paramount for the learning, planning, adaptation and skilled performance of sophisticated movements as they are performed by athletes or dancers, the criteria applied to define the complexity of motor actions are still manifold. Wulf and Shea (2002) argue that no one continuum can be satisfactory for defining complexity with regards to task or action, but that a number of different context-dependent continua and their interactions, as well as the demands placed on the learner's capacity must be taken into account. The degree of complexity then depends on the choice of criteria applied, which might vary according to the context. Dance can be regarded as a domain in which actions require specific criteria for complexity. Tempel et al. (2015)refer to dance moves as definitely complex because they involve the whole body, have a hierarchical structure (i.e., they can be combined to higher level phrases), and they follow spatiotemporal rules (i.e., they have to be executed in a special order, corresponding to a given rhythm, and according to predefined spatial patterns). Furthermore, dancing requires practice and is embedded in a social and cultural context. An aspect that differentiates dance from most other action domains is that the absence of obvious external action goals is rather common. In contrast, dance movements often possess internal goals that are directly related to the movement itself, its trajectory, dynamics and expression. It has even been proposed that working memory might contain a kinaesthetic-spatial system in which body configurations act as goals, comparable to targets in external space (Smyth and Pendleton, 1990; Cortese Rossi-Arnaud and Rossi-Arnaud, 2010). Even though many motor actions performed in dance contexts also have external goals (depending on the choreography), these commonly do not supervene the movement-related goals that are typical for, and constitutive for, dance moves. Schachner and Carey (2013) refer to motor actions that do not possess external but movement-related goals as "dance-like," even if these actions are not performed in a dance context, and state that dance-like actions are primarily characterized by their movement-based goals, whereas other "rational" actions have obvious external goals. Such dance-like actions, even though they do not easily comply to all criteria that have been identified for actions in general (such as an external goal), can be highly complex. With regards to learning, they are likely to depend more strongly on dynamic, movement-related representational formats than actions with external goals, and therefore to rely more strongly on internal simulation processes. Dance-like actions are definitely controlled by volition, require learning and practice, and can have a complex hierarchical structure, but the actual action goal is often hard to recognize for the naive observer, as it is predominantly related to movement parameters. Such actions can hardly be learned by emulation, but rather have to be acquired by imitation (i.e., copying results vs. copying actions, see e.g., Tennie et al., 2006), involving the direct route that is based on motor resonance rather than understanding of action goals and action semantics (Gonzalez Rothi et al., 1991; Rumiati et al., 2005).

Movement learning in dance therefore represents a specific type of motor learning that is characterized by a strong engagement of motor simulation processes, as well as by cognitive processes and strategies that depend on skill level. Novel movement material is typically taught in a multimodal manner, based on visual observation of a human demonstrator, supported by language, gesture, and body language providing kinematic, artistic, expressive and spatiotemporal cues, as well as the dancer's own advanced motor and imagery skills (e.g., Stevens and McKechnie, 2005; Stevens, 2017). It has been shown that the use of language (in terms of verbal cues) can facilitate or enhance motor learning by guiding attention toward relevant features of the movement and making these aspects explicit (see Wulf and Prinz, 2001). In dance and sports training, observational learning from a visual model is often supported by verbal cue-giving. Evidence from practice suggests that explicit verbal instructions and movement descriptions play an important supportive role in movement learning by providing conceptual clarity with regards to kinematic and spatiotemporal aspects and thereby fostering the understanding, simulation and performance of movement phrases. The role of language in motor learning, and in particular in the learning of complex movement skills, has been discussed mainly with a focus on verbal feedback (e.g., Magill, 1993) and attentional focus (e.g., Wulf and Prinz, 2001; Al-Abood et al., 2002). In addition to action execution, simulation and observation, the use of language (e.g., verbal instructions, verbal cues for imagery, explanation of complex moves, verbal feedback and error correction) can facilitate or enhance motor learning by guiding attention toward relevant features of the movement and thus making them explicit (Landin, 1994; Wulf and Prinz, 2001), or by adding semantic content to support the creative adaptation or expressive quality. Examples of such verbal comments could be: "make sure that your arms are stretched out in direct opposition, let the hands pull away from each other to the sides until you feel the pull in your shoulder blades," or "don't just walk toward the front, but imagine you were approaching a long missed friend you have just spotted in the back of the audience," respectively. Verbal instruction or feedback, however, can also interfere with motor learning by putting too much pressure on certain aspects and distracting from the movement flow (Wulf and Weigelt, 1997). While evidence from practice clearly shows that verbal cue giving plays an important role in dance on many levels (Waterhouse et al., 2014), it seems that the full potential of the use of language in the context of learning complex motor actions has not been exploited by research in the field.

In most dance disciplines, a repertoire of movement elements is built up through training that can then be combined into increasingly long and complex combinations. In modern and contemporary dance, especially on an advanced level, the creation of choreography goes far beyond the aligning of predefined steps and moves elements—in this case, learning of a new dance phrase would only mean to learn by heart the new sequence in which the familiar elements have to be concatenated; this, obviously, is not the case. Even in classical ballet, in which an extensive canon of more than 430 clearly defined movement elements is trained systematically (Puttke, 2018), learning new dance sequences on a higher skill level is far more than just lining up these elements, or "moving from pose to pose"—it is rather the acquisition and practice of a holistic movement "gestalt" that is characterized by its special dynamics, spatial and temporal parameters, expression and semantic and emotional content, as much as the body postures involved. In fact, advanced dancers are often less concerned with the poses or postures than with the transitions between them, making the flow of the movement progress continuously even at node points that look like breaks or goal postures to the observer. This "flow of energy" with its spatial, temporal dynamic and expressive features is what has to be learnt together with the movement "as such" when learning a dance phrase. In particularly in modern and contemporary dance, the composition of choreographed movement is less strictly bound to a predefined movement repertoire, but choreographers and dancers strive to explore and create novel ways of expressing themselves through the body. Modern dance of course has its own movement repertoire that it builds upon, but the emphasis is particularly strong on the flow of energy that characterizes the movement, making it novel, expressive, and special. Many choreographers therefore do not expect the dancers to simply reproduce movement phrases in adequate form, but to develop movement material on their own, in accordance with a given idea, description or instruction, aiming at a personal expression and special artistic quality of the developed movement material. In contemporary dance training, in particular, dancers are expected not only to reproduce movement material, but also to shape and develop movement material on their own, to achieve a more personal expression and higher artistic quality. Dance pedagogues educating future professional dancers emphasize the importance of their students achieving skills that enable them to interpret it with their own artistic quality, thus claiming ownership of the novel movement material rather than reproducing it. An assumption that had evolved in the context of dance training practice at the Palucca Hochschule für Tanz Dresden was that even though observational learning of dance movement had proved to lead to the best results in terms of correct movement reproduction, the presence of the visual model would at the same time interfere with the dancers development of movement ownership that would become visible in the expression of the performed movement as creative transformation of the learned movement.

To elucidate the roles of different modes of learning in dance in this context, we conducted a study with a group of second year dance students in which we compared the respective benefits of observational learning from a human model and learning from verbal movement description. Learning success was evaluated in terms of movement recall, performance of the learned material and personal preference. The dance students learned two comparable movement phrases via two different modes that were applied exclusively and in real-time: observation of a human model displayed in a video clip, and listening to a spoken movement description presented as audio recording. In a second step, the complementary mode of presentation was added. Between and after both learning steps, the students' physical performance of the learned material was recorded on video. A third video recording was produced during a retention test applied 10–14 days after the learning session to evaluate long-term effects. We expected that the dance students would reproduce the learned material more precisely after observational learning (hypothesis 1), but that their individual interpretation and personal liking for the learned material might be better for the material first learned from verbal description, due to a stronger embodiment and identification with the movement (hypothesis 2). The latter was hypothesized on the basis of evidence from practice gained by the teachers, who argued that due to the superiority of vision over the other senses, the students' perception of the movement would be drawn away from their own and toward the demonstrator's motor system, which would result in more precise reproduction but less personal adaption of the movement material. In contrast, when learning from verbal description, the students would have to rely more strongly on their own motor system to re-create the movement, thus giving it a more personal note and experiencing a stronger feeling of engagement. Furthermore, we expected that performance after the second learning step would be improved compared to performance after the first learning step in both conditions, due to beneficial effects of a combination of different modes of presentation and an increased amount of practice (hypothesis 3).

# METHODS

### Participants

Eighteen students (age: 18.39 ± 1.04 years, range 17–20 years; one ambidextrous; 11 female) from the BA Dance study program (early second year) at the Palucca Hochschule für Tanz Dresden took part in the study without compensation or course credit. According to the Edinburgh Handedness Inventory applied before the experiment, 17 participants were righthanded and one was ambidextrous. The group of participants included native speakers of German (9), Portuguese (5), English (2), French (2), and Italian (2); two participants were bilingual. English was the commonly used working language in class, and all participants who were not native speakers of English indicated that their knowledge of English was at least good. Five of the participants engaged in recreational sports activities other than dance-related, including swimming, basketball, volleyball, soccer, and table tennis. All participants reported having normal or corrected-to-normal vision, and were naive with regard to the purpose of the experiment. This study was carried out in accordance with the recommendations of the ethics committee of Bielefeld University. A prospective ethics approval was not required in agreement with the institutional institution's guidelines and national regulations. All subjects gave written informed consent in accordance with the Declaration of Helsinki.

#### Material

Two dance phrases were created as material for the learning task by two dance pedagogues teaching at the Palucca Hochschule für Tanz Dresden (co-authors Jenny Coogan and José Biondi). The dance phrases were choreographed in such a way that they were similar in length and complexity, each including a range of defined elements (such as turns, jumps, walks, changes of direction and height level). Both phrases were recorded on video in a dance studio at the Palucca Hochschule für Tanz Dresden, both danced by dancer and former Palucca student Robin Jung (**Figure 1**). In the study, the dance phrases were presented as video clips of 26 s each. For each of the two phrases, a verbal description was created by the choreographers that described full-body movements and movements of body parts in detail using every-day language (no particular dance-specific terms), including spatial and temporal cues. The following text represents an example from the movement description of Phrase 2: "Stand facing the left front diagonal of the room in parallel position. Feel the wind from the back that shifts your weight forward; let your upper body respond. Allow your body to move back and take the impulse again to move forward, allowing your weight to transfer from your heels to your toes. Once again shift back, this time falling onto your left leg, and follow with another step back, long and grounded, ending in a low lunge position, torso diagonal. Staying low, kick your right leg forward and your arms outwards to the sides as you twist your torso in opposition to the kick. Quickly bend your leg and arms into your center with a half turn to the right. Let the weight of your arms and center sink down on your left leg as your torso melts in a sidebend to the left and your leg extends sideways in opposition. Shift your weight onto the extended leg while your left arm describes a horizontal surface in front of you, reaching your torso over to the right side and bringing your left foot to the knee."

The verbal descriptions were recorded as spoken audio files at the Palucca Hochschule für Tanz Dresden, the speaker was Alex Simkins. Durations of the audio files were 149 and 156 s for Phrases 1 and 2, respectively. Video clips and verbal movement descriptions as well as audio files of the recorded verbal descriptions are available as **Supplementary Material**.

#### Procedure

The experimental learning sessions were carried out with all participants at the biomechanics lab at CITEC, Bielefeld University; during four consecutive days. Each participant was tested individually; the experimental session with each participant lasted ∼1 h. After finishing the experimental session in the lab, the participant was asked to fill out a postexperimental questionnaire and was verbally interviewed. Before the experimental learning sessions started, each participant was assigned to one of four experimental groups (see **Table 1**: experimental design), with attention toward a balance with regards to gender and language background.

At the beginning of the individual experimental learning session, the participant entered the biomechanics lab and was introduced to the experimenters and the technical set-up. Both dance pedagogues were also present during the entire session. The participant was then equipped with 42 retro-reflective markers, as parts of the experimental session were recorded with a Vicon motion capture system (12 infra-red camera), in addition to the two video cameras positioned at different locations in the lab (in this article, only video-based results are presented; results of the motion-capture will be presented separately). Before the experimental session started, the participant was asked to fill out the necessary forms (e.g., consent) as well as a pre-experimental learning type questionnaire. The participant was then asked to enter the recording space in the middle of the lab and instructed how to use the space to allow for optimal visibility for the Vicon system and the video cameras. Subsequently, the participant was verbally given the learning task instructions (depending on the experimental design, see **Table 1**) by the main experimenter. The participant was instructed to learn two dance phrases of similar length and complexity, each through a combination of visual observation and verbal movement description. Depending on the group the participant had been assigned to, the individual participant's session either started with verbal or visual learning

FIGURE 1 | Images illustrating parts of the two dance phrases used in this study. Top panel: Phrase 1, choreographed by Jenny Coogan; the individual images correspond to movement elements 1–8 and 11 (end pose), as described in the Methods (completeness scores). Bottom panel: Phrase 2, choreographed by José Biondi; the individual images correspond to movement elements 1–6 and 8–10. Video clips of both dance phrases are provided as Supplementary Material. Dancer: Robin Jung (2013). Written informed consent was obtained from Robin Jung for the publication of this image in this article and the corresponding video clips in the Supplementary Material.


#### TABLE 1 | Design of the experimental learning task.

Dark gray, visual-first condition; Light gray, verbal-first condition.

of either Phrase 1 or Phrase 2 (see **Table 1**). In the following, the learning of a particular dance phrase starting with visual learning (observation of the video clip) will be referred to as Visualfirst condition, and the learning task starting with learning from verbal description (listening to the audio clip) will be referred to as Verbal-first condition.

For better comprehensibility, the procedure is described here for a participant assigned to group 1A, as example; this participant learned Phrase 1 in the Verbal-first condition and Phrase 2 in the Visual-first condition. This means that the participant first learned Phrase 1 from verbal description only (Step 1) with visual information added subsequently (Step 2). Then, the participant learned Phrase 2 from visual observation only (Step 1) with verbal information added subsequently (Step 2).

In the first learning step, an audio clip of the spoken verbal description of Phrase 1 was played five times consecutively. The participant was instructed to learn the movement sequence and was allowed to move or mark the movement as required while listening in order to support the learning process. After the fifth time listening to the audio file, the participant was allowed to try out the learnt movement phrase once; then s/he was recorded performing the learnt phrase. Up to three trials of the phrase were recorded and the best performance was marked, according to the participant's decision and preference. In the second learning step, the same dance phrase was presented as video clip twice, and the participant watched while being allowed to move or mark ad libitum. After the second watching, the participant was again allowed to try out and then was recorded again up to three times, as before. Two repetitions were chosen at this stage because no learning of new movement material was involved; instead, the novel information was supposed to be used only to compare and update the previously learned and practiced material. After a short break, the participant was presented Phrase 2 five times consecutively as video clip (Step 1), and the participant watched while moving as required. After trying out, the participant was recorded up to three times performing the new phrase. Then the verbal description of Phrase 2 was played twice while the participant listened and moved ad libitum (Step 2), and the phrase was recorded again, as before. After the markers were removed from the participant's body, s/he was debriefed and led to another room to fill out the post-experimental questionnaire and, finally, was interviewed about his or her personal impression of the experiment (the interviews were recorded for teachingrelated purposes and are therefore not considered here).

#### Retention Test

Ten days after the last day of learning sessions (i.e., 10–13 days after the individual participants' learning sessions), a retention test was carried out at the Palucca Hochschule für Tanz Dresden. The retention test took ∼10 min for each participant; all 13 participants were tested consecutively on the same day. Five out of the 18 participants were not present on that day due to illness or injury and therefore did not participate in the retention test. The remaining thirteen dance students (8 women) were called individually from the on-going training session to a free dance studio and were asked to dance the two dance phrases from the experimental learning session as completely and perfectly as possible. Each participant was allowed to practice for several minutes and then was instructed to dance the phrases in arbitrary order while being recorded with a video camera. The participants were not given any verbal or other assistance in reproducing the phrases, and they had not been informed in any way that there would be a retention test or that the movement material learned during the experimental session would be needed later on. After the recording, the participant filled out a post-retention questionnaire and was taken back to the dance class after being instructed not to convey any information to the other participants.

# Evaluation and Analysis of the Video Material

From each participant, six video clips were used for the evaluation, three for each dance phrase. Each video clip was representative for the student's performance of the phrase at a particular recording time: after learning Step 1 (one modality), after learning Step 2 (two modalities), and after the retention test. If two or three trials had been recorded at a given recording time, the trial that the participant had indicated as the preferred one was chosen. For the evaluation, the six selected video clips of each participant were named assigned names that included the phrase and an abstract code (e.g., P1\_xyz) that did not give away the recording time (Step 1, Step 2, Retention) or the learning condition (Visual, Verbal). In total, 98 video clips (4 × 18 = 72 from the experimental learning session, 2 × 13 = 26 from the retention test) were used for the evaluation.

# Completeness Scores

All video clips were annotated for their completeness by two independent annotators, both advanced students of sport science who were experienced with analyzing human motor action from video material, who had learned each movement phrase in detail from both the video and the verbal description. As basis for the evaluation of completeness, each dance phrase was segmented into eleven sub-phrases or elements (note that the phrases had been choreographed to resemble each other in complexity, duration and structure). Both annotators independently watched all 98 video clips in randomized order and rated for each of the eleven movement elements if it was present and correctly executed (1.0 points), missing (0.0 points), or in between (e.g., half present/correct: 0.5 points; almost correct: 0.8). Each annotator produced a score between 0.0 and 11.0 for each video clip. This way, a minimum score of 0.0 would indicate that the phrase was not danced at all, or was not at all recognizable, whereas a maximum score of 11.0 would indicate that the phrase was performed in completeness and without error. The ratings of the two annotators were then averaged.

# Expert Ratings

Two professional dance pedagogues teaching on the level of professional dance education comparable to the level of the participants who were not involved in the study otherwise and were naive toward the conditions and instructions and the experimental design were assigned as independent experts. Both experts to learned each movement phrase in detail from both the video and the verbal description and then independently watched and rated the 72 video clips from the experimental learning session in randomized order using a standard score sheet that contained a pre-defined list of criteria. The score sheet contained 15 questions (14 six-point Likert-type questions) assigned to two main categories, Approximation of the Model (AMo) and Individual Interpretation (IndI). Nine questions, or criteria, were addressed explicitly with regards to the approximation of the model phrase (AMo), namely:

A1.1 Clarity of movement initiation and pathway through the body

A1.2 Spatial orientation in external space (allocentric)

A1.3 Spatial orientation in external body-related space (egocentric)

A1.4 Temporal differentiation (proportion of the parts of the sequence in relation to one another)

A1.5 Connectedness, fluency of the movement

A1.6 Performance, over-all in relation to the model

A2.1 How much does Part 1 of the Phrase resemble the model?

A2.2 How much does Part 2 of the Phrase resemble the model?

A2.3 How much does Part 3 of the Phrase resemble the model?

The three parts addressed in questions A2.1-3 corresponded to elements 1–5, 6–7, and 8–11 (Phrase 1) and elements 1–4, 5–9, and 10–11 (Phrase 2) used as basis for the completeness scores, respectively.

Six criteria were addressed with regards to the Individual Interpretation (IndI), independent of the model phrase:

B1.1 Clarity of movement initiation and pathway through the body

B1.2 Spatial orientation in external body-related space (egocentric)

B1.3 Phrasing, temporal differentiation

B1.4 Connectedness, fluency of the movement

B1.5 Performance quality

B2 Did the phrase include the following elements?

Pause / suspension / successional movement / simultaneous movement

For each clip, the ratings of the two experts were averaged. Additionally, ratings for questions A1.1-6 (A1), for questions A2.1-3 (A2) and for questions B1.1-5 (B1) were averaged to achieve over-all ratings for each category.

#### Questionnaires

Before the experiment, participants filled out a questionnaire to determine their learning type (e.g., Kirby et al., 1988). The questionnaire was shaped to mainly differentiate visual from verbal learners, it was based on the more extended Index of Learning Styles Questionnaire by Litzinger et al. (2007). Eight out of the 16 questions focused on this difference, the other eight questions were mainly added to make this purpose less obvious for the participants. Questions were phrased in the following way (example): "When I think about what I did yesterday, I am most likely to get: (a) words (b) a picture," with one option always referring to the category "verbal" and the other one to the category "visual." For the eight relevant questions, one point was added for the category the participant had chosen; if the participant had marked both answers, each category scored 0.5 points. Each participant was assigned to the category in which s/he had scored two or more points more than in the other category; if the difference was smaller than two points, the participant was defined as mixed-type learner.

After the experimental procedure and after the retention test, participants filled out questionnaires evaluating their impressions of the task and of their own performance. The post-experimental questionnaire included learning task specific questions for each condition (e.g., how confident did you feel when dancing the sequence after learning it from observation/from verbal instruction? How clear did you find the video demonstration/verbal instruction? How clear did you find the additional verbal/visual information?) and general questions (e.g., how competent do you consider yourself at learning from observation/from verbal instruction? How much do you enjoy learning from observation/from verbal instruction?). The retention questionnaire included only learning task specific questions for each condition (e.g., how difficult did you find this dance phrase? How difficult did you find it to remember the sequence? How much did you like this dance phrase? How confident did you feel when dancing the sequence?).

#### RESULTS

Completeness scores given by the two annotators were highly correlated (Steps 1 and 2: r = 0.903, p < 0.001; Retention: r = 0.957, p < 0.001), therefore completeness scores of the two annotators were averaged for the further analysis. After confirming normal distribution of the data (Shapiro-Wilk test), a 2 × 3 ANOVA with factors CONDITION (Verbal-first, Visualfirst) and TIME (Step 1, Step 2, Retention) revealed main effects of CONDITION [F(1,12) = 9.286; p = 0.010] and TIME [F(1.16,13.97) = 11.702; p = 0.003], but no interaction. A 2 × 3 ANOVA with factors PHRASE (Phrase 1, Phrase 2) and TIME (Step 1, Step 2, Retention) revealed a main effect of TIME [F(1.16,13.97) = 11.702; p = 0.003], but no effect of PHRASE and no interaction. Violation of the sphericity assumption resulted in a correction of the p-values and degrees of freedom according to Greenhouse-Geisser. As post-hoc comparison, paired T-tests were used to compare the averaged completeness scores between learning conditions (Visual-first, Verbal-first), learning steps (Step1, Step 2), and dance phrases (Phrase 1, Phrase 2). After the first learning step, completeness scores were better for the phrase learned in the Visual-first condition (mean completeness score: 8.10) than for the phrase learned in the Verbal-first condition [mean completeness score: 6.36; t(17) = 2.905, p = 0.010], whereas no difference between the conditions was found after the second learning step [Visual-first: 9.29; Verbal-first: 8.27; t(17) = 2.010, p = 0.061]. After the retention, completeness scores were again better for the phrase learned in the Visual-first condition (mean completeness score: 8.30) than for the phrase learned in the Verbal-first condition [mean completeness score: 6.19; t(12) = 2.526, p = 0.027]. Within both learning conditions, completeness increased from Step 1 to Step 2 [Visual-first: t(17) = −3.591, p = 0.002; Verbal-first: t(17) = −5.191, p < 0.001]. Only in the Verbalfirst condition, completeness dropped significantly from Step 2 to the retention [t(12) = 6.832, p < 0.001]. Results for completeness scores are displayed in **Figure 2**.

Comparison of the completeness scores for the eleven individual elements of both phrases revealed higher scores for the first 3 and 2 elements in the Verbal-first condition and the Visual-first condition, respectively (see **Figure 3**; **Table A**, **Supplementary Material**), which points toward a primacy effect

FIGURE 2 | Completeness scores for video recordings of participants' performance of the two dance phrases learned in the experimental task at three occasions. Light gray: Visual-first condition; dark gray: Verbal-first condition. Step 1: participant's performance recorded after learning from either visual (Visual-first condition) or verbal (Verbal first condition) information (five repetitions). Step 2: participant's performance recorded after receiving the complementary (verbal or visual) information (two repetitions). Retention: participant's performance 10–13 days after the experimental learning session (unprepared test). Numbers on the y-axis refer to scores given by the two annotators (averaged) for the performance of 11 elements, or sub-phrases, of the dance phrases; for each element, each annotator could give a score between 0.0 (element missing) and 1.0 (element performed completely and without error), resulting in a maximum score of 11.0 for the entire phrase. Asterisks refer to significance levels of comparison of means: \*p ≤ 0.05; \*\*p ≤ 0.01; \*\*\*p ≤ 0.001.

that was more pronounced in the Verbal-first than in the Visual-first condition.

#### Expert Ratings

Ratings of the two experts were positively correlated for both main categories (AMo: r = 0.528; p < 0.001; IndI: r = 0.513; p < 0.001). Non-parametric tests (Wilcoxon signed-rank tast) were used to compare the averaged ratings for each individual question and categorial ratings A1, A2 and B1 between the four conditions (Visual-first Step 1, Visual-first Step 2, Verbalfirst Step 1, Verbal-first Step 2). In the Verbal-first condition, all individual AMo ratings (questions A1.1-6 and A2.1.3) were higher after Step 2 than after Step 1 (see **Table 2**; **Table B**, **Supplementary Material**). In the Visual-first condition, no differences between the Step 1 and Step 2 were found. After Step 1 and after Step 2, AMo ratings for the Visual-first condition were generally better than for the Verbal-first condition (exception: Step 2 A1.6; tendencies for A1.1). Comparison of the ratings for the first, middle and last part of each dance phrase (questions A2.1-3) revealed that AMo ratings for the first part were better than for the middle part in all conditions (Visual-first Step 1: p = 0.007; Visual-first Step 2: p = 0.001; Verbal-first Step 1: p < 0.001; Verbal-first Step 2: p < 0.001) and better than the last part in three

between 0.0 (element missing) and 1.0 (element performed completely and without error).

out of the four conditions (Visual-first Step 2: p = 0.011; Verbalfirst Step 1: p = 0.001; Verbal-first Step 2: p = 0.006), whereas no difference was found between the ratings for the middle and last part (see **Figure 4**). Averaged AMo ratings for A1 and A2 are displayed in **Figure 4**.

IndI ratings did not differ between learning steps in either condition. After Step 1, IndI ratings for the Visual-first condition were better than for the Verbal-first condition for questions B1.2- 4, but not for the average B1 rating. After Step 2, only tendencies were found for questions B1.2 and B2. Medians for all categories and results of all tests are displayed in **Table 2**. Results of the averaged IndI ratings (B1) are displayed in **Figure 4**.

## Questionnaires

Post-experimental and post-retention questionnaires were analyzed comparing participants' mean responses for the two learning conditions (Wilcoxon signed-rank) and correlating responses to each other and to completeness scores and expert ratings (Spearman's rho). According to the post-experimental questionnaires, the students perceived the information provided by the video clip as clearer than the information provided by the audio text, both in Step 1 (Z = −3.002, p = 0.003) and in Step 2 (Z = −2.547, p = 0.011). In the Visual-first condition, feeling confident dancing after Step 1 was negatively correlated to finding the additional verbal information in Step 2 clear (r = −0.556, p = 0.018) and useful (r = −0.577, p = 0.012), and finding the additional information ins Step 2 useful was positively correlated to finding it useful (r = 0.762, p < 0.001) and feeling confident dancing afterwards in Step 2 (r = 0.494, p = 0.037). In the Verbal-first condition, feeling confident dancing after Step 1 was positively correlated to finding the verbal instruction clear (r = 0.650, p = 0.003) and feeling confident dancing after Step 2 (r = 0.499, p = 0.035). In general (i.e., independent of the experimental task) participants enjoyed learning from observation better than learning from verbal instruction (Z = −2.084, p = 0.037), they considered learning from observation as more useful than learning from verbal instruction in dance (Z = −3.028, p = 0.002) and they were more familiar with learning dance movement from observation than from verbal instruction (Z = −3.458, p = 0.001). Enjoying learning was positively correlated to feeling competent for both learning modes (verbal: r = 0.737, p < 0.001; visual: r = 0.623, p = 0.006). Being familiar with learning from verbal instruction was positively correlated to feeling competent for it (r = 0.725, p = 0.001), enjoying it (r = 0.638, p = 0.004) and finding it useful (r = 0.464, p = 0.052).

The retention questionnaires revealed that in both conditions (marginal for the Verbal-first condition), liking a phrase was negatively correlated to finding it difficult to remember (Visualfirst: r = −0.746, p = 0.003; Verbal-first: r = −0.550, p = 0.051). In the Verbal-first condition, liking a phrase was negatively correlated to finding it difficult to dance (r = −0.599, p = 0.030), and feeling confident dancing a phrase was negatively correlated to finding it difficult to remember (r = −0.871, p < 0.001) and difficult to dance (r = −0.612, p = 0.026). For both conditions, finding it difficult to remember a phrase was negatively correlated to the retention test completeness scores (Visual-first: r = −0.557, p = 0.029; Verbal-first: r = −0.621, p = 0.024). In the Verbal-first condition, feeling confident dancing a phrase was positively correlated to the retention test completeness scores (r = 0.628, p = 0.028).

## Learning Type Questionnaires

According to the learning type questionnaire, seven participants (four females) were assigned to the visual learners and six (two females) were assigned to the verbal learners; the remaining five participants were mixed-type learners. Completeness scores, expert ratings and questionnaire results of visual (N = 7) and verbal (N = 6) learners were compared, however, no significant differences between the learning-type groups were found.

#### TABLE 2 | Medians of expert ratings.


AMo, approximation of the model; IndI, individual interpretation; cells with categorical ratings A1 (A1.1-6), A2 (A2.1-3), and B1 (B1.1-5) are marked in gray.

## DISCUSSION

In a study with 18 second year dance students, the participants' performance of two dance phrases learned under different conditions was compared. Each participant learned one phrase initially via listening to a verbal movement description (Verbalfirst condition) and the other one via observation of a human model in a video clip (Visual-first condition). In a second learning step, the complementary modality of information was presented. In a retention test ∼10 days after learning, students were asked unexpectedly to recall and perform both learned dance phrases. Completeness of the dance phrases performed by the participants was evaluated on the basis of video clips recorded at three points in time, as measures of learning success in terms of recall. The three recordings were produced after the first and the second learning step of the experimental learning task and at the retention test. Additionally, expert ratings for two main criteria, approximation of the model phrase (AMo) and individual interpretation (IndI), were used to evaluate the quality of the performance after the first and the second learning step from a dance-pedagogical perspective. After the experimental learning task and after the retention test, questionnaires were applied to evaluate the participants' personal impressions.

Completeness scores showed that recall was generally better after the second learning step than after the first (i.e., after learning from both modalities compared to only one modality). This finding can be interpreted as supporting the view that information from different modalities is beneficial for the

FIGURE 4 | Expert ratings for participants' recorded performance according to the criteria Approximation of the model (AMo) and Individual Interpretation (IndI). Black: Verbal-first condition, Step 1; dark gray: Verbal-first condition, Step 2; white: Visual-first condition, Step 1; light gray: Visual-first condition, Step 2. Numbers on the y-axis refer to experts' ratings on a six-point Likert scale used to evaluate the performance of the dance phrases (note that for B2, 4 is the maximum value). Labels on the x-axis: A1 refers to AMo criteria A1.1-6: Clarity of movement initiation and pathway through the body; Spatial orientation in external space (allocentric); Spatial orientation in external body-related space (egocentric); Temporal differentiation (proportion of the parts of the sequence in relation to one another); Connectedness, fluency of the movement; Performance, over-all in relation to the model). A2 refers to AMo criteria A2.1-3: ("How much does Part 1/Part 2/Part 3 of the Phrase resemble the model?"). B1 refers to IndI criteria B1.1-5: Clarity of movement initiation and pathway through the body; Spatial orientation in external body-related space (egocentric); Phrasing, temporal differentiation; Connectedness, fluency of the movement; Performance quality. B2 refers to question B2: "Did the phrase include the following elements: pause/suspension/successional movement/simultaneous movement?" For each clip, the ratings of the two experts were averaged. Additionally, ratings for questions A1.1-6 (A1), for questions A2.1-3 (A2), and for questions B1.1-5 (B1) were averaged to achieve over-all ratings for each category. Asterisks refer to significance levels of comparison of means: \*p ≤ 0.05; \*\*p ≤ 0.01; \*\*\*p ≤ 0.001; dashed line: tendency (p = 0.052).

learning of a motor task. In real world dance learning situations, movement is hardly ever learnt through one modality alone, but from visual observation of movement typically demonstrated by the teacher, complemented by verbal cue-giving and instruction and supported by the dance student's own motor action. An explanation for the superiority of combined learning modes compared to learning through one modality alone is provided by the perspective that during motor learning and practice, information from all sensory modalities is integrated and merged into rich action representations that are perceived as consistent and meaningful (Zacks et al., 2007; Barsalou, 2008; Nomikou et al., 2016). Such representations in long-term memory are thought to comprise declarative and non-declarative information that is updated with every new access, and underlie the execution and imagery of the action (Land et al., 2013; Schack et al., 2014). This suggests that involving two or more modalities in the learning process might result in a richer representation that involves more complementary information and therefore leads to a better learning outcome. In support of this view, Rosenblum et al. (2017) propose that the architecture of the brain implies perceptual parity between the senses, in particular between audition and vision, and that cross-sensory integration occurs completely and early in the perceptual stream. In the current study, the second learning step in fact consisted of more than just presentation of the complementary mode of information. In addition to the additional observation or listening to the verbal instruction, the participants had already practiced and performed the movement phrase several times. The second step thereby contained more physical practice and performance in addition to the additional information. It can thereby not be concluded that the development from Step 1 to Step 2 was entirely due to richer information. Adding another condition with the same modality in Step 1 and Step 2 would have helped to clarify this issue.

After the first learning step and in the retention test, the phrase initially leaned from visual observation was reproduced more completely than the phrase initially learned from verbal instruction. Crucially, the phrase initially learned from verbal instruction was reproduced less completely in the retention test than after the second learning step, whereas no such difference was found for the phrase initially leaned from visual observation. These results clearly indicate that initial learning from observation (complemented later by verbal information) was more successful in terms of recall and reproduction of the learnt material than initial learning from verbal instruction (complemented later by visual information). The superiority of visual observation as initial source of information is in line with previous findings supporting the view that learning from observation is the major learning mode for motor actions and is most successful in terms of the time spent learning and accuracy of the outcome (e.g., Schmidt, 1975, 2003; Schmidt and Lee, 1998; Hodges et al., 2007). Observational learning is considered to be mediated through the activation of shared neural correlates of action execution, observation and simulation (Jeannerod, 1995, 2004) as well as through the involvement of visual pathways for action perception in working memory processes (Vicary and Stevens, 2014; Vicary et al., 2014). The finding that AMo ratings were better for the first part of each phrase than for the middle and last part corresponds to the primacy effect found in the completeness scores; in both cases, the effect was more prominent in the Verbal-first than in the Visual-first condition. These findings support previous results according to which primacy effects, but no recency effects were found with regards to the learning of action sequences (Allard and Starkes, 1991; Wachowicz et al., 2011).

Corroborating the results for the completeness scores, expert ratings for AMo were generally better for the Visual-first condition than for the Verbal-fist condition after both learning steps. In the Verbal-first condition, expert ratings for AMo were better after the second than after the first learning step, whereas no difference between learning steps was found in the Visual-first condition. According to the dance experts' evaluation, approximation of the model phrase was generally better after visual learning than after learning from verbal instruction. In contrast, expert rating for the participants' individual interpretation of the learned movement phrases did not improve from learning Step 1 to Step 2 in either condition, and only for a subset of the questions, Visual-first ratings were slightly better than Verbal-first ratings after Step 1. The individual movement interpretation was obviously less sensitive to the learning mode than model approximation and not depending on the availability of complementary information, showing that participants' ability to dance and interpret the phrases did not depend on the information they had received, and that it therefore was more than plain reproduction of the movement, but rather a personal creative process.

In contrast to our practice-based expectations, expert ratings for individual interpretation were not better for the Verbalfirst condition than for the Visual-first condition. Additionally, according to the questionnaires, students did not show a general preference or stronger feeling of ownership for the verbally learned compared to the visually learned movement, but, in contrast, expressed a general preference (more enjoyable, more useful) and higher familiarity for learning dance movement from observation. With regards to the experimental task, participants perceived the visual information as clearer than the verbal information, and they liked the phrase learned in the Visual-first condition better and felt more confident dancing it. Generally, liking a phrase was linked to finding it easy to remember and to dance, and feeling confident was linked to recall performance. Together with the finding that students in general preferred dance learning from observation and found it more familiar, and the lack of effects of learning type (according to the learning type questionnaire), these results suggest that learning preference and performance might strongly depend on habit, or being used to learning dance movement in a specific way. Taking the dance pedagogue's observations into account according to which learning dance movement in absence of a visual model might lead to a stronger identification and better interpretation of the movement, it might be the case that the stressful situation of the experimental learning task (an unfamiliar lab setting, restricted time, the teachers and other people watching) might have worked against the less familiar and therefore potentially more cognitively demanding way of leaning dance movement without a visual model. Indeed, most students expressed after the experimental task that they considered the situation as slightly stressful and perceived some kind of stage fright, in particular because their teachers were watching them during the learning task. It can be speculated that in a more relaxed situation with more time and less pressure, learning dance movement from verbal instruction might have been a more rewarding experience for the students. In dance training, experimenting with learning and teaching modes and approaches to movement learning, including the variation of available sensory information, can be a promising means to break habits, broaden the students' spectrum of experiences and induce creative processes. Another aspect that might play a role here is the type of verbal material used. In the present study, we used verbal instructions that mainly described the movement (e.g., "Extend your left leg forward and your two arms sideways to the horizontal. Allow your right hand to continue moving until it arrives to a high diagonal."), and only very few instances of metaphorical language or images ("Feel the wind from the back that shifts your weight forward; let your

upper body respond."). There is some evidence that metaphorical language might work better in dance-related contexts than pure movement description (e.g., Sawada et al., 2002). In this study, no clear distinction was made between descriptive and metaphorical language, however, it might be promising to compare the use of both types of language in a similar learning scenario.

Taken together, the results of the study support our first hypothesis, as well as findings from previous research on observational learning of movement, showing clearly that initial observation of a human model is superior with respect to the recall and reproduction of the movement compared to initial learning movement from verbal instruction or movement description. It has to be pointed out that due to the design of this study we have not tested pure learning from observation to pure learning from acoustically presented verbal instruction, but two different approaches to learning a dance phrase based on either visual observation or verbal instruction as initial mode, later on followed by complementary verbal or visual information, respectively. Therefore, conclusions about learning from on or the other source of information exclusively can only be drawn with regards to the completeness scores and experts' ratings after Step 1, but not after Step 2 or the retention test. Accordingly, with regards to the retention test, we cannot draw any conclusion about the comparison of learning from visual vs. verbal information exclusively, but we compare the two modes as initial source of information.

In contrast to our expectations that were based on experience from dance training, initial learning from verbal instruction in absence of a visual model (that was presented only in the later learning step) did neither result in better ratings for the individual interpretation in movement performance, nor in a stronger personal preference for, or identification with, the learnt material, leading to the rejection of our second hypothesis. The third hypothesis, namely that performance after the second learning step would be improved due to the combination of different modes of presentation and an increased amount of practice, was supported by the completeness scores for both conditions, whereas the expert ratings of approximation of the model, but not for individual interpretation supported this hypothesis for the material initially learned through verbal instruction. These findings again support the superiority of visual observation as initial learning mode with regard to the approximation of the model phrase.

To round up the discussion of the presented results, it has to be admitted that the findings are potentially limited by the rather small number of 18 participants, as is often the case with participants who are experts in particular fields of practice, such as dance or sports (this issue was increased further by the unforeseen and unfortunate reduction to N = 13 at the time of the retention test). Due to this limitation, a withingroups design was chosen two comparable dance phrases, with each participant learning one phrase in one condition and the other one in the other condition. A between-groups design (with only one to-be-learnt dance phrase) would have resulted in a cleaner design, but would have required a larger group of participants. On the other hand, a clear (and rare) advantage of the available group of participants was that as they were all studying dance together in the same class, with the same teachers, and therefore were as similar in their dance experience as it might be possible in the real world. Another way of keeping the design simpler and thus easier to interpret would have been to leave out the second learning step and to concentrate on the effects of purely observational vs. verbal learning. This would have made the retention test easier to interpret and potentially more meaningful (in particular with the full number of participants). Movement learning in dance is hardly ever based on one learning mode alone, but is commonly supported by a combination of visual observation, verbal instructions and feedback, and kinesthetic information achieved through physical movement, typically applied at the same time. In the present study, our aim was to disentangle visual and verbal information about a to-be-learnt dance phrase without removing too much of the "normal" dance learning context. A simpler design that separates visual and verbal learning throughout the study and controls more rigorously for the participants' actions during the learning task would have made the experimental conditions clearer for interpretation, in particular at the time of retention.

# CONCLUSIONS

In general, our results corroborate the superiority of visual observation of a human model as means for learning complex movement sequences in dance. More precisely, visual observation as initial learning mode (followed by verbal instruction) was found to result in better outcomes than verbal instruction as initial learning mode (followed by observation of a visual model). Given the findings of this study and arguments brought forward in the literature in support observational learning (e.g., Vicary and Stevens, 2014; Vicary et al., 2014), it might seem surprising that the participants were well able to learn the complex movement sequences exclusively from verbal information. Even though the participants of this study were students still at the beginning of their professional dance education, they clearly demonstrated remarkable abilities in movement learning that can be considered a specific feature of dance expertise. Dancers' specific learning and memory skills have been addressed in several studies (e.g., Stevens and McKechnie, 2005; Bläsing et al., 2009; Wachowicz et al., 2011; Bläsing and Schack, 2012), and growing evidence exists that dance experts differ from non-dancers not only with regards to quantitative aspects, such as working memory capacity, but also qualitative ones, including specific modes and strategies of storage and retrieval, as well as the interaction between memory processing and motor performance (see Sevdalis and Keller, 2011; Bläsing et al., 2012; Stevens, 2017). Furthermore, dancers' enhanced skills in motor imagery have been found to contribute substantially to their learning and performance skills by increasing the efficiency of kinaesthetic sensations and making images more complex and vivid (e.g., Nordin and Cumming, 2007; Golomer et al., 2008; Fink et al., 2009), and might thus have supported the students' performance in learning movement from verbal instruction in the present study. Additionally, dancers acquire specific strategies and techniques to support movement learning and recall, such as marking dance movements by hand gestures or reduced full-body movements (Kirsh, 2011; Warburton et al., 2013). Marking can be considered a cognitive tool that makes use of the same cognitive functions as executing, observing and mentally simulating motor actions (Jeannerod, 1995, 2004) and that can thereby serve as a kind of (partly) externalized memory (Allard and Starkes, 1991; Stevens, 2017) or as cognitive offline-strategy in which the body is used to reduce cognitive load (Wilson, 2002). With regards to the current study, it can be reported as qualitative observation that students used a lot of different marking while watching or listening during the experimental learning tasks. Analysing the individual learning strategies (including marking techniques) applied by the participants of this study could be a promising next step to increase or understanding of movement learning in dance.

## AUTHOR CONTRIBUTIONS

BB, JC, JB, and TS have planned the study together. BB, JC, and JB have conducted the study and collected the data. BB has analyzed the data and written the manuscript. TS have provided input to the manuscript and feedback to several previous versions.

#### REFERENCES


#### ACKNOWLEDGMENTS

We thank the Dance engaging Science Network, in particular Scott deLahunta for initializing this project, and Liane Simmel for her valuable contributions and feedback during the planning phase of this study. We are very grateful to Manuela Poß for her support during the experiment and her excellent work on the tamed science blog, to two external dance pedagogues for their effort producing the expert ratings, and to Dima Volchenkov, Meike Allerdissen and Robert Stojan for their intensive help with the data collection, data processing and video annotation. Thanks to Robin Jung for being our video demo dancer and Alex Simkins for lending his voice for the movement instruction. We thank our participants for their excellent work and patience and the Palucca Hochschule für Tanz Dresden for their interest and support. This work was supported by the Volkswagen Foundation and by the Cluster of Excellence Cognitive Interaction Technology CITEC (EXC 277) at Bielefeld University, which is funded by the German research foundation (DFG).

#### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg. 2018.02371/full#supplementary-material


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Bläsing, Coogan, Biondi and Schack. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# The Limitations of Being a Copycat: Learning Golf Putting Through Auditory and Visual Guidance

Marta M. N. Bienkiewicz ´ 1 \*, Lionel Bringoux <sup>1</sup> , Franck Buloup<sup>1</sup> , Matthew Rodger <sup>2</sup> , Cathy Craig3,4 and Christophe Bourdin<sup>1</sup>

<sup>1</sup> Aix-Marseille Université, CNRS, ISM, UMR 7287, Marseille, France, <sup>2</sup> School of Psychology, Queen's University of Belfast, Belfast, United Kingdom, <sup>3</sup> INCISIV Ltd, Belfast, United Kingdom, <sup>4</sup> School of Psychology at Ulster University, Coleraine, United Kingdom

The goal of this study was to investigate whether sensory cues carrying the kinematic template of expert performance (produced by mapping movement to a sound or visual cue) displayed prior to and during movement execution can enhance motor learning of a new skill (golf putting) in a group of novices. We conducted a motor learning study on a sample of 30 participants who were divided into three groups: a control, an auditory guide and visual guide group. The learning phase comprised of two sessions per week over a period of 4 weeks, giving rise to eight sessions. In each session participants made 20 shots to three different putting distances. All participants had their measurements taken at separate sessions without any guidance: baseline, transfer (different distances) and retention 2 weeks later. Results revealed a subtle improvement in goal attainment and a decrease in kinematic variability in the sensory groups (auditory and visual) compared to the control group. The comparable changes in performance between the visual and auditory guide groups, particularly during training, supports the idea that temporal patterns relevant to motor control can be perceived similarly through either visual or auditory modalities. This opens up the use of auditory displays to inform motor learning in tasks or situations where visual attention is otherwise constrained or unsuitable. Further research into the most useful template actions to display to learners may thus still support effective auditory guidance in motor learning.

Keywords: auditory-visual perception, motor learning and control, movement guidance, golf putting, kinematic template

# HIGHLIGHTS


Edited by:

Mauro Murgia, University of Trieste, Italy

#### Reviewed by:

Nina Schaffert, Universität Hamburg, Germany Ambra Bisio, Università di Genova, Italy

> \*Correspondence: Marta M. N. Bienkiewicz ´ mbienia@gmail.com

#### Specialty section:

This article was submitted to Movement Science and Sport Psychology, a section of the journal Frontiers in Psychology

Received: 18 September 2018 Accepted: 14 January 2019 Published: 01 February 2019

#### Citation:

Bienkiewicz MMN, Bringoux L, ´ Buloup F, Rodger M, Craig C and Bourdin C (2019) The Limitations of Being a Copycat: Learning Golf Putting Through Auditory and Visual Guidance. Front. Psychol. 10:92. doi: 10.3389/fpsyg.2019.00092

# INTRODUCTION

Motor learning can be described as a lasting improvement in performance compared to a baseline measure that can be attributed to training (Shmuelof et al., 2012; Sigrist et al., 2013). Fitts and Posner (1967) described motor learning processes as passing through three stages: from the first stage of very attentive and effortful movement, to the second stage of fine tuning of the action to the final stage of automation, or at least partial automation, of the movement. When a skill is mastered we observe successful goal attainment, but also reductions in the variability of movement across repetitions and an increase in movement smoothness. Those mechanisms provide evidence for efficient feedback control mechanisms (Shmuelof et al., 2012), which allow the performer to fine-tune previously performed movements at the next opportunity (Yousif and Diedrichsen, 2012). For example, in a golf swing study by Lai Ab et al. (2011) skilled golf players demonstrated more consistent swing patterns in their pelvis movements than beginners. In this study we examined the effects of sensory guidance on motor learning in a golf putting task. We assessed levels of motor learning by measuring both putting success and reductions in variability, which may be independent of each other when refining putting technique (Richardson et al., 2018).

Contrary to the popular belief that a fixed number of hours are required to learn a new skill, research has shown that the speed with which people learn will depend on both practice effort and personal abilities (Hambrick et al., 2014). For example, learning how to play golf, like any other complex motor behavior, is effortful, prone to error and frustration, and requires external guidance to efficiently control the different kinematic parameters. Teachers and coaches use a variety of methods to facilitate learning. Verbal instruction is usually given along with a visual demonstration of the movement from another person (usually a coach). The coach will also offer further instruction on which specific aspects of the movement the player needs to focus on to improve performance. However, verbal instruction alone is not sufficient to improve performance in complex skills like golf putting. For instance, a posteriori verbal instruction seems inappropriate and too non-specific to guide the desired timing of learner's movements to create an "ideal" putt.

#### Describing the Golf Putting Action

The putting action can be broken down into four principal phases: backswing, downswing, impact, and follow through (see **Figure 1**). There are a few major factors that have been found to be linked to the consistency and repeatability of the golf putting swing: namely movement velocity, velocity through the swing motion path immediately surrounding impact and the temporal ratio of the backswing to the downswing (Burchfield and Venkatesan, 2010). The ideal ratio is considered to be 2:1 (backswing phase being twice as long as the downswing) (Grober, 2009; Kooyman et al., 2013) regardless of the target distance of the putt (Grober, 2011). Other non-golf related studies have demonstrated that the human motor system generates a spontaneous movement tempo to use the least force to generate motion (Bove et al., 2009; Avanzino et al., 2015; Bisio et al., 2015). The "ideal ratio" was found to lead to good control improving the accuracy and distance of the putt. The ratio also allowed the random errors caused by the magnitude of the applied forces to be minimized and the velocity of the club head at ball impact to be kept relatively constant<sup>1</sup> (Grober, 2009). Players can feel their natural tempo by swinging the club back and forth and are often observed doing it almost instinctively before hitting the ball. In a study by Kooyman et al. (2013), it was found that golfers who received visual feedback on their temporal ratio of their putting action over three different putting distances using a custommade GUI, improved their putting motion and decreased shot variability for both the experienced and inexperienced golfers.

Accurate golf putting requires that a golfer exhibits the finest degree of control of both the spatial and temporal parameters of the movement. The putter allows for the efficient transfer of energy generated by the movement dynamics of the golfer to the ball so that it travels the required distance. It is important to note, however, that this is the case only if the center of the putter face hits the ball. Golfers who showed high levels of putting ability were found to show reduced variability of the movement (Burchfield and Venkatesan, 2010). The seminal study by Craig et al. (2000) found a linear relationship between the putting distance and clubhead velocity at ball impact.

## Feedback and Motor Learning in Golf

Based on the features mentioned above, we chose a golf putting task as an example of a complex motor task. The aim of this study was to see if sound could be used to convey the dynamics of an expert's motion and help accelerate the learning of a putting task in a group of novices. There is a growing body of research that is examining the efficacy of sensory guidance and action observation to improve motor performance, which has relevance not only for sport, but also for the recovery of motor function (Krakauer, 2006). When using sensory guidance, the learner is presented with a template which provides information about how to perform an action. This approach differs from augmented feedback that is directly connected to the learner's own movement (see Sigrist et al., 2013 for a review).

In the context of providing sensory guidance to enhance motor learning in golf putting, it is mandatory to consider the specificity of the skill to be learned. In golf, instructors ask players to keep their eye on the ball whilst swinging the club. Such instructions make it difficult to use visual guidance to improve movement as following a visual guide would compromise the ability to focus their visual attention on the target that needs to be hit. In this study, we decided to examine the difference between the efficacy of auditory information compared to visual information as a way of helping novices improve their performance in a golf putting task.

An auditory signal can provide information about clubhead velocity and the temporal ratio of the backswing to the downswing (Murgia et al., 2012), allowing the golfer to visually

<sup>1</sup>This is why the resulting velocity remains "insensitive to the exact shape of the force profile, so long as the force remains rooted in the second harmonic of the resonance" (Grober, 2009, p.22). The force applied in the backswing phase should equal, in magnitude, to the force applied in the downswing, with the length of the backswing defining the speed of impact.

attend to the spatial aspects of the task (i.e., assessing the putt distance and keeping their eyes on the ball whilst swinging the club). We transformed movement data into auditory information, using a process called "movement sonification" (defined in broad terms as the mapping of movement data onto pre-defined sound parameters). Sound may not only be more effective for conveying temporal information than vision (Hirsh and Watson, 1996; Murgia et al., 2017), but also uses fewer attentional resources and is more portable (Secoli et al., 2011). A few studies have already demonstrated that sonification can be used to guide motion in simple tasks. Young et al. (2014) found that both healthy controls and Parkinson's disease patients

the backswing duration is more than double the duration of the downswing (in this example temporal ratio of 2.5:1).

are able to re-enact step lengths from recorded sounds of the footsteps of a neurologically intact person when walking on gravel. Both groups were able to adapt their gait irrespective of whether they heard actual sounds or recalled them from memory. This study provided evidence that sound is a powerful carrier of the kinematic features of movement, at least for this clinical population. Interestingly, the actual information that was relayed by the environmental sound (natural recordings) was reported to be a richer source of information than the synthesized sounds, possibly due to the fact they depicted the motor action in a more holistic (Gestalt) way (Koffka, 1999; Kennel et al., 2014). A similar effect was observed by Murgia et al. (2016) when they studied the natural recording of breathing sounds vs. engineered sounds conveying the same temporal structure. Improving motor behavior (learning) by employing auditory displays has also been reported in sports related contexts. For example, Agostini et al. (2004), designed an experiment where athletes were performing hammer throws while listening to the natural recording of their best throw the previous day. It led to increase in the throw length and a decrease in the throw variability. Schaffert and Mattes (2014) used augmented acoustic feedback from a boat's acceleration-time trace in a rowing experiment with high-performance squads. The presence of an auditory display enhanced mean boat speed when compared to the baseline performance of each squad and immediate retention effect was also present after the withdrawal of the feedback. In addition, athletes reported auditory feedback as beneficial in providing additional information to supplement the already available visual feedback relating to their performance. A study by Effenberg et al. (2016) demonstrated that four dimensional sonification of rowing movement parameters (grip force, sum of footrest forces, grip pull-out length, and sliding seat position) with a modulation in frequency and amplitude (combined with video instruction and recording of sonification from an expert) can enhance motor learning. The effects observed with sonified stimuli were beyond enhancement observed with the use a pacemaker sound, or natural sound guidance in comparison. Interestingly, the effects were still present at a 3-week retention measurement test.

# Sensory Guidance and Motor Learning: A Theoretical Perspective

In terms of trying to understand why sensory guidance may help skill acquisition, a variety of different yet converging perspectives have been put forward. From an ecological psychology perspective, motor skill acquisition can be defined as an improved use and handing of informational variables available in the environment (Jacobs and Michaels, 2007; Huys et al., 2009; Gray, 2010; Huet et al., 2011). In that sense, novices can be described as perceivers with pre-existing skills for perception and action learning who adapt their performance in response to training of their attention (Dyer et al., 2017). Alternatively, the concept of perception guiding action can be referred to as a feed-forward model of human motor control. Humans are believed to internally simulate the movement prior to execution and then correct it during action performance based on feedback (external and proprioceptive) (Wolpert et al., 1995). The same feed-forward can be applied to conceptualizing what happens when our own actions are organized with regards to external movement patterns, both biological or non-biological (de Wit and Buxbaum, 2017). In other words, our brain is designed for perception to guide and correct action, but also to understand the actions of others via the same neural networks (Rizzolatti and Craighero, 2004). Regardless of different theoretical approaches that link perception and action, neuroimaging studies show that humans exhibit an affinity for human velocity patterns in motion (Stadler et al., 2011, 2012), even if it is reduced to a display comprising of a few points of light (Johansson, 1973). Moreover, the detection of patterns of human action is likely a "supramodal" process, that is, independent of whether the movement is perceived visually or auditorily (Rosenblum et al., 2017).

Many studies show that visual guidance can facilitate motor learning of a new skill. In a study investigating the effects of observational learning on golf swing performance in a group of novices, the results showed that participants benefited when their attention was being visually guided to specific aspects of the movement (D'Innocenzo et al., 2016). Directing attention straight to accentuated points in the display was more beneficial than observing the movement of an expert alone or replaying their own performance on a video recording. Another study that looked at the effects of observational learning when learning to bowl a delivery in cricket, found that point light displays improved interlimb coordination during the movement and helped participants recreate a movement that resembled the model movement in the full body display (Breslin et al., 2009). Similar results were reported for video and point light displays in learning to kick a soccer ball in a group novices. Results again showed that there was a convergence toward the kinematics demonstrated in the model movement, without any impact on success or accuracy of the kicks (Horn et al., 2002).

### Research Questions

The core research question in this study is to investigate whether people can achieve better learning outcomes if a perfect "copy" of the movement dynamics and tempo is made available to them via an auditory channel. We call this approach the "copycat" approach as it aims to imitate someone else's behavior. Our idea is based on how skills are learned in real-life settings: people often try to track a particular motion template, or rhythm, presented in a single sensory domain—usually visual. Occasionally coaches haptically guide the movement of students by using their own motion to convey the template information via the proprioceptive channel. In this study, we adopt a novel approach, where a novice is presented with an expert's kinematic template of movement that is encapsulated in patterns of sound. This sound contains temporal information to guide movement just before (feedforward) and concurrent to the execution. In doing so, both the relative spatial and temporal characteristics of the movement are conveyed via sound so they can be re-enacted (Young et al., 2013). To explore this novel approach, we recorded the putting performance of a professional golfer when putting to three distances to provide the kinematic pattern for both an auditory and visual display (**Figures 2**, **3**, respectively) that could be used later to assist learning in groups of students learning to putt a golf ball.

We posed three research questions:


# METHODS

#### Participants

Thirty right-handed Sport Science students at Aix-Marseille University took part in the experiment (mean age:19.6 ± 2.4 years). Participants were asked not to take up any golf related practice outside of the training for the duration of the study. None of the participants had previous experience playing golf or putting. All participants had normal or corrected to normal vision and no hearing impairments. All participants provided written and informed consent to voluntarily participate in the study, in exchange for student course credits. All participants

FIGURE 2 | Sound stimuli for the GS condition. The spectograms of the sound stimuli used in the generation of the GS with the original velocity curves derived from the motion capture recordings of the professional player (for 3, 6, 9 m successful putts). See Supplementary Information to listen to the sounds used in the Experiment.

FIGURE 3 | An illustration of the visual display. (Top) A subject from the GV in the process of learning using the visual guide (written informed consent was obtained from the depicted individual for the publication of this image). The ball is aligned to the starting position of the display. The participant waits for the display to launch, observes the first loop of the display and then moves along with the second loop. (Bottom) Flowchart depicting experimental procedure in each trial respective of participant's group. See Videos 1, 2 to see LED guide used in the Experiment.

were informed of their right to withdraw at any time. This study was performed in accordance with the ethical standards of the Declaration of Helsinki (Salako, 2006). The protocol was approved by the Ethics Committee of Aix-Marseille University.

# Protocol

After the baseline measurements were collected from all participants (ten putts to three distances: 3, 6, 9 m), they were pseudo-randomly assigned to one of three experimental groups (n = 10) such that there were two females per group (mean age Control: 19.9 ± 2.2, Sound: 20.1 ± 3.1, Visual: 19.3 ± 1.8 years). The three experimental groups were:


The number of sessions and timeline of the study is depicted in **Table 1**. Participants were asked to train by putting a golf ball a certain number of times (as determined by the session requirements) to each of the distances (See **Table 1**). During the learning sessions, the first five putts were made to each target distance and were recorded as retrieval trials (i.e., performed without any sensory display (sound or vision) being made available). A further fifteen putts were also recorded as learning trials where the sound or visual display was made available to the GS and GV respectively, with no display for the GC. The order of putting distances was randomized in each session using custom made software (Docometre).

Participants performed 120 practice shots to each putt length (360 in total across three lengths) with 40 retrieval trials (120 in total across three lengths) over eight learning sessions (4 weeks). The breakdown of each session is available in **Table 1**. Baseline measurements were conducted 2 weeks prior to the start of the training and the retention measures were taken 2 weeks after the end of the training. Transfer tests were conducted immediately after the last learning session (8th) for each participant and comprised of two new putting distances: 4.5 and 7.5 m.

Each trial had two phases (see **Figure 3**, bottom panel). The first was a preparation phase where the participant was instructed to focus on the ball and the putting distance, and second was a putting gesture phase where the participant was instructed to hit the ball as soon as s/he felt ready. Each phase was preceded with three metronome beeps (60 bpm, 500 ms inter-beep-duration, 440 Hz) to control the general timing instructions to putt in each trial. Participants were instructed to move after the last beep of the metronome in the Gesture putting phase. For the GS and GV participants, they either listened to the sound or observed the LED display after three metronome beeps. In the GC and GV a continuous pink noise (duration matching sound duration in


GC, control group; GS, group with sound display; GV, group with visual display, BS, Baseline Session; LS 1-8, Learning Sessions; RS, Retention Session; TS, Transfer Session; RT, Retrieval Trials; PT, Practice Trials.

GS for each length) was played after each metronome display to match the presence of sound in GS. In the Baseline, Transfer and Retention tests for all of the groups (GC, GS, GV) they performed the shots with a metronome followed by a continuous pink noise (duration 1.5 s).

# Apparatus

A 2 × 0.03 × 15 m (W × H × L) artificial golf green was positioned on wooden planks with a 10.8 cm hole cut out 1.5 m from the wall end in a dedicated golf putting lab. Five black painted marks on the artificial green were made to determine five distances to the hole (**3**, **4.5**, **6**, **7.5**, **9 m**). Although we chose three distances (**3**, **6**, **9 m**) to manipulate the difficulty of the task, we are aware that the typical putt in the game of golf does not normally exceed 7.5 m (Burchfield and Venkatesan, 2010). A Logitech Camera (HD Video Camera- Pro Webcam C930e) was mounted on an extended mechanical arm parallel to the green and overlooked the putting hole (2.5 m above the putting green) and allowed us to measure the accuracy of each putt (**Figure 4**). The camera was controlled using custom-made software that recorded ball movement at 30 Hz. The recording was triggered at the start of each trial and was stopped by the researcher when the ball was stationary near the hole. An Oddysey White Ice putter for right handers was used for the task, along with a set of Titleist balls PROV1X (60 balls for each session). All putting movements were recorded using the CodaMotion system. One CX1 camera was placed parallel to the starting position of the putt on the putting green, with infra-red active markers being placed near the top of the putter shaft and on the club head of the putter. Positional data of the movement of the putter were exported to Matlab for processing. The launch of trials and all the devices connected were controlled using the Adwin Gold system (©JAGER GmbH) piloted via our inhouse Docometre software. Sound was delivered by a Raspberry Pi and custom-made program based on the ALSA software. Participants in all groups were wearing Sennheiser headphones to provide them with an auditory cue to signal the launch of the trials.

# Design

#### Copycat Approach

For the GS and GV, we designed the sensory displays based on the performance of an expert golfer (copycat approach). To do so, we invited a professional player to putt a golf ball to three distances (3, 6, 9 m) during the pilot stage of this study and recorded his movement using the CodaMotion motion capture camera CX1 and two infra-red active markers placed near the top of the putter shaft and the club head (see **Table 2**). The sound of ball impact was also recorded with a portable microphone (ZOOM H4 handy microphone) placed on the putting green 15 cm from the golf ball at each putting distance. We chose the best putt across the expert golfer's successful trials (ball going in the hole), based on the visual inspection of the velocity curve and personal feedback from the player. We chose the first derivative of the spatial position to create the pattern of information presented in the sensory displays—auditory (GS) and visual (GV) and also determine the time of ball impact in the action.

#### Auditory Guidance for GS

Many studies select a sonification method a priori without considering what is important for the design of the sound stimuli (Sigrist et al., 2013). In fact, there is a need for research to map properties of sound, such as amplitude, brightness, or loudness, onto movement parameters. To convey the motion in sound in the best possible way, we ran two pre-tests to decide on the best sound design to use (see O'Brien et al., 2018). The sounds implemented in this study (**Figure 2**) were synthesized using a tailor-made script as white noise with the center of a band-pass filter mapped to velocity ("whoosh" sound designed to resemble the aural consequence of metal club cutting through the air). We used a psychometric conversion to the Mel scale incorporating a linear mapping of the velocity signal. We added a stereo effect reflecting the positional changes of the golf club with respect to the midline axis of the body. To convey the changes in the energy levels necessary to putt to longer distances (effectively increasing the movement velocity) the sound for the **3 m** putt was scaled on a band from 56 to 252 Hz (with peak velocity of the movement of the pro player being 0.56 of the value of the **9 m** peak velocity); the **6 m** putt was scaled on the band of 158–358 Hz (with peak velocity of movement of the pro player being 0.80 the value of the **9 m** peak velocity); and the **9 m** putt was scaled to 250–450 Hz. The pre-recorded sound of impact was embedded into the sound to correspond to the point of impact between the club and ball and was based on the kinematic recordings.

#### Visual Guidance for GV

To depict motion visually, we used a LED guide consisting of a series of 400 linearly aligned LEDs (1.2 m long) fully programmable and mounted in a portable, rectangular unit, with a PIC board inside (**Figure 3**, top panel). The connection was set up via a PCB USB adapter to the external computer, which allowed us to trigger the display using a User Datagram Protocol (UDP) predesignated signal (Unicode character). The custom software made in C++ meant we could load any artificial, biological motion profile allowing us to control the number of LEDs involved in the display and the time each was lit for. Using a custom-made script in Matlab, we translated the position on the x axis into the LED display scaling the amplitude of the movement to the amplitude of the display (see **Table 2** for information on speed and amplitude of movements across different putt distances). The congruency between the display and the original kinematic was previously validated using video tracking method in a prototype of the used LED guide in a study by Bienkiewicz et al. (2013). The original motion capture profile of the expert golfer was translated into the LED display using a custom MATLAB script that translated positional data into the amplitude and time that each LED was lit up for. This programme has full functionality to determine the direction and the timing of the LED display. This way, the visual motion of the expert player was depicted as a point of light moving in a linear fashion on a predesignated path conveyed



MT, movement time; STD, standard deviation.

the movement of a club head in a golf putting movement (See **Video 1**).

We validated the span of the display with the actual physical measurements of the swing from the motion capture and observed differences of ± 5 mm due to the small gaps between blocks of LEDs. The UDP character was sent via a LAN connection to launch the guide in sync with the other devices.

#### Calibration Method for Video Acquisition

For each participant, and each trial, camera images from the experimental sessions were captured at a frequency of 30 Hz and saved in a separate folder. Post session, all images were processed using the automatic custom-made ball trajectory recognition software Eclipse RCP and OpenCV technologies. Algorithms were able to detect the contrast between the background putting green and the ball, tracking the point that corresponded to the center of the ball. The coordinates of the ball in each frame were extracted and saved as a text file. Each trial was visually inspected to verify that the automatic tracking was correct. If there was too much light or an alien object was present in the camera view distorting the recognition, relevant masks were applied and the trial was reprocessed.

The video calibration was applied to the post-processed text files to translate the pixel coordinates into the physical metric coordinates of the experimental space. This was done using a custom written Python script that incorporated static and dynamic calibration using an A3 print-out of a chessboard panel (calibration image). Firstly, the camera's intrinsic parameters and distortion coefficients were computed using 32 images taken at different perspectives. This allowed us to transform the image obtained using coefficients that could account for the light modification due to hardware properties. Secondly, perspective projection was computed using homography of a pixel position mapping onto the experimental metric space in a reference calibration image. The origin was placed in the center pixel of the putting hole. After the calibration processing, each trial had a text file with a metric position for the ball in each trial. This data was used for further analysis.

#### Data Extraction

Trials where participants failed to smoothly strike the ball (i.e., when two or more peaks were detected in the velocity profile around ball impact point due to the participant hitting the putting green before the ball) were excluded from any further kinematic analysis. We chose to run an analysis that calculated the linear velocity relative to the putt-direction axis rather than the angular velocity as the latter can misrepresent impact dynamics if the movement is not performed by a professional (i.e., if the participant is a novice and has a putting action that does not follow a semi-circular movement path).

For the velocity calculation, we applied a low-pass Butterworth filter of 20 Hz, 8th order based on the RMSE method to ensure minimal data loss of the 20 randomly selected recordings of positional data from the data pool. The beginning of the movement was automatically detected as being when the movement velocity exceeded 2% of backswing peak velocity (x axis), and the end of the gesture was denoted as the point when the velocity fell below 2% of follow-through peak velocity (x axis).

#### Statistical Analysis

To explore if there are differences in the way all three groups learned the task, we divided the analysis into three parts: (1) The spatial accuracy of the putts (percentage of successful putts and distance from the hole), (2) Kinematic variability (standard deviation of impact velocity across trials), and (3) Temporal ratio (time spent in backswing movement divided by time spent in downswing movement). To account for the variability in the initial performance between groups we normalized (standardized) the learning sessions and retention data for all of the presented variables to the baseline performance for each individual. For the learning sessions we analyzed separately the retrieval trials (first five shots during the learning sessions, for sensory groups performed without guidance, see **Table 1**) and practice trials (fifteen putts following retrieval, for sensory groups performed with guidance, see **Table 1**).

The analysis presented in the results section compares the performance of three different groups of learners over an eightweek period. The learners were divided into three groups and received (i) auditory, (ii) visual or (iii) no sensory guidance when learning to putt a ball in golf.

For all outcome variables, mixed ANOVAs were carried out with group as a between-subjects factor and both target distance and session number as within-subject factors, unless otherwise indicated. Where main effects were detected, posthoc Bonferroni-adjusted t-tests were carried out. Where the assumption of sphericity was violated, Greenhouse-Geisser adjustments to degrees of freedom are reported.

To estimate the effect size of factors we used partial etasquared (η<sup>p</sup> 2 ) calculations, and complied with the interpretation of indexes proposed by Cohen (1988) (0.01 = small effect; 0.06 = medium effect; 0.14 = large effect). Statistical significance was set at the 5% level.

### RESULTS

# Results Referring to Research Questions 1 and 2

#### Spatial Accuracy of Golf Putts

**Figure 5** top panel represents the overall number of successful putts (defined as ball going into the target hole) per group per round normalized for the baseline for the first five putts at the beginning of each learning session. The bottom panel represents number of successful hits for practice trials across learning sessions. **Figure 6** illustrates performance of participants for transfer and retention sessions.

#### **Retrieval trials (1:5)**

In the In the retrieval trials, the first five trials (without any display for GS and GV), showed improvement across sessions when hitting to **3 m** compared to **6** and **9 m**. We found the following main effects: **learning session number** on the gain in success rate F(7, 189) = 5.1, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.16, main effect of **target distance** F(2, 30.7) = 10.17, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.28 and interactions: **target distance**<sup>∗</sup> **learning session number** F(14, 201.78) = 2.49, p = 0.01, η<sup>p</sup> <sup>2</sup> = 0.08. Bonferroni corrected pairwise comparisons revealed significant differences (p < 0.05) between T1 (0.03 ± 0.01) and learning sessions T6 (0.11 ± 0.03), T7 (0.13 ± 0.02), and T8 (0.13 ± 0.16). There was a significant difference between performance at the target putt distance **3 m** (0.16 ± 0.03) and **9 m** (0.03 ± 0.01), p < 0.01, and between **6 m** (0.07 ± 0.01) and **9 m**, p < 0.01.

To further investigate this relationship, we looked at how the radial distance from the hole changed over sessions. This variable was derived using the coordinates of the final ball position and the origin of the hole in metric units and was normalized with respect to baseline data for each participant. **Figure 8** shows changes over the sessions for retrieval trials (top panel). In the retrieval trials we found significant main effects for putting **target distance** [F(2, 54) = 6.12, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.18], and **learning session number** on the distance from the hole [F(4.2, 115.8) = 4.46, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.14]. Bonferroni corrected pairwise comparisons revealed a significant difference within learning sessions T1 (0.93 ± 0.05) and T7 (0.74 ± 0.05), p = 0.02, and T1 and T8 (0.70 ± 0.06) p < 0.01. There was a significant difference between performance at target distance **3 m** (0.74 ± 0.07) and **9 m** (0.93 ± 0.05), p = 0.01, and between **6 m** (0.71 ± 0.05) and **9 m**, p < 0.01.

#### **Practice trials (6:20)**

In the practice trials that included fifteen putts to each target distance that directly followed retrieval trials, all groups improved with time, but the improvement in GS and GV was more pronounced. We found a main effect of **learning session number** at the hit rate F(7, 189) = 9.09, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.25, **target distance** F(1.18, 30.7) = 38.18, p < 0.01, η 2 <sup>p</sup> = 0.59 and interactions: **target distance** <sup>∗</sup> **learning session number** F(14, 201.78) = 2.49, p = 0.01, η<sup>p</sup> <sup>2</sup> = 0.08, and target distance <sup>∗</sup> learning session

the retrieval trials (no guidance in GS and GV). The graphs show the first five shots of each session, and the practice trials when the sensory groups had acoustic and visual guides, respectively. All groups performed better with the progression of the sessions. (Bottom) The GV condition had a visible dissonance effect between the retrieval and practice conditions suggesting a greater level of sensory dependency.

number<sup>∗</sup> group F(14.95, 201.78) = 2.17, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.14. **Figure 5** depicts a more pronounced increment in the success rates at **3 m** for GS and GV than for GC. Bonferroni corrected pairwise comparisons revealed significant differences (p < 0.05) between T1 (0.09 ± 0.01) and learning sessions T5 (0.15 ± 0.02), T6 (0.16 ± 0.02), T7 (0.18 ± 0.01), T8 (0.17 ± 0.02). There were significant differences (p < 0.01) between performance at target distance **3 m** (0.27 ± 0.03) and **9 m** (0.04 ± 0.01), and **6 m** (0.1 ± 0.01), and **9 m**. No group differences were found.

**Figure 7** shows changes in radial distance from the hole over the sessions for practice trials (bottom panel). For practice trials we found significant main effects for **target distance** [F(2, 54) = 13.6, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.33] and **learning session** on ball distance to the hole [F(4.06, 109.7) = 12.4, η<sup>p</sup> <sup>2</sup> = 0.31] and also a significant interaction between **target distance** <sup>∗</sup> **learning sessions number** <sup>∗</sup> **group** [F(12.74, 172) <sup>=</sup> 1.6, <sup>p</sup> <sup>=</sup> 0.03, ηp <sup>2</sup> = 0.11]. Bonferonni corrected pairwise comparisons revealed significant differences (p < 0.01) between T1 (0.93 ± 0.06) and learning sessions T3 (0.72 ± 0.04), T4 (0.67, ± 0.04), T5 (0.65 ± 0.05), T6 (0.65 ± 0.05), T7 (0.64 ± 0.03), T8 (0.63 ± 0.04). There was a significant difference between performance at target distance **3 m** (0.55 ± 0.07) and **9 m** (0.89 ± 0.04), p < 0.01,

bottom panel shows the hit rates observed during the transfer test. The control group performed better than the sensory groups at the 4.5 m distance, but not at the longer 7.5 m distance.

and **6 m** (0.69 ± 0.04) and **9 m,** p = 0.01. No group differences were found.

#### **Retention**

At the retention test (top panel of **Figure 6**) there was a significant effect of putt target distance on the number of successful putts normalized to baseline F(1.39, 37.6) = 14.46, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.35, with Bonferonni corrected pairwise comparisons demonstrating differences between on distances 3 m (0.24 ± 0.03) and 6 m (0.09 ± 0.02) rate p < 0.01, and 3 m and 9 m (0.06 ± 0.01) p < 0.01.

There was also a main effect change in radial distance to the target of **target distance** [F(2, 4) = 12.90, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.32], with differences between **3 m** (0.54 ± 0.04) and **9 m** p < 0.01 (0.91 ± 0.07) and between **6 m** (0.65 ± 0.05) and **9 m** (p = 0.01).

#### **Transfer**

The bottom panel of **Figure 6** depicts performance at the transfer test in all groups putting to the 4.5 and 7.5 m distances. Due to a violated assumption of normality for the residuals we ran a Wilcoxon Signed Ranks Test, Z = −3.5, p < 0.01 for two conditions. A Kruskal-Wallis Test revealed no differences between groups in performance at the transfer test.

# Results Referring to Research Questions 2 and 3

#### Kinematic Variability (Impact Velocity)

We have pooled together all practice trials from all participants across all lengths and learning sessions (30 participants × 8 learning sessions × 3 distances × 15 practice trials for each distance) to verify if the key factor in kinematic performance that influenced the distance of ball traveled was impact velocity. We found, using a linear model, that impact velocity explained 82% (Adjusted R-Squared 0.82 p < 0.01) of the distance the ball traveled (measured as a function of putting metric distance). Therefore, to quantify the kinematic variability of performance across trials we extracted, for each participant, a standard deviation across trials [separately retrieval (1:5) and practice trials (6:20)] for impact velocity.

#### **Retrieval trials (1:5)**

A significant main effect of **learning session number** on **impact velocity variability** (standard deviation) was found F(4.80, 129.72) = 4.65, p < 0.01, η 2 <sup>p</sup> = 0.15 in the retrieval trials (normalized to baseline performance), indicating some form of learning and skill acquisition associated with practice (see **Figure 8** for reference). Bonferroni corrected pairwise comparisons revealed a significant difference within learning sessions T1 (0.84 ± 0.05) and T7 (0.62 ± 0.03), p < 0.01, and T1 and T8 (0.58 ± 0.03), p < 0.01.

#### **Practice trials (6:20)**

In the practice trials, the main effect **learning session number** on **impact velocity variability** was noted [F(4.65,125.76) = 10.06, p < 0.01, η 2 <sup>p</sup> = 0.27). Main effect of interaction between T1 and T8 **target distance**<sup>∗</sup> **learning session**∗**group on variability** was found F(12.74,172) = 2.7, p = 0.04, η<sup>p</sup> <sup>2</sup> = 0.16. Bonferroni corrected pairwise comparisons revealed significant differences (p < 0.05) between T1 (0.83 ± 0.04) and learning sessions T3 (0.67 ± 0.03), T5 (0.63 ± 0.04), T6 (0.63 ± 0.04), T7 (0.58 ± 0.02), T8 (0.59 ± 0.02). No other effects were found.

#### **Retention**

At retention there was a trend toward main interaction of **target distance** <sup>∗</sup> **group** on **impact velocity variability** <sup>F</sup>(2, 4) <sup>=</sup> 2.3, p = 0.07, η<sup>p</sup> <sup>2</sup> = 0.14.

#### Timing Variability (Temporal Ratio)

In this section we present findings with reference to:

Professional players keep their temporal ratio between the duration of the backswing to forward swing constant across putts to different target distances. In our study we found that participants show a different pattern of behavior.

#### **Retrieval trials (1:5)**

For retrieval trials—we found a significant effect of the putt **target distance on** the temporal ratio F(1.4, 37.8) = 11.94 p < 0.01, ηp <sup>2</sup> = 0.3) suggesting that people adapted their putting timing pattern to accommodate different distances (see **Figure 9**). No learning session number or group effects were found. Bonferroni corrected pairwise comparisons revealed significant

distance to the hole (normalized for baseline performance) for each subject (where 1 stands for performance equal to baseline, and 0 reaches the hole). The top panel depicts the retrieval trials (first five trials across each learning session) compared to retention. The bottom panel depicts practice trials (fifteen putts following retrieval trials) across each session compared to retention trials.

differences (p < 0.01) between the temporal ratios at all putt target distances **3 m** (2.1 ± 0.34), **6 m** (2.20 ± 0.39), **9 m** (2.26 ± 0.41).

**For the standard deviation of the temporal ratio normalized to the baseline data** (**Figure 10**) we observed no main effects in the retrieval trials.

#### **Practice trials (6:20)**

For practice trials—we found a significant main effect for putt **target distance** on the temporal ratio [F(2, 54) = 16.27 p < 0.01, ηp <sup>2</sup> = 0.37] again suggesting that people can adapt the putting timing pattern to achieve different putt distances (see **Figure 9**).

Bonferroni pairwise comparison found significant differences between temporal ratio for putt target distance between **3 m** (2.1 ± 0.34) and **9 m** (2.21 ± 0.43), p < 0.01, and **6 m** (2.15 ± 0.39) and **9 m,** p = 0.01.

We did find a significant interaction in the practice trials between **target distance** <sup>∗</sup> **learning session number** <sup>∗</sup> **group** [F(10.09, 131.23) = 1.72, p = 0.01 η<sup>p</sup> <sup>2</sup> = 0.11) and **standard deviation of temporal ratio** (see **Figure 10**). Bonferroni corrected pairwise comparisons did not reveal significant differences for any of the factors.

#### **Retention**

For retention we found a main effect of target distance F(1.47, 39.8) = 8.22, p < 0.01, η<sup>p</sup> <sup>2</sup> = 0.23 (see **Figure 9**). Bonferroni corrected pairwise comparisons revealed differences between **3 m** (2.20 ± 0.06) and **6 m** (2.27 ± 0.07), p = 0.01, and **3** and **9 m** (2.33 ± 0.08), p = 0.01.

No main effects or interactions were present for standard deviation of temporal ratio at the retention measurement (see **Figure 10**).

#### DISCUSSION

In this study, we wanted to investigate whether people can achieve better performance outcomes if a model template of the movement dynamics and tempo are made available to them through either an auditory or visual display. When compared to the performance of a control group, our data show that

both groups exposed to sensory guidance showed improved task performance during learning.

Our first research question investigated whether novices can "learn" a golf putting task better when compared to a control group, where success is measured in terms of goal attainment. We found an interaction between groups at each learning session during the putts performed with assistance of sensory guides. However, those performance advantages were not present during the retrieval trials performed without sensory guides, or in the retention tests two weeks after the end of training sessions. We also did not find a difference between groups in the transfer test between trials.

With respect to our second research question, we wanted to see if the sensory guidance resulted in differences between the groups in terms of kinematic variability (standard deviation of impact velocity across trials) and timing variability (standard deviation of temporal ratio between backswing and downswing movement). We found significant interactions of group for both factors when putts were performed in the presence of sensory guides.

With regards to our third research question, we found that a sound guide that delivers the spatio-temporal characteristics of expert motion can influence the learning of a new, and complex motor task in a similar way to a visual display. This is particularly interesting considering that the acoustic display was representing information participants were not accustomed to having since they had no prior experience of golf putting. We did not observe differences between sensory guide groups in terms of performance suggesting that people were able to pick up information relating to the movement dynamics of a professional player from environmental sounds. This is consistent with Rosenblum et al.'s "Supramodal Brain Theory" (2017), mentioned in the introduction, according to which external events may be equally well perceived through visual or auditory channels, providing that the relevant information patterns are specified in either sense modality. However, the observed advantage compared to the control group was not significant in post-hoc tests.

Taken together, our findings suggest that sensory guidance during learning might lead to an enhancement of performance,

but is limited to the presence of the guide. This phenomenon was previously described in the literature as sensory dependency (improvement present only when the guide is available) and has been reported in other studies (Anderson et al., 2005; Maslovat et al., 2009). We found the performance advantage was not retained 2 weeks after the end of training, with no specific transfer to other distances (**4.5** and **7.5 m**). Therefore, it seems that although a sound display improves real-time performance when learning a complex task, it does not carry over to performance in the absence of any sensory guidance. Below we will discuss important lessons that have been learned from this study and will suggest other ways in which sensory guidance could be used in a more practical and meaningful way to improve motor performance.

# Lessons Learned From the "Copycat" Approach

Auditory and visual guidance have been repeatedly reported in the literature to be efficient in modifying parameters of human movement in a directive way (Sigrist et al., 2013; Young et al., 2013; Schmitz and Bock, 2014; Danna et al., 2015; Effenberg et al., 2016; Bringoux et al., 2017). However, the majority of previous studies did not look into the use of sound guidance in a motor learning context. In our study, we confirmed that it can bring immediate benefits to performance, but we did not observe these benefits to be retained over time.

Our results suggest that the "copycat" approach we have explored in this study does not bring a long term advantage in performance when compared to learning without guidance. We see the issue regarding this observation as 3 fold. Firstly, sensory guidance has been demonstrated before to lead toward sensory dependency. Adams et al. (1972) described this as a "guidance hypothesis" and explained it as learners becoming over-reliant on the external sensory information and neglecting task-intrinsic, proprioceptive feedback. Therefore, when the guidance is no longer present (i.e., during retention tests) performance drops due to the underdevelopment of internal motor task representation during learning; caused by a neglect

horizontal line on both panels.

of proprioceptive feedback due to the attentional resources being deployed during sensory guidance (Anderson et al., 2005; Maslovat et al., 2009). In this respect, many researchers consider retention performance as a more accurate assessment of learning outcomes than the learning curve during training (Salmoni et al., 1984). The majority of the evidence in the literature about effectiveness of auditory signals in guiding motion comes from studies looking at concurrent real-time auditory feedback tracking parameters of a person's own movement, which is different from the "copycat" approach that tries to imitate the template of an expert's movement. For example, in a study looking at bimanual learning, Dyer et al. (2016) did not observe "guidance reliance" in an immediate retention test, with participants being better than controls when they previously trained with concurrent sonification feedback. The authors of this study hypothesized that extra auditory information might have enhanced the proprioceptive perception of the task goal timing pattern, rather than lead to the neglect of it. However, the observed advantage was diminished at the 24 h post-retention test. Dyer et al. (2017) postulate that the "guidance effect" can be avoided if sonification focuses on enhancement of the naturally occurring task feedback. This stance follows the proposal by Jacobs and Michaels (2007) that motor learning is in fact the training of attention to attend to streams of information that are relevant to task performance. In a similar vein, Buchanan and Wang (2012) demonstrated that if the feedback displayed is not juxtaposed spatially with the movement zone it does not hinder development of the spatial representation of the task. This does not only relate to visual guidance, but also auditory guidance. Arnott and Alain (2011) state that auditory pathway can feed information to action processing in the dorsal pathway (the headquarters of motor action guidance and navigating around space), especially with regards to directing attention to a designated space. Our results did not show any differences in retention between groups. Interestingly, the neuro-imagining study by Ronsse et al. (2011) in a concurrent feedback experiment suggested that the overreliance on visual guidance is stronger than auditory guidance, with the sensory areas being activated during task performance and decrease in auditory conditions. The design of that study, however, could not control for whether participants could memorize the task and the rhythm during practice, and this perhaps influenced their findings. We have found no evidence for this being the case in our study and we are also aware that the translation from studies using concurrent feedback to guidance paradigm (feedforward template of the expert's movement as in this study) is not straightforward. In our study, there was no difference at the retention phase between the performance of groups who used sensory guides when learning the task and those who did not.

Secondly, it is not completely clear how well humans can decode a kinematic template of movement from an auditory signal when it pertains to an environmental sound. Other studies have attempted to investigate the perception of biological motion in healthy adults using sound only (Murgia et al., 2012; Cesari et al., 2014; Kennel et al., 2014; Young et al., 2014). In our piloting phase (O'Brien et al., 2018) we demonstrated that people are able to distinguish between different speeds of golf swing via an auditory signal. This is in line with previous study of Murgia et al. (2012), which found that golfers can recognize their own swing motion via sound recording using two temporal parameters: temporal ratio and overall duration of the swing. Previous research in the visual domain has demonstrated that visual sensitivity to biological motion patterns seems to play a crucial function with links to cognition. For example, research has shown that there is a relationship between our ability to predict the outcomes of an unfolding of action and whether we have executed it before (Knoblich and Flach, 2001; Makris and Urgesi, 2015). Professional athletes demonstrated that they were able to distinguish whether a free throw shot was successful or not having only a point light display of the movement (Aglioti et al., 2008). In one study carried out by the authors, access to the visual point light display (depicting biological movement of healthy adults) resulted in the improvement of the temporal characteristics of an upper arm extension movement in a small sample of Parkinson's disease patients (Bienkiewicz et al., 2013). The brain activity unique for perception of such patterns has been identified by brain imaging studies to be a small area of the superior-temporal sulcus, more precisely the ventral bank of the occipital extent and a small region in the medial cerebellum (Grossman et al., 2000). This neurological circuitry is linked to the ability of animals to understand the action of others and imitate it (Rizzolatti and Craighero, 2004). Despite our reservations, both sensory groups developed their putting skills in a comparable fashion, suggesting that similar information was detectable through both the visual and auditory displays. This has implications for future studies investigating scenarios where sound might be a better fit for providing performance feedback as it is a portable, and relatively easy to implement as a stimuli, without burdening visual attention necessary to control spatial aspects of the task.

#### Limitations

It should be noted that in our study, we did not test how the learned skills of golf putting would transfer to the performance on an actual putting green on a golf course. In addition, we are aware that participants in a non-lab setting would practice a more variable selection of shooting distances instead of **3**, **6,** and **9 m** during all trials. Also, running this experiment in a more ecological setting than a designated lab space could yield entirely different results. Therefore, the "copycat" approach in our laboratory study cannot be generalized to training in a real-life setting.

We also acknowledge that we did not test concurrent sonification in our study, but a feed forward movement template of sonified velocity of a professional player. This leads us to question whether velocity was the right parameter to sonify in this study. The current developments in our lab are focused on investigating motor learning with concurrent auditory feedback with different parameters of sound mapping. We hypothesize that different concurrent sonification methods could reinforce the proprioceptive feedback from movement and perhaps enhance learning to a greater extent than exposure to a template of the movement. In addition, both the sound and visual displays were artificially synthesized/engineered, which might have failed to convey the movement pattern as accurately as actual recordings of the movement (ecological sound, and/or video). Our analysis has been limited to a few of the variables that we deemed most interesting. In future research it is important to consider other factors that influence the precision of the golf ball's trajectory and speed: such as the face, loft and lie angles of the club, the location of impact on the club face (close to the "sweet spot") along with the ratio of the shift of the center of pressure during the movement (Burchfield and Venkatesan, 2010).

## AUTHOR CONTRIBUTIONS

MB wrote the manuscript, collected the data, and conducted the data analysis. FB was involved in setting up the experiment in the laboratory and reviewed the manuscript. MR, CC, LB and CB conducted conceptual work for this study and reviewed the manuscript.

# ACKNOWLEDGMENTS

Authors would like to thank Mr. Rudy Lucas (professional golf trainer) for his expertise and guidelines, and time spent coming to the lab on the numerous occasions for motion capture recordings; Dr. Jorge Ibáñez-Gijón for his help and providing the video calibration protocols in Python for analysis of the spatial accuracy of the shots and assistance with various MATLAB problems; Dr. Stuart Ferguson for providing his help regarding the custom designed visual display–LED guide and help with adapting it to work in the apparatus used in this study. Dr. Benjamin O'Brien for his helpful suggestions on the draft versions of this manuscript and providing an expertise in acoustics. Dr. Mario LaFortune for his insights and expert knowledge in golf shared with us. We also would like to acknowledge our project partners at PRISM CRNS Marseille in providing input, equipment and guidance regarding the sound design: Richard Kronland-Martinet, Solvi Ystad, Mitsuko Aramaki, Gaëton Parseihian for their contributions for the sound design. We would like to thank student assistants for the help with data collection: Pascaline Lantoine, Benjamin Mathieu, Tom Auclair, Nicolas Leclere. We would like to acknowledge the contribution of Dr. Stuart Ferguson and the know-how and hardware from the TEMPUS-G project, funded by the European Research Council under the European Union's Seventh Framework Programme, (FP7/2007-2013) ERC grant agreement n◦ 21 0007-2. Furthermore, we are grateful to the Editor and to the two referees for their valuable comments that helped us to improve the quality and the clarity of our

#### REFERENCES


work. The manuscript is part of the SONIMOVE project, which was fully funded and supported by ANR agency, project ID ANR-14-CE24-0018.

#### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg. 2019.00092/full#supplementary-material


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Bienkiewicz, Bringoux, Buloup, Rodger, Craig and Bourdin. This ´ is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# A Review on the Relationship Between Sound and Movement in Sports and Rehabilitation

#### Nina Schaffert<sup>1</sup> \*, Thenille Braun Janzen<sup>2</sup> , Klaus Mattes<sup>1</sup> and Michael H. Thaut<sup>2</sup>

<sup>1</sup> Department of Movement and Training Science, Institute for Human Movement Science, University of Hamburg, Hamburg, Germany, <sup>2</sup> Music and Health Science Research Collaboratory, Faculty of Music, University of Toronto, Toronto, ON, Canada

The role of auditory information on perceptual-motor processes has gained increased interest in sports and psychology research in recent years. Numerous neurobiological and behavioral studies have demonstrated the close interaction between auditory and motor areas of the brain, and the importance of auditory information for movement execution, control, and learning. In applied research, artificially produced acoustic information and real-time auditory information have been implemented in sports and rehabilitation to improve motor performance in athletes, healthy individuals, and patients affected by neurological or movement disorders. However, this research is scattered both across time and scientific disciplines. The aim of this paper is to provide an overview about the interaction between movement and sound and review the current literature regarding the effect of natural movement sounds, movement sonification, and rhythmic auditory information in sports and motor rehabilitation. The focus here is threefold: firstly, we provide an overview of empirical studies using natural movement sounds and movement sonification in sports. Secondly, we review recent clinical and applied studies using rhythmic auditory information and sonification in rehabilitation, addressing in particular studies on Parkinson's disease and stroke. Thirdly, we summarize current evidence regarding the cognitive mechanisms and neural correlates underlying the processing of auditory information during movement execution and its mental representation. The current state of knowledge here reviewed provides evidence of the feasibility and effectiveness of the application of auditory information to improve movement execution, control, and (re)learning in sports and motor rehabilitation. Findings also corroborate the critical role of auditory information in auditory-motor coupling during motor (re)learning and performance, suggesting that this area of clinical and applied research has a large potential that is yet to be fully explored.

Keywords: acoustic feedback, movement sonification, rhythmic auditory stimulation, sports, motor rehabilitation, Parkinson's disease, stroke

# INTRODUCTION

In the last decades, research in the fields of sport, neuroscience, and psychology, has sought to better understand the role of sounds on perceptual-motor processes from multiple angles of investigation. In applied research, there has been a great interest in how auditory information affect the production of complex movements and how it may be used in sports training and movement

#### Edited by:

Mauro Murgia, University of Trieste, Italy

#### Reviewed by:

Gerd Schmitz, Leibniz University Hannover, Germany Ivan Camponogara, New York University Abu Dhabi, United Arab Emirates

> \*Correspondence: Nina Schaffert nina.schaffert@uni-hamburg.de

#### Specialty section:

This article was submitted to Movement Science and Sport Psychology, a section of the journal Frontiers in Psychology

Received: 09 July 2018 Accepted: 24 January 2019 Published: 12 February 2019

#### Citation:

Schaffert N, Braun Janzen T, Mattes K and Thaut MH (2019) A Review on the Relationship Between Sound and Movement in Sports and Rehabilitation. Front. Psychol. 10:244. doi: 10.3389/fpsyg.2019.00244

**178**

rehabilitation to improve motor performance in athletes, healthy individuals, and patients affected by neurological or movement disorders (e.g., Dubus and Bresin, 2013; Sigrist et al., 2013; Murgia et al., 2015; Pizzera and Hohmann, 2015; Sors et al., 2015; Thaut et al., 2015; Ghai and Ghai, 2018; Ghai et al., 2018a,b). However, this body of research is scattered both across time and scientific disciplines. Therefore, the aim of this paper is to provide an overview about the interaction between movement and sound and review the current literature regarding the effect of acoustic information to improve movement execution, control, and (re)learning in sports and motor rehabilitation.

The first section of the paper (Key topic 1) focuses on sports movements and presents an overview of studies investigating the effect of natural movement sounds and sonification in athlete performance enhancement. Natural movement sounds refer to real-time and naturally occurring acoustic information in the form of auditory reafferences, such as the sound resulting from the contact phase of the feet meeting the ground or the physical impact of limbs or equipment with air/ground/water/ball (Kennel et al., 2015; Pizzera and Hohmann, 2015). Natural auditory signals provide a large amount of information about movements that are readily available to the listener (Gaver, 1993a,b) and may be used in sport training to inform or enhance task-intrinsic feedback (Dubus and Bresin, 2013; Sigrist et al., 2013; Sors et al., 2015). Another line of research is dedicated to the development of perceptual strategies based on auditory information to assist movement execution and control through sonification. Sonification involves the transformation of kinematic and dynamic movement parameters into nonspeech artificially produced sounds in order to improve motor perception and performance (Effenberg, 2005).

The second section (Key topic 2) addresses the use of sounds in motor rehabilitation. Firstly, we focus on rehabilitation methods that administer auditory rhythmic cues to improve motor function in Parkinson's disease (PD) and post-stroke, such as Rhythmic Auditory Stimulation (RAS) (Thaut and Hoemberg, 2014; Murgia et al., 2015). Secondly, we consider studies investigating the application of movement sonification (i.e., real-time artificially produced sounds or musical sonification) to assist in the rehabilitation of motor functions in PD and stroke. Note that musical sonification differs methodologically from music-supported therapy, where the former is a relatively novel approach that uses measuring systems (e.g., inertial sensors) to map different movement parameters using musical components, and the latter involves repetitive exercises using musical instruments to retrain motor functions, thus not providing continuous real-time movement feedback (see for review, De Dreu et al., 2012; Zhang et al., 2016; Sihvonen et al., 2017). Studies addressing background music or applying music as auditory feedback are beyond the scope of this review.

In the third section (Key topic 3), we provide an overview of current evidence regarding the neural mechanisms involved in auditory-motor coupling. Particularly, we describe brain regions involved in auditory-motor coupling and address the role of mechanisms such as auditory-motor entrainment, auditory mirror neurons, and sensorimotor integration. By organizing and providing a critical appraisal of the current research, we attempt to develop a framework for future applied and clinical research on the effects of auditory information for motor control and (re)learning.

# METHODS

# Search Strategy

The systematic searches included numerous electronic literature databases (e.g., MEDLINE, EMBASE) and trial registers, as well as hand-searching of major journals, abstract books, conference proceedings and reference lists of retrieved publications. Also, potentially relevant texts known to the reviewers were included.

# Study Selection

The search and screening process for relevant literature is shown in **Figure 1**. The titles of all retrieved publications were checked, duplicates were removed, and those publications related to other fields of research were excluded. The initial screening resulted in 345 remaining publications, which were further screened for eligibility based on the following criteria: (a) the work must be published in full in English language, (b) must be based on original data, and (c) must be related to the field of auditory information within the context of sport or sport-related activities, and rehabilitation. Publication abstracts and full texts were used to perform a thorough check of these criteria. After this step, 222 publications were identified and included in this paper, of which 131 papers are clinical or applied studies investigating the effect of auditory information in sports and motor rehabilitation.

# RESULTS

#### Key Topic 1: Natural Movement Sounds and Movement Sonification in Sports Natural Movement Sounds

The role of natural movement sounds in auditory actionperception coupling has been studied in sports domains and daily physical activity as part of more general research. Among the topics investigated, studies have examined the influence of natural movement sounds on movement execution (Agostini et al., 2004; Kennel et al., 2015), sense of agency (self vs. other) (Murgia et al., 2012a; Kennel et al., 2014a,b), action anticipation (Cesari et al., 2014; Allerdissen et al., 2017; Camponogara et al., 2017; Sors et al., 2017, 2018a,b; Cañal-Bruland et al., 2018), and motor learning (Pizzera et al., 2017).

Natural movement sounds carry rich auditory information that has direct physical correspondence to their referent event(s), providing crucial information that may be used to inform or enhance task-intrinsic feedback (Dubus and Bresin, 2013; Sigrist et al., 2013; Sors et al., 2015). One of the direct effects of the presence of natural movement sounds is improving athletes' movement execution, as shown in a study investigating hammer throwing (Agostini et al., 2004). The role of auditory information on movement execution has also been investigated by manipulating the amount or the temporal

features of feedback provided to athletes. It has been shown, for instance, that deprivation of auditory feedback hindered the performance of experienced tennis players by decreasing receiving service precision (Takeuchi, 1993). Kennel et al. (2015) examined whether the sounds of the steps during running would influence hurdling performance and found that temporally delayed auditory feedback decreased athletes' performance by slowing down the time to complete the track and affecting the motion sequence during the first trials where the manipulation was presented. However, there were no differences in movement execution when comparing normal real-time auditory feedback condition and white noise.

Natural movement sounds also provide fundamental information about agency and facilitate the discrimination of one's own from another person's movement. The role of specific sound features on the sense of agency has been recently investigated in sports such as golf (Murgia et al., 2012a) and hurdling (Kennel et al., 2014a,b). Murgia et al. (2012a) found that expert golfers could identify the recorded sounds of their own golf swings from those of other athletes based on the temporal features of the movement sound, such as the overall action duration (i.e., how long the swing movement lasted from beginning to end) and the rhythmic patterns of the backswing and downswing movement. Kennel et al. (2014a,b) also found that athletes could distinguish between their own hurdling movements from those of others' on the basis of the auditory information, using a variety of sound characteristics (e.g., hurdling step structure, amplitude of the sounds) to build a holistic representation of their own and others' movements.

Research has also shown that athletes are able to extract relevant information from the sounds generated by their own or others' movements to predict and anticipate actions based on changes in the environment or the opponents' behaviors. It has been demonstrated, for example, that expert basketball players can detect the movement intentions of an opponent and prediction their running direction based on the sounds generated by the opponent's movements (Camponogara et al., 2017). Cesari et al. (2014) found that the ability to precisely anticipate and reproduce a skateboarding jump based only on movement sounds was superior for experienced athletes than for non-experts. Specifically, only experts were able to modify their underfoot force and apply muscle synergies that were essentially similar to those used during a real jump on a skateboard only by hearing the movement sounds. Similarly, studies have also demonstrated that auditory information generated by movements may be used to predict a attack movement in

fencing (Allerdissen et al., 2017), the shot power in soccer (Sors et al., 2017, 2018a), and the length of volleyball serves (Sors et al., 2018b). These behavioral data collectively indicate that the auditory-motor coupling generated during extensive training significantly interacts with athletes' internal motor simulation as experienced athletes are not only able to extract highly specific information from action-related sounds but also use this information to anticipate another person's movements based on action prediction mechanisms.

The short- and long-term effects of acoustic reafference to improve movement control and learning of complex movements has been recently investigated. Pizzera et al. (2017) tested a training protocol where natural step sounds produced during hurdling were recorded and presented before each trial with modulated tempo in relation to baseline: faster tempo, slower tempo, or normal tempo. Results showed that the presentation of the auditory information increased overall performance for all groups at short-term, enhancing running time and movement technique. When considering the long-term effects, findings suggested that only the groups that received acoustic information with modulated tempo (faster or slower) further increased performance at a 10-week retention test, whereas the performance of the group who trained with normal auditory feedback declined. These results indicate that, while acoustic information during training have immediate effects on athletes' performance, repeated training with modified temporal acoustic information may be more effective and contribute to the development of a richer internal representation of the movement.

#### Movement Sonification

Sonification, as the transfer of movement data into non-speech audio signals, refers to the mapping of physiological and physical data onto psychoacoustic parameters (i.e., loudness, pitch, timbre, harmony and rhythm) in order to provide onand/or offline access to biomechanical information otherwise not available (for an overview see Effenberg et al., 2011, 2016; Dubus and Bresin, 2013; Sigrist et al., 2013; Kos et al., 2015; Pizzera and Hohmann, 2015). Movement sonification thus aims to assist movement control, execution, and planning by improving self-awareness of physiological processes underlying movement execution and optimizing movement regulation and control (Effenberg, 2005).

The potential use of real-time movement sonification has motivated researchers to investigate the effectiveness of sonification as additional real-time acoustic information in sport training to enhance athletic performance in a wide range of sports (see **Supplementary Table 1**), including: running (Eriksson and Bresin, 2010; Bolíbar and Bresin, 2012; Boyd and Godbout, 2012; Sanderson and Hunt, 2016), aerobics (Hermann and Zehe, 2011), rowing (Dubus and Bresin, 2010; Schaffert and Mattes, 2011; Wolf et al., 2011; Cesarini et al., 2014b), swimming (Hermann et al., 2012; Cesarini et al., 2014a), sailing (Tarnas and Schaffert, 2017), cycling (Sigrist et al., 2016; Schaffert et al., 2017), speed skating (Godbout and Boyd, 2010; Stienstra et al., 2011; Boyd et al., 2012; Godbout et al., 2014), skiing (Kirby, 2009; Hasegawa et al., 2012), golf (Kleiman-Weiner and Berger, 2006; Nylander et al., 2014), juggling (Bovermann et al., 2007), German wheel (Hummel et al., 2010), squat jumps (Newbold et al., 2017), motorsport (Powell and Lumsden, 2015), recreational sports (Barrass et al., 2010), postural control (Avissar et al., 2013), slackline (Anlauff et al., 2013), handball (Höner et al., 2004), basketball (Ramezanzade et al., 2014), elastic trampoline (Pugliese and Takala, 2015), and manual wheelchair training and operation (Almqvist Gref et al., 2016).

Investigations examining the use of sonification in elite or high-performance sports have demonstrated that the presentation of artificially generated sounds optimize movement control and execution (e.g., stability, velocity, pattern and force symmetry) in sports such as swimming (Chollet et al., 1988, 1992), rowing (Schaffert et al., 2010, 2011; Schaffert and Mattes, 2011, 2015b, 2016), and cycling (Sigrist et al., 2016; Schaffert et al., 2017). For instance, Chollet et al. (1988, 1992) examined the effects of the presentation of concurrent auditory signals of hydrodynamic pressure exerted by the athlete's hand during the propulsive action in crawl swimming. Movement data were transformed into auditory signals of equal amplitude and mapped on to pitch so that higher pressure was displayed as a higher pitch. The study results indicated that real-time sonification allowed swimmers to maintain stroke velocity improving movement stability and control (Chollet et al., 1988, 1992). Schaffert and colleagues investigated the influence of acoustic feedback in elite rowing (Schaffert et al., 2010, 2011; Schaffert and Mattes, 2011, 2015b, 2016) and elite para-rowing (Schaffert and Mattes, 2015a). For that, we measured the propulsive boat acceleration trace and converted this information into pitch changes so that athletes perceived an increase in pitch the more the boat accelerated. These studies repeatedly found that movement sonification led to faster boat speeds, increased distances traveled per stroke, and improved crew synchronization compared to training without additional auditory information (Schaffert et al., 2010, 2011; Schaffert and Mattes, 2011, 2015a,b, 2016). In cycling, Schaffert et al. (2017) demonstrated that the continuous real-time auditory information allowed cyclists to perceive fluctuations in forces applied on the pedals and consequently adapt muscle activation to maintain a consistent movement execution pattern and symmetry.

Real-time auditory signals may also enhance athletes' selfawareness during movement execution by providing auditory feedback otherwise not available. This has been shown in studies evaluating the effect of sonification on exerted muscle power in resistance training and weightlifting (Murgia et al., 2012b; Yang and Hunt, 2013, 2015), precision rifle shooting (Konttinen et al., 2004) and inter-limb coordination in gymnastics (Baudry et al., 2006). Yang and Hunt (2013, 2015) examined the potential of real-time sonification to improve the quality of resistance training. Muscular activity (biceps curl) was measured with electromyographic sensors and sonified in relation to the biceps contractions and extensions so that the more effort was exerted the brighter the tone of the sound. The results showed that the auditory information provided concomitant with the movement helped athletes to maintain the pacing of their movement and improve exercise metrics with greater average repetition range and total effort. Murgia et al. (2012b) also showed that

high-intensity sounds presented during the bench-press phase of weightlifting facilitated exerted mean muscle power compared with no sonification. Konttinen et al. (2004) investigated the effects of augmented auditory feedback on precision rifle shooting. The auditory signal informed shooters about rifle alignment by mapping the distance between their aiming point and target center. The study results showed improved shooting performance in the post- and retention tests (after 10 and 40 days) compared to a control group, suggesting that the auditory feedback enhanced shooters' ability to detect errors in body alignment and modify their movements to improve rifle stability and shooting precision. The presentation of auditory information not usually available to athletes' also improved interlimb coordination in gymnastics. Baudry et al. (2006) examined the effects of auditory concurrent feedback on body segmental alignment and inter-limb coordination on experienced male gymnasts during the performance of a circle on a pommel horse. A two-part device (with one piece placed on the upper back and the other – a spring – placed on the knee's backside, both linked with a cable) informed gymnasts about the bent position of the body with an auditory signal. Positive effects on body segmental alignment were found after 2 weeks of training, with gymnasts in the experimental group improving their percentage of maximum body segmental alignment whereas no gains in body alignment were observed for the control group.

Sonification has also been applied during sports training to inform athletes about performance error/deviation in realtime. Collectively, studies in sports training such as handball (Höner et al., 2004), sailing (Tarnas and Schaffert, 2017), speed skating (Godbout and Boyd, 2010; Godbout et al., 2014), and basketball (Ramezanzade et al., 2014) indicate that the availability of real-time auditory feedback enhances online error-correction mechanisms during movement execution and facilitate the learning of a new motor skill. In speed skating, for instance, Godbout and Boyd (2010) provided corrective sonic feedback to an elite athlete with difficulties to perform the cross-over stride movement. The skating stride was matched to a model skater and the differences were sonified. Based on this sonification model it was possible to provide warning cues, timing, and body position information in real-time, allowing the athlete to make corrections and adjustments during movement execution. Sonification modeling was also tested to improve jump shot in basketball with 20 novice participants (Ramezanzade et al., 2014). For that, one group received visual information from a professional player (model) as well as additional auditory information derived from the angular speed of the elbow joint of the player, whereas the second group only received visual information from the player. The findings indicated that the group who received audiovisual information outperformed the group that received only visual information in both the acquisition and retention tests, suggesting that auditory information may facilitate the acquisition and retention of a new motor skill.

Research indeed suggests that real-time auditory feedback supports the learning and retention of new motor skills. Studies collectively indicate that the acquisition of a new skill or movement technique (e.g., swimming stroke technique, precision shooting, inter-limb coordination in gymnastics, rowing technique, and basketball jump shots) is facilitated when auditory information is provided during the acquisition of a new motor skill (e.g., Chollet et al., 1992; Konttinen et al., 2004; Baudry et al., 2006; Ramezanzade et al., 2014; Schaffert and Mattes, 2014). Moreover, with ongoing training, the sonification of the movement is integrated into an internal representation of that skill, thus enhancing the efficacy of motor learning (Effenberg et al., 2016).

# Key Topic 2: Rhythmic Auditory Stimulation and Movement Sonification in Rehabilitation

#### Rhythmic Auditory Stimulation (RAS)

Rehabilitation programs use rhythmic auditory cues as a means to enhance auditory-motor synchronization and promote sustained functional changes to movement (e.g., Thaut, 2005; Thaut and Hoemberg, 2014; Murgia et al., 2015). In particular, rhythmbased techniques use rhythmic patterns to prime the motor system by providing continuous time references that generate expectations for when auditory events will occur or when a movement needs to be performed. The foreknowledge of the duration of the cues allows movement anticipation and motor preparation, hence increasing the quality and precision of the motor responses (Thaut et al., 2015). Specifically, RAS is a rehabilitation technique that involves the utilization of rhythmic cues (metronome or rhythmically accentuated music with embedded metronome clicks) to facilitate rehabilitation of intrinsically rhythmical movements (Thaut et al., 1999; Thaut and Hoemberg, 2014). RAS can be used as an immediate entrainment stimulus providing rhythmic cues during movement or as a facilitating stimulus for training to achieve more functional movement patterns. This technique typically uses simple metronome beats matched to the patient's baseline gait, but walking cadence can also be facilitated by using metronome beats embedded in musical patterns that are 5–10% faster than baseline (Thaut et al., 1996). Alternative versions of RAS include metronome sounds embedded in expert-selected (McIntosh et al., 1997) or patient-selected music (Thaut et al., 1996). In these studies, it was proposed that the musical texture would provide additional timing information compared with metronome alone, thus facilitating detection, anticipation, and synchronization to the beat (Thaut et al., 1997). A modification of RAS can also be found in the literature as Rhythmic Auditory Cueing (RAC), which is defined as the application of repetitive isochronous beats. Although the terminology may differ in different disciplines, the basic underlying principle of these techniques is the same.

There is robust evidence of the effectiveness of RAS to improve movement in PD patients (reviewed in Rubinstein et al., 2002; Lim et al., 2005; Rochester et al., 2010; Thaut and Abiru, 2010; Spaulding et al., 2013; Wittwer et al., 2013; Rocha et al., 2014; Schaefer, 2014; Murgia et al., 2015; Thaut et al., 2015; Ghai et al., 2018b), stroke (for review, Thaut and Abiru, 2010; Yoo and Kim, 2016), traumatic brain injury (e.g., Hurt et al., 1998), multiple sclerosis (e.g., Conklyn et al., 2010; Shahraki et al., 2017; reviewed in Ghai and Ghai, 2018), and cerebral palsy (e.g., Kwak, 2007;

Kim et al., 2011; Baram and Lenger, 2012; Kwak and Kim, 2013; Ghai et al., 2018a for an overview). As the scope of this paper does not allow for a thorough description of all relevant clinical literature using RAS on motor rehabilitation, here we provide a brief overview of representative clinical studies using RAS in PD and post-stroke.

#### **Parkinson's disease**

Gait disturbances such as shuffling, freezing of gait, instability (asymmetry and variability between steps) and general difficulties in walking movements and posture are among the most apparent symptoms of PD (Bloem et al., 2004; Rodger and Craig, 2016). Typically, PD patients with impaired gait have difficulty in regulating stride length (Morris et al., 1996) and tend to walk with reduced velocity and increased cadence or step rate (Knutsson, 1972). One probable origin of gait impairment in PD is deficient internal motor timing mechanisms due to basal ganglia dysfunction. Studies have also suggested that the irregular timing of walking pace may be associated with disturbances of coordinated rhythmic locomotion (Ebersbach et al., 1999; Thaut et al., 2001; Skodda et al., 2010) and sensorimotor synchronization (Bienkiewicz and Craig, 2015, 2016 ´ ).

Thaut et al. (1996) first described the effect of rhythmic entrainment on gait patterns in PD by demonstrating that patients who underwent 30 min of daily home-based gait training with RAS significantly improved their gait velocity, stride length, and step cadence after 3 weeks of intervention in relation to controls. These findings were later confirmed by several studies (e.g., McIntosh et al., 1997; Freedland et al., 2002; Del Olmo et al., 2006; Nieuwboer et al., 2007; Arias and Cudeiro, 2008, 2010; Hove et al., 2012; Song et al., 2015; Pau et al., 2016). Studies have also found that RAS training can have positive carry-over effects on movement from a few minutes to up to 4 weeks (McIntosh et al., 1997, McIntosh et al., 1998; Nieuwboer et al., 2009). Other beneficial outcomes include increase in the symmetry of muscle activation in upper and lower limbs (Malcolm et al., 2009; Bailey et al., 2018), and reduction of timing variability (Miller et al., 1996), resulting in more stable walking (Thaut et al., 1999; Hausdorff et al., 2007; Hove et al., 2012). A recent study also found positive effects of RAS for the facilitation of gait relearning (Uchitomi et al., 2013). Additionally, there are indications that RAS is superior in maintaining gait performance during dualtasks due to low cognitive attentional load (Baker et al., 2008). This robust body of literature has been recently summarized and analyzed in systematic and meta-analysis studies, which concluded that rhythmic auditory information is generally an effective therapeutic tool for treating gait disturbances in PD (see Spaulding et al., 2013; Schaefer, 2014; Ghai et al., 2018b).

Although the application of rhythmic auditory information in gait training is well-established, the use of rhythm-based interventions to improve PD symptoms such as freezing of gait and risk of falls is still under investigation. In relation to freezing of gait, Willems et al. (2006) found no beneficial effects of RAS on freezing of gait in patients with less severe symptoms, but Delval et al. (2014) and Plotnik et al. (2014) reported positive effects of RAS on gait initiation and freezing of gait in PD patients. A recent review (Ginis et al., 2017) also concluded that cue-augmented training can reduce the severity of freezing in PD patients, but limitations in long-term consolidation and transfer of the effects to untrained tasks need to be considered in this population. RAS has been also recently applied to reduce falls or risk of falls in healthy elderly (Hurt-Thaut, 2014) and PD patients (Thaut et al., 2018). These studies collectively found that RAS training significantly reduced the number of falls in healthy individuals and PD patients by modifying key kinematics in gait control, thus suggesting that RAS may be beneficial to address the risk of falls.

A recent line of research has focused on whether specific parameters of the acoustic cues can influence the results of rhythm-based interventions for PD by comparing, for instance, differences between music and isochronous sounds (i.e., metronome) with interactive cueing systems that adapt to the patient's gait (see review in Ashoori et al., 2015; Hove and Keller, 2015). For instance, Murgia et al. (2018) compared whether the nature of the stimulus presented would influence the effectiveness of RAS by providing ecological footstep sounds as auditory information. For that, one group of PD patients completed 5 weeks of supervised rehabilitation training that included walking while listening to ecological footsteps sounds, whereas the second group of patients walked listening to artificial stimuli (e.g., metronome). The overall conclusion of the study was that biological motion sounds such as footsteps are as effective as the metronome, but exploratory analyses of biomechanical measures suggested that there may be some differences in improvement linked to the type of auditory stimuli. Similarly, Dotov et al. (2017) tested biological variability in auditory stimulus vs. isochronous cues and found superiority of biologically variable auditory cues in fostering natural gait variability in PD patients; however, the authors limited their analysis to only immediate and likely transient effects of cueing. Young et al. (2016) found that action-relevance was a more dominant factor in facilitating improvements in gait parameters than acoustic continuity. Finally, Baram et al. (2016) examined the effectiveness of a device that provided a clicking sound generated in response to every step taken by the patient and found that closed-loop auditory feedback produced better results than open-loop auditory cues (e.g., metronome) in relation to gait speed. It is important to note, however, that the use of closed-loop auditory feedback information stands in contrast to metronomebased approaches in relation to a critical component, that is, the use of external auditory cues as predictable feedforward information transmitted by the steady rhythmic information.

Negative effects of RAS were reported when auditory cues were presented at rates much slower (e.g., 20%) or much higher than the patient's preferred gait (Del Olmo and Cudeiro, 2005; Nombela et al., 2013). Arias and Cudeiro (2008) and Dalla Bella et al. (2017) also suggested that RAS efficacy may depend on individual characteristics, including severity of disease symptoms and impaired ability to synchronize to the beat. However, there are indications that beat perception may be of lesser importance due to evidence of the primacy of period entrainment over phase/beat entrainment during small tempo perturbations (e.g., Thaut et al., 1998a,b; Roberts et al., 2000; Thaut and Kenyon, 2003).

Overall, research using RAS and rhythmically enhancedmusic show consistent evidence of the improvement of motor function in PD. Moreover, recent studies have extended the application of RAS to other non-motor functions (for review, see Thaut and Abiru, 2010). For instance, studies have indicated that RAS training enhances patients' performance in both motor timing (movement synchronization, tapping) and in perceptual timing tasks (duration discrimination, beat detection in music), supporting the hypothesis that RAS engages brain networks involved in both perceptual and motor timing (Benoit et al., 2014; Dalla Bella et al., 2015).

#### **Stroke**

Motor impairment is one the most widely recognized consequences of stroke, which include reduced movement coordination, decreased postural control, and decreased upper-limb function (Langhorne et al., 2009). Such significant impairments in locomotive function can lead to limitations in independent mobility, thus strongly affecting patients' quality of life (Michael et al., 2005).

There is strong evidence that RAS can be effectively applied for timely motor control during gait training for stroke patients (Thaut et al., 1993, 1997, 2007; Hayden et al., 2009; for review see Thaut and Abiru, 2010; Hollands et al., 2012; Thaut and McIntosh, 2014; Nascimento et al., 2015; Yoo and Kim, 2016). Thaut et al. (1993) found that patients who walked with RAS matched to their baseline gait cadence showed decreased stride time variability and more balanced muscular activation pattern between the paretic and non-paretic limbs. Recent studies also indicate significant effects of RAS on standing balance (Suh et al., 2014), and gait coordination and symmetry during normal overground walking (Prassas et al., 1997; Roerdink et al., 2011; Lee et al., 2012; Yang et al., 2016) and treadmill training (Roerdink et al., 2007; Park et al., 2015; Yoon and Kang, 2016; Mainka et al., 2018). Immediate effects of RAS training with tempo changes were also found on gait kinematics (Cha et al., 2014) and in relation to the lesion site (Kobinata et al., 2016). Finally, there is growing support for the use of RAS in gait training during the chronic phase of stroke to improve walking speed and functional mobility (e.g., Shin et al., 2015; Ko et al., 2016; Wright et al., 2016, 2017).

Studies have also reported significant improvements in upper limb function after training with RAS (e.g., Whitall et al., 2000; Thaut et al., 2002a,b; Luft et al., 2004; Jeong and Kim, 2007; Malcolm et al., 2009; Chen et al., 2016). For instance, Malcolm et al. (2009) reported a significant decrease in compensatory reaching movements after a 2-week RAS training program, which consisted of patients moving between at least two targets by touching the digits of their affected hand to the assigned targets in synchrony with the auditory rhythmic stimuli. Another line of interventions has used RAC to prime and facilitate bilateral arm training, also known as BATRAC (for review Wolf et al., 2014; Choo et al., 2015). As an example of BATRAC training, in Whitall et al. (2000) participants pushed and pulled bilaterally two independent bar handles in synchrony or alternation with rhythmic auditory cues. The authors found significant improvement in isometric strength, range of motion, and functional motor performance of the paretic arm after 6 weeks of intervention and also at an 8-week followup assessment. Additionally, there are indications that musicsupported training using musical instruments can improve motor recovery of arm movements after stroke by inducing auditorysensorimotor co-representation of movements (e.g., Thaut et al., 2002a,b; Schneider et al., 2007, 2010; Rodríguez-Fornells et al., 2012; Altenmüller et al., 2009; Altenmüller and Schlaug, 2013; Amengual et al., 2013; for a review on music-support training, see Zhang et al., 2016).

#### Movement Sonification

Technology-assisted therapy and rehabilitation seek to help patients in regaining the ability to independently perform daily activities and to facilitate their reintegration into social and domestic life by using advances in smart technologies or robotics (for reviews on robotic-assisted therapy, see Lum et al., 2002; Prange et al., 2006; Kwakkel et al., 2008; Marchal-Crespo and Reinkensmeyer, 2009; Secoli et al., 2011; Pennycott et al., 2012; Rosati et al., 2013).

One of the first applications of sonification in a rehabilitation context was developed by Pauletto and Hunt (2006, 2009). They sonified muscular activity using the temporal patterns in electromyography (EMG) by converting electrical impulses from muscles into auditory information. The goal of this sonification approach was to assist therapists to audibly analyze the complex signals originating from multiple EMG-sensors during physical activity. Several sonification methods and system prototypes have been developed in recent years (e.g., Chiari et al., 2005; Dozza et al., 2005, 2007; Vogt et al., 2010; Tissberger and Wersenyi, 2011; Matsubara et al., 2012; Franco et al., 2013; Torres et al., 2013; Ghai et al., 2018c; see **Supplementary Table 2** for details). A growing body of recent research generally agrees that sonification is a promising feedback tool for patients and therapists, complementing existing analytical components in therapy (such as visual displays) (for an overview, see Huang et al., 2006; Dubus and Bresin, 2013). The following sections present an overview of current investigations in sonification for movement rehabilitation in PD and stroke.

#### **Parkinson's disease**

There has been growing interest in the application of sonification systems in neurologic rehabilitation focusing on improving gait in PD patients (e.g., Batavia et al., 2001; Miyake, 2009; Torres et al., 2013; Contreras Lopez et al., 2014; Young et al., 2014; Horsak et al., 2016; Schedel et al., 2016; see **Supplementary Table 2** for details). A sonification system that has received significant attention in recent years is the use of instrumented footwear (Bresin et al., 2010; Fischer et al., 2017; see review in Maculewicz et al., 2016). These systems comprise of interactive shoes with embedded sensors that collect gait information (e.g., cadence, velocity, stride length), which are then used to trigger auditory cueing stimuli to inform both the therapist and the patient about the user's current state. Recently, Gorgas et al. (2017) tested the effect of an instrumented shoe-insoledevice for real-time sonification of gait (SONIGait; see also Horsak et al., 2016). This sonification system mapped individual

walking characteristics on to musical notes in order to provide gait spatiotemporal information. Results indicated that a 5-min practice phase with sonification increased gait velocity and cadence, opening the possibility for further testing of this realtime sonification device in large controlled trials. Rodger et al. (2014) tested two sonification systems using synthesized walking sounds to guide and improve gait coordination in PD. The first approach used computer-generated sounds of footsteps on gravel in order to convey ecological information regarding step lengths and duration, whereas the second approach involved real-time sonification of the swing-phase of gait by using motion-capture and audio processing software. Study results suggested that both methods had an effect on step length variability but did not alter step duration variability, suggesting that the presentation of auditory information within the patient's normal step duration range had an effect only on spatial characteristics of gait rather than temporal parameters.

A recent innovative line of motor learning based interventions have combined action observation and sonification to improve freezing of gait (see Gilat et al., 2018 for review). For instance, Mezzarobba et al. (2018) presented videos showing an actor performing gait-related gestures while simultaneously presenting the sonification generated by the kinematics of each gesture. Patients were then asked to imitate the movements shown. This training protocol was completed twice a week for a total of 8 weeks by a group of 12 patients, whereas the control group practiced the motor gestures by means of visual (stripes on the floor) or auditory cues (metronome). Assessments conducted after the intervention and 3 months after the treatment suggested that the multisensory treatment significantly reduced the number of episodes and duration of freezing facilitating the priming effect generated by action observation, whereas no significant difference was observed for all mobility indices in the control group.

#### **Stroke**

External real-time auditory feedback has been extensively applied in upper-limb rehabilitation post-stroke (e.g., Maulucci and Eckhouse, 2001; Chen et al., 2006; Wallis et al., 2007; Dailly et al., 2012; Immoos et al., 2013; Bruckner et al., 2014; Fujii et al., 2016; reviewed in Ghai, 2018; see **Supplementary Table 2** for details). For instance, Chen et al. (2006) and Wallis et al. (2007) tested a real-time multimodal sonification system which provided visual and auditory information in order to motivate arm reaching training for stroke patients. Specifically, arm movements triggered musical feedback that provided information about movement smoothness/jerkiness and speed of reach such that the acceleration of the motion during reaching and returning changed the musical intervals and harmonic progressions presented. Test results with three stroke patients reported in Wallis et al. (2007) suggested the feasibility of such sonification systems, opening new avenues for the application of this system in large-scale studies.

Scholz et al. (2015, 2016) investigated the effectiveness of a musical sonification therapy protocol to train gross motor function of upper extremities. For that, patients' arm movements were sonified in real-time using two inertial sensors placed at

the wrist and upper-arm of the affected side. The 3D-movement data were transformed into sounds so that upward movements resulted in an ascending C major scale, vertical movements into changes in brightness/timbre of the sounds, and sagittal movements into changes in loudness. The final goal of the training was to teach patients to play simple melodies by moving their arm in a 3D-sonification space. Patients received an average of 10 days of musical sonification therapy or a sham sonification training that did not include auditory feedback. The study results indicated that patients in the music group improved in measures of motor function relating to the smoothness of reaching but no significant changes were observed in other arm-function measures. Additionally, findings suggested a reduction of joint pain in a subgroup of patients who presented lower pain scores prior to the commencement of the musical sonification therapy.

Schmitz et al. (2014) tested an expanded concept for sonification in upper-limb stroke rehabilitation which included a mobile sonification system that provided 4D information about arm positions and trajectories as captured by inertial sensors. Specifically, hand position was mapped onto four acoustic parameters: arm velocity was mapped onto amplitude; elevation angle onto frequencies between 133.3 and 266.6 Hz; radial arm amplitude changed the impression of sound brightness; and azimuth angle determined the interaural intensity difference. Test results with seven patients indicated the potential application of this sonification system in larger clinical trials (see Schmitz et al., 2018).

Robertson et al. (2009) investigated the effect of sonification on upper limb movements after stroke. Patients performed a reaching task that involved reciprocal pointing to 9 targets while a sensor fixed to the hand processed online kinematic data and modulated the auditory feedback presented during movement. The study reported that the sonification had a positive effect on movement performance such as movement smoothness and trajectory curvature for patients with right hemisphere damage, while it worsened the performance of patients with left hemisphere damage. This result thus suggests that responses to auditory feedback may differ when the side of the lesion after stroke is taken into consideration.

# Key Topic 3: Cognitive Mechanisms and Neural Correlates Underlying Auditory-Motor Coupling

There is robust evidence from multiple lines of inquiry that auditory information has a profound effect on the motor system. Physiological and neuroimaging research has demonstrated that one of the factors underlying this strong interaction is the widely distributed neuroanatomical network connecting the auditory and motor systems at the spinal cord, subcortical and cortical levels (Nayagam et al., 2011; Theunissen and Elie, 2014; Bizley, 2017). For instance, studies investigating reflexive motor responses to sound have described neural pathways formed by descending (efferent) fiber tracts originating in the ventral cochlear nucleus that project bilaterally to sensorimotor tracts in the spinal cord via reticulospinal connections (Rossignol and Melvill Jones, 1976; Huffman and Henson,

1990; Delwaide and Schepens, 1995; Marinovic et al., 2014; Marinovic and Tresilian, 2016). Neuroimaging research has also identified rich neuroanatomical interconnectivity between several distant cortical and subcortical brain areas, including the cerebellum, basal ganglia, thalamus, supplementary motor area (SMA) and pre-SMA, premotor cortex, and the auditory cortex (for review, see Teki et al., 2012; Chauvigné et al., 2014; Merchant et al., 2015; Lusk et al., 2016; Petter et al., 2016; Braun Janzen and Thaut, 2018; Koshimori and Thaut, 2018). Specifically, corticocerebellar networks have been shown to be predominantly engaged in movement synchronization to externally cued stimuli (Buhusi and Meck, 2005; Brown et al., 2006; Del Olmo et al., 2007; Thaut et al., 2008, 2009; Witt et al., 2008; Manto et al., 2012; Chauvigné et al., 2014), whereas basal ganglia-thalamo-cortical networks seem particularly involved in beat-based timing and self-paced or internally driven rhythmic movements (Halsband et al., 1993; Cunnington et al., 1996; Rao et al., 1997; Grahn and Rowe, 2009, 2013). Furthermore, recently emerging evidence also indicate that auditory and motor areas have direct routes of communication at cortical level via the arcuate fascicle, a white matter fiber tract with direct projections from the auditory cortex to motor areas, including primary motor cortex and premotor cortex (Fernández-Miranda et al., 2015; Wang et al., 2016).

Another crucial aspect is that the functional and structural architecture of the auditory system is built to rapidly detect temporal patterns of periodicity in acoustic signals. There is considerable evidence that the temporal resolution of the auditory system is superior to other sensory modalities (e.g., Repp and Penel, 2002, 2004; Grondin and McAuley, 2009; Shelton and Kumar, 2010; Grahn et al., 2011; Stauffer et al., 2012; Ammirante et al., 2016). Recent electrophysiological research has demonstrated that the temporal information of acoustic signals is highly preserved at all levels of the auditory processing stream and elicit a periodic neural response at the exact same frequency of the stimuli (for review, see Nozaradan, 2014). Moreover, listening to auditory rhythmic stimuli primes the motor system, increasing the neural efficiency of the motor cortex through a process of auditory-motor entrainment (Crasta et al., 2018). That is, the firing rates of auditory neurons triggered by auditory rhythmic information, such as the beat of the music or a metronome, entrains the firing patterns of neurons in the motor cortex. The oscillatory coupling of neural impulses in the cortical loop between auditory and motor regions generates temporal predictions that are crucial for the perception of, and entrainment to, auditory rhythms (Large and Snyder, 2009; Fujioka et al., 2012; Large et al., 2015; Merchant et al., 2015; Ross et al., 2016, 2017; Morillon and Baillet, 2017). Therefore, the continuous time reference of the rhythmic auditory cues provides predictable feedforward information that allows movement anticipation and motor preparation (Thaut et al., 2015). Additionally, it has been shown that external rhythmic auditory input also changes the pattern of muscle activation through changes in corticospinal excitability (Thaut et al., 1992, 1999; Miller et al., 1996; Wilson and Davey, 2002; Stupacher et al., 2013), modulates beta (β) brain oscillations (Fujioka et al., 2012; Merchant et al., 2015; Ross et al., 2016, 2017), and promotes neural-plasticity (Luft et al., 2004). Collectively, these findings provide strong evidence of the neurobiological mechanisms underlying the effects of RAS on motor planning and execution.

The use of real-time movement information extends the benefits of discrete rhythmic auditory stimuli by adding an auditory component to the movement cycle either with natural movement sounds or movement sonification (Effenberg, 2005; Sigrist et al., 2013; Effenberg et al., 2016; Bevilacqua et al., 2016; Dyer et al., 2017a). Robust evidence suggests that merely listening to action-related sounds activates the neural processes necessary to produce those sounds (e.g., Aziz-Zadeh et al., 2004; Lewis et al., 2005; Pizzamiglio et al., 2005; Aziz-Zadeh et al., 2006; Gazzola et al., 2006; Caetano et al., 2007; Pazzaglia et al., 2008; Alaerts et al., 2009; Engel et al., 2009; Ticini et al., 2012; reviewed in Aglioti and Pazzaglia, 2010). Kohler et al. (2002) provided the first empirical evidence that premotor neurons in monkeys respond to the sound of a familiar action, expanding the notion that movements and their perceptual consequences are intrinsically coupled in the brain (Fadiga et al., 1995; Rizzolatti and Craighero, 2004; Schütz-Bosbach and Prinz, 2007; Rizzolatti and Sinigaglia, 2010). In humans, research shows that acoustic information are sufficient to evoke accurate representations of complex movements (Repp and Knoblich, 2004; van der Zwan et al., 2009; Lewis et al., 2011; Murgia et al., 2012a; Sevdalis and Keller, 2014; Kennel et al., 2014a; reviewed in Pizzera and Hohmann, 2015), activating superior and medial posterior temporal regions involved in human motion recognition (Bidet-Caulet et al., 2005; Baumann and Greenlee, 2006; Saarela and Hari, 2008; Scheef et al., 2009; Schmitz et al., 2013). Importantly, motor resonance is associated with and strengthened by one's experience and familiarity with the actions observed/perceived, as demonstrated by studies comparing expert and novice responses to specific sports- or dance-related sounds (e.g., Agostini et al., 2004; Hohmann et al., 2011; Tomeo et al., 2012; Woods et al., 2014; Murgia et al., 2017). Further evidence of the role of learning and expertise has been provided by research showing that a network comprising areas such as dorsolateral and inferior frontal cortex (including Broca's area), superior temporal gyrus, and motor areas including supplementary motor and premotor areas, is engaged when experienced musicians listen to wellrehearsed music (Haueisen and Knösche, 2001; Bangert et al., 2006; D'Ausilio et al., 2006; Harris and De Jong, 2014; see also Proverbio et al., 2014) or watch silent video recordings of known music pieces (Lotze et al., 2003; Hasegawa et al., 2004; Baumann et al., 2007; Bianco et al., 2016; reviewed in Maes et al., 2014; Novembre and Keller, 2014). Activation of this network was also found when non-musicians listened to a music piece they had learned to play after a short period of training (Lahav et al., 2007; see also Bangert and Altenmüller, 2003). These findings thus suggest that strong auditory-motor associations are developed during sound-making experiences, providing support for the use of real-time auditory feedback to enhance sensorimotor representations and facilitate movement (re)-acquisition.

It is also thought that the continuous availability of information provided by mapping different dynamic or kinematic movement parameters onto distinct sound

components (e.g., pitch, loudness, rhythm, timbre) improves movement quality and motor (re)learning through the integration of multiple congruent perceptual streams (Scholz et al., 2015; Effenberg and Schmitz, 2018; Ghai et al., 2018c), resulting in a richer and more effective internal representation of the movement (Shams and Seitz, 2008; Wolpert et al., 2011; Schmitz et al., 2013; Effenberg et al., 2016). Furthermore, the availability of real-time auditory feedback also enhances online error-correction mechanisms (Dyer et al., 2015; Hossner et al., 2015; Sigrist et al., 2015; van Vugt and Tillmann, 2015), increases cognitive-emotional functioning (Van Vugt et al., 2014; Altenmüller and Schlaug, 2015; Sihvonen et al., 2017), and may supplement perceptual deficits (Tinazzi et al., 2002; van Vugt and Tillmann, 2015; Danna and Velay, 2017; Ghai et al., 2018c).

# DISCUSSION

The studies here reviewed examined the relationship between sound and movement in the context of sports training and movement rehabilitation. Our narrative synthesis focused specifically on the literature regarding the effect of natural movement sounds, movement sonification, and rhythmic auditory information. The current state of knowledge here summarized provides promising evidence of the effect of auditory information on sporting performance and motor (re)learning.

The availability of auditory information in the form of natural sounds occurring as a byproduct of a movement or as additional real-time acoustic feedback driven by movement dynamic or kinematic parameters has significant implications for motor execution and control of skilled performances. The large body of research here reviewed indicates that auditory information provides crucial information about agency (Murgia et al., 2012a; Kennel et al., 2014a,b), movement control and timing (e.g., Chollet et al., 1988, 1992; Schaffert and Mattes, 2011, 2016; Sigrist et al., 2016; Schaffert et al., 2017), movement execution (e.g., Agostini et al., 2004; Konttinen et al., 2004; Baudry et al., 2006; Murgia et al., 2012b; Yang and Hunt, 2013, 2015; Kennel et al., 2015), and performance error/deviation (e.g., Höner et al., 2004; Godbout and Boyd, 2010; Wolf et al., 2011; Godbout et al., 2014; Ramezanzade et al., 2014; Tarnas and Schaffert, 2017). Behavioral data also suggest that the auditory-motor coupling generated during extensive training significantly interacts with athletes' internal motor simulation (Murgia et al., 2012a; Kennel et al., 2014a,b; Pizzera et al., 2017), as shown by studies demonstrating that skilled athletes are able to extract highly specific information from actionrelated sounds (e.g., Roberts et al., 2005) and predict another person's movements based on action prediction mechanisms (e.g., Cesari et al., 2014; Camponogara et al., 2017; Allerdissen et al., 2017). These findings corroborate a robust body of neuroimaging and neurophysiological studies indicating that the mirror neuron system and a widely distributed neuroanatomical network is involved in the processing of action sounds (e.g., Fadiga et al., 1995; Kohler et al., 2002; Aziz-Zadeh et al., 2004, 2006; Bidet-Caulet et al., 2005; Lewis et al., 2005; Pizzamiglio et al., 2005; Pazzaglia et al., 2008; Ticini et al., 2012; Schmitz et al., 2013).

Studies also demonstrated positive effects of auditory information on motor (re)learning in sports and rehabilitation. Research findings revealed that real-time auditory feedback facilitates learning and improves retention of new motor skills (e.g., Chollet et al., 1992; Konttinen et al., 2004; Baudry et al., 2006; Ramezanzade et al., 2014; Schaffert and Mattes, 2014; Pizzera et al., 2017). There is growing support for the application of movement sonification to increase upper-limb functions after stroke (e.g., Wallis et al., 2007; Immoos et al., 2013; Schmitz et al., 2014, 2018; Scholz et al., 2015, 2016; Ghai, 2018), and to improve gait in PD patients using, for instance, instrumented footwear (e.g., Batavia et al., 2001; Rodger et al., 2014; Horsak et al., 2016; Maculewicz et al., 2016; Gorgas et al., 2017). These sonification approaches rely on the transformation of dynamic and kinematic movement parameters onto distinct sound components (e.g., pitch, loudness, rhythm, timbre) to increase cross-modal stimulation (Scholz et al., 2015, 2016; Ghai et al., 2018c) and sensorimotor representation of the movement to be (re)learned (Shams and Seitz, 2008; Schmitz et al., 2013; Effenberg et al., 2016).

On the other hand, another line of clinical studies summarized in this review focuses primarily on the rhythmic patterns of sound, making use of metronome or beat-enhanced music to facilitate rehabilitation of intrinsically rhythmical movements (Thaut, 2005; Thaut and Hoemberg, 2014; Murgia et al., 2015; Ghai et al., 2018b). This robust body of research evidence indicates that RAS has immediate effects on gait velocity, step cadence, and stride length (e.g., Thaut et al., 1996; McIntosh et al., 1997; Freedland et al., 2002; Nieuwboer et al., 2007; Arias and Cudeiro, 2008, 2010; Hove et al., 2012; Song et al., 2015; Pau et al., 2016), reducing gait variability (Miller et al., 1996) and improving walking stability in PD (Thaut et al., 1999; Hausdorff et al., 2007; Hove et al., 2012) and stroke (Thaut et al., 1993, 1997, 2007; Hayden et al., 2009; for review see Thaut and Abiru, 2010; Hollands et al., 2012; Thaut and McIntosh, 2014; Nascimento et al., 2015; Yoo and Kim, 2016). Studies have also demonstrated that auditory cueing significantly improves upperlimb function after stroke by reducing movement variability and reliance on compensatory movements (e.g., Whitall et al., 2000; Thaut et al., 2002a,b; Luft et al., 2004; Jeong and Kim, 2007; Malcolm et al., 2009; Chen et al., 2016). It has been proposed that the continuous time reference provided by the rhythmic auditory cues facilitates movement retraining by priming the motor system, allowing movement anticipation and motor preparation (Thaut et al., 2015), and potentially bypassing damaged areas through the activation of alternative pathways (Hoemberg, 2005; Dalla Bella et al., 2017; Braunlich et al., 2018).

We also identified a small number of studies that evaluated other variables influencing the effect of auditory information on motor performance, such as physiological arousal and motivation (Murgia et al., 2012b; Bood et al., 2013; Immoos et al., 2013; Pugliese and Takala, 2015; Scholz et al., 2016; Sanderson and Hunt, 2016; Newbold et al., 2017). Murgia et al. (2012b) found that athletes' maintained peak performance and reduced performance variability in trials

where high-intensity sounds were presented during the pressing phase of weightlifting exercises, and Bood et al. (2013) reported changes in psychophysical and physiological outcome measures due to the motivational aspects of the stimuli during running. Novel therapeutic approaches, such as musical sonification (Scholz et al., 2016), also considered the motivational aspects of adding real-time auditory feedback to stimulate patients and improve treatment compliance, thus opening new avenues to systematically examine the role of physiological arousal, motivation, reward, and mood in larger clinical trials. The potential use of interactive sonification systems in sports and rehabilitation has motivated researchers and engineers to develop applications and system prototypes for exercise and physical activity, rehabilitation, and entertainment (e.g., Barrass et al., 2010; Lécuyer et al., 2011; Franco et al., 2013; Bruckner et al., 2014; Contreras Lopez et al., 2014; Pugliese and Takala, 2015; Newbold et al., 2017; see **Supplementary Material**). These studies explore a wide range of devices and applications where the playful character of music or the competitive component of sports has inspired new technology-enabled forms of play (e.g., exertion or computer games) and therapy. Future applications of this technology in sports, recreation, and rehabilitation are yet to be fully explored.

The large body of literature here reviewed clearly shows an emerging area of clinical and applied research. However, there are important research gaps that need to be addressed in future research. Firstly, there is a clear need to better understand what auditory components and amount of information are most relevant in motor training and rehabilitation. This is not trivial, particularly in sonification applications, as research suggests that an overload of auditory information has detrimental effects on task performance (e.g., Wolf et al., 2011) and that taskirrelevant auditory stimuli are strong distractors (Parmentier, 2014). The use of meaningful auditory information is, therefore, determinant for the user's experience (Effenberg et al., 2016; Dyer et al., 2017b) and needs to be considered in a clear framework for sonification mapping derived from a better understanding of the processes underlying motor learning/control from a basic research perspective (Dyer et al., 2015). Research in the field of auditory information processing has great potential to promote active crosstalk between basic and applied research, with findings generated in the laboratory providing insights for the application in real-life situations, that being in sports training or therapy and rehabilitation, and vice-versa. Secondly, we have identified few studies using natural movement sounds or sonification in elite or high-performance sports. A challenge

## REFERENCES


for future investigations is to evaluate novel applications in ecologically valid and real-life situations that closely resemble the athlete's movement technique and training conditions in order to better identify what type of information is most relevant and improve equipment setup, thus acquiring more reliable results. This depends directly on the development of procedures that are feasible for the systematic use in daily training. In addition, future research should also consider the way in which auditory information is presented to athletes and patients (loudspeaker vs. earplugs) in order to avoid, for instance, perceptual overload, and to ensure that the feedback information is properly delivered. From the clinical perspective, although there is growing attention on the application of sonification systems to improve motor function in PD and post-stroke, we have identified a relatively small number of controlled trials, revealing the need to further examine the effectiveness and feasibility of sonification methods and devices in larger controlled clinical studies.

# CONCLUSION

This review examined the relationship between sound and movement in the context of sports training and movement rehabilitation. The findings here summarized provide evidence of the effect of natural movement sounds, movement sonification, and rhythmic auditory information on sporting performance and motor (re)learning. This emerging area of clinical and applied research demonstrates large underutilized potential, warranting further investigation of the promising application of auditory feedback information in sports and rehabilitation.

# AUTHOR CONTRIBUTIONS

All authors listed have made a substantial intellectual contribution to the work and approved it for publication. NS conceived and drafted the first version of the manuscript. TBJ contributed to the writing and revision of the manuscript. MT and KM supervised and revised the manuscript.

# SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg. 2019.00244/full#supplementary-material




entrainment. Neuropsychologia 117, 102–112. doi: 10.1016/j.neuropsychologia. 2018.05.017









**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Schaffert, Braun Janzen, Mattes and Thaut. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Haptic Error Modulation Outperforms Visual Error Amplification When Learning a Modified Gait Pattern

Laura Marchal-Crespo1,2 \*, Panagiotis Tsangaridis<sup>2</sup> , David Obwegeser<sup>2</sup> , Serena Maggioni2,3,4 and Robert Riener2,3

<sup>1</sup> Gerontechnology and Rehabilitation Group, ARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland, <sup>2</sup> Sensory-Motor Systems (SMS) Lab, Institute of Robotics and Intelligent Systems (IRIS), Department of Health Sciences and Technology (D-HEST), ETH Zürich, Zurich, Switzerland, <sup>3</sup> Reharobotics Group, Spinal Cord Injury Center, Balgrist University Hospital, Medical Faculty, University of Zurich, Zurich, Switzerland, <sup>4</sup> Hocoma AG, Volketswil, Switzerland

#### Edited by:

Mauro Murgia, University of Trieste, Italy

#### Reviewed by:

Herbert Heuer, Leibniz Research Centre for Working Environment and Human Factors (IfADo), Germany Reinoud J. Bootsma, Aix-Marseille Université, France

#### \*Correspondence:

Laura Marchal-Crespo laura.marchal@artorg.unibe.ch; laura.marchal@hest.ethz.ch

#### Specialty section:

This article was submitted to Perception Science, a section of the journal Frontiers in Neuroscience

Received: 23 May 2018 Accepted: 21 January 2019 Published: 19 February 2019

#### Citation:

Marchal-Crespo L, Tsangaridis P, Obwegeser D, Maggioni S and Riener R (2019) Haptic Error Modulation Outperforms Visual Error Amplification When Learning a Modified Gait Pattern. Front. Neurosci. 13:61. doi: 10.3389/fnins.2019.00061 Robotic algorithms that augment movement errors have been proposed as promising training strategies to enhance motor learning and neurorehabilitation. However, most research effort has focused on rehabilitation of upper limbs, probably because large movement errors are especially dangerous during gait training, as they might result in stumbling and falling. Furthermore, systematic large movement errors might limit the participants' motivation during training. In this study, we investigated the effect of training with novel error modulating strategies, which guarantee a safe training environment, on motivation and learning of a modified asymmetric gait pattern. Thirty healthy young participants walked in the exoskeletal robotic system Lokomat while performing a foot target-tracking task, which required an increased hip and knee flexion in the dominant leg. Learning the asymmetric gait pattern with three different strategies was evaluated: (i) No disturbance: no robot disturbance/guidance was applied, (ii) haptic error amplification: unsafe and discouraging large errors were limited with haptic guidance, while haptic error amplification enhanced awareness of small errors relevant for learning, and (iii) visual error amplification: visually observed errors were amplified in a virtual reality environment. We also evaluated whether increasing the movement variability during training by adding randomly varying haptic disturbances on top of the other training strategies further enhances learning. We analyzed participants' motor performance and self-reported intrinsic motivation before, during and after training. We found that training with the novel haptic error amplification strategy did not hamper motor adaptation and enhanced transfer of the practiced asymmetric gait pattern to free walking. Training with visual error amplification, on the other hand, increased errors during training and hampered motor learning. Participants who trained with visual error amplification also reported a reduced perceived competence. Adding haptic disturbance increased the movement variability during training, but did not have a significant effect on motor adaptation, probably because training with haptic disturbance on top of visual and haptic error amplification decreased the participants' feelings of competence. The

**198**

proposed novel haptic error modulating controller that amplifies small task-relevant errors while limiting large errors outperformed visual error augmentation and might provide a promising framework to improve robotic gait training outcomes in neurological patients.

Keywords: motor learning, motor adaptation, haptic guidance, error amplification, force disturbance, visual feedback, robotic gait-training, rehabilitation robotics

# INTRODUCTION

The interest in using robotic devices to provide more intensive and cost-effective gait training has increased during the last years (Marchal-Crespo and Reinkensmeyer, 2009). During robotic gait training, patients are physically assisted by a robotic device in order to move their legs into a physiological gait pattern (Marchal-Crespo and Riener, 2018). Robotic gait training has the potential to increase the training intensity while keeping patients in a safe and enjoyable environment (e.g., by using virtual reality) (Brütsch et al., 2010; Donati et al., 2016). However, robot-guided movements might, in some cases, decrease patients' physical and mental effort during training (Israel et al., 2006). This could explain the limited functional gains observed after robotic gait training up to date (Dobkin and Duncan, 2012; Mehrholz et al., 2017).

It is generally accepted in the field of neurorehabilitation that recovery is a form of motor learning (Krakauer, 2006), and that understanding the underlying mechanisms during motor learning may facilitate the design of novel strategies to improve neurorehabilitation (Dietz and Ward, 2015). Active participation is thought to be an essential driving factor to elicit motor plasticity (Lotze et al., 2003; Behrman et al., 2006). Therefore, robotic rehabilitation could potentially hamper recovery if it promotes a decrease in cognitive and physical effort during training (Scheidt et al., 2000). "Challenge-based" controllers have been proposed in order to promote trainees' participation. These challenging controllers, unlike guiding controllers that reduce errors during movement training, make motor tasks more difficult or challenging to perform (Marchal-Crespo and Reinkensmeyer, 2009).

Challenging controllers are based on the motor learning research that state that errors are fundamental signals to drive motor learning (Emken and Reinkensmeyer, 2005; Reisman et al., 2013). There is evidence that amplifying trajectory errors during walking using robotic forces accelerates the adaptive processes in healthy participants (Emken and Reinkensmeyer, 2005). Training with error amplification also enhanced learning of a complex locomotor task in initially more skilled healthy participants (Marchal-Crespo et al., 2017b). Error amplification during locomotion training resulted in more robust aftereffects than assistive training (Yen et al., 2012). However, only few studies have tested for long-term retention (Helm and Reisman, 2015), and therefore, conclusions on the effect of error augmentation on motor learning of locomotor tasks should be taken cautiously. Furthermore, there are also studies that found that challenge-based controllers have a negative effect on participants' motivation (Duarte and Reinkensmeyer, 2015), suggesting that error amplification might limit motor learning if it increases participants' frustration during training.

Errors can also be visually augmented (i.e., the presented error on the display is distorted). In a relatively recent study, participants were asked to perform planar point-topoint reaching movements under a visuomotor rotation while holding the handle of a robotic device. Their arms were hidden by a screen showing them the reference trajectory as well as their current position, which was distorted in the experimental groups. The groups that had visual error amplification resulted in better learning outcomes than those who trained without augmented errors (Patton et al., 2013). Research in visual error amplification is quite recent. Exploration of visual error amplification has mainly focused on the upper limbs (Brewer et al., 2008; Celik et al., 2009; Patton et al., 2013; Basalp et al., 2016), although recent work has started to explore the possibility of using visual distortions on gait rehabilitation (Tobar et al., 2018). The use of visual error amplification is attractive, because it does not apply forces, and therefore, it does not create potential unsafe environments. Furthermore, it involves the use of virtual reality (VR), which has been shown to increase motivation and active participation during rehabilitation (Zimmerli et al., 2013; Bergmann et al., 2017). Including visual feedback (i.e., a VR representation of the desired and actual trajectory of the participants' ankle), during training with a patient-cooperative minimally assistive robotic controller using the gait rehabilitation Lokomat (Hocoma AG, Switzerland), enhanced motor adaptation of a new gait pattern and resulted in improvements in locomotor function in stroke patients (Krishnan et al., 2012, 2013).

A recent study found that participants with more variable movements during baseline could more rapidly adapt to a perturbation and learn a new skill than participants with low movement variability (Wu et al., 2014). This is in line with recent research that states that during the first stages of learning, error exploration (i.e., the active exploration of new motor tasks) is crucial to boost motor learning (Huberdeau et al., 2015). Therefore, increasing movement variability during training might result in better motor learning. A possible approach to increase movement variability is to apply randomly varying feedforward forces (i.e., haptic disturbance) during training (Rüdt et al., 2016). In a motor learning study on upper limbs, adding haptic disturbance while training a tracking task resulted in better tracking skills than training with haptic error augmentation and training without disturbances (Lee and Choi, 2010). We recently found that adding random haptic disturbance during training a locomotion task increased muscle activation

and seemed to enhance attention and motor learning (Marchal-Crespo et al., 2014a,b, 2017b).

Haptic guidance seems to be particularly helpful for initially less skilled participants (Marchal-Crespo et al., 2010), while error amplification was found to be more beneficial for more skilled participants (Cesqui et al., 2008; Milot et al., 2010). This is in line with the challenge point theory, which states that optimal learning is achieved when the difficulty of the task is appropriate for the participant's level of expertise (Guadagnoli and Lee, 2004). Therefore, matching the robotic training strategy to the trainee's skill level may provide the greatest opportunity for learning (Metzger et al., 2014).

Motivation has been suggested to play a key role during motor learning and neurorehabilitation (Reinkensmeyer and Housman, 2007; Novak et al., 2014). Several studies have shown that increasing participants' perceived competence and intrinsic motivation during training can enhance the acquisition of new motor skills (Ávila et al., 2012; Saemi et al., 2012; Widmer et al., 2016). Motivation may, in some training situations –e.g., when it is associated with high reward– improve learning consolidation (Trempe et al., 2012). The close relationship between task difficulty and motivation has been extensively studied since the early 20-th century [e.g., difficulty law of motivation (Ach, 1935)]. According to the Flow theory and Self-determination theory, the maximum intrinsic motivation is achieved when the difficulty of the task optimally challenges the participant. It has been suggested that the relationship between perceived task difficulty and motivation follows an inverted U-shaped function (Ma et al., 2017). The optimal level (i.e., the apex of the curve, named flow) is described as "an intrinsically motivating and fully engaging state of consciousness" (Csikszentmihalyi, 2014). Interestingly, recent studies have shown that participants report higher levels of intrinsic motivation when they slightly underperform in the tasks (the so-called 'close missing'), compared to performing perfectly (boredom channel) or far worse than required (anxiety channel) (Abuhamdeh et al., 2015; Ma et al., 2017). Therefore, although reward might enhance motor skill learning, performing systematically well might decrease motivation, compared to closely missing the target.

An important concept in error-based motor learning theory is the idea that participants must explore the task by themselves (exploration) and exploit current reliable knowledge (exploitation). These two processes are often considered to be antagonistic (the so-called 'exploration-exploitation tradeoff'). However, this tradeoff might be bypassed using robotic devices. An optimal framework for motor learning might consist in limiting unsafe and frustrating large errors, which might result in participants stumbling or falling, by using robotic haptic guidance (i.e., favoring exploitation), while augmenting movement variability and awareness of small learning-relevant errors by using error amplification and random haptic disturbance (i.e., enforcing exploration). This optimal framework might influence not only the exploration and exploitation processes that have a direct effect on motor learning, but might also increase motivation. Bringing participants to practice in an area close to the flow apex (in the close missing area) might increase participants' motivation and enjoyment.

In this experiment, we investigated the effect of training with novel visual and haptic error modulating strategies on motivation and learning of a modified gait pattern. Learning with three different error modulating strategies was evaluated: (i) no disturbance: no robot disturbance/guidance was applied, (ii) haptic error amplification: unsafe and frustrating large errors were limited with haptic guidance, while haptic error amplification enhanced awareness of task-relevant errors, and (iii) visual error amplification: visually perceived errors were amplified in a virtual reality environment. We also evaluated whether increasing the movement variability during training by adding randomly varying haptic disturbances on top of the other training strategies further enhanced learning. Thirty healthy young participants walked in the robotic gait trainer Lokomat while performing an ankle target-tracking task which required an increased hip and knee flexion in the dominant leg. We analyzed participants' motor performance and self-reported intrinsic motivation before, during and after training. We hypothesized that training with error amplification, either visually or haptically, would result in better motor learning than training without error amplification. We hypothesized that limiting large errors during haptic error amplification would increase self-perceived competence, enjoyment and, therefore, motivation. Finally, we expected that adding haptic disturbance on top of the other training strategies would increase movement variability during training and would further enhance motor learning.

# MATERIALS AND METHODS

#### Lokomat

Even though the error modulating controllers presented in this paper are applicable to different robotic gait-training systems, the presented experiment was performed with the Lokomat <sup>R</sup> (Hocoma AG, Switzerland). The Lokomat is a commercial available robotic gait trainer that consist of two leg orthoses, a body weight support system and a treadmill (**Figure 1**) (Riener et al., 2010). Each orthosis can induce flexion and extension movements in the hip and knee joints in the sagittal plane through linear drives. Ankle dorsiflexion during the swing phase can be supported through passive foot lifters. Each leg orthosis is fixed to a frame that allows for passive vertical translations and keeps the orientation of the pelvis segment constant. A sophisticated combination of passive elastic and active dynamic systems, the "Lokolift," allows for a constant unloading of patient's weight during treadmill walking (Frey et al., 2006).

## Experimental Task

The experimental task consisted in tracking with the dominant ankle a desired trajectory presented on a visual display, while the non-dominant leg was fully guided by the robot. Participants' legs were visually displayed as an avatar on a large screen in front of them (**Figure 1**). Participants were requested to track a blue dot (reference position), which moved along the reference trajectory, with the ankle of their dominant leg. In order to facilitate the task, an orange dot indicated the participant's dominant ankle actual

FIGURE 2 | Example of the ankle trajectories that resulted from applying forward kinematic analysis to a participant's average hip and knee joints (θave), the Lokomat original (θLok) and adjusted Lokomat references (θ<sup>ˆ</sup> Lok), and the final reference trajectory (θref) in the non-dominant (A), and dominant legs (B). The final trajectory is the result of increasing the hip and knee angle ROM of the dominant leg by 20%. The final reference ankle trajectory was shown on the screen (C) together with the actual ankle position and an avatar representation of the legs (with dominant leg on top).

position, and an orange trace showed the path followed by the ankle for a certain time (**Figure 2C**).

The target trajectory corresponded to a gait pattern that required a 20% increase in the dominant leg's average hip and knee joint angular range of motion (ROM). A similar gait pattern was successfully employed in previous experiments performed with the Lokomat (Krishnan et al., 2013; Ranganathan et al., 2016). The individual average hip and knee angles across the gait cycle for both legs for each participant (θave) were calculated after a 2 min calibration test, where participants freely walk in the Lokomat without any haptic guidance/disturbance. The recorded time series were then split into single steps. The start of every step was defined as the point when the left hip joint angle reached a maximum. The default joint references loaded in the Lokomat (θLok) are based on pre-recorded standard joint trajectories (Colombo et al., 2000). These trajectories can be manually modified to better fit the participant's particular gait pattern by changing the Gain and Offset parameters. We developed an algorithm that calculates the Gain and Offset parameters that better fit the pre-recorded Lokomat joint references to each participant' specific gait pattern.

The Gain is calculated for each joint and leg independently by dividing the ROM (i.e., the difference between the maximum and minimum joint angles) of the measured average joint trajectory (ROMave) over the ROM of the Lokomat reference joint trajectory (ROMLok).

$$Gain = \frac{ROM\_{\text{ave}}}{ROM\_{\text{Lok}}} \tag{1}$$

The Offset was calculated by computing the average deviation between the average joint trajectories (θave) and the Lokomat default trajectory (θLok) multiplied by the Gain (calculated with Eq. 1).

$$\text{Offset} = \frac{\sum\_{i=1}^{m} \left(\theta\_{\text{ave},i} - \theta\_{\text{Lok},i} \bullet \text{Gain}\right)}{\text{m}} \tag{2}$$

where m is the number of data points in each gait cycle (m = 250). The fitted Lokomat reference joint trajectories for each joint (θˆ Lok) are then calculated as:

$$\hat{\theta}\_{\text{Lok}} = \left( \text{Gain} \bullet \theta\_{\text{Lok}} + \text{Offset} \right) \tag{3}$$

We computed the final desired joint trajectories by increasing the gain of the fitted Lokomat reference θˆ Lok by 20% in the dominant leg. The desired and actual ankle trajectories presented on the visual display were then calculated by employing forward kinematics analysis of the hip and knee joint angles (θˆ Lok/hip, <sup>θ</sup><sup>ˆ</sup> Lok/knee), and the measured segment lengths of the thighs and shanks lthigh, lshank for each participant (**Figure 2**).

#### Training Strategies

We developed new training strategies to haptically or visually modulate movement errors in order to enhance motivation and learning of a modified gait pattern. The design and evaluation of the haptic disturbance and error modulating haptic controllers for the Lokomat were described in detail in Rüdt et al. (2016). Here, only a brief summary is given for completeness. Similar haptic disturbance (Marchal-Crespo et al., 2017a) and error modulating controllers (Marchal-Crespo et al., 2017c) were developed to perform motor learning experiments with ARMin, a 7 degree-of-freedom (DoF) robotic exoskeleton for upper limb rehabilitation. The haptic controllers employed in the present experiment, however, were developed in joint coordinates and were not based on the end-effector trajectories. The visual error amplification in joint coordinates was newly developed for the current experiment.

#### No Disturbance

In no-disturbance mode, the robotic device does not help nor disturb the participants during walking. The robot works with zero-torque control, in such a way that the interaction torques between robot and human are minimized by letting the robot follow the participant's self-selected movements (Riener et al., 2005). Friction and gravity are compensated to improve the transparency of the orthoses.

#### Error Modulating Haptic Controller

In order to haptically augment errors, we developed a controller that provides amplifying torques (Tamp) that direct the joint angles away from the desired position (Rüdt et al., 2016). These amplifying torques are calculated using a proportional controller in joint space of the form:

$$T\_{\rm amp} = \lambda\_{\rm amp} \bullet (\theta\_{\rm act} - \theta\_{\rm ref}) = \lambda\_{\rm amp} \bullet e \tag{4}$$

where θref is the reference joint angle, and θact is the measured angle (θhip or θknee). With this formula, the error amplification torques would increase proportionally to the tracking errors. However, participants are able to apply only a certain maximum torque to correct their movements. Therefore, come back to the desired joint position would be challenging if the error is especially large. In order to limit the amount of torque amplification and limit large errors that can be unsafe and frustrating for the participants, we realize a conversion toward haptic guidance when the error is larger than a predefined allowed error (eturn). In this way, the system amplifies small errors (e < eturn) but prevents participants from performing large errors (e > eturn). This is achieved by making the proportional gain λamp a function of the participants' ongoing error (Rüdt et al., 2016):

$$
\lambda\_{\text{amp}} = \lambda\_{\text{max}} \bullet \left( \frac{2}{1 + \exp\left(k \bullet (|e| - e\_{\text{turn}})\right)} - 1 \right) \tag{5}
$$

The impedance gain follows the superposition of two sigmoid functions (**Figure 3**, up). The gain is maximal (λmax) when the error (e) is equal to zero. The controller applies error amplification as far as the error is within the allowed error (eturn) (**Figure 3**, bottom). For larger errors, participants are directed back to the desired position with haptic guidance. The impedance gain saturates for big errors (−λmax). The error range in which the impedance gain remains constant, and the transition between error amplification and haptic guidance around eturn can be tuned using the parameter k. Small values of k result in slow soft transitions between control modes but reduce the width of the impedance gain saturation area (**Figure 3**, up). In order

(e > eturn) with haptic guidance (HG).

to provide a relative soft transition between controllers, while keeping an impedance gain saturation area relatively large, the slope k of the sigmoid was set to 10. In order to allow a certain tracking accuracy without excessive participants' physical effort, a small λmax gain of 25 Nm/◦ for the hip and 5 Nm/◦ for the knee were selected. The predefined allowed error (eturn) was fixed to 8◦ . Participants' legs are attached to the Lokomat through fabric cuffs fastened with Velcro, and therefore, small relative movement between the participants' and robot links can be minimized, but not totally canceled (i.e., keeping the error exactly at zero is hardly possible). In order to avoid amplifying errors that are not directly related to the participants' own performance, errors smaller than a threshold (2◦ ) are not amplified. The error amplification torque is then multiplied by a sigmoid function derived from the safety constraints described in Section "Constraints," and input to a close-loop torque controller.

#### Visual Error Augmentation

The visual error augmentation algorithm displays the participant's actual ankle position farther away from the reference point than it actually is. The visual error amplification algorithm works in joint coordinates to resemble the haptic error amplification algorithm. The hip and knee joints represented by the VR avatar when visual error amplification is applied (θshown, hip, θshown, knee), are calculated as follows:

$$
\begin{bmatrix}
\theta\_{shown,hip} \\
\theta\_{shmm,kne}
\end{bmatrix} = \begin{bmatrix}
\theta\_{act,hip} \\
\theta\_{act,kne}
\end{bmatrix} + \begin{bmatrix}
\theta\_{act,hip} - \theta\_{\text{ref, hip}} \\
\theta\_{act,kne} - \theta\_{\text{ref, knee}}
\end{bmatrix} \bullet \begin{pmatrix}
\alpha\_{\text{amp}} \\
\end{pmatrix} \tag{6}
$$

The visual error amplification gain (αamp) was set to 0.2 (i.e., 20% amplification). This value was selected based on previous experiments that showed that gains bigger than 40% result in participants' confusion when tracking a continuous repetitive trajectory (Basalp et al., 2016). With these new calculated joint angles, forward kinematic analysis was employed to compute the position of the ankle shown on the VR screen.

We added some saturation constraints on the visual amplification algorithm in order to keep the VR avatar realistic. In an obvious case, we prevented the knees from going into hyperextension (θshown, knee ≥ 0 ◦ ). We also saturated the amount of amplified error. In preliminary tests, we observed that an added 15◦ error was the upper limit where a participant could still believe that the movement shown on the VR was his or her own movement.

#### Haptic Disturbance

fnins-13-00061 February 15, 2019 Time: 19:20 # 7

The idea of the random haptic disturbance algorithm is to increase the participants' error variability while training with their main training strategy (i.e., no disturbance, haptic or visual error amplification). By increasing the variability, we seek to push the participants away from their "comfort zone," encouraging the exploration of the task environment (Rüdt et al., 2016).

The haptic disturbance controller generates torque pulses that last for 0.15 s. The occurrence of the pulses is pseudorandomized. Every 0.1 s, a random number is drawn from a standard uniform distribution. If this value is smaller than 0.2, a disturbance is applied. This results in an average pulse frequency of 2 Hz. However, the algorithm also enforces a minimum idle time between pulses of 0.1 s and a maximum of 0.5 s, and therefore, the timing of the pulses cannot be considered totally random. The magnitude of the disturbance torques are drawn from a uniform distribution in the range from −1 to 1. This magnitude is then multiplied by the maximum possible disturbing torque (10 Nm). The Lokomat safety control shuts down the system when a sudden change in the joint angles is detected. Therefore, a rate limiter was implemented to guarantee that the disturbance torque takes 0.05 s to reach its full value, maintains it for 0.05 s, and returns to zero in 0.05 s. The disturbance torques are then multiplied by a sigmoid function –derived from the safety constrains described in next section– and applied simultaneously to the hip and knee joints of the dominant leg on top of the other training strategies.

#### Constraints

The goal of the novel haptic error modulating training strategies is to increase movement variability and kinematic errors in a motivating and safe way. In order to achieve a safe environment for gait training, different constraints were introduced that prevented participants from stumbling (Rüdt et al., 2016).

#### **Constraint 1: no application during stance phase**

The stance phase starts when the heel strikes the ground and last until the toe leaves the ground (i.e., determines the time when the foot is in contact with the ground). The application of haptic error amplification and haptic disturbance during stance might make the leg buckle and result in stumbling. Therefore, the use of these disturbing torques is limited to the swing phase. A swing phase detection algorithm that uses the online measurements of both legs' hip (θhip) and knee (θknee) angles, and the lengths of each participant's thighs (lthigh) and shanks (lshank) was developed (see Rüdt et al., 2016 for more detailed information). This algorithm calculates the vertical distance from the hip joint to the heel (yheel) and the toe (ytoe) for each leg using forward kinematics.

The maximum yheel and ytoedistances for each leg are selected and compared between legs (ydiff). If their difference ydiff is greater than an ad hoc selected threshold (ythreshold = 0.02 m), the leg with the smallest distance is considered to be in swing phase.

#### **Constraint 2: continuous transition**

It is important to guarantee a smooth transition between the application of the haptic error amplification and disturbing torques during swing phase and the disturbance-free stance phase enforced by the first constraint. In order to apply a soft transition between swing and stance phases, we calculate a sigmoid function that changes its value from one to zero depending on the difference between the two legs' vertical distance (ydiff) (Rüdt et al., 2016). The disturbance torques are then multiplied by this sigmoid function in order to limit their magnitude in the swing-stance and stance-swing transitions. In order to preserve participants' safety during training, the constraints are only applied to the haptic error amplification and disturbing torques, but have no effect on the haptic guidance. Haptic guidance would always be applied during the stance phase, if the error is larger than the predefined allowed error (eturn).

## Experimental Protocol

The study was approved by the ethical committee of ETH Zürich and conducted in compliance with the Declaration of Helsinki. Thirty young healthy participants (15 females), 26.0 ± 2.8 years old, gave written informed consent to participate in the study. All participants, except for one, were right footed, as determined by their preferred leg to kick a ball as far as possible (Krishnan and Williams, 2009). Participants were randomly allocated to one of three training groups (Parallel design: no disturbance [Control], haptic error amplification [HEA], and visual error amplification [VEA]). Within these groups, participants were again split into two groups (cross-over design), depending whether they started training with haptic disturbance (HD) added on top of their main training strategy (HD1), or on the contrary, they started training without haptic disturbance (HD2). An overview of the study protocol is depicted in **Figure 4**. Participants were not informed in which group they were allocated but were informed about the possibility that the robot could help or disturb them while executing the task.

Participants were positioned in the Lokomat using the usual Velcro cuffs at the pelvis, thighs and shanks and the length of each robot segment was adjusted to correctly align the hip and knee joints of the exoskeleton with the participants' joints. We employed passive foot lifters to support ankle dorsiflexion during the swing phase. The experiment was performed with a treadmill speed of 1.5 km/h at a pace of around 57 steps per minute and with 30% body weight support, provided through a harness. A relative low speed was selected to ensure that the task was not too challenging for the participants, and to allow them to correct the errors through the step cycle.

The experiment consisted of two experimental days which were 3–4 days apart. On day 1, participants started to freely walk in the Lokomat for 2 min to get used to the robot in no disturbance mode (warm-up). Participants were then requested to freely walk for another 2 min (calibration) in order to determine the ankle target template (see section "Experimental Task"). Participants were verbally instructed to walk as naturally as possible. Once the target trajectory for each joint and each leg was computed, we turned on the VR game and participants were informed that the experiment would start. During baseline (2 min), participants were instructed to follow a blue dot that moved on the target ankle trajectory presented on the screen in front of them, with their dominant leg (signaled with an

orange dot) whenever the visual feedback was available. Their non-dominant leg was fully guided by the robot, while they could freely move their dominant leg.

After a short break (<1 min), participants started the first training block. A training block consisted of: 2 min of training (training 1.1), 2 min of free walking (FW1), and again 2 min of training (training 1.2). The VR game and the participant's specific training controller (Control, HEA or VEA in the dominant leg and full guidance on the non-dominant leg), were active during training, and turned off during free walking. Participants were verbally instructed to walk naturally during the free walking test. Participants performed two training blocks of 6 min each. Half of the participants (HD1) trained with haptic disturbance on top of their specific training strategy during the first training block. Between the first and second training blocks, a 2-min mid-training retention (MTR) test took place. During the second training block (training 2.1, FW2, and training 2.2), participants in the HD2 group trained with haptic disturbance on top of their specific training controller, while participants in HD1 trained without haptic disturbance. After the second training block and a 5-min break, a short-term retention (STR) test was performed. The total experimental time was around 1 h. Participants were invited to return after 3–4 days to perform the long-term retention test (LTR). All the retention tests followed the same structure as baseline.

In total, participants performed the tracking task (with or without error augmentation strategies) during seven time intervals of 2 min each in the first experimental day. This number was selected based on the limited previous experimental results that showed significant performance improvements in a very similar task after performing the tracking task during four training intervals of 2 min each (Krishnan et al., 2013).

Performance in the tracking task saturated after performing the task for more than five training intervals of 2 min each (Ranganathan et al., 2016). Although the previous experimental results assessed performance during training, rather than learning, we hypothesized that the task was relatively easy and could be mastered in a short time (Marchal-Crespo et al., 2010; Winstein et al., 1994).

We assessed participants' subjective experience with the experimental task after baseline, after the first and second training blocks, and after the short and long retention tests (**Figure 4**). We employed six statements (**Table 1**) from the wellestablished Intrinsic Motivation Inventory (IMI, Ryan, 1982). The IMI has been successfully employed in several motor learning experiments to assess intrinsic motivation (Duarte and Reinkensmeyer, 2015; Marchal-Crespo et al., 2017c). The full questionnaire assesses seven motivational subscales (with a total of 45 questions). In the present study, we focused in assessing interest/enjoyment, perceived competence, and effort/importance. Participants ranked their agreement with each of the six statements using a Likert scale between 1 and 7 points; 1 indicated "I disagree completely" and 7 indicated "I agree completely." The questions were presented in German and English. Answers from the same questions at different experimental times were always visible.

#### Data Processing

The knee and hip angles were recorded by the robot at 50 Hz. All data was processed with Matlab (MathWorks, Natick, MA, United States). The recorded angles were smoothed using a moving average filter with a span of five. The actual and reference ankle positions were determined using forward kinematics analysis of the hip and knee joint angles along with segment lengths of the thigh and shank measured for each participant. For the analysis, the time series collected from each participant were segmented into single steps using a heel strike detection algorithm (Bernhardt et al., 2005). The ankle trajectories were then normalized to 250 discrete points via interpolation in order to have equal number of time frames for each gait cycle. In average, during free walking, the first time frame would

TABLE 1 | IMI Questionnaire.


The six statements selected from the Intrinsic Motivation Inventory in order to assess participants' interest/enjoyment, perceived competence, and effort/importance (IMI, Ryan, 1982).

correspond to heel strike (start of stance phase), the pre-swing phase would start around the time frame 100 (40% of the gait cycle) and swing phase would start around the time frame 150 (60% of gait cycle) (see **Supplementary Figure A1**). In order to avoid transitory effects (i.e., participants needed time to synchronize their gait with the robot), the first five steps recorded for each participant during a training or retention test were removed.

Different variables were extracted from the ankle position to evaluate the participants' spatio-temporal performance (error) and movement consistency (variability). The tracking error (ei,t) in each time frame (t) of a gait cycle (i) was obtained by calculating the absolute distance between the actual and the desired ankle position at each specific time frame (250 discrete points per gait cycle). The average trajectory tracking error is then calculated by averaging the tracking error in each time frame over all gait cycles. The experimental task consisted in tracking a desired position over time, therefore, the tracking error includes timing and spatial mismatches between desired and actual positions.

We also evaluated the spatial errors using dynamic time warping (DTW) with the weighting of the temporal shift set to zero (Giese and Poggio, 2000). The spatial error provides important information regarding how close was the performed ankle trajectory to the desired ankle reference trajectory. This information is valuable to assess whether participants learned to perform the asymmetric desired trajectory, contrary to the tracking error, which is employed to assess how precise were the participants in tracking the desired ankle movement at each time frame. The spatial error in each time frame of the gait cycle was obtained using the MATLAB built-in function dtw. During DTW, the total distance between the two temporal sequences is computed as the minimum sum of the Euclidean distances between the column vectors of these sequences. The reader is referred to (Giese and Poggio, 2000) for a detailed description of the DTW algorithm. In our case, column vectors represent the time frame during the gait cycle (a total of 250 time frames, where the first value corresponds to heel strike), while row vectors are the Cartesian coordinates (x and y) of the ankle's position. Hence, we extracted the spatial error at each time frame (t) of a gait cycle (i) by comparing the measured trajectory of each step with the corresponding reference trajectory.

In order to investigate whether adding haptic disturbance increased the movement variability, we also calculated the variability of the spatial error. The variability is defined as the average trajectory spatial error from each step to each other step. The trajectory spatial error of each step to each other step is calculated using the dtw MATLAB function, creating an n × n symmetric matrix. The trajectory variability is then obtained by calculating the average trajectory of one half of the matrix.

In order to evaluate whether training the asymmetric gait pattern modified the gait pattern during free walking tests (transfer), we calculated the asymmetry between trained and untrained legs for the hip and knee joints during calibration and free walking blocks (FW1 and FW2). The asymmetry performance metric was defined as the percentage difference between the ROM of the trained and untrained leg

joints. We calculated the asymmetry for the hip and knee joints independently.

$$Asymmetry\_{\text{hip}/\text{knee}} = \frac{ROM\_{\text{training hip}/\text{knee}} - ROI\_{\text{untrained hip}/\text{knee}}}{ROM\_{\text{untrained hip}/\text{knee}}}$$

(7) Positive asymmetry values imply bigger ROM in the trained joints, while big divergence from zero indicates high asymmetry between the legs. The asymmetry variable is a discrete value calculated for each single step. In order to evaluate differences between trained and non-trained joints within a continuous gait cycle, we calculated the difference between the joint trajectories of the trained and the untrained legs (θtrained hip/knee − θuntrained hip/knee) within each gait circle.

#### Statistical Analysis

We excluded one outlier from the analysis (from the HEA group). We detected one participants who performed systematically worse than the others [his/her performance variables systematically lied out of the 1.5 inter quartile range (IQR) in most test and training blocks]. We noted that the calibration process in this particular participant resulted in exceptional large joint ROMs. This probably led to a target trajectory which was too challenging to reach, as the target trajectory is proportional to the ROMs calculated during calibration (a 20% increase in the ROM). After the exclusion of the outlier, the normality of the data was confirmed using Kolmogorov–Smirnov tests.

We used linear mixed effects (LME) analysis to evaluate the effect of the different training groups (Control, HEA, and VEA), time (e.g., baseline, mid-training retention, short-term retention) and HD factor (addition of haptic disturbance in the cross-over design) on the performance variables. We employed the absolute mean values of each performance variable during a gait cycle as dependent variables (i.e., for each gait cycle we took the mean of the 250 time frames). We used the lme4 package (Bates et al., 2015) for R (R Core Team, 2017)). Initially, we entered as fixed effects the interaction between training groups, time and HD factor into the LME model, while participants were modeled as a random factor to account for the by-subject variation. With backward elimination of the non-significant fixed effects using model comparison analysis with the Akaike Information Criterion (AIC), the HD factor was eliminated from the original model, since the addition of haptic disturbance did not have any significant effect on any of the error performance variables. Therefore, the final model employed to fit our data had the form:

$$\text{Performance\\_variable} \sim \text{group}^\* \text{time} + \text{(1\\_subject)} + \text{s} \tag{8}$$

The lmerTest package in R (Kuznetsova et al., 2017) was used to test significance of the effects while it provides degrees of freedom and p-values for the t and type III F-tests with Satterthwaite degrees of freedom approximation. We report the estimates (β), standard errors (SE), confident interval (CI) with parametric bootstrapping, and significant levels (p).

Participants were repositioned in the Lokomat when they returned for the second experimental day. Although the examiner employed the same individual participant-based parameters in both days (e.g., length of segments in the orthoses, cadence, etc.), it is challenging to precisely reposition participants in an exoskeleton (Marchal-Crespo and Riener, 2018). Different alignments between the anatomical human and robotic joint axes within the two experimental days may have an impact in the end-effector kinematics (Bartenbach et al., 2015) and introduce variability into the data that could potentially influence the power of our statistical model. Thus, the performance in long-term retention compared to baseline and short-term retention was separately investigated using two independent two-level LME models (baseline-LTR and STR-LTR). We employed ANOVAs in order to test whether the performance variables during baseline were different between training groups. Post hoc comparisons were performed with Tukey corrections. In order to investigate whether the performance of the participants trained with HEA significantly changed between training and retention blocks (i.e., whether participants relied on the provided torques during training), we compared the performance between training 1.2 and mid-training retention and between training 2.2 and shortterm retention with paired t-tests. We investigated the effect of adding haptic disturbance on the variability of the spatial error during training using linear mixed effects models with the interaction of training groups and haptic disturbance groups as the fixed effects.

For safety reasons, the training controllers are only active during swing phase. Therefore, no difference between training groups were expected during the stance phase. In order to get a better insight into participants' performance during the continuous gait cycle, we also performed statistical analysis on the continuous performance variables using Statistical Parametric Mapping (SPM). SPM is suitable for the analysis of smooth continuum changes in biomechanical data and allows for topological analysis of the data. The SPM main advantage over the mean performance variable approach, is that statistical results are presented directly in the original sampling space without any need for data reduction and discretization of the dependent variables (Friston et al., 2007). Therefore, the differences between/within training groups can be localized within a gait cycle (i.e., vectors with 250 time frames). The SPM analysis was performed using the open-source spm1d package (Pataky, 2012) in Python (Python Software Foundation, version 2.7<sup>1</sup> ). Two-way ANOVAs with repeated measures on the time factor (baseline – training 1.1 – training 1.2 – training 2.1 – training 2.2 when analyzing training performance, and baseline – mid-training retention – short-term retention when analyzing motor adaptation), and main effect of training group (Control, HEA, and VEA) were performed with the continuous tracking error as dependent variables. A two-way ANOVA with repeated measures on the time factor (calibration – free walking 1 – free walking 2), and main effect of training group (Control, HEA, and VEA) was performed with the continuous difference between dominant and non-dominant knee trajectories. Oneway ANOVAs were used in further comparisons if the two-way ANOVA was significant.

<sup>1</sup>www.python.org

We evaluated the effects that the different training strategies had on the three IMI subscales: interest/enjoyment (Q1, Q6), perceived competence (Q2, Q5), and effort/importance (Q3, Q4). We used non-parametric independent samples Kruskal–Wallis tests in order to evaluate potential differences between training groups in the responses to each IMI subscales after baseline. We compared the responses to each IMI subscales after the first and second training blocks, and after the short- and long-retention tests relative to the responses after baseline using Kruskal–Wallis test with training group as the main factor. If the Kruskal– Wallis test was significant, Mann–Whitney Test range was used to perform pairwise comparisons. A Mann–Whitney Test was also employed to test the effect of adding haptic disturbance on the changes of each IMI subscale scores from baseline to first training block, and from first to second training blocks.

Statistical analysis of the IMI questionnaire responses was performed in IBM <sup>R</sup> SPSS <sup>R</sup> Software (version 21, Chicago, IL, United States). The significance level of all statistical test was set to α = 0.05.

# RESULTS

We did not find a significant effect of adding haptic disturbance (HD) during training in the error reduction from baseline to mid-training retention. As discussed above, using backward elimination and model comparison analysis with the AIC, the HD factor was eliminated from the LME model (Eq. 8).

# Performance During Training With Different Training Strategies

We used a LME model (Eq. 8) with five levels in time factor (baseline, training 1.1, training 1.2, training 2.1, and training 2.2), three levels in training group (Control, HEA, and VEA) and their interaction in order to analyze the participants' performance during training. We selected the Control group as the reference level for group factor and baseline for the time factor in the contrast analysis.

When employing spatial error as dependent variable, we found a significant time effect [**Figure 5A**, F(4,104) = 21.002, p < 0.001] and interaction between time and training group [**Figure 5A**, F(8,104) = 3.192, p = 0.003]. In particular, participants in the VEA group increased the spatial error systematically more than the Control group from baseline to training 1.1 (**Table 2**, p = 0.007), and training 1.2 (**Table 2**, p < 0.001). Similarly, the VEA increased the spatial error systematically more than the HEA group from baseline to training 1.1 [**Figure 5A**, β = 0.002, t(104) = 2.175, p = 0.031]. We also found that participants in the Control group reduced the error from baseline to training 1.2 in a significantly greater amount than the HEA group (**Table 2**, p = 0.006). However, we also found differences between training groups in the spatial error during baseline [**Figure 5A**, ANOVA: F(2,26) = 2.74, p = 0.083]. Post hoc comparisons revealed that participants in the HEA group performed systematically better than the Control group during baseline, although the difference did not reach significance (p = 0.102). Similar results were observed within the tracking error (see **Figure 5B** and **Supplementary Table A1**).

Results from the two-way ANOVA SPM analysis with the continuous tracking error as dependent variable confirm these observations. We found a significant time effect, indicated by two supra-threshold clusters in the test statistic trajectory (SPM{F}, at time frames 76–133 and 141–184 in the gait cycle) that exceeded the critical threshold of F <sup>∗</sup> = 4.131 with p < 0.001. We also found a significant interaction between time and training group (time frames 105–159, F <sup>∗</sup> = 3.090, p < 0.001). Differences across groups were found at the expected start of the swing phase (around the 100–150 time frames of the gait cycle, see **Supplementary Figure A1**). We note that usually there is a delay between the reference and actual ankle positions. Therefore, the differences noted around these time frames suggest that participants changed the timing of transitions between gait phases.

We found a significant main effect of adding haptic disturbance on top of the other training strategies in the variability of the spatial error during the first training block (LME with training groups [Control, HEA, and VEA], haptic disturbance groups [HD1, HD2] and their interaction as fixed effects; **Supplementary Figure A2**, training 1.2: F(1,23) = 6.928, p = 0.015). In particular, participants trained with visual or haptic error augmentation showed larger variability when haptic disturbance was added during training 1.2 compared to participants without haptic disturbance [VEA: β = −0.009, t(23) = −2.637, p = 0.015; HEA: β = −0.0067, t(23) = −1.842, p = 0.078]. During the second training block, the haptic disturbance was removed in the HD1 group and was added on top of the HD2 group (i.e., to the participants who were not trained with haptic disturbance during the first training block). We observed again that the variability was larger in participants trained with haptic disturbance on top of their main strategy [**Supplementary Figure A2**, training 2.1: F(1,23) = 6.554, p = 0.018]. In particular, adding haptic disturbance on top of the Control group significantly increased the variability during the second training block [training 2.1: β = 0.008, t(23) = 3.072, p = 0.005; training 2.2: β = 0.008, t(23) = 2.799, p = 0.01]. The differences in spatial variability between participants trained with and without haptic disturbance did not completely faded at short term retention [**Supplementary Figure A2**, F(1,23) = 5.34, p = 0.03].

### Effect of the Training Strategies on Motor Adaptation and Learning

Motor adaptation was evaluated using a LME model with the training groups (Control, HEA and VEA), time (baseline, mid-training retention, and short-term retention) and their interaction as fixed effects (Eq. 8). We selected the Control group as the reference level for group factor and baseline for the time factor in the contrast analysis.

In general, participants improved their performance, as suggested by a significant main time effect on spatial error [**Figure 5A**, F(2,52) = 29.04, p < 0.001]. We also found that the interaction between training group and time almost reached significance [**Figure 5A**, F(4,52) = 2.204, p = 0.061]. In particular,

participants trained with VEA increased the spatial error from baseline to mid-training retention, while participants in the Control group reduced the error (**Table 3**, p = 0.008). This difference was also significant at short-term retention (**Table 3**, p = 0.016). We found that participants trained with HEA reduced significantly the spatial error when the HEA torques were removed from training 1.2 to mid-training retention [**Figure 5B**, paired t-test: t(8) = 2.66, p = 0.029]. However, we did not find significant differences in the tracking error between the last training test of each block and the following retention test. Similar results were observed within the tracking error (see **Figure 5B** and **Supplementary Table A2**).

TABLE 2 | Results from the linear mixed-effects model with training blocks as time factors (Baseline, Training 1.1, Training 1.2, Training 2.1, and Training 2.2) and spatial error as dependent variable.


SE, standard error; CI, confidence interval using parametric bootstrapping. Reference level for group factor is Control and for time factor is Baseline. ∗∗∗p ≤ 0.001, ∗∗p ≤ 0.01, <sup>∗</sup>p ≤ 0.05, ·p ≤ 0.1.

TABLE 3 | Results from the linear mixed-effects model with retention blocks as time factors [baseline, mid-training retention (MTR), short-term retention (STR)] and spatial error as dependent variable.


SE, standard error; CI, confidence interval using parametric bootstrapping. Reference level for group factor is Control and for time factor is Baseline. ∗∗∗p ≤ 0.001, ∗∗p ≤ 0.01, <sup>∗</sup>p ≤ 0.05, ·p ≤ 0.1.

Results from the two-way ANOVA SPM analysis of the continuous tracking error confirm these observations. We found a main effect of training group (time frames 96–120 exceeded the critical threshold of F <sup>∗</sup> = 7.125 with p = 0.012), and time effect (two clusters exceeded the threshold of F <sup>∗</sup> = 6.282, at 74–120 time frames with p < 0.001 and 231– 249 time frames with p = 0.025). The interaction between training group and time almost reached significance (α = 0.1, time frames 139–149, F <sup>∗</sup> = 3.905, p = 0.083). In particular, we found differences between training groups in the error reduction from baseline to mid-training retention (**Figure 6**, ANOVA, time frames 111–133, F <sup>∗</sup> = 7.238, p = 0.0122). As observed during training, the differences between training groups were mostly found at the time frames around the first part of the swing phase, where the training strategies become active.

In general, participants reduced the spatial error from baseline to long-term retention (**Figure 5A**; LME with time [baseline and long-term retention], training group and their interaction as factors; main effect of time: F(1,26) = 5.827, p = 0.023). Not all training groups seemed to learn the target trajectory at the same extent (e.g., participants in the VEA did not reduced the spatial error at long-term). However, the interaction between training group and time did not reach significance [F(2,26) = 2.573, p = 0.096]. Participants showed a significant performance deterioration between the short- and long-term retention tests sessions (**Figure 5A**, LME with time [short- and long-term retention], training group and their interaction as factors; main effect of time: F(1,26) = 13.066, p = 0.001). On the other hand, we did not find a significant reduction of the tracking error from baseline to long-term retention. Participants showed a significant deterioration of their tracking performance between short- and long-term retention tests [**Figure 5B**, F(1,26) = 9.864, p = 0.004]. However, not all training groups seemed to worsen at the same extent (**Figure 5B**; time [short- and long-term retention] × group effect: F(2,26) = 2.46, p = 0.105). In particular, the tracking performance in the VEA and Control groups seemed to

deteriorate more than the performance of participants trained with HEA (**Figure 5B**).

## Effect of the Training Strategies on Free Walking

We analyzed the effect of the different training strategies on gait asymmetry during the free walking tests performed in the middle of each training block (**Figure 4**) using LME models with training groups (Control, HEA and VEA), time (calibration, free walking 1 [FW1], and free walking 2 [FW2]) and their interaction as fixed effects (Eq. 8). The asymmetry of knee and hip joints between legs were employed as dependent variables.

We found a main time effect in the asymmetry between knees [**Figure 7A**, F(2,52) = 9.83, p < 0.001] and in the interaction of time and training group [**Figure 7A**, F(4,52) = 2.56, p = 0.049]. In particular, participants trained with HEA increased their knee asymmetry in a greater amount than participants in the Control group from calibration to the second free walking test (**Table 4**, p = 0.021) and participants who trained with VEA [β = −0.118, t(52) = −2.101, p = 0.041]. We did not find interaction effects between time and training groups in the hip asymmetry (see **Figure 7B** and **Supplementary Table A3**).

We further investigated the differences between the trained and non-trained joint trajectories using SPM. Two-way ANOVA with time effect (Calibration, free walking 1 and free walking 2) as repeated measures factor and training group as fixed effect (Control, HEA, and VEA) with knee trajectory differences between legs as the dependent variable showed significance on the time effect (at time frames 170–242, F <sup>∗</sup> = 5.957, p < 0.001), indicating a significant increase of asymmetry between the knees during the free walking tests. The observed differences in the SPM plots occur mainly in the areas of maximum flexion (around the 170–210 time frames of the gait cycle, **Supplementary Figure A1**).

We performed a one-way ANOVA to evaluate the change from calibration to free walking 2 in the differences between knee trajectories (i.e., [θknee trained − θknee untrained]Calibration − [θknee trained − θknee untrained]FW2). Although not significant, we observed that during the region of maximum knee flexion (170– 210 time frames) participants trained with HEA showed a higher asymmetry (more negative values in knee trajectory differences) compared to the other training groups. In fact, only subjects trained with HEA significantly increased the knee asymmetry between calibration and the second free walking test (**Figure 8**, paired t-test, supra-threshold cluster at time frames 174–208 exceeding critical threshold of t <sup>∗</sup> = −4.266 with p = 0.001). Positive values in this region indicate higher flexion on the trained knee compared to the untrained knee.

# Effect of Training Strategies on Motivation

We found a significant main effect of training strategy on several subscales of the intrinsic motivation inventory. We found an almost significant effect in interest/enjoyment increase during training (**Figure 9A**, p = 0.058), a significant effect after short retention (**Figure 9A**, p = 0.042), and

a one-side significant effect at long term (p = 0.090). In particular, participants in the VEA group reported a higher interest/enjoyment increase than participants in the HEA group (training: p = 0.028; short retention: p = 0.028), and participants in the Control group (retention: p = 0.043). We also found a significant effect of training strategy in the perceived competence during training (**Figure 9B**, p = 0.039). In particular, participants trained with VEA reported a lower perceived competence level compared to participants trained with HEA (p = 0.035) and the Control group (p = 0.043). We did not find a significant effect of the training strategy on effort/importance (**Figure 9C**).

Participants, who trained with haptic disturbance showed no significant changes in scores compared to those trained without


TABLE 4 | Results from the linear mixed-effects with free walking blocks as time factors [Calibration, free walking 1 (FW1), free walking 2 (FW2)] model of impact in knees asymmetry.

SE, standard error; CI, confidence interval using parametric bootstrapping. Reference level for group factor is Control and for time factor is Calibration. <sup>∗</sup>p ≤ 0.05, ·p ≤ 0.1.

disturbance in any of the IMI subscales (**Figure 9D**). However, when analyzing the effect of adding haptic disturbance only in participants trained with VEA and HEA, we found that participants trained with error amplification alone increased their perceived competence in a significant greater amount than participants who trained with error augmenting strategies plus haptic disturbance (**Supplementary Figure A3**, p = 0.017). This significant difference was not visible after the second training block.

### DISCUSSION

## Training With Visual Error Amplification Hampered Performance and Motivation During Training

Based on the idea that errors are fundamental signals that drive motor adaptation (Emken and Reinkensmeyer, 2005), we expected better performance during training, since visually amplified movement errors would increase the detection and correction of small errors (Wei et al., 2005). However, training with visual error amplification was especially challenging, as suggested by the larger movement errors observed during the first training block, compared to the Control and haptic error amplification groups. This is corroborated by the limited reported perceived competence by the visual amplification group, contrary to the increased perceived competence reported by participants in the other training groups.

A possible explanation for this unexpected performance degradation might be originated in the value of the visual amplifying gain. Although we selected a relatively small gain (αamp = 0.2), maybe the gain was still too large for participants to correctly interpret their performance loss during training. Previous experiments showed that doubling the visual errors (i.e., αamp = 1), during training a reaching task resulted in faster adaptation (Patton et al., 2013) and better motor learning (Celik et al., 2009). It has been recently suggested that in order to accelerate learning of a point-to-point reaching task with visuomotor rotation, the gain should be 0.92, and for the fastest learning in combination with the best post-training performance, the gain should be decreased from 0.92 to 0 throughout training (Parmar and Patton, 2015). However, previous research aimed to amplify spatial errors in reaching point to point tasks –i.e., discrete simple movements– while in the current experiment, participants were requested to perform a continuous tracking task with their ankle –i.e., a rhythmic continuous task– while we amplified tracking errors (i.e., spatio-temporal errors). Therefore, it cannot be ensured that visual amplification gains that successfully work in learning simple point-to-point reaching task would also help learning more complex tasks. In fact, in a recent experiment, gains bigger than 0.4 confused participants when tracking a complex rowing stroke –i.e., a rhythmic continuous movement (Basalp et al., 2016). The gain of 0.2 was probably too large in this specific complex task, especially during the first training block. A possible solution might be to employ an adaptive visual amplification gain that is augmented based on participants' ongoing errors (Rauter et al., 2011).

Training with haptic error amplification, on the other hand, did not result in poor performance during training, probably because large errors were limited with haptic guidance. A well-known potential limitation of haptic strategies is that participants might rely on the haptic guidance during training, and therefore, might fail to actively perform the task by themselves (Reinkensmeyer et al., 2009). However, we did not find a performance degradation when the haptic error amplification strategy was removed during the retention tests. In fact, participants performed significantly better when the haptic error amplification was removed during the mid-training retention test, suggesting that small tracking errors were, indeed, amplified.

# Training With Visual Error Amplification Hampered Motor Adaptation of the Locomotor Task

In general, all participants improved their performance already after the first training block. However, when comparing between training strategies, we found that participants trained with visual error amplification reduced their errors after the first training block (mid-training retention) significantly less than participants in the Control group. This difference was maintained after the second training block (at short-term retention). The motor adaptation limitation observed in the visual error

amplification group could be explained by the poor performance and motivation observed during training. Probably, participants did not benefit from the large errors created during training because they failed to understand the reason behind their performance loss. Based on these results, it is essential to reexamine the simplistic interpretation of error-based theories in motor learning, i.e., that larger errors drive faster adaptation. It is crucial to evaluate with greater detail under what task conditions, and for what kind of errors, visual error amplification may benefit motor learning. An optimal framework might be, similarly to the haptic error modulating controller here presented, to visually amplify medium-sized errors that might be optimal for learning, while reducing large errors that can be frustrating to the participants. We note that in the visual error amplification strategy presented here we limited the amount of error amplified (to a maximum of 15◦ ), but no error reduction was implemented.

Training with haptic error amplification, on the other hand, did not hamper the adaptation process. We found a smaller tracking error reduction after the second training block (at short-term retention), compared to the Control group. However, this difference might be originated in the initially better performance observed in the haptic error amplification group during baseline. Probably their potential to further improve was limited (ceiling effect). In previous studies we found that the specific characteristics of the motor task to be learned might play an important role on the effectiveness of robotic training (Marchal-Crespo et al., 2015b). In particular, we found that training with haptic guidance seemed to hamper learning of continuous rhythmic tasks (Marchal-Crespo et al., 2015a). Although haptic guidance was applied during training of the walking task presented here (i.e., a continuous rhythmic task) when errors were larger than a preselected threshold, this did not hamper the learning of the continuous rhythmic task. Probably, the addition of the haptic error amplification when the errors were sufficiently small prevented participants to rely on the guidance and promoted motor adaptation.

The statistical parametric mapping analysis revealed that the differences across training groups were mainly found at the time frames around the first part of the swing phase, where the training strategies become active. Note that increasing the joints' ROM to create the reference ankle trajectory resulted in longer and higher steps along with longer swing phases. Therefore, the differences noted around the beginning of the swing phase suggest that participants in the different training groups adapted differently how to time the transition between gait phases of the reference trajectory. This is in line with several studies that have suggested that haptic demonstration of optimal timing, rather than movement magnitude, may facilitate skill transfer (Heuer and Lüttgen, 2015; Milot et al., 2018).

Nevertheless, in a recent experiment we found that the most effective robotic training condition depended on the characteristics of the task to be learned. We employed a similar haptic error amplification strategy in a 7 DoF robotic exoskeleton for upper limb rehabilitation (Marchal-Crespo et al., 2017a). In an experiment with thirty healthy participants, we evaluated the effectiveness of three error-modulating training strategies -no guidance, haptic error amplification and haptic guidance- on self-reported motivation and learning of continuous and discrete tasks. We found that training with haptic error amplification seemed to be especially suitable to enhance learning of discrete tasks, but did not result in better learning of a continuous task. This is in line with the results reported here. Participants probably benefited from the haptic error amplification provided during the transition between stance and swing phase to better time the gait cycle phases (i.e., time discrete task), but the benefit was limited in the overall continuous task (tracking a desired trajectory presented on a visual display with the ankle is a continuous task). We speculated that the lack of improvement when training continuous tasks with haptic error amplification might be linked to the specificity-of-learning hypothesis, which states that learning is most effective when training is performed involving

the most crucial sensory information source needed to perform the motor task in retention tests. In both experiments, concurrent visual information was crucial in order to perform the continuous task, and therefore, maybe other sources of sensory information -for example proprioception- were neglected (Proteau, 2005).

In general, participants learned to perform the asymmetric desired trajectory (i.e., they reduced the spatial errors at longterm retention). However, not all training groups seemed to learn the target trajectory at the same extent. In fact, participants in the visual error amplification did not reduced the spatial error at long-term. However, the interaction between training group and error reduction from baseline to long-term retention did not reach significance. In general, participants did not learn how to precisely track the desired ankle trajectory (the tracking error reduction al long-term retention was non-significant). We observed tracking performance differences between short and long- term retention tests. Participants trained with haptic error amplification seemed to retain the improved tracking performance at long term, while participants trained with visual error amplification showed a significant tracking performance deterioration at long term. However, caution must be taken when driving conclusions from long-term retention results. Participants, in general, did not reduce the tracking errors at long-term retention. These lack of lasting effects on tracking error at long retention might be due to the too long time between experimental days -retention tests are usually performed after only 1–2 days (Heuer and Lüttgen, 2014; Duarte and Reinkensmeyer, 2015)- and due to the relative short training time (four training intervals of 2 min each).

As discussed above, the selection of the visual gain might play a crucial role on the effectiveness of visual error amplification in motor learning (Parmar and Patton, 2015; Basalp et al., 2016). Thus, we cannot categorically conclude that visual error amplification hampers motor adaptation and learning. Other visual amplifying gains (e.g., gains that are depended on the participants' ongoing error) should be systematically evaluated in order to define the values that might improve adaptation and learning of a complex locomotor task.

# Training With Haptic Error Amplification Enhanced Transfer of the Practiced Asymmetric Gait Pattern to Free Walking

Training with haptic error amplification facilitated transfer of the practiced asymmetric gait pattern, as suggested by the more prominent gait asymmetry observed during the free walking tests, compared to the other training groups. Participants trained with haptic error amplification significantly increased their knee asymmetry by 16% (just below the 20% ROM increase employed to create the new gait pattern), even if they were instructed to walk naturally. In particular, we observed that during the region of maximum knee flexion, participants trained with haptic error amplification showed a higher asymmetry compared to the other training groups. In fact, only participants trained with haptic error amplification showed a significant change in asymmetry after training. This difference was more evident during the second training block, when participants already trained the task for 6 min with haptic error amplification. This finding is of special relevance in the field of robotic gait training. The aim of gait rehabilitation is that the gains observed during training are transferred to overground walking when the haptic and/or visual feedback employed during training is removed. Interestingly, training with only visual feedback (Control), and visual error amplification did not result in transfer of the practiced gait pattern, suggesting that the addition of robotic torques on top of the visual feedback had a positive effect on transfer.

This is in line with previous studies that found that robotic gait training with resistive forces applied during the swing phase results in improvements in walking function in post-stroke (Savin et al., 2014; Yen et al., 2015) and spinal cord injured subjects (Houldin et al., 2011; Yen et al., 2012). These walking improvements have been associated to the after-effects that appear when external forces –to which subjects have already adapted– are suddenly removed (Reisman et al., 2013). Furthermore, exposure to resistive forces may enhance muscle activation (Marchal-Crespo et al., 2014b). An additional explanation for the outperformance of the haptic error amplification strategy is that by adding forces on top of the experienced concurrent visual feedback, participants could benefit from more sensory inputs and improve motor adaptation (Wei and Patton, 2004). Some studies have suggested that multimodal feedback (i.e., the simultaneous addition of several sensory channels, such as haptic, auditory, and visual) enhances perception and action (Carson and Kelso, 2004; Seitz and Dinse, 2007), and may enhance learning of specially complex tasks (Sigrist et al., 2013, 2015).

# The Addition of Haptic Disturbance Increased Movement Variability During Training, but Had No Effect on Motor Adaptation

We found that, as expected, adding haptic disturbance increased the movement variability during training, especially in participants in the visual and haptic error amplification groups. However, the increased variability did not have a significant effect on motor adaptation. This is contrary to our hypothesis and to previous research that found a positive effect on adaptation when training with random feedforward torques (Lee and Choi, 2010; Marchal-Crespo et al., 2014b, 2017b). A possible rationale for this inconsistency is the relative short training duration under haptic disturbance (only two training intervals of 2 min each). A longer training duration with the addition of haptic disturbance might have resulted in different learning outcomes when compared to training without haptic disturbance.

Another rationale is that, in our experiment, haptic disturbance was added on top of the other training strategies that further augmented errors. Therefore, the effect of the haptic disturbance was augmented when applied on top of the error amplification strategies, independently whether the augmentation was done visually or haptically. This explanation is supported by the motivation results. When taking all participants together, we did not find a significant effect of adding haptic disturbance in any motivation subscale. However, we did find

that participants trained with haptic disturbance on top of visual and haptic error amplification during the first training block (HD1 group) exhibited larger movement variability, compared to participants in the error amplification groups without haptic disturbance. Participants in the Control group, however, did not show larger variability when the disturbances were added during the first training block. The contrary effect was observed in the second training block: when haptic disturbance was added during the second training block (HD2 group), only participants in the Control group exhibited larger variability. Therefore, the way and order in which the haptic disturbance was added on top of the other training strategies had an impact on the error variability. Whether this has also an effect on motor learning needs further investigation in future work.

A decrease of perceived competence during training with haptic disturbance was also observed in a previous study (Marchal-Crespo et al., 2017a). Adding randomly varying disturbance torques during training complex 3D arm movements hampered learning and resulted in a decrease of feeling of competence when the haptic disturbance was applied. We hypothesized that this decrease in self-perceived competence probably reduced participants' motivation to perform the task, and therefore, limited motor learning (McAuley et al., 1989). This is in line with a recent study which found that haptically amplifying errors reduced participants' motivation and did not improve learning of a golf putting task (Duarte and Reinkensmeyer, 2015). Therefore, the positive effect of adding haptic disturbance to increase variability during training might have been limited by the negative effects of a decrease in perceived competence, especially in the groups trained with error augmentation. However, further experiments are needed to further evaluate the effect of different forms of haptic disturbance (e.g., different frequency and magnitude parameters) on motor learning.

# Training With Haptic Error Amplification Maintains Levels of Interest and Enjoyment and Leads to an Increase in Perceived Competence

As hypothesized, since the haptic error amplification strategy combined simultaneously haptic guidance and error amplification, it did not impact negatively on participants' motivation, compared to the Control group. Participants trained with haptic error amplification maintained the level of interest/enjoyment during training and retention. Participants in the visual error amplification group, on the other hand, increased their interest/enjoyment during training in a greater amount that participants in the other training groups. This difference was more evident after the short-term retention test. At short-term retention, all participants performed the task without any guidance or disturbance from the robot, therefore the observed significant difference at retention might be related to the increase of perceived competence reported when the visual error amplification was removed. In fact, during training, participants in the visual error amplification group reported significantly lower values of perceived competence than participants in the Control and haptic error amplification groups. This difference, however, vanished once the visual error amplification was removed at retention tests.

Participants trained with the novel haptic error amplification strategy that combines haptic guidance and error amplification did not show significant differences in the evolution of the perceived competence with respect to the Control group. Participants probably benefited from the effect that error amplification had on keeping the interest and enjoyment during training, while the haptic guidance helped to increase the perceived competence as training progressed. Therefore, the novel designed haptic error amplification strategy kept the participants' interest and enjoyment during training without negatively affecting their perceived competence.

# Experimental Design Limitations

The experimental design suffers from some limitations. First, the number of training blocks seemed to be insufficient to drive learning of some aspects of the motor task. Participants learned to perform the asymmetric desired trajectory (as suggested by a significant spatial error reduction at long-term retention) but did not learn how to precisely track the desired ankle movement at each time frame (the tracking error reduction al long-term retention was non-significant). The number of training blocks was decided after previous experiments that showed that performance of a similar locomotor task reached a plateau after five training intervals (Krishnan et al., 2013; Ranganathan et al., 2016). However, these previous experiments only accounted for a change in the performance, rather than learning effects (i.e., no changes in performance were tested at long term). Future research should include a larger number of training blocks during the first experimental day, or at different time points (e.g., after 1, 3, and 7 days) to evaluate whether learning of the tracking task (and differences between training groups) can also be observed at long-term retention.

Second, the effect of the haptic disturbance was augmented when applied on top of the error amplification strategies. Therefore, the analysis of the effect of haptic disturbance on motor adaptation is limited, as its effect on the participants trained with error augmenting strategies differs from that of participants in the control group. Finally, while haptic error amplification limited large errors while augmenting smallmedium errors, visual error amplification augmented the errors, independently of their size (although we saturated the amplification at a certain error level). An interesting direction for future research is to perform further studies to evaluate a visual error amplification paradigm that visually amplifies medium-sized errors that might be optimal for learning, while reducing large errors that can be frustrating to the participants.

# Implications for Robot-Aided Gait Rehabilitation

During the last years, few studies have evaluated the use of resistive training strategies during robotic gait training. Robotic

training with resistive forces applied during the swing phase resulted in improvements in walking function in individuals post-stroke (Savin et al., 2014; Yen et al., 2015) and spinal cord injured subjects (Houldin et al., 2011; Yen et al., 2012) compared to training with assistance. Similar outcomes have been observed in stroke patients when increasing participants' walking asymmetry through a split-belt treadmill intervention (Reisman et al., 2013). Although training with resistive forces seems to improve motor function, training with these challenging strategies might also be associated with a long-term decrease on perceived competence and motivation (Duarte and Reinkensmeyer, 2015). Furthermore, applying external forces which reduce the patients' performance during training might result in dangerous conditions, such as undesired stumbling.

Motor recovery is associated with brain plasticity induced by active training (Cramer et al., 2011). Similar cortical changes have been observed during the acquisition of new motor skills (Lotze et al., 2003). In fact, it is commonly accepted that recovery is a form of motor learning (or relearning) (Dietz and Ward, 2015). The novel haptic error amplification strategy presented in this paper, contrary to prior resistive training strategies, was developed based on well-established motor learning theories. The novel error modulating strategy limited dangerous and frustrating large error, while augmented smaller task-relevant errors. We found that training with this controller did not hamper adaptation and, in fact, resulted in good transfer of the practiced task to free walking. Furthermore, the haptic guidance limited performance errors during training, avoided participants to rely on the guidance and did not hamper the self-reported level of perceived competence, neither reduced the reported interest and enjoyment during training. Taking all this into account, we hypothesize that this novel haptic error amplification strategy might be a good framework to improve robotic gait training in neurological patients.

# CONCLUSION

We have shown that training with visual error amplification is specially challenging, as suggested by a performance degradation and decrease in the reported perceived competence during training. Training with visual error amplification also hampers motor learning of the locomotor task. Training with haptic error amplification facilitates transfer of the new asymmetric gait pattern during free walking, as suggested by a more prominent asymmetry between the legs after training. Adding haptic disturbance on top of the other training strategies increases the movement variability during training. However, increasing the variability during training does not improve motor adaptation,

## REFERENCES


probably because the unforeseen random torques reduce the selfreported motivation level, especially in participants trained with visual and haptic error amplification. The differences observed between training strategies are predominantly localized during the first half of the swing phase.

The novel haptic error amplification strategy presented in this paper, which limits unsafe and frustrating large errors with haptic guidance while haptically augmenting small errors by means of error amplification, was developed considering wellestablished motor learning theories. Therefore, we hypothesize that the proposed haptic error amplification strategy might be a promising framework to improve robotic gait training in neurological patients. Further investigations with neurological patients are needed to corroborate this hypothesis.

# AUTHOR CONTRIBUTIONS

LM-C, SM, and RR contributed to the experimental design and project supervision. LM-C and DO participated in the study design and data acquisition. LM-C and PT performed the data analysis and interpretation of the results. LM-C, PT, and RR prepared the manuscript. All authors read and approved the final manuscript.

# FUNDING

This work was partially supported by the Swiss National Science Foundation (SNF) through the grants numbers PMPDP2\_151319 and PP00P2\_163800, and the National Centre of Competence in Research (NCCR) Robotics.

# ACKNOWLEDGMENTS

The authors gratefully acknowledge the contribution of Dr. Marc Bolliger, Simon Rüdt, Marco Moos, Solange Seppey, and Dr. Jaime Duarte. The authors thank the Statistical Consulting service at ETH, Zürich, for their assistance in the statistical analysis. Parts of this work have been published earlier in conference papers (see Rüdt et al., 2016; Tsangaridis et al., 2018).

# SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnins. 2019.00061/full#supplementary-material




Robots and Systems, (Piscataway, NJ: IEEE), 3068–3073. doi: 10.1109/IROS. 2011.6094832



#### **Conflict of Interest Statement:** SM is employed by Hocoma.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Marchal-Crespo, Tsangaridis, Obwegeser, Maggioni and Riener. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# The Role of Quiet Eye Timing and Location in the Basketball Three-Point Shot: A New Research Paradigm

#### Joan N. Vickers<sup>1</sup> \*, Joe Causer<sup>2</sup> and Dan Vanhooren<sup>1</sup>

<sup>1</sup> Faculty of Kinesiology, University of Calgary, Calgary, AB, Canada, <sup>2</sup> Research Institute for Sport and Exercise Sciences, Liverpool John Moores University, Liverpool, United Kingdom

We investigated three areas of uncertainty about the role of vision in basketball shooting, the timing of fixations (early, late), the location of fixations (hoop centre, non-centre) and the effect of the defender on performance. We also sought to overcome a limitation of past quiet eye studies that reported only one quiet eye (QE) period prior to a phase of the action. Elite basketball players received the pass and took three-point shots in undefended and defended conditions. Five sequential QE periods were analyzed that were initiated prior to each phase of the shooting action: QE catch, QE arm preparation, QE arm flexion, QE arm extension, and QE ball release. We used a novel design in which the number of hits and misses were held constant by condition, thus leaving the timing and location of QE fixations free to vary across the phases during an equal number of successful and unsuccessful trials. The number of QE fixations accounted for 87% of total fixations. The greatest percent occurred during QE catch (43.6%), followed by QE arm flexion (34.1%), QE arm extension (17.5%) and QE ball release (4.8%). No fixations were found prior to QE arm preparation, due to a saccade made immediately to the target after QE catch. Fixation frequency averaged 2.20 per trial, and 1.25 during the final shooting action, meaning that most participants had time for only one fixation as the shot was taken. Accuracy was enhanced when: (1) an early QE offset occurred prior to the catch, (2) an early saccade was made to the target, (3) a longer QE duration occurred during arm flexion, and (4) QE arm flexion was located on the centre of the hoop, rather than on non-centre locations. Overall, the results provide evidence that vision of the hoop was severely limited during the last phase of the shooting action (QE ball release). The significance of the results is explored in the discussion, along with a QE training program designed to improve three-point shooting. Overall, the results greatly expand the role of the QE in explaining optimal motor performance.

Keywords: vision, motor control, attention, perception-action, expertise, eye tracking, training

# INTRODUCTION

The quiet eye (QE) is defined as the final fixation or tracking gaze that is located on a specific location or object in the task environment within 3◦ of visual angle (or less) for a minimum of 100 milliseconds (ms). The onset of the QE occurs prior to a critical phase of the movement and the offset occurs when the gaze deviates off the location for a minimum of 100 ms

#### Edited by:

Mauro Murgia, University of Trieste, Italy

#### Reviewed by:

Sérgio Tosi Rodrigues, Universidade Estadual Paulista Júlio de Mesquita Filho (UNESP), Brazil Christopher Janelle, University of Florida, United States

> \*Correspondence: Joan N. Vickers vickers@ucalgary.ca

#### Specialty section:

This article was submitted to Movement Science and Sport Psychology, a section of the journal Frontiers in Psychology

Received: 03 July 2019 Accepted: 11 October 2019 Published: 30 October 2019

#### Citation:

Vickers JN, Causer J and Vanhooren D (2019) The Role of Quiet Eye Timing and Location in the Basketball Three-Point Shot: A New Research Paradigm. Front. Psychol. 10:2424. doi: 10.3389/fpsyg.2019.02424

**222**

(Vickers, 1996a,b, 2007). Extensive research shows the QE of elite performers is significantly earlier and longer than that of near-elite, or lower skilled performers (Mann et al., 2007; Lebeau et al., 2016; Rienhoff et al., 2016). Since only one final QE period has been reported in most studies to date, critics rightly mention that the majority of fixations may be ignored that play a critical role in performance (Gonzalez et al., 2015). To date, QE studies have defined the QE period relative to a previously identified single critical or final motor phase that has been derived from past QE studies, available biomechanical research, and/or applied technical knowledge. However, it was never intended that only one QE period be considered, but that all QE periods be isolated within a motor task and the one underlying higher levels of performance empirically determined (Vickers, 1992, 2007). We therefore determined five QE periods in the three-point shot, with each having an onset prior to a critical biomechanical phase of the shooting action: QE catch, QE arm preparation, QE arm flexion, QE arm extension, and QE ball release. Our goal was to determine which of these QE periods was most important in contributing to high levels of accuracy in the three-point shot.

Theoretically, the QE is grounded in one of the oldest findings from psychology and neuroscience, which shows there is a delay (later called a latency or reaction time period) that proceeds the initiation of a movement, or phase of the movement (James, 1890/1982; Wundt, 1904; Ladd and Woodworth, 1911). For decades researchers were mystified by the delay, and what may be happening in the brain during this time. For example, Woodworth (1958) commented that "we know–not merely assume–that states of readiness exist in the nerve centers, even though at the present time we cannot do much in the way of describing what goes on in the brain (p. 41)". With the advent of mobile eye trackers synchronized to external motor cameras in the 1980's, the fixations of athletes became available for analysis, thus providing insight to what athletes see during the delay period and the effect this has on their performance. Given the complex nature of most motor tasks, multiple sub-phases exist that together combine to carry out the overall task (Schmidt and Lee, 2014). Theoretically, a QE period could exist before each of the sub-phases, with each providing the task information needed to perform effectively and efficiently. Our goal in this paper was to further our understanding of perceptual-motor coordination by empirically isolating which QE period contributed most to high levels of performance in the three-point shot. Our basic hypothesis is that in order for high levels of success to occur in a motor task, a fixation or tracking gaze must be initiated for a long duration on a specific location in the task environment prior to a specific phase of the movement. It is during this time the brain receives the task specific visual information that it needs to organize the extensive neural networks underlying the planning, initiation and on-going control of the movement.

In selecting the three-point shot, we were motivated by the remarkable performance of Stephen Curry, a National Basketball Association (NBA) player who has not only broke previous records in the three-point shot, but also changed how the game of basketball is played. Only rarely does a single athlete emerge who possesses unique abilities that may be physical, visual, or a combination of both. Curry made more three-point shots than any other player during five NBA seasons (2013, 2014, 2015, 2016, and 2017), received two MVP awards (2015 and 2016), and led his team to three NBA championships (2015, 2017, and 2018). He also made more three-point shots in a single season than any other player (402). His three-point accuracy from 2012 to 2019 was 39.9% (NBA, 2019a). Prior to the emergence of Curry, basketball was a big player's game with the outcome dominated by tall players (NBA average is 6<sup>0</sup> 7 <sup>00</sup>) who took shots from near the basket. The three-point shot is taken further from the basket than any other shot (range 220–23.9<sup>00</sup> in the NBA), thus allowing a relatively small player like Curry (6<sup>0</sup> 3 <sup>00</sup>) to take shots that previously were attempted on few occasions. In 2010 there were only 16 NBA players who made more than 150 three-point shots per season, while in 2018 there were 50 (NBA, 2019c). Clearly the skill needed to shoot from that distance can be acquired, but little is known about the role of vision in the shot. The threepoint shot is unique not only because it is taken further from the basket than any other shot in the game, but it is also very fast. From the moment the ball is received until it leaves the finger tips, players at Curry's level release the ball in 600–800 ms, making it an exceedingly difficult to defend (Waters, 2017). We had elite players with season statistics similar to Curry receive a pass and take three-point shots during an equal number of hits and misses in undefended and defended conditions.

# Timing of Vision in Basketball Shooting

Despite extensive research carried out in basketball shooting (Okazaki et al., 2015; Marques et al., 2018), uncertainty exists about the role of vision in three areas: the timing of vision (early or late), the location of vision (hoop centre, non-centre) and the effect of the defender (undefended, defended). Eye-tracking studies in basketball have resulted in two schools of thought regarding the timing of vision. QE studies report fixations that occur early in the shooting action are most important (Vickers, 1996a,b, 2017; Harle and Vickers, 2001; Wilson et al., 2009; Vine and Wilson, 2011; Klostermann et al., 2017), while ecological, dynamic system studies stress the importance of late "looking" before the ball is released (de Oliveira et al., 2006, 2007, 2008). Theoretically, the results are important, as an early QE supports motor program, open-loop control in which a well learned neural network or motor program is activated and the movement carried out without the use of on-going visual feedback of the target. In contrast, the ecological approach argues that perceived structures in the optic flow field are sufficient to guide motor behavior in an ongoing manner, without reference to internal neural structures or networks. The first QE study was carried out in the basketball free throw and found that elite players fixated the hoop early for an average of 972 ms on hits and 806 ms on misses, while their near-elite teammates averaged 400 ms on hits and 250 ms on misses (Vickers, 1996a,b). Subsequent studies have confirmed these results for high and lower skilled athletes, under conditions of anxiety and in QE training studies where novices are taught the QE characteristics of experts (Vickers, 1996a,b; Harle and Vickers, 2001; Wilson et al., 2009; Rienhoff et al., 2013; Fischer et al., 2015; Klostermann et al., 2017). A number of perceptual/cognitive and/or neural models have been proposed to explain these findings, for example, attention

control theory (Eysenck et al., 2007; Wilson et al., 2009; Causer et al., 2011), ventral and dorsal processing (Vickers, 2012; Vickers and Willams, 2017), the inhibition hypothesis (Klostermann et al., 2014; Klostermann, 2019), and EEG/QE/ocular activity (Janelle et al., 2000a,b; Mann et al., 2011; Muraskin et al., 2016; Gallicchio et al., 2018).

The ecological/dynamic systems approach is based on the work of Bernstein (1967) and Gibson (1979) who state that humans perceive action environments directly, unaided by inference, memories, or internal perceptual/cognitive processes. Highly skilled actors, such as elite athletes, directly perceive the affordances in the environment and organize their movements as they move using "the optic flow field, which is the pattern of motion visible at the eye, (which) also informs about motion and immobility, direction of heading, and steering" (de Oliveira, 2016, p. 260). de Oliveira (2016) citing a study by Carlton (1992) also mentions there is a visuomotor delay period (which is the duration it takes for visual information to be used in motor control), but this is due to a physiological delay and not to higher mental processes. The strongest early evidence supporting optic flow came from Lee (1976, 2009) who found that time-to-contact information, or tau (the inverse of the rate of dilation of the object on the retina) was sufficient to guide motor behavior. A number of ecological studies have been carried out in basketball shooting, with one of the first by Oudejans et al. (2002) who identified two styles of shooting, a high style that used information from the basket to the release of the ball, in contrast to a low style similar to that found in QE studies. de Oliveira et al. (2008) in an eye tracking study found that during the low style, the "expert low-style shooters looked comparatively long at the target area when taking free throws, as was the case in previous research" (Vickers, 1996a,b, p. 403). However, when players used a high style they raised the ball above their head and acquired late visual information from the target prior and during ball release. Results showed late "looking" was critical for the successful completion of the shot. Two caveats apply to the approach of de Oliveira et al. (2008). First, they did not differentiate between fixations or saccades, which play a different role in vision. During fixations the gaze remains stable on a location within 1–3◦ of visual angle for a minimum of 100 ms allowing the brain to process the information being viewed, while during saccades vision is suppressed (Liversedge et al., 2006; Nystrom and Holmqvist, 2010; Marsman et al., 2012). de Oliveira collected data at 50 Hz, therefore each sample of "looking" had a duration of 20 ms, irrespective of being a fixation or a saccade. A second caveat surrounded how the location of "looking behavior" was determined. Looking behavior was coded using a 0 to 1 system, in which looking at the rim was given a score of 1, the net or the small square on the backboard 0.8, the backboard 0.6, all other locations 0.4, and no gaze behavior 0. In their final analysis, they defined looking using all the gaze with scores less than or equal to 6, thus the target encompassed a very large area that included not only the hoop, but the net and backboard as well.

## Location of Fixations

Studies have varied in how they have detected and analyzed the location of fixation during the basketball shot, with no consensus about which location is most critical to success. Vickers (1996a,b) determined the location of fixations relative to seven areas (ball, hands, floor; front hoop, middle hoop, backboard, out of range (outside the backboard) and reported that the location of fixation had no relationship to where the ball eventually landed. In a QE training study, Harle and Vickers (2001) identified the QE relative to five locations (front rim, back rim, left rim, right rim, backboard) and found players increased the percent of QE on the back rim after QE training. de Oliveira (2016) defined looking as described above, while Klostermann et al. (2017, p. 3) defined the QE as the last fixation anchored "for at least 100 ms at the basketball hoop". We determined the location of the QE on seven locations: the ball, passer, backboard, net and the hoop divided into three locations, centre hoop, left hoop, right hoop, with each section being 600/15.24 cm wide (**Figure 1**). We divided the hoop into three areas as our goal was to determine if ego-centric control of the gaze was critical in achieving success. Perception of direction includes both allocentric and ego-centric perception of space (Coren et al., 2004). Allo-centric vision encodes spatial information about objects relative to one another, for example the location of players on the court relative to one another, while ego-centric vision is defined as the perceived location of an object in space with respect to the observer as origin (Morgan, 1978). Three type of ego-centric perception have been identified, body-centric, headcentric, and gaze centric (Li et al., 2013). Although all three are important when performing a basketball shot, we concentrate on gaze-centric vision. When applied to the three-point shot, gaze-centric vision occurred when the QE was located on the centre of the hoop, versus non-centre locations. Since QE training

FIGURE 1 | The experimental set-up showing the start position of the participant and the passer/defender relative to the three-point line. The hoop from the perspective of the participant is inset, showing the hoop left, hoop centre, and hoop right locations. The green circle shows the location of the gaze cursor subtending the target by 1.25◦ of visual angle from a distance of 23 ft (7.01 meters).

studies that have emphasized focusing on the centre portion of the hoop have led to improved performance, we expected the egocentric control of the gaze on the centre of the hoop would be a characteristic of higher performance (Harle and Vickers, 2001; Vine and Wilson, 2011).

# Effect of the Defender

fpsyg-10-02424 October 31, 2019 Time: 12:28 # 4

National Basketball Association statistics show the average accuracy of the top 50 NBA players in the 2018 season in the undefended free throw was 92% compared to 37.8% for the defended three-point shot revealing the profound effect the defender has on accuracy (NBA, 2019b). Early eye tracking studies in the free throw and jump shot did not include a defender, while only a few studies have included a defender, leading to mixed results in terms of the defenders effect on accuracy and the duration of fixations. Gorman and Maloney (2016) and Klostermann et al. (2017) found the defender reduced accuracy, while van Maarseveen et al. (2018) found the defender was not a significant factor. Klostermann et al. (2017) also found no differences in the QE duration during the undefended condition, but a longer QE duration in the defended condition, while van Maarseveen et al. (2018) found players who had the highest accuracy scores had a longer final fixation duration in the defended and undefended condition, while the lower scoring group had a longer duration only in the defended condition.

#### Hypotheses

To date, the eye tracking literature in basketball and other motor tasks, does not suggest the number or percent of QE periods should differ by motor phase. We therefore hypothesized that there would be no significant difference in the number or percent of QE periods due to phase. Consistent with past QE studies, we expected the participants to have a longer QE duration during successful trials, and that greater success would occur during an early phase of the shot (QE catch, QE arm preparation, or QE arm flexion), rather than during a latter phase (QE arm extension, QE ball release). We also expected ego-centric control of the QE on the centre of the hoop to contribute to better performance than fixations on non-centre locations. Finally, we predicted the defender would have a negative impact on shooting accuracy, consistent with competitive results.

# MATERIALS AND METHODS

#### Participants

Twelve elite basketball players (8 male, aged 22.4 ± 2.2 years) were recruited with a combined average above 30% in the two and three-point during the previous season (**Table 1**). All played at the university or semi-professional level; eight were members of the team that won the Canadian university men's championship the following season. The research protocol was approved prior to data collection by the Conjoint Ethics Committee of the University of Calgary, and all participants gave consent.

## Equipment

Gaze was recorded using an ASL Mobile Eye 5 eye tracker (Applied Sciences Laboratory, Bedford, MA, United States), and an external motor camera (Canon Vixia HF R42) that recorded the phases of the shooting action in the sagittal plane. The Mobile Eye is a light (76 g), glasses-mounted, monocular corneal reflection system that measures point of gaze with an accuracy and precision of 0.5◦ of visual angle. Both the gaze and motor videos were recorded at a rate of 30 Hz (33.33 ms/frame of video).

# Task and Protocol

Shots were taken from behind the three-point line on a regulation basketball court used in competition from a distance of 22– 23 ft from the hoop (**Figure 1**). All shots were one-time shots, which occur when the player takes the shot immediately after receiving the pass without any attempt to dribble or take

TABLE 1 | Percent accuracy for each participant in the two and three-point jump shot in the previous season, and in the four tests (pre-test, undefended, defended, and post-test).


The average for all participants is shown at the bottom.

other evasive actions. One-time shots require precise timing and are among the most difficult and advanced shots in basketball. During the 2016 NBA season, approximately one half of the 402 three-point shots that Curry took were one time shots (FreeDawkins, 2019). Participants were instructed to step forward to receive the ball and shoot as quickly and accurately as possible from behind the three-point line. The pass was delivered by a highly skilled player/coach, who also acted as the defender. After a warm-up, a pre-test was performed without the eye tracker, followed by fitting the eye tracker and taking 3–5 practice trials until comfortable. Continuous shots were taken in counterbalanced conditions (undefended, defended) until 10 hits and 10 misses were recorded in each condition. A maximum of 40 shots were taken per condition and percent accuracy determined. During the defended condition, the defender actively challenged the participant, using an outstretched hand that was visible in the participant's visual field during the defended trials. The post-test followed and was performed without the eye tracker. Total testing time was approximately 45 min. During data collection, the gaze and motor data were observed in real time on monitors to ensure calibration on each trial.

# Data Coding and Processing

The gaze and motor videos were synchronized using the Quiet Eye Solutions software (quieteyesolutions.com, 2010). A total of 430 trials were coded of the maximum 480 possible. Fifty trials were not included due to technical difficulties with the eye tracker and/or camera during data collection. An equal number of hits and misses were included for each participant per condition. The final data set consisted of 215 hits and 215 misses and 218 undefended and 212 defended trials. Trial onset (0 ms, 0%) was similar in each trial for the motor and gaze data. Trial onset occurred when the ball left the hand of the passer, and trial offset (100%) when the ball was released from the participant's fingertips. The pass phase began with the first frame of video showing the ball leave the hands of the passer and ended with the frame prior to the ball first contacting the hands of the participant. The arm preparation phase began with the first frame showing the angle at the elbow increase [also called the dip (Penner, 2018), or loading the ball]. Arm flexion began when the angle at the elbow decreased as the ball was raised through the mid-line of the body and above the head. Arm extension began with the first frame showing the angle at the elbow increase until the ball left the finger-tips. Arm extension offset was similar to ball release, as beyond this point the participants had no control over the outcome of the shot. The occlusion phase began when the ball/hands/arm of the shooter entered the visual field and the target was no longer visible. The occlusion period ended when the target was visible.

Once the motor phases were entered into the Quiet Eye Solutions program, fixations and saccades were entered, in order, beginning at time 0. A fixation occurred when the participant's gaze dwelled on a location for a minimum of 100 ms (3 frames of video) within 1.25◦ of visual angle (width of the cursor on the hoop shown in **Figure 1**). Each section of the hoop subtended a visual angle of 1.25◦ from a distance of 23 ft (7.01 meters) from the hoop as calculated by the Visual Angle Calculator available at Ellis (2009). The hoop was divided into three equal parts, each having an equal centre width of 6<sup>00</sup> (15.24 cm). Within each third of the hoop, the athlete normally fixated the front, middle or back of the hoop. If the gaze cursor was located on an area between the three target areas, or on the edge of the rim, it was assigned to the area in which more than half the cursor was located (which was within the 0.5◦ of precision and accuracy of the eye tracker). For example, **Figure 1** (inset) shows that more than half of the gaze cursor was located on the centre of the hoop, therefore it was coded as a fixation on hoop centre assuming three consecutive frames were located in the hoop centre area. If more than half the gaze cursor was located on the rim, then it was coded on the backboard or net.

A saccade occurred when the gaze moved rapidly between locations in two or more frames. Seven locations were coded: passer, ball, hoop centre, hoop right, hoop left, net, and backboard. Coding was carried out by two independent coders, and intra-class correlations were determined for the motor phases and QE onset, offset and duration. R-values ranged from 0.88 to 0.92.

# Isolating the Five QE Periods

The five QE periods were isolated using the Quiet Eye Solutions software, which has a function that detects the onset of the final fixation prior to the onset of a motor phase and automatically outputs the QE location, onset, offset, and duration. Each QE period was isolated separately, and then combined into a single data file. QE catch onset was the final fixation prior to the catch, and had an offset that occurred when the final fixation deviated off a location by more than 1.25◦ of visual angle or 3 frames (100 ms), a standard applied to all the QE offsets. QE arm preparation onset was the final fixation on a location prior to the angle at the elbow increasing. QE arm flexion onset began on a location prior to the angle at the elbow decreasing. QE arm extension onset was the final fixation on a location prior to the angle at the elbow increasing. QE ball release was initiated during arm extension prior to ball release. One limitation of the Quiet Eye Solutions software is that it duplicates a QE period when it extends across two or more motor phases. All duplicate QE periods were removed and the first was one retained, as it provided the most immediate visual guidance to the motor phase immediately following.

## Data Analysis

Data were analyzed and the results graphed using JMP 14.3 (JMP/SAS, 2019). Season and experimental accuracy were analyzed using ANOVA for test (pre-test, undefended, defended condition, post-test) and condition (undefended, defended). The number and percent of QE were analyzed by phase using nominal logistic regression. Motor phase onset, offset, duration, and QE phase onset, offset, and duration were analyzed in absolute (ms) and relative time (%) using a full-factorial repeated-measures linear mixed-effects ANOVA. Relative time was calculated by

determining the percent (%) of a motor or QE variable as a function of total trial time. A mixed model ANOVA was most appropriate for the current study as it is a powerful method for handling missing observations and unbalanced designs, leading to more reliable conclusions, as well as accounting for repeated measures (Bagiella et al., 2000; Baayen et al., 2008). Fixed effects were condition (undefended, defended), outcome (hits, misses), location (hoop centre, non-centre), and participants (n = 12) were the random effect. Contrast of means was used to determine interaction effects. Effect sizes were calculated using partial η 2 in accordance with Cohen's d, with 0.10 considered a low effect, 30 a moderate effect, and 0.50 a large effect. The significance level was set at 0.05.

#### RESULTS

#### 1.0 Percent Accuracy

**Table 1** shows the percent accuracy for the two and threepoint in the previous season, and for the pre-test, defended and undefended tests, and post-test. A significant difference was found for test, F(3,33) = 3.04, p < 0.04, d = 0.22. Pretest accuracy did not differ significantly in the undefended and defended conditions, and was lower confirming the eye tracker did not affect accuracy. Post-test accuracy did not differ from the undefended and defended conditions, confirming fatigue was not a factor. Undefended accuracy was higher (58%) than defended (50%), contrast of means, F(1,33) = 4.55, p < 0.001.

#### 2.0 Trial Duration, Motor Phase Onsets, Offsets and Durations and Occlusion

Trial duration was longer in the undefended than defended condition, F(1,11.06) = 92.86, p < 0.0001, η<sup>P</sup> <sup>2</sup> = 0.89, M undefended = 1417.96 ms (SE = 36.23 ms), M defended = 1270.65 ms (SE = 36.24). **Table 2** presents the mean motor phase onsets, offsets and durations in absolute and relative time. No significant differences were found related to outcome by phase or condition. Significant differences were found for phase and also condition. Motor phase onsets differed in absolute time, F(3,33.05) = 523.26, p < 0.0001, η<sup>P</sup> <sup>2</sup> = 0.94, as did phase offsets, F(3305) = 235.11, p < 0.0001, η<sup>P</sup> <sup>2</sup> = 0.85 and durations, F(3,33.03) = 18.01, p < 0.0001, η<sup>P</sup> <sup>2</sup> = 0.37. Similar differences were found for relative time onset, F(3,33.05) = 725.05, p < 0.0001, η<sup>P</sup> <sup>2</sup> = 0.95, offset, F(3,33.05) = 384.64, p < 0.0001, ηP <sup>2</sup> = 0.92, and duration, F(3,33.05) = 19.45, p < 0.0001, η<sup>P</sup> <sup>2</sup> = 0.37. The interaction of phase by condition was significant for onset in absolute time, F(3,33.05) = 31.37, p < 0.0001, η<sup>P</sup> <sup>2</sup> = 0.49, as well as offset, F(3,33.05) = 14.21, p < 0.0001, η<sup>P</sup> <sup>2</sup> = 0.31, and duration, F(3,33.05) = 5.58, p < 0.003, η<sup>P</sup> <sup>2</sup> = 0.15. Only phase duration differed in relative time, F(3,33.05) = 11.02, p < 0.0001, ηP <sup>2</sup> = 0.25. The pass was delivered more slowly by the passer in the undefended than defended condition, F(1,11.23) = 52.64, p < 0.0001, η<sup>P</sup> <sup>2</sup> = 0.82, M undefended = 506.71 ms (SE = 7.93) and M defended = 438.97 ms (SE = 7.94). Shot release time included the arm preparation, arm flexion, and arm extension durations combined, and occurred earlier in the defended than undefended condition, F(1,11) = 23.44, p < 0.02, η<sup>P</sup> <sup>2</sup> = 0.005, and was slower in the undefended condition, M undefended = 860.92 (SE = 32.79 ms) and M defended = 780.94 (SE = 32.94 ms).

Occlusion onset occurred earlier in the defended than undefended condition, F(1,11) = 85.51, p = < 0.0001, d = 0.88, M defended = 1025.86 (SE = 44.06) and M undefended = 1174.60 (SE = 44.05). Occlusion offset was earlier in defended versus undefended condition, F(1,11) = 9.93 p < 0.009, d = 0.52, M defended = 1372.57 (SE = 83.25) and M undefended = 1469.80 (SE = 83.23). Occlusion duration did not differ by condition, M undefended M = 294.30 ms (SE = 80.22) and defended M = 364.93 (SE = 81.80). Outcome was significantly affected

TABLE 2 | Mean motor onsets, offsets and durations (ms, %) for the (1) the pass (as delivered by the passer), (2) arm preparation, (3) arm flexion, and (4) arm extension by condition.


by occlusion offset, F(1,11) = 5.60 p < 0.04, d = 0.52. Hits occurred when occlusion offset was earlier, M hits = 1403.38 (SE = 82.42), and M misses, 1440.46 (SE = 82.42). The interaction of condition by outcome was not significant. In order to determine if occlusion may have played a role in initiating arm extension, arm extension onset was subtracted from occlusion onset. Occlusion onset occurred before arm extension in the undefended by 74.00 and 1.59 ms before in the defended, suggesting it may have played a role in initiating arm extension (however, see the results for QE duration **Figure 2**). A similar comparison for occlusion offset relative to ball release indicated that the target was occluded for 38.87 ms beyond ball release in the undefended condition, and 147.00 ms beyond in the defended condition.

# 3.0 Number and Percent of QE Fixations by Phase

A total of 944 QE fixations were found, which accounted for 87.08% of all fixations. **Table 3** shows the percent of QE declined across the motor phases, with the highest percent occurring during QE catch, followed by QE arm flexion, QE arm extension, and finally QE ball release. There were no QE fixations during arm preparation, due to a rapid shift of gaze (saccade) to the target made by all participants immediately following QE catch offset. Our expectation that the proportion of QE fixations would be equal in each motor phase (25% per phase) was not upheld and differed significantly from the hypothesized values, χ 2 (3,427.73) p < 0.0001. The one-way ANOVA was significant for the number of QE by phase, F(3,47) = 19.91, p < 0.0001, η<sup>P</sup> <sup>2</sup> = 0.75, indicating the target was increasingly difficult to fixate across the motor phases. **Table 3** shows the percent of QE initiated in each phase that resulted in a hit or miss. The highest percent of hits were initiated during QE arm flexion (61%), followed by QE arm extension (31%), QE ball release (8.6%) and least for QE catch (0.003%).

TABLE 3 | Number and percent of quiet eye fixations by motor phase, and percent of quiet eye initiated in a phase during hits and misses.




#### Individual Frequency of QE

**Table 4** presents the mean frequency of QE per participant during the complete trial (four phases), and during the final three shooting phases. Frequency of QE averaged 2.22 per trial and did not differ due to condition, M undefended = 2.23 (SE = 0.03) and M defended = 2.20 (SE = 0.03), or outcome, M hits = 2.23 (SE = 0.03), and M misses = 2.20 (SE = 0.03). QE frequency averaged 1.25 fixations during the final three phases and did not differ by condition, M undefended = 1.26 (SE = 0.03), M defended = 1.23 (SE = 0.03), or outcome, M hits = 1.26 (SE = 0.03), M misses = 1.24 (SE = 0.03). One participant (A9) had a higher frequency of QE fixations, averaging 3.11 over all phases and 2.11 as the shot was taken. Overall, the results show that 11/12 participants' averaged one opportunity to fixate the target after the pass was received.

#### Percent of QE by Phase by Participant

The mosaic plot in **Figure 2** shows the percent of QE initiated by each participant during QE catch, QE arm flexion, QE arm extension and QE ball release. During QE catch, all participants had a consistently high percentage, ranging from 35.2 to 50.9% of the total QE. During the final three phases, participants initiated a QE during the arm flexion or extension phase, but given the mean QE frequency was 1.25, there was not enough time for a fixation to occur in both phases in one trial. Nine participants (P1, P2, P3, P4 P7, P8, P9, P10, and P11) primarily fixated the target during arm flexion (red), initiating a minimum 31.3% of their total QE during this phase, and a low percent of QE during arm extension (green). Three participants (P5, P6, and P12) had a later QE during arm extension, minimum 37%. Three participants (P4, P9, and P12) initiated a QE during all three shooting phases, and were the only athletes to initiate a QE prior to ball release (purple). P9 was unique in consistently initiating a QE prior to ball release, accounting for 35 of the total 45 QE periods observed. Due to the low number of fixations during QE ball release, a formal analysis could not be carried out, but a descriptive analysis is provided after the ANOVA results for QE catch, QE arm flexion and QE arm extension are presented.

# 5.0 QE Location

The mosaic plot shown in **Figure 3** shows the percent of QE fixations by phase on seven locations in the top panel **(A)**, and two locations (hoop centre, non-centre) in the bottom **(B)** by condition and outcome. During QE catch, the ball accounted for 93.1–96.2% of the total. Fixations were also found on the passer and the hoop, but these accounted for a low percent of the data. During QE arm flexion and QE arm extension, the primary locations fixated, in order, were hoop centre, net, backboard, hoop right, and hoop left.

Percent of QE on hoop centre and non-centre locations (**Figure 3B**) were analyzed using nominal logistic regression. The probability of fixating the hoop centre versus non-centre locations was affected by phase, χ 2 (3,427.73) p < 0.0001, but not by outcome or condition. **Figure 4** presents the prediction profile for location by phase. During QE catch (A), the probability of fixating non-hoop locations was 0.99 and these fixations were primarily on the ball. During QE arm flexion (B) the probability of fixating the hoop centre was 0.587 and 0.413 for non-centre locations, while during QE arm extension (C), the probabilities declined to 0.527 and 0.473, respectively.

#### QE Onset, Offset and Duration by Phase

**Table 5** presents the mean QE onset, offset and duration by phase and condition in both absolute (ms) and relative (%) time. QE catch was analyzed separately from QE arm flexion and QE arm extension, due to its different functions performed, i.e., to catch the ball versus taking the shot. The QE catch data were analyzed using a repeated mixed-effects ANOVA by condition, location and outcome.

#### QE Catch

No differences were found for QE onset, but QE offset differed by condition by outcome in relative time, F(1,10.23) = 5.06, p < 0.002, ηP <sup>2</sup> = 0.33, contrast of means, F(1,29.29) = 18.35, p < 0.05, and neared significance in absolute time, F(1,10.72) = 3.71, p < 0.08. **Figure 5A** shows that during hits QE offset (%) occurred earlier in the defended than the undefended condition. QE catch duration (%) was shorter in the defended than undefended condition, in both relative F(1,10.89) = 10.89, p < 0.006, η<sup>P</sup> <sup>2</sup> = 0.52, and absolute (ms) time, F(1,11.01) = 59.31, p < 0.0001, η<sup>P</sup> <sup>2</sup> = 0.85. During defended hits, participants ceased tracking on the ball an average of 109 ms before the catch in the undefended condition, and 89 ms in the defended.

#### Saccade to the Target

Immediately following QE catch offset, a saccade was made to the target. Saccade onset differed by condition and outcome in both absolute F(1,8.48) = 4.51, p < 0.03, d = 0.34, and relative time, F(1,8.48) = 5.71, p < 0.04, d = 0.40. **Figure 5B** show the saccade onset occurred earlier during hits in the defended condition than undefended. Saccade duration was

longer in the defended than undefended condition, in both absolute, F(1,7.43) = 9.61, p < 0.02, d = 0.56, and relative time, F(1,8.90) = 18.01, p < 0.002, d = 0.67, M undefended = 195.66 ms (SE = 23.39), M defended = 231.28 ms (SE = 23.98).

#### QE Arm Flexion and QE Arm Extension

Since a goal of the study was to determine which QE period was most effective, the QE arm flexion and QE arm extension data were analyzed using a repeated mixed-effects ANOVA by QE phase, condition, location and outcome.

#### QE Onset

Significant main effects were found for QE phase onset in absolute, F(1,10.57) = 31.46, p < 0.0002, η<sup>P</sup> <sup>2</sup> = 0.75, and relative time, F(1,10.86) = 40.19, p < 0.0001, η<sup>P</sup> <sup>2</sup> = 0.79. Then main effect for condition differed in absolute time, F(1,12.02) = 20.86, p < 0.0006, η<sup>P</sup> <sup>2</sup> = 0.63; and for the interaction of condition x phase x location in both absolute, F(1,168.4) = 4.08, p < 0.04, ηP <sup>2</sup> = 0.03, and relative time, F(1,219.4) = 4.54, p < 0.03, ηP <sup>2</sup> = 0.02, **Figure 6A** shows the participants initiated a fixation earlier during QE arm flexion than QE arm extension in both the undefended and defended conditions on hoop centre and non-centre locations.

#### QE Offset

A significant difference was found for QE phase in absolute, F(1,10.45) = 34.57, p < 0.0001, η<sup>P</sup> <sup>2</sup> = 0.76, and relative time, F(1,10) = 11.37, p < 0.008, η<sup>P</sup> <sup>2</sup> = 0.53. The interaction of QE phase × condition × location was significant in absolute time (ms), F(1,144.3) = 5.19, p < 0.02, η<sup>P</sup> <sup>2</sup> = 0.03, and relative time, F(1,147.3) = 7.22, p < 0.006, η<sup>P</sup> <sup>2</sup> = 0.05. **Figure 6B** shows the QE offset was maintained later on the hoop centre during QE arm flexion and QE arm extension in the undefended condition, but occurred significantly earlier in both phases in the defended condition. The earlier QE offsets during the defended condition could have been caused by ball occlusion, or by pressure from the defender. Since ball occlusion also occurred in the undefended condition, then the defender was the most likely cause for the early termination of fixations on hoop centre and non-centre locations. The QE offsets in both conditions occurred well before ball release, which occurred at 1430.28 ms in the undefended condition and 1291.37 ms in the defended (**Table 2**).

#### QE Duration

A significant difference was found for QE phase in absolute, F(1,11.52) = 36.79, p < 0.0001, η<sup>P</sup> <sup>2</sup> = 0.76, and relative time, F(1,10.34) = 48.76, p < 0.0001, η<sup>P</sup> <sup>2</sup> = 0.80. The interaction of condition × location × outcome was significant in absolute time, F(1,58.09) = 4.37, p < 0.04, η<sup>P</sup> <sup>2</sup> = 0.08, and neared significance for relative time, F(1,173.50) = 3.22, p < 0.07. **Figure 6C** shows the QE duration was longer during hits on hoop centre (M = 275.53 ms) than on non-centre locations (M = 187.39 ms) in the undefended condition (contrast of means, F(1,28.40) = 12.21, p < 0.03), but that a shorter duration QE occurred during hits in the defended condition (M = 190.72 ms). The shorter QE durations in the defended condition could have been due to occlusion, or pressure from the defender. Since occlusion occurred in both conditions, the shorter QE duration was most likely caused by pressure from the defender.

#### QE Ball Release

Participant P9 was unique in using three QE periods per trial, one during the pass, one during arm flexion and one prior to ball release (**Figure 2**). In both the defended and undefended condition his trial duration was longer than the other participants, 1805.92 ms in the undefended condition, and 1546.78 in the defended, compared to 1430.26 and 1291.37 ms, respectively, for the other participants. Review of his video data showed his gaze deviated off the hoop centre to the left hoop after QE arm flexion, followed by a saccade back to hoop centre, and a final QE on hoop centre prior to ball release. P9 had the highest accuracy (80%) during the undefended condition, but was

6th overall at 48% during the defended condition (**Table 1**). Given he consistently fixated the target prior to ball release, then he would be classified as a high-style shooter as defined by Oudejans et al. (2002).

## DISCUSSION

Our goal in this study was to investigate three areas of uncertainly about the role of vision in basketball shooting, specifically the timing of QE fixations, their location, and the role of the defender. We also sought to overcome a criticism of past QE studies, which have reported only one QE period. We sought to alleviate this problem by analyzing five QE periods, with each initiated before a biomechanical phase of the one-time basketball shot: QE catch, QE arm preparation, QE arm flexion, QE arm extension, and QE ball release. At the outset, we expected a longer QE duration during successful trials, which would be initiated during the early phases of the shot rather than during the latter phases. We also expected that ego-centric control of the QE on the centre of the hoop would contribute to better performance than fixations on non-centre locations, and that the defender would have a negative impact on shooting performance. We did not expect a significant difference in the number or percent of QE fixation in each phase. For the most part our expectations were upheld, but with some notable exceptions discussed below.

#### Percent Accuracy

Percent accuracy was lower in the defended condition than undefended, 50% compared to 58%, which agrees with competitive statistics, and also studies by Gorman and Maloney (2016) and Klostermann et al. (2017)

#### Effect of the Defender

The defender not only negatively affected accuracy, but also the duration of the motor phases and QE periods. The duration of the arm preparation, flexion, and extension motor phases

were lower in the defended versus the undefended condition (**Table 2**), as were the QE durations, whether calculated in absolute or relative time (**Table 5**). Release time was also lower in the defended condition (780.94 ms) than in the undefended (860.92). These results agree with those reported by Waters (2017), who determined the release time of five top NBA shooters from catch to ball release ranged from a low of 770 ms to a high of 820 ms.

#### Percent of QE Across the Motor Phases

A total of 944 QE fixations were found, which accounted for 87.1% of total fixations. At the outset we expected to find no significant differences in the percent of QE fixations by phase. Contrary to our expectation, we found the percent of QE fixations differed significantly across the phases, with the highest percent/number occurring during QE catch (43.6%; 412 QE fixations out of 430 trials), followed by QE arm flexion (34.1%, 322 QE fixations), QE arm extension (17.5%, 165 QE fixations) and least for QE ball release (4.8%, 45 QE fixations). We found that no QE fixations were initiated prior to the arm preparation phase due to a saccade made by the participants to the target immediately after tracking on the ball ceased. The exceptionally low number of fixations during QE ball release was unexpected, and showed that the threepoint shot is taken under such extreme time and defensive pressure that sustaining a fixation until the ball is released is very difficult.

#### Participant Frequency of QE

Frequency of QE averaged 2.22 per trial, and 1.25 per participant during the final three shooting phases, meaning most participants had only one opportunity to fixate the target after the ball was caught. Percent of QE initiated by each participant by phase, showed that nine participants initiated their QE primarily during arm flexion, and three primarily during arm extension (**Figure 2**). Three participants were able to initiate a QE prior to ball release, and most of these were taken by one participant (P9), who was unique in consistently initiating a QE prior to ball release. A review of P9's gaze videos showed that he used three QE periods, the first during QE catch, the second during QE arm flexion, and the final during QE ball release. He had a very long duration QE on hoop centre during arm flexion, but his gaze drifted to the left hoop during arm extension, followed by refixating the hoop centre prior to ball release. Since it takes time to re-fixate a location, this increased his trial duration to an average of 1803.92 ms in the undefended condition and 1546.78 ms in the defended, compared to 1430.26 and 1291.37 ms, respectively, for the other participants (**Table 2**). During the undefended trials he had exceptional accuracy (80%, 1st), but during the undefended condition his accuracy fell to 48%, 6th overall. In many respects, P9 exhibited the visual behavior described by Oudejans et al. (2002) and de Oliveira (2016) for high style shooters, as he did maintain fixation on the target through to the release of the ball, during both the defended and undefended trials. Overall, P9 shot was slower in delivering the shot than the other participants, his release time averaged 1023.10 ms in the defended condition, compared to 756.72 ms for the other participants.

For 11/12 participants, it is important to consider the offset of their QE arm flexion and QE arm extension (**Table 5**) relative to ball release (**Table 2** arm extension offset). In the defended condition, QE arm flexion offset occurred at 893.08 ms, or 70.80% of trial time, and QE arm extension offset at 993.01 ms, or 78.12% of trial time (**Table 5**). Since ball release occurred at 1291.37 ms in the defended condition, this meant the target was not visible for 398.29 ms if QE arm flexion was the final fixation, and 298.36 ms if QE arm extension was used. These results also show that, except for P9, most participants used a low style of shooting rather than the high style as described by Oudejans et al. (2002).

TABLE 5 | Mean QE onset, offset and duration (with standard error) by phase and condition in both absolute (ms) and relative time (%).


#### Effect of QE Location

fpsyg-10-02424 October 31, 2019 Time: 12:28 # 13

During QE catch, participants primarily fixated the ball, and had very few fixations on the passer or the hoop. They also immediately made a saccade to the hoop thus lying to rest the idea that players can see the target during the pass, since during a saccade vision is suppressed. They also did not track the ball to their hands, but ceased tracking 109 ms before the catch in the undefended condition, and 89 ms in the defended. Critically, a long duration QE during hits occurred when the final fixation was located on the centre of the hoop, versus non-centre locations. The probability of being able to fixate the centre of the hoop was highest during QE arm flexion, rather than during QE arm extension or QE ball release (**Figure 4**). A weight of evidence therefore shows that ego-centric control of the QE during arm flexion was a factor in accuracy in the three-point shot as 61% of hits were initiated on the target during arm flexion, compared to 0.007% during the catch, 31% during arm extension, and 7.70% during ball release.

Why was a long duration ego-centric QE during arm flexion a factor in performance? Nakashima et al. (2015, p. 2) defines ego-centric spatial perception as " the perception of direction or position of oneself based on visual information acquired in the visual field." Results showed that the participant's focus was entirely on the ball prior to the catch, followed by a saccade during which vision was suppressed. Therefore vision of the target did not become possible until the onset of QE arm flexion, which occurred half way through the trial between 650 and 700 ms, or 50% of trial time. Following this, on-going feedback of the target did not occur for 11 of 12 participants during the final 300 ms before ball release. A weight of evidence therefore suggests the three-point shot is under automatic motor program control, in which a highly developed neural network is activated and the movement carried out without on-going visual feedback of the target during arm extension and ball release. The brief duration of the extension motor phase, which averaged 148–149 ms, and therefore qualifies as a ballistic movement and adds evidence in support of open loop control and motor program control (**Table 2**). We also must conclude that since all the participants ceased tracking the ball before the catch, and 11/12 ceased fixating the target during the final 300 ms of the shooting action, then optic flow and tau, as defined by Lee (1976, 2009), does not explain how accurate shots were made in the three point shot.

Since 11/12 participants did not fixate the target during the final 300 ms of the shooting action, then how was accuracy maintained during this time. Research shows that long-term memory of spatial locations persists when ego-centric vision is used to locate targets in space. Henriques et al. (1998) and Schutz et al. (2013) found that ego-centric target representation persisted in long term memory for up to 12 s when a reach movement to delayed targets was made. They concluded that "ego-centric target representations can persist for at least several seconds instead of becoming unavailable immediately after the target vanishes" Schutz et al. (2013, p. 46). Their results may also explain why ego-centric vision was more successful than noncentric vision. The QE non-centric fixations were much briefer than fixations on hoop centre, and were also to more locations. This meant there was less time to encode the location of the target, leading to non-optimal target commands to the shooting arm and hands as the ball was released.

## How Do the Five QE Periods Add to Our Existing Knowledge?

Given that five QE periods were isolated, it is interesting to speculate a few ways they add to our existing knowledge about the technical and strategic requirements needed to perform the three point shot effectively. What does knowing about the timing of the QE periods, their onset, offset, and location during successful and unsuccessful trials add to the game and future research? During QE catch, all the athletes fixated the ball, followed by a saccade to the hoop before the ball was caught, on average 109 ms before the catch in the undefended condition, and 89 ms in the defended. We can therefore speculate, that consistent with coaching (Wissel, 2011) and research (Ripoll et al., 1986; Marques et al., 2018) that the function of QE catch was to prepare the hands to catch the ball as early as possible, followed by an early offset of E catch and a rapid shift in gaze (saccade) to the hoop. But consider the effect of an alternate gaze strategy? What would have happened if the participants had fixated the target during the early pass, and then fixated the ball up to the moment it was caught, a strategy often recommended by coaches (USA Basketball, 2010). In the current study, the participants did not look the ball into their hands, but instead gained approximately 100 ms by shifting their QE to the hoop early before the catch, results that were directly related to accuracy. During QE arm flexion, the participants fixated the hoop for the first time using a long duration QE fixation that was initiated during the latter part of the arm preparation phase. Since this occurred relatively late in each trial this meant the location of the hoop had to be stored in memory for half a second or more. It therefore may be advisable to teach athletes to visually locate the hoop before the pass begins. This requires the athlete develop the decision-making and footwork skills to move into position and acquire information about the target before the pass begins. Gaze that occurs prior to critical events in basketball and other sports is an area that is receiving increased research attention (Okazaki et al., 2015, p. 12). Vater et al. (2019) provide a meta-analysis of the role and importance of peripheral vision across various sports, while in basketball specifically van Maarseveen et al. (2018), p. 250) found that "peripheral vision may serve a significant role in decision making in situ, whereas players mainly relied on central vision to execute an action." The results of the current study and that of other eye tracking studies show the sequence of gaze matters in daily life (Henderson, 2003, 2017) and in QE training studies in sport and medicine (Causer et al., 2014a,b; Miles et al., 2015a,b; Lebeau et al., 2016). Elite athletes have found the best way to time the onset of their QE fixations and saccadic movements to optimally acquire task information at critical times during the movement. Lower level performers benefit from knowing about the experts' sequence of gaze and optimal QE timing. Finally, from the defensive perspective, strategies that disrupt specific QE periods may reduce the effectiveness of the three-point shooter. Defensive strategies should include disruptive defense prior to and during QE catch, and defensive pressure in the visual field of the shooter during QE arm flexion and QE arm extension (Okazaki et al., 2015, p. 12–13; Rojas et al., 2000).

# QE Training Program

fpsyg-10-02424 October 31, 2019 Time: 12:28 # 14

**Figure 6C** shows the QE arm extension duration in the defended condition during hits was very brief (average 190.72). Since this may lead coaches and athletes to attempt to develop a short duration QE in practice, we wondered if participants who had a low QE duration during the undefended condition had a higher percentage during the past season than those with a longer QE duration. We created two sub-groups (Low QE and High QE) based on their quiet eye duration in the undefended condition. Those classified as Low QE had a quiet eye duration ≤250 ms, and the High QE eye group >250 ms. Participants in the High QE group (n = 6) had a higher three-point average the previous season, average 35% than those in the Low QE group (n = 6) average 31%, suggesting the ability to focus for a longer duration during undefended practice conditions leads to better performance under the extreme pressure of competition. Athletes who develop a long duration QE in practice may have developed a neural network that is more easily adapted to handle high pressure conditions, but the opposite may not apply when the only QE duration an athlete possesses is very brief. Athletes who develop a low duration QE in practice may be less able to increase or decrease the quiet eye duration in response to the variable conditions of competition. Based on the results of this study, a QE training program in the basketball one-time three-point shot is recommended as follows:


## LIMITATIONS AND RECOMMENDATIONS

The main limitation in the current study is the low number of participants with three-point shooting averages high enough over a full season to be classified as experts (N = 12). This is a common problem in expertise research, where the number of experts is usually relatively low. We expect this problem to improve in the coming years as more athletes perfect the three-point shot. A second limitation is that the results apply to the three-point shot, but we encourage studies in other basketball shots, as well as other motor tasks in which multiple quiet eye periods can be isolated, each prior to a biomechanical phase of the movement. It is critical an equal number of successful and unsuccessful trials be included, while leaving free to vary other conditions, such as the timing of the quiet eye periods, and the location of the quiet eye in each phase, as was done in current study. We realize this is a new experimental paradigm, but one we feel it will open up new avenues of understanding not only the nature of motor expertise, but also motor learning and control, and the importance of empirically defining the moment of optimal focus and attention.

# CONCLUSION

The results of this study greatly expand the explanatory power of multiple QE periods, with each initiated prior to the onset of a specific phase of the movement. Not only is perception-action coupling investigated relative to more biomechanical phases, but other factors are considered across the entire trial, such as the timing of QE fixations across the motor phases, the location of the QE in each phase, and the onset, offset and duration of each QE period by phase. To our knowledge this is the first study to show that the ability to fixate a target declines with the phases of the movement when extreme pressure is encountered. This is also the first eye tracking study to show that aiming to a far target is aided when ego-centric gaze control of the QE occurs prior to a specific phase of the movement. We also show that in the three-point shot there is an optimal moment when a fixation on the hoop centre had its greatest impact, and this occurred during QE arm flexion. We can therefore speculate that an elite athlete like Stephen Curry tracks the ball closely as it leaves the passers hand, followed by an early QE catch offset, an early saccade to the target, and an early QE fixation on the centre of the hoop before and during arm flexion for a duration of around 300 ms, and a rapid, automatic shooting action that is oblivious to the actions of the defender.

## DATA AVAILABILITY STATEMENT

The datasets generated for this study are available on request to the corresponding author.

# ETHICS STATEMENT

The studies involving human participants were reviewed and approved by the research protocol was approved prior to data collection by the Conjoint Ethics Committee of the University of Calgary, and all participants gave consent. The patients/participants provided their written informed consent to participate in this study.

### AUTHOR CONTRIBUTIONS

fpsyg-10-02424 October 31, 2019 Time: 12:28 # 15

JV, JC, and DV made substantial contributions to conception and design, acquisition of data, and interpretation of data. JV carried out the analysis and drafted the manuscript. JC and DV participated in revising it critically for important content and final approval of the version submitted. All authors accepted the final manuscript.

#### REFERENCES


#### ACKNOWLEDGMENTS

The authors wish to thank the elite basketball players and coaches who volunteered their time to participate in this study. Thanks to Jordyn Vinneau, Fabian Hoitz, and Rob Vickers who helped with the data collection and processing of the data, and to Benno and Sandro Nigg for their on-going support. Our sincerest thanks to Julian Parris of JMP (SAS), who assisted with the statistical analysis.


Lebeau, J. C., Liu, S., Saenz-Moncaleano, C., Sanduvete-Chaves, S., Chacon-Moscoso, S., Becker, B. J., et al. (2016). Quiet eye and performance in sport: a meta-analysis. J. Sport Exerc. Psychol. 38, 441–457. doi: 10.1123/jsep.2015-0123



**Conflict of Interest:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The reviewer SR declared a past collaboration with one of the authors, JV, to the handling Editor.

Copyright © 2019 Vickers, Causer and Vanhooren. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.