Intra-Auditory Integration Improves Motor Performance and Synergy in an Accurate Multi-Finger Pressing Task

Introduction

For nearly every real-world case of motor behavior, the central nervous system (CNS) simultaneously receives information from a variety of sensory modalities, including visual, tactile, and auditory signals (Ernst and Banks, 2002; Stein and Stanford, 2008; Berniker and Kording, 2011), and is required to coordinate multiple effectors (e.g., motor units, muscles, joints, limbs) to effectively perform the movement (Babinski, 1899; Castiello, 1997; Alexandrov et al., 1998; Shim et al., 2003b). Many theories of motor control argue that integration of these multiple sensory modalities with the synergistic control of multiple effectors is critical for successful or optimal motor performance (Ernst and Banks, 2002; Berniker and Kording, 2011; Pantall et al., 2012; Scholz et al., 2012). In effectively integrating multi-sensory information and controlling multi-effector system, the CNS faces at least two challenges: it must estimate the state of the system with sensory information that is corrupted by noise and transmission delays (Körding and Wolpert, 2004; Faisal et al., 2008), and it must solve the motor redundancy problem when deciding how effectors are used from a theoretically infinite set of possibilities (Bernstein, 1967; Gel'Fand, 1971; Ting and Macpherson, 2005; Latash et al., 2007).

Concerning the first problem (state estimation), previous studies on inter-sensory integration (e.g., visual and auditory modalities) have suggested that the CNS combines multiple sensory modalities to produce minimum variance in the state estimate. Bayesian inference is a promising approach to interpret integration of multiple sensory modalities and to define the statistically optimal way to maximize accuracy of the state estimation (Ernst and Banks, 2002; Berniker and Kording, 2011). Relatedly, each individual sensory modality consists of multiple components that can be perceived distinctively by the CNS, such as the color and shape of an object in vision, or the frequency and intensity of a sound in audition (Helmholtz and Ellis, 1885; Bloj et al., 1999). However, our knowledge of how the CNS integrates these different components within a modality (i.e., intra-sensory integration) is very limited. For example, most previous studies on intra-sensory integration were on vision (Jacobs, 1999; Knill and Saunders, 2003; Hillis et al., 2004), with relatively little known about intra-sensory integration of auditory feedback. Sound is physically a pressure wave transmitted through the air, and humans can normally hear sound waves with frequencies up to ~10,000 Hz and intensities up to ~120 dB. These two physical quantities (frequency, intensity) are the most salient features of sound that contribute to its perception, and they appear to be perceived independently by the CNS (Helmholtz and Ellis, 1885; Zagorski, 1975). However, it remains unclear if and how the CNS integrates auditory frequency and intensity for state estimation.

Concerning the second problem (motor redundancy), the control of multiple motor effectors has been studied at the level of muscles (Babinski, 1899; Smith, 1993; Ting and McKay, 2007), joints (Castiello, 1997; van der Kamp and Steenbergen, 1999; Latash and Jaric, 2002), and body segments (Alexandrov et al., 1998; Kim et al., 2012), including fingers (Latash, 2000; Latash et al., 2002, 2007; Shim et al., 2003b, 2005b, 2008). From these studies, the notion of motor synergy has been developed to describe the interactions between multiple motor effectors for the successful completion of a given motor task (Latash et al., 2007). One of our previous simulation studies suggested a central back coupling hypothesis showing that it was theoretically possible to generate the synergistic actions between motor effectors in a feedforward system without an active involvement of sensory feedback (Latash et al., 2005). However, several previous studies suggested that sensory feedback plays an important role in synergistic interactions between motor effectors (Ranganathan and Newell, 2008; Shim et al., 2012; Koh et al., 2015). Ranganathan and Newell (2008) found that the removal of visual sensory feedback from combined visual and tactile feedbacks resulted in decreased synergistic actions between finger forces in both within-trial (online) and between-trial (offline) during multi-finger pressing. In addition, our recent work showed that the removal of tactile feedback in multi-finger pressing tasks resulted in decreases in multi-finger synergies for online control (Koh et al., 2015). These studies suggest that sensory feedback plays critical roles in multi-finger synergies and sensory integration may also systematically influence the synergistic interactions between motor effectors. Especially, the results of these studies imply that sensory integration may lead to improvements in multi-finger synergy (i.e., decrease covariance between finger forces).

Previous studies have shown that multi-finger actions are controlled in a hierarchical manner with at least two hierarchical levels: individual finger actions at the lower level, and virtual finer (VF: an imagined finger producing the same mechanical effect as all fingers together) actions through the combined mechanical outcomes of individual finer actions at the higher level (Arbib, 1985; MacKenzie and Iberall, 1994; Baud-Bovy and Soechting, 2001; Shim et al., 2005a,b), at the higher level. In our recent work, we developed hierarchical variability decomposition (HVD) model to quantify the hierarchical organization of multi-finger actions with the separate analysis of within-trial (online) and between-trial (offline) motor behaviors (Koh et al., 2015). The HVD model decomposes the variability in the motor system into mathematically independent components each of which quantifies a distinct motor behavior. In the HVD model, the systematic error, consistency, and repeatability are quantified at the higher level. In a constant force production task, the systematic error reflects the CNS's ability to estimate the target force, and consistency reflects the CNS's ability to perform the task on a moment-to-moment basis (i.e., online variability), while repeatability reflects the ability to reproduce the same task goal on trial-to-trial basis (i.e., offline variability). The analysis of online and offline variability were previously used to infer different aspects of the control mechanism in redundant motor systems (Scholz et al., 2003; Ranganathan and Newell, 2008; Koh et al., 2015). In the HVD model, both consistency and repeatability can be explained by the sum of individual finger force variances and the amount of error compensation (or amplification) among individual finger forces (i.e., motor synergy; Latash et al., 2005; Martin et al., 2009) at the lower level. Here, we tested how the intra-auditory integration affects these several aspects of multi-finger actions quantified through the HVD model.

In the present study, we therefore investigated the effect of intra-auditory integration between frequency and intensity of sound on motor performance and motor synergy during a constant multi-finger force production. We hypothesized that motor task errors would decrease when both frequency and intensity components are presented to subjects as compared to the conditions that present only one of these auditory components to them, which would support Bayesian integration between frequency and intensity of sound. We also hypothesized that intra-auditory integration would be associated with enhanced multi-finger synergy, which would indicate an evidenced role of intra-sensory integration in the interactions between redundant motor effectors in humans.

Methods

Participants

Ten right-handed male volunteers (age 24.2 ± 1 years) participated in the study. Participants were free of neurological, psychiatric, speech-language, and motor impairments. No participant had more than 1 year of musical training. Participants provided written informed consent. All procedures were approved by the University of Maryland College Park Institutional Review Board.

Experimental Setup

Finger pressing forces at the distal phalanges were collected using load cells (Nano 17, ATI Industrial Automation, Apex, NC, US) at a sampling frequency of 1000 Hz with data acquisition hardware (6024E, National Instruments Corporation, Austin, TX, US) using a custom LabVIEW program (LabVIEW 8.2, National Instruments Corporation, Austin, TX, US). The program interfaced with a function generator (Agilent 33522A, Keysight Technologies, Inc., Santa Rosa, CA, US) to register the individual finger forces and calculate the virtual finger (VF) force as the sum of individual finger forces. The program also generated auditory signals played through the left and right ears of headphones worn by the subjects (AE2, Bose Corporation, Framingham, MA, US).

In order to minimize distortion of sound due to headphone frequency response characteristics (Jackson and Vinegar, 1979), the auditory signal was calibrated to produce a constant intensity across all frequencies. Calibration was performed in a soundproof room by manipulating frequency from 20 to 10,000 Hz in 1 Hz increments and normalizing intensity at each increment.

Procedures

Subjects sat on a chair, wore the headphones, and placed the tips of their right hand's fingers (index, middle, ring, and little) on the load cells (Figure 1). The subjects were asked to use these fingers to produce a constant VF force of 18 N over 20 s while they received auditory feedback tones on the reference force (left ear) and on the VF force (right ear). The tone for the reference force (i.e., the reference tone) had frequency of 1000 Hz and intensity of 70 dB. The tone for VF force (i.e., the tracking tone) varied in three different experimental conditions:

1) Frequency condition (F): the frequency of the tracking tone was modulated with the deviation of the subject's VF force from 18 N, while the intensity of the tracking tone was kept constant at 70 dB.

2) Intensity condition (I): the intensity of the tracking tone was modulated with the deviation of the subject's VF force from 18 N, while the frequency of the tracking tone was kept constant at 1000 Hz.

3) Frequency and Intensity condition (FI): both frequency and intensity of the tracking tone was modulated with the subject's VF force.

FIGURE 1

Figure 1. Experimental setup. The subjects sat and placed their right hand finger tips on the sensors, wearing the headphones (A). During the task, the reference force (18 N) was provided as an auditory signal, a sinusoid signal with frequency 1000 Hz and intensity 70 dB (i.e., reference tone), played to the subject's left ear (B in red). The force generated by the subject was also provided as a sinusoid signal (i.e., tracking tone) to their right ear according to three feedback conditions: frequency (F) in which the frequency of the tracking tone changed, intensity (I) in which the intensity of the tracking tone changed, and frequency and intensity condition (FI) in which both the frequency and the intensity of the tracking tone changed according to the force the subject generated (B in blue).

The frequency of 1000 Hz for the reference tone was used to minimize the influence of binaural beats (Oster, 1973; Wahbeh et al., 2007), and the intensity of 70 dB for the reference tone was selected as mid-level hearing. The feedback gains of frequency and intensity were 7 Hz/N and 0.7 dB/N, respectively, according to “just noticeable differences” of frequency and intensity previously reported assuming a change in force of 1 N (Moore, 1973; Ozimek and Zwislocki, 1996). Prior to the data collection, participants were given five familiarization trials that were excluded from analysis. From each 20-s trial, the 11-s window from 6 to 17 s where the VF force was relatively constant was extracted for analysis (Koh et al., 2015) in order to avoid the initial force stabilization in the beginning of each trial and premature cessation of force production at the end. Subjects completed 15 trials and with rest 30 s rest between trials to prevent fatigue. The order of conditions was balanced across the subjects.

Analysis and Bayesian Model

The Bayesian approach in sensory integration research describes how sensory information can be used to update our knowledge of some quantity of interest (Jacobs, 1999; Ernst and Banks, 2002; Berniker and Kording, 2011). Our study had the following assumptions for analysis:

1. The Bayesian model for the sensory integration appropriate for static performance can also be used for dynamic performance (Ronsse et al., 2009). Let the random variable S be the true unknown VF force. If S has the specific value s, then (i) the estimated VF force based on frequency feedback, Ŝ_F, and estimated VF force based on intensity feedback, Ŝ_I, are independent Gaussian random variables with means s+b_F and s+b_I and variances ${\sigma }_F^2$ and ${\sigma }_I^2$, respectively, where b_F and b_I and are biases; (ii) the estimated VF force based on both frequency and intensity feedback, Ŝ_FI, is the Bayesian estimate assuming a uniform prior over the range of feasible VF forces and ignoring the biases b_F and b_I (Scarfe and Hibbard, 2011; Shi et al., 2013). Let s+b_FI and ${\sigma }_{FI}^2$ denote the mean and variance, respectively, of the random variable Ŝ_FI, where b_FI is the bias of the combined estimate.

2. The variances of VF force in the F, I, and FI conditions primarily reflect the estimation variances, ${\sigma }_F^2$, ${\sigma }_I^2$, and ${\sigma }_{FI}^2$, respectively. Additional variability in VF force caused by noise in other sensory systems and noise in the motor system is assumed to be small in all experimental conditions. The biases b_F, b_I, and b_FI equal f_T minus the mean VF force in the F, I, and FI conditions, respectively, where f_T is the reference force of 18 N.

Based on these assumptions of the Bayesian model, the posterior probability density describing our knowledge of S given specific measurements Ŝ_F = ŝ_F and Ŝ_I = ŝ_I is:

$\begin{array}{l}p({Ŝ}_{FI}=s)=p\left(S=s|{Ŝ}_F={ŝ}_F,{Ŝ}_I={ŝ}_I\right)\\ =\frac{p\left({Ŝ}_F={ŝ}_F|S=s\right)p\left({Ŝ}_I={ŝ}_I|S=s\right)}{{\displaystyle {\int }_{-\infty }^{\infty }}p\left({Ŝ}_F={ŝ}_F|S=s\right)p\left({Ŝ}_I={ŝ}_I|S=s\right)ds}.\end{array}$

The posterior has a Gaussian distribution. Its mean, ŝ_FI, is a weighted sum of the estimates ŝ_F and ŝ_I:

$\begin{array}{l}{ŝ}_{FI}={\text{w}}_{\text{F}}{ŝ}_F+{\text{w}}_{\text{I}}{ŝ}_I,\end{array}$

where ${\text{w}}_{\text{F}}=\frac{{\sigma }_I^2}{{\sigma }_F^2+{\sigma }_I^2}$ and ${\text{w}}_{\text{I}}=\frac{{\sigma }_F^2}{{\sigma }_F^2+{\sigma }_I^2}$. The variance of the posterior, ${\sigma }_{FI}^2$, is smaller than the variances of the frequency estimate, ${\sigma }_F^2$, and intensity estimate, ${\sigma }_I^2$:

$\begin{array}{l}{\sigma }_{FI}^2=\frac{{\sigma }_F^2{\sigma }_I^2}{{\sigma }_F^2+{\sigma }_I^2}\end{array}$

The combined bias b_FI can also be expressed as a weighted average of the F condition bias, b_F, and the I condition bias, b_I with the weights, w_F and w_I (Scarfe and Hibbard, 2011; Shi et al., 2013):

$\begin{array}{l}{b}_{FI}={\text{w}}_{\text{F}}{b}_F+{\text{w}}_{\text{I}}{b}_I\end{array}$

To test whether intra-auditory components, frequency and intensity, are integrated for the motor performance enhancement, we calculated the overall mean-squared error (OMSE) from the experimental data, the averaged squared deviation of the VF force from the reference force as a measure of motor performance:

$\begin{array}{l}OMSE=\frac{1}{N}{\displaystyle \sum _{i=1}^N}\left\{\frac{1}{\tau }\int {\left[f_T-y_i\left(t\right)\right]}^{2}dt\right\}\end{array}$

where y_i(t) is VF force at trial i, and τ is the duration of y_i(t).

In order to examine if the intra-auditory integration follows the Bayesian model, OMSE_FI was also estimated according to Bayesian model using the bias and variance values we experimentally obtained from the F condition and I condition:

$\begin{array}{l}{OMSE}_{FI}={\sigma }_{FI}^2+b_{FI}^2=\frac{{\sigma }_F^2{\sigma }_I^2}{{\sigma }_F^2+{\sigma }_I^2}+{\left({\text{w}}_{\text{F}}{b}_F+{\text{w}}_{\text{I}}{b}_I\right)}^2\end{array}$

OMSE_FI estimated from the model was then compared to OMSE obtained from the experiment (Figure 2).

FIGURE 2

Figure 2. Left figures show VF force time courses for all 15 trials from a representative subject for F condition (red), I condition (green), and FI condition (blue). Right Figure shows best fitted Gaussian distribution of VF force for all conditions and a Gaussian distribution calculated by the Bayesian model (black).

Hierarchical Structure and Decomposition of Motor Variability

The VF force for trial i, y_i(t), was modeled as the sum of three components:

$\begin{array}{l}{y}_i\left(t\right)=X_i\left(t\right)+E_i+m\end{array}$

where X_i(t) is the demeaned VF force for trial i, m is the mean VF force after averaging over all time steps of all 15 trials, and E_i is the difference between the mean VF forces for trial i and m.

OMSE was partitioned into three error components that influence the motor performance following the HVD model (Koh et al., 2015): the “online variance” $\overline{{\sigma }_X^2}$, defined as the variance within a trial, averaged over trials, the “offline variance” ${\sigma }_E^2$, defined as the variance between trials, and the “systematic error,” b², defined as squared deviation between the reference force and m. The sum of online and offline variances is the variance of the VF force (${\sigma }^2=\overline{{\sigma }_X^2}+{\sigma }_E^2$), and the systematic error is the squared bias of the VF force ($b_{FI}^2={(18N-m)}^2$). The online and offline variances can also be defined in terms of the individual finger force variances and covariances as shown below.

$\begin{array}{l}OMSE=\overline{{\sigma }_X^2}+{\sigma }_E^2+b^2=\overline{{\displaystyle \sum _{i=1}^n}{\sigma }_{x_i}^2}+\overline{{\displaystyle \sum _{i\ne j}^n}{\sigma }_{x_i,x_j}}+{\displaystyle \sum _{i=1}^n}{\sigma }_{e_i}^2\\ +{\displaystyle \sum _{i\ne j}^n}{\sigma }_{e_i,e_j}+b^2\end{array}$

where n is the number of task fingers (n = 4), ${\sigma }_{x_i}^2$ and σ_xi, _xj are a variance of within-trial ith finger force and a covariance between within-trial ith and jth finger forces, respectively, and ${\sigma }_{e_i}^2$ and σ_ei, _ej are a variance and covariance for between-trial finger forces, respectively. The overhead bars indicate means over trials.

According to the HVD model (Figure 3), the online and offline errors can be further defined as the sum of the individual finger force variances, plus between-finger covariances. Here, we use the covariances between finger forces, $\overline{\sum {\sigma }_{x_i,x_j}}$ and ∑σ_ei, _ej, as indices of multi-finger synergy for online and offline, respectively (Koh et al., 2015). Negative covariance indicates that the individual finger forces synergistically act to compensate each other errors and attenuate the VF force error (i.e., error compensation), while positive covariance means that individual finger forces act synergistically, but each finger's errors are accumulated and the VF force error increases (i.e., error amplification; Shim et al., 2003a; Latash et al., 2007). The covariances between finger forces, $\overline{\sum {\sigma }_{x_i,x_j}}$ and ∑σ_ei, _ej that we use as an index of multi-finger synergy are mathematically similar to the motor synergy indices used in other previous studies (Shim et al., 2004; Latash, 2008, 2010; Ranganathan and Newell, 2008; Karol et al., 2011). One advantage of using covariances is that the total variance can be linearly decomposed into individual finger variances and inter-finger covariances (Koh et al., 2015). Therefore, covariances were used in this study as indices of motor synergies. We define motor synergy as task-specific interactions between motor effectors employed by CNS for enhancement of motor performance and utilization of degrees of freedom in a redundant motor system (Latash et al., 2007; Shim et al., 2008). The sum of individual finger force variances is the total variance in the motor task, while the between-finger covariance reflects synergistic interactions of individual finger forces. The total variance of the VF force (i.e., the sum of individual finger variances) was also decomposed into task-relevant and task-irrelevant variances according to the Uncontrolled Manifold theory (Scholz and Sch¨oner, 1999; Shim et al., 2004; Latash, 2010) to examine the task-relevant and task-irrelevant variances related to motor synergy.

FIGURE 3

Figure 3. Hierarchical organization of multi-finger force variability. The overall mean squared error (OMSE) is composed of or the linear sum of the variance of the VF force (σ²) and the systematic error (b²). The variance of the VF force (σ²) is composed of or the linear sum of the intra-trial moment-to-moment variance (online variance; $\overline{{\sigma }_X^2}$), the time-averaged trial-to-trial variance (offline variance; $\overline{{\sigma }_E^2}$), and at the virtual finger (VF) level where the task is performed with the sum of all finger forces (VF force). The online and offline variances at the VF level are composed of or the linear sum of individual finger (IF) force variances $\left(\overline{{\displaystyle \sum _{i=1}^n{\sigma }_{x_i}^2}}\text{ and }{\displaystyle \sum _{i=1}^n{\sigma }_{e_i}^2}\right)$ and between-finger force covariances $\left(\overline{{\displaystyle \sum _{i\ne j}^n{\sigma }_{x_i,x_j}^2}}\text{ and }{\displaystyle \sum _{i\ne j}^n{\sigma }_{e_i,e_j}}\right)$ at the IF level.

Statistical Analysis

Paired t-test was used to compare OMSE_FI experimentally obtained and OMSE statistically estimated. One-way repeated-measures ANOVA with Fisher's post hoc pairwise multiple comparisons were used to compare three feedback conditions. A simple regression analysis was performed to determine whether either of mean or variance of VF force at each trial increase or decrease as a function of trial. The level of statistical significance was set at p = 0.05.

Results

Effects of Intra-Auditory Integration on Motor Performance

We found that intra-auditory integration followed Bayesian model. The OMSE_FI calculated from the Bayesian model did not differ from the experimentally obtained OMSE [t₍₉₎ = 0.896, p = 0.393]. There were also no differences between the model and the experimental data for VF variance [t₍₉₎ = 0.614, p = 0.554] or the systematic error [t₍₉₎ = 2.10, p = 0.065; Figure 4]. We also found that intra-auditory integration enhanced motor performance quantified as OMSE. Repeated-measures ANOVA showed that OMSE experimentally obtained from the FI condition was smaller than both OMSE_F and OMSE_I from the F condition and the I condition, respectively [F_{(2, 9)} = 12.76, p < 0.001; FI vs. F: p = 0.003, and FI vs. I: p = 0.004]. Using HDV model, OMSE decomposed into online and offline variance of the VF force, and the systematic error, which are reflected as consistency, reproducibility, and accuracy of motor task, respectively. We found that the enhanced motor performance achieved by improving consistency and reproducibility of the VF force with unchanged accuracy. The variance of the VF force in the FI condition was also smaller than those from the F condition and the I condition for both online control [F_{(2, 9)} = 8.92, p = 0.002; FI vs. F: p = 0.009, and FI vs. I: p = 0.002] and offline control [F_{(2, 9)} = 10.07, p = 0.001; FI vs. F: p = 0.003, and FI vs. I: p = 0.015]. The systematic errors did not differ across the feedback conditions [F_{(2, 9)} = 2.088, p = 0.153]. Thus, the decreases in OMSE were mainly due to the reduction of both online and offline variances, rather than systematic errors.

FIGURE 4

Figure 4. VF level variables: overall mean square error (OMSE), varaince, sestematic error, and online and offline variances at the VF level for F condition (red), I condition (green), and FI condition (blue). These variables are associated with task performance. Error bars represent s.e.m. across subjects. OMSE calculated from the Bayesian model is also shown. Asterisk indicates statistical significance (P < 0.05).

Effects of Intra-Auditory Integration on Motor Synergy

The VF variance in this model is mathematically equivalent to the sum of individual finger variances and covariances for both online and offline control, which allows us to investigate the source of changes in VF variances. We found that intra-auditory integration enhanced both online and offline motor synergy. Repeated-measures ANOVA revealed that online individual finger variance [F_{(2, 9)} = 6.86, p = 0.006; FI vs. F: p = 0.019, and FI vs. I: p = 0.016] and covariance [F_{(2, 9)} = 7.38, p = 0.005; FI vs. F: p = 0.011, and FI vs. I: p = 0.003] were lower in the FI condition than in the F and I conditions (Figure 5). In contrast, offline IF variance in the FI condition was smaller than the F condition, but there was no difference between the FI condition and the I condition [F_{(2, 9)} = 9.92, p = 0.001; FI vs. F; p = 0.002, and FI vs. I; p = 0.064]. Offline individual finger covariance was smaller in the FI condition than the F and I conditions [F_{(1.276, 9)} = 9.85, p = 0.006; FI vs. F: p = 0.004, and FI vs. I: p = 0.019]. Note that smaller individual finger covariance represents smaller error amplification between individual finger forces (Koh et al., 2015). In analyses of task-relevant and task-irrelevant spaces, both online and offline individual finger variances in the task-relevant space were smaller in the FI condition as compared to the F and I conditions [online: F_{(2, 9)} = 8.92, p = 0.002; FI vs. F: p = 0.009, and FI vs. I: p = 0.002, and offline: F_{(2, 9)} = 10.07, p = 0.001; FI vs. F: p = 0.003, and FI vs. I: p = 0.015], while both online and offline individual variances in task-irrelevant space remained unchanged throughout the feedback conditions [online: F_{(2, 9)} = 2.725, p = 0.093, and offline: F_{(2, 9)} = 1.001, p = 0.387]. Online individual finger covariance in both F and I conditions was statistically greater than zero [F: t₍₉₎ = 3.020, p = 0.014 and I: t₍₉₎ = 3.997, p = 0.003] while online covariance in FI condition did not significantly differ from zero [t₍₉₎ = 1.429, p = 0.187]. Offline individual finger covariance in all conditions was statistically greater than zero [F: t₍₉₎ = 6.056, p < 0.001, I: t₍₉₎ = 6.845, p < 0.001, and FI: t₍₉₎ = 4.326, p = 0.002].

FIGURE 5

Figure 5. Multi-finger force covariance and individual finger force variance for online and offline controls for F condition (red), I condition (green), and FI condition (blue), at the individual finger (IF) level. The individual fingers' contributions to the overall covariance and variance values are shown in the 4-dimensional diamond graphs (A,C,E,G). Error bars represent s.e.m. across subjects. The accompanying bar graphs show the overall covariance and variance values (B,D,F,H). The covariance and variance, only in online control, showed statistically significant differences between FI and either F or I conditions. The overall variance values are decomposed into task-relevant and task-irrelevant variance values (I–L). Task-relevant variance for both online and offline controls showed statistically significant differences between FI and either F or I condition. Asterisk indicates statistical significance (p < 0.05).

Learning Effects

We also tested whether a learning effect exists using a simple linear regression. The mean and variance of VF force across time at each trial was calculated to test if these values significantly increase or decrease as a function of trial. No significant regression was found for mean of VF force in any of the conditions [F: F_{(1, 13)} = 4.296, p = 0.059, I: F_{(1, 13)} = 0.050, p = 0.827, and FI: F_{(1, 13)} = 0.108, p = 0.748], along with no significant regression for variance [F: F_{(1, 13)} = 4.515, p = 0.053, I: F_{(1, 13)} = 2.313, p = 0.152, and FI: F_{(1, 13)} = 2.392, p = 0.146].

Discussion

Previous studies on sensory integration have mainly focused on the CNS mechanisms of inter-sensory integration, e.g., visual and auditory, visual and tactile, or visual and proprioceptive (Ernst and Banks, 2002; Alais and Burr, 2004; Helbig and Ernst, 2007; Butler et al., 2010; Fetsch et al., 2010; Reuschel et al., 2010). The current study, for the first time, showed how intra-sensory integration of frequency and intensity in auditory feedback affected motor performance and multi-finger synergy in a constant finger force production task. We reported that motor performance could be enhanced through intra-auditory integration as evidenced by improved tracking of the target force by the VF force. In addition, we report that multi-finger synergy in producing that VF force is enhanced through intra-auditory integration as evidenced by decreased covariance between individual finger forces.

Our study employed the hierarchical organization of multi-finger actions and used the HVD model previously proposed by our group for the separate analysis of within-trial (online) and between-trial (offline) motor behaviors (Koh et al., 2015). We found that providing feedback on both intensity and frequency was associated with the decreased covariances among individual finger forces in both online and offline control. This result indicates that the CNS responded to the presentation of feedback on multiple auditory components by decreasing error amplification among individual finger forces, resulting in both greater consistency and greater repeatability of the VF force.

Positive covariance between individual finger forces indicates that individual finger forces synergistically interact (Latash, 2010), but amplify the VF force error in the constant multi-finger pressing task, while negative covariance implies that individual finger forces interact to attenuate the VF force error, or to maintain a relatively constant VF force when individual finger forces are fluctuating. The ability to produce a particular VF with fluctuating individual finger forces can be viewed as a sign of versatility or flexibility within the motor system. The current study showed that the intra-auditory integration was associated with decreased positive covariance between individual finger forces, i.e., decreased error amplification at VF level. Many previous studies that employed visual force feedback often reported negative covariance between individual finger forces in similar tasks of constant VF force production (Shim et al., 2003b, 2004; Latash, 2010; Karol et al., 2011). Contrary to these studies, our study showed positive covariance between individual finger forces. This covariance difference may be due to the differences in the level of feedback uncertainty in sensory feedback the CNS received, leading to relatively high task uncertainty introduced by the auditory feedback used in our study as compared to the visual feedback used in the previous studies. This speculation is consistent with the study by Ranganathan and Newell (2008) in which they demonstrated that the covariances between individual finger forces could be systematically modulated by the uncertainty of the sensory feedback. Reduction in positive covariance between individual finger forces with intra-auditory integration would also be consistent with the principle of minimal interactions (Gelfand and Tsetlin, 1966), where interactions between low-level elements of a hierarchy are organized such that the inputs from one element to another (i.e., interactions) are minimized in an optimal system.

In addition, it was observed that offline individual finger variances and covariances were 2–5 times greater than online counterparts. The large differences between online and offline control suggest that the individual finger forces fluctuated more and the coupling between individual finger forces were stronger between trials as compared to within a trial. This observation indicates that the CNS used larger “workspaces” of individual finger forces and the inter-finger interactions were greater between individual finger forces in offline control than in online control. The large fluctuation of individual fingers forces in offline may be contributed by two different mechanisms: active utilization of the redundant degrees of freedom in the multi-finger system in offline control, and/or greater noise in the offline part of the control system. According to the Bayesian model, prior knowledge about the relative likelihood of different stimulus values can be integrated with new sensory information to improve estimates of stimulus values (Körding and Wolpert, 2004). Prior knowledge is task-specific and achieved through experience with that particular task. Although our analysis assumes no prior knowledge, it may be possible that prior knowledge gained from previous moments within a trial or previous completed trials may have contributed to the motor outcomes. If it is indeed the case, it is more likely that the prior knowledge influenced motor behaviors more within a trial as the sensory feedback is continuously provided online contrary to the offline counterpart. Future research specifically on prior knowledge and its impact on both online and offline control may enhance our understanding of the role of intra-auditory integration in motor performance and motor synergy.

Sensory integration research has often used congruent or incongruent multi-sensory information to provide sensory feedback for the same or a different state of physical property, respectively. For example, sound and light signals can come from a same position (congruency) or different positions (incongruency) in an experiment where a subject is instructed to identify the location of the sensory source (Battaglia et al., 2003). The previous studies showed that both congruent and incongruent feedback conditions follow the Bayesian model (Knill and Pouget, 2004; Körding and Wolpert, 2004). Our research investigated only the congruent auditory feedback where we provided the information about the same state of sound due to the technical difficulties to provide different reference force levels required for incongruent auditory feedback. According to the previous studies, force magnitude is associated with force variability (Schmidt et al., 1979; Galganski et al., 1993; Enoka et al., 1999). In other words, the force variability can affect motor performance and potentially motor synergy as well. Because of these reasons, our experiment did not involve incongruent auditory feedback in the effort to prevent adding additional complexity in the study design.

The feedback gains used in the current study were set according to “just noticeable differences” previously reported assuming a change in force of 1 N. The use of the feedback gains through individual auditory sensitivity test may have provided more accurate measures into the intra-auditory integration. In addition, we manipulated two physical quantities, frequency, and intensity. These quantities are transformed as physiological quantities, pitch, and loudness. The psychophysical transformation from frequency-intensity space to pitch-loudness space may not be a homeomorphism. However, the sensitivity test during a constant force production task demands a new method for accurate quantification through careful experiments and modeling because the just noticeable difference may be subject to changes during the dynamic process of finger force production.

In conclusion, we found that both motor performance and motor synergy during a constant multi-finger force production were enhanced through intra-auditory integration, which supports the idea that the CNS can integrate multiple sensory components within the auditory modality to enhance motor outputs consistent with the Bayesian model. In addition, our results provide new evidence that the intra-auditory integration is also associated with improved motor synergy. Although the central back coupling hypothesis proposed in our previous study demonstrated the theoretical possibility of generating synergistic actions between motor effectors without an active involvement of sensory feedback (Latash et al., 2005), the current study provides evidences that the auditory feedback and intra-auditory integration can indeed play an important role multi-finger synergy.

Author Contributions

Study conception and design: KK, HK, and JS. Acquisition of data: KK, HK. Analysis and interpretation of data: KK, HK, and JS. Drafting of manuscript: KK and JS. Critical revision: YP, TK, RM, YK, and JS.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

This study has been supported by National Research Foundation of Korea (NRF-2012R1A6A3A04040457 and NRF-2013R1A1A2004812) and the Translational Research Center for Rehabilitation Robots, Korea National Rehabilitation Center, Ministry of Health and Welfare, Korea (NRCTR-EX15007).

References

Alais, D., and Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal integration. Curr. Biol. 14, 257–262. doi: 10.1016/j.cub.2004.01.029

PubMed Abstract | CrossRef Full Text | Google Scholar

Alexandrov, A., Frolov, A., and Massion, J. (1998). Axial synergies during human upper trunk bending. Exp. Brain Res. 118, 210–220. doi: 10.1007/s002210050274

PubMed Abstract | CrossRef Full Text | Google Scholar

Arbib, M. A. (1985). Coordinated control programs for movements of the hand. Hand Funct. Neocortex 83, 111–129. doi: 10.1007/978-3-642-70105-4_7

CrossRef Full Text | Google Scholar

Babinski, J. (1899). De l'asynergie cérébelleuse. Rev. Neurol. 7, 806–816.

Battaglia, P. W., Jacobs, R. A., and Aslin, R. N. (2003). Bayesian integration of visual and auditory signals for spatial localization. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 20, 1391–1397. doi: 10.1364/JOSAA.20.001391

PubMed Abstract | CrossRef Full Text | Google Scholar

Baud-Bovy, G., and Soechting, J. F. (2001). Two virtual fingers in the control of the tripod grasp. J. Neurophysiol. 86, 604–615.

PubMed Abstract | Google Scholar

Berniker, M., and Kording, K. (2011). Bayesian approaches to sensory integration for motor control. Wiley Interdiscip. Rev. Cogn. Sci. 2, 419–428. doi: 10.1002/wcs.125

PubMed Abstract | CrossRef Full Text

Bernstein, N. A. (1967). The Co-ordination and Regulation of Movements. Oxford: Pergamon Press.

Google Scholar

Bloj, M. G., Kersten, D., and Hurlbert, A. C. (1999). Perception of three-dimensional shape influences colour perception through mutual illumination. Nature 402, 877–879.

PubMed Abstract | Google Scholar

Butler, J. S., Smith, S. T., Campos, J. L., and Bülthoff, H. H. (2010). Bayesian integration of visual and vestibular signals for heading. J. Vis. 10:23. doi: 10.1167/10.11.23

PubMed Abstract | CrossRef Full Text | Google Scholar

Castiello, U. (1997). Arm and mouth coordination during the eating action in humans: a kinematic analysis. Exp. Brain Res. 115, 552–556. doi: 10.1007/PL00005726

PubMed Abstract | CrossRef Full Text | Google Scholar

Enoka, R. M., Burnett, R. A., Graves, A. E., Kornatz, K. W., and Laidlaw, D. H. (1999). Task- and age-dependent variations in steadiness. Prog. Brain Res. 123, 389–395. doi: 10.1016/S0079-6123(08)62873-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Ernst, M. O., and Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429–433. doi: 10.1038/415429a

PubMed Abstract | CrossRef Full Text | Google Scholar

Faisal, A. A., Selen, L. P., and Wolpert, D. M. (2008). Noise in the nervous system. Nat. Rev. Neurosci. 9, 292–303. doi: 10.1038/nrn2258

PubMed Abstract | CrossRef Full Text | Google Scholar

Fetsch, C. R., Deangelis, G. C., and Angelaki, D. E. (2010). Visual-vestibular cue integration for heading perception: applications of optimal cue integration theory. Eur. J. Neurosci. 31, 1721–1729. doi: 10.1111/j.1460-9568.2010.07207.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Galganski, M. E., Fuglevand, A. J., and Enoka, R. M. (1993). Reduced control of motor output in a human hand muscle of elderly subjects during submaximal contractions. J. Neurophysiol. 69, 2108–2115.

PubMed Abstract | Google Scholar

Gel'Fand, I. M. (1971). Models of the Structural-Functional Organization of Certain Biological Systems. Cambridge, MA: MIT Press.

Gelfand, I. M., and Tsetlin, M. L. (1966). On mathematical modeling of the mechanisms of the central nervous ysstem, in Models of the Structural-Functional Organization of Certain Biological Systems, eds I. M. Gelfand, V. S. Gurfinkel, S. V. Fomin, and M. L. Tsetlin (Cambridge, MA: MIT Press), 9–26.

PubMed Abstract | Google Scholar

Helbig, H. B., and Ernst, M. O. (2007). Optimal integration of shape information from vision and touch. Exp. Brain Res. 179, 595–606. doi: 10.1007/s00221-006-0814-y

PubMed Abstract | CrossRef Full Text | Google Scholar

Helmholtz, H. V., and Ellis, A. J. (1885). On the Sensations of Tone as a Physiological Basis for the Theory of Music. London: Longmans, Green.

Google Scholar

Hillis, J. M., Watt, S. J., Landy, M. S., and Banks, M. S. (2004). Slant from texture and disparity cues: optimal cue combination. J. Vis. 4, 967–992. doi: 10.1167/4.12.1

PubMed Abstract | CrossRef Full Text | Google Scholar

Jackson, A. C., and Vinegar, A. (1979). A technique for measuring frequency response of pressure, volume, and flow transducers. J. Appl. Physiol. 47, 462–467.

PubMed Abstract | Google Scholar

Jacobs, R. A. (1999). Optimal integration of texture and motion cues to depth. Vision Res. 39, 3621–3629. doi: 10.1016/S0042-6989(99)00088-7

PubMed Abstract | CrossRef Full Text | Google Scholar

Karol, S., Kim, Y. S., Huang, J., Kim, Y. H., Koh, K., Yoon, B. C., et al. (2011). Multi-finger pressing synergies change with the level of extra degrees of freedom. Exp. Brain Res. 208, 359–367. doi: 10.1007/s00221-010-2486-x

PubMed Abstract | CrossRef Full Text | Google Scholar

Kim, M. J., Karol, S., Park, J., Auyang, A., Kim, Y. H., Kim, S., et al. (2012). Inter-joint synergies increase with motor task uncertainty in a whole-body pointing task. Neurosci. Lett. 512, 114–117. doi: 10.1016/j.neulet.2012.01.072

PubMed Abstract | CrossRef Full Text | Google Scholar

Knill, D. C., and Pouget, A. (2004). The Bayesian brain: the role of uncertainty in neural coding and computation. Trends Neurosci. 27, 712–719. doi: 10.1016/j.tins.2004.10.007

PubMed Abstract | CrossRef Full Text | Google Scholar

Knill, D. C., and Saunders, J. A. (2003). Do humans optimally integrate stereo and texture information for judgments of surface slant? Vision Res. 43, 2539–2558. doi: 10.1016/S0042-6989(03)00458-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Koh, K., Kwon, H. J., Yoon, B. C., Cho, Y., Shin, J. H., Hahn, J. O., et al. (2015). The role of tactile sensation in online and offline hierarchical control of multi-finger force synergy. Exp. Brain Res. 233, 2539–2548. doi: 10.1007/s00221-015-4325-6

PubMed Abstract | CrossRef Full Text | Google Scholar

Körding, K. P., and Wolpert, D. M. (2004). Bayesian integration in sensorimotor learning. Nature 427, 244–247. doi: 10.1038/nature02169

PubMed Abstract | CrossRef Full Text | Google Scholar

Latash, M. (2000). There is no motor redundancy in human movements. There is motor abundance. Motor Control 4, 259–261.

PubMed Abstract | Google Scholar

Latash, M. L. (2008). Synergy. Oxford; New York, NY: Oxford University Press.

Google Scholar

Latash, M. L. (2010). Motor synergies and the equilibrium-point hypothesis. Motor Control 14, 294–322.

PubMed Abstract | Google Scholar

Latash, M. L., and Jaric, S. (2002). Organization of drinking: postural characteristics of arm-head coordination. J. Mot. Behav. 34, 139–150. doi: 10.1080/00222890209601936

PubMed Abstract | CrossRef Full Text | Google Scholar

Latash, M. L., Scholz, J. P., and Schöner, G. (2002). Motor control strategies revealed in the structure of motor variability. Exerc. Sport Sci. Rev. 30, 26–31. doi: 10.1097/00003677-200201000-00006

PubMed Abstract | CrossRef Full Text | Google Scholar

Latash, M. L., Scholz, J. P., and Schöner, G. (2007). Toward a new theory of motor synergies. Motor Control 11, 276–308.

PubMed Abstract | Google Scholar

Latash, M. L., Shim, J. K., Smilga, A. V., and Zatsiorsky, V. M. (2005). A central back-coupling hypothesis on the organization of motor synergies: a physical metaphor and a neural model. Biol. Cybern. 92, 186–191. doi: 10.1007/s00422-005-0548-0

PubMed Abstract | CrossRef Full Text | Google Scholar

MacKenzie, C. L., and Iberall, T. (1994). The Grasping Hand. Amsterdam: Elsevier.

Google Scholar

Martin, J. R., Latash, M. L., and Zatsiorsky, V. M. (2009). Interaction of finger enslaving and error compensation in multiple finger force production. Exp. Brain Res. 192, 293–298. doi: 10.1007/s00221-008-1615-2

PubMed Abstract | CrossRef Full Text | Google Scholar

Moore, B. C. (1973). Frequency difference limens for short-duration tones. J. Acoust. Soc. Am. 54, 610–619. doi: 10.1121/1.1913640

PubMed Abstract | CrossRef Full Text | Google Scholar

Oster, G. (1973). Auditory beats in the brain. Sci. Am. 229, 94–102. doi: 10.1038/scientificamerican1073-94

PubMed Abstract | CrossRef Full Text | Google Scholar

Ozimek, E., and Zwislocki, J. J. (1996). Relationships of intensity discrimination to sensation and loudness levels: dependence on sound frequency. J. Acoust. Soc. Am. 100, 3304–3320. doi: 10.1121/1.416993

PubMed Abstract | CrossRef Full Text | Google Scholar

Pantall, A., Teulier, C., and Ulrich, B. D. (2012). Changes in muscle activation patterns in response to enhanced sensory input during treadmill stepping in infants born with myelomeningocele. Hum. Mov. Sci. 31, 1670–1687. doi: 10.1016/j.humov.2012.04.003

PubMed Abstract | CrossRef Full Text | Google Scholar

Ranganathan, R., and Newell, K. M. (2008). Motor synergies: feedback and error compensation within and between trials. Exp. Brain Res. 186, 561–570. doi: 10.1007/s00221-007-1259-7

PubMed Abstract | CrossRef Full Text | Google Scholar

Reuschel, J., Drewing, K., Henriques, D. Y., Rösler, F., and Fiehler, K. (2010). Optimal integration of visual and proprioceptive movement information for the perception of trajectory geometry. Exp. Brain Res. 201, 853–862. doi: 10.1007/s00221-009-2099-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Ronsse, R., Miall, R. C., and Swinnen, S. P. (2009). Multisensory integration in dynamical behaviors: maximum likelihood estimation across bimanual skill learning. J. Neurosci. 29, 8419–8428. doi: 10.1523/JNEUROSCI.5734-08.2009

PubMed Abstract | CrossRef Full Text | Google Scholar

Scarfe, P., and Hibbard, P. B. (2011). Statistically optimal integration of biased sensory estimates. J. Vis. 11:12. doi: 10.1167/11.7.12

PubMed Abstract | CrossRef Full Text | Google Scholar

Schmidt, R. A., Zelaznik, H., Hawkins, B., Frank, J. S., and Quinn, J. T. Jr. (1979). Motor-output variability: a theory for the accuracy of rapid motor acts. Psychol. Rev. 47, 415–451. doi: 10.1037/0033-295X.86.5.415

PubMed Abstract | CrossRef Full Text | Google Scholar

Scholz, J. P., Kang, N., Patterson, D., and Latash, M. L. (2003). Uncontrolled manifold analysis of single trials during multi-finger force production by persons with and without down syndrome. Exp. Brain Res. 153, 45–58. doi: 10.1007/s00221-003-1580-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Scholz, J. P., Park, E., Jeka, J. J., Schöner, G., and Kiemel, T. (2012). How visual information links to multijoint coordination during quiet standing. Exp. Brain Res. 222, 229–239. doi: 10.1007/s00221-012-3210-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Scholz, J. P., and Schöner, G. (1999). The uncontrolled manifold concept: identifying control variables for a functional task. Exp. Brain Res. 126, 289–306. doi: 10.1007/s002210050738

PubMed Abstract | CrossRef Full Text | Google Scholar

Shi, Z., Ganzenmüller, S., and Müller, H. J. (2013). Reducing bias in auditory duration reproduction by integrating the reproduced signal. PLoS ONE 8:e62065. doi: 10.1371/journal.pone.0062065

PubMed Abstract | CrossRef Full Text | Google Scholar

Shim, J. K., Hsu, J., Karol, S., and Hurley, B. F. (2008). Strength training increases training-specific multifinger coordination in humans. Motor Control 12, 311–329.

PubMed Abstract | Google Scholar

Shim, J. K., Karol, S., Kim, Y. S., Seo, N. J., Kim, Y. H., Kim, Y., et al. (2012). Tactile feedback plays a critical role in maximum finger force production. J. Biomech. 45, 415–420. doi: 10.1016/j.jbiomech.2011.12.001

PubMed Abstract | CrossRef Full Text | Google Scholar

Shim, J. K., Latash, M. L., and Zatsiorsky, V. M. (2003a). The human central nervous system needs time to organize task-specific covariation of finger forces. Neurosci. Lett. 353, 72–74. doi: 10.1016/j.neulet.2003.08.079

PubMed Abstract | CrossRef Full Text | Google Scholar

Shim, J. K., Latash, M. L., and Zatsiorsky, V. M. (2003b). Prehension synergies: trial-to-trial variability and hierarchical organization of stable performance. Exp. Brain Res. 152, 173–184. doi: 10.1007/s00221-003-1527-0

PubMed Abstract | CrossRef Full Text | Google Scholar

Shim, J. K., Latash, M. L., and Zatsiorsky, V. M. (2005a). Prehension synergies in three dimensions. J. Neurophysiol. 93, 766–776. doi: 10.1152/jn.00764.2004

PubMed Abstract | CrossRef Full Text | Google Scholar

Shim, J. K., Latash, M. L., and Zatsiorsky, V. M. (2005b). Prehension synergies: trial-to-trial variability and principle of superposition during static prehension in three dimensions. J. Neurophysiol. 93, 3649–3658. doi: 10.1152/jn.01262.2004

PubMed Abstract | CrossRef Full Text | Google Scholar

Shim, J. K., Lay, B. S., Zatsiorsky, V. M., and Latash, M. L. (2004). Age-related changes in finger coordination in static prehension tasks. J. Appl. Physiol. 97, 213–224. doi: 10.1152/japplphysiol.00045.2004

PubMed Abstract | CrossRef Full Text | Google Scholar

Smith, A. M. (1993). [Babinski and movement synergism]. Rev. Neurol. (Paris). 149, 764–770.

PubMed Abstract | Google Scholar

Stein, B. E., and Stanford, T. R. (2008). Multisensory integration: current issues from the perspective of the single neuron. Nat. Rev. Neurosci. 9, 255–266. doi: 10.1038/nrn2331

PubMed Abstract | CrossRef Full Text | Google Scholar

Ting, L. H., and Macpherson, J. M. (2005). A limited set of muscle synergies for force control during a postural task. J. Neurophysiol. 93, 609–613. doi: 10.1152/jn.00681.2004

PubMed Abstract | CrossRef Full Text | Google Scholar

Ting, L. H., and McKay, J. L. (2007). Neuromechanics of muscle synergies for posture and movement. Curr. Opin. Neurobiol. 17, 622–628. doi: 10.1016/j.conb.2008.01.002

PubMed Abstract | CrossRef Full Text | Google Scholar

van der Kamp, J., and Steenbergen, B. (1999). The kinematics of eating with a spoon: bringing the food to the mouth, or the mouth to the food? Exp. Brain Res. 129, 68–76. doi: 10.1007/s002210050937

PubMed Abstract | CrossRef Full Text | Google Scholar

Wahbeh, H., Calabrese, C., and Zwickey, H. (2007). Binaural beat technology in humans: a pilot study to assess psychologic and physiologic effects. J. Altern. Complement. Med. 13, 25–32. doi: 10.1089/acm.2006.6196

PubMed Abstract | CrossRef Full Text | Google Scholar

Zagorski, M. (1975). Perceptual independence of pitch and loudness in a signal detection experiment: a processing model for 2ATFC (21FC) experiments. Percept. Psychophys. 17, 525–531. doi: 10.3758/BF03203963

CrossRef Full Text | Google Scholar