Evaluating Electromyography and Sonomyography Sensor Fusion to Estimate Lower-Limb Kinematics Using Gaussian Process Regression

Introduction

The continued progression of robotic (i.e., powered) lower-limb assistive devices with enhanced sensing, control and actuation has the ability to improve personalized rehabilitation strategies, restore independence and improve quality of life for individuals with mobility disorders and lower-limb loss. Significant improvements of these devices include reduction in their overall weight, improved energy-transfer capabilities via injection and recycling, and the increase in overall number of powered joints (Price et al., 2019). However, when considering their physical human-robot interaction, the increased complexity of these devices, including the number of powered joints and their capabilities, is met with an increase in the degrees-of-freedom of the overall control problem. Therefore, sensing modalities that can accurately detect motion of multiple degrees-of-freedom for precise mapping to multiple joints’ control parameters are critical for device translation and clinical adoption.

A breadth of wearable sensing modalities and control schemes have been explored for the accurate recognition of user movement intent. The primary control schemes include classifier-based control and volitional control, as well as some combination of the two, often referred to as shared or indirect volitional control. Classifier-based control typically identifies the mode of ambulation or specific task (Hargrove et al., 2011; Huang et al., 2011; Young et al., 2014). Conversely, volitional control allows the user to execute voluntary movements by more directly mapping user intent to a device output, such as angle, velocity or torque (Ha et al., 2011; Kannape and Herr, 2014). Recent studies that evaluated online performance of shared volitional (i.e., semi-volitional) control of powered lower-limb prostheses have proven successful for providing additional capabilities to the user, such as crossing over obstacles during level walking (Mendez et al., 2020). However, to enable users the access to the full range of capabilities of powered assistive devices, such as use during weight-bearing and non-weight-bearing, it may be best to enable a more fully-volitional control scheme, and one that incorporates intent recognition of multiple degrees-of-freedom. When there are multiple powered joints, such as knees and ankles in a transfemoral prostheses or both hips and knees in an exoskeleton, volitional control over multiple degrees-of-freedom simultaneously would be ideal.

Traditionally, researchers have relied on mechanical sensors as well as neural sensors for classifier-based control and volitional control techniques. Mechanical sensors are used for extracting joint kinematics and kinetics with great success, and are advantageous for aiding in shared volitional control (Mendez et al., 2020). However, these sensors record kinematic or kinetic signals as the movement occurs and thus do not always precede impending motion. Non-invasive neural sensors, e.g. surface electromyography (EMG), are commonly used to extract muscle activation information that precedes joint movement (Farina et al., 2004). However, surface EMG is susceptible to muscle crosstalk and accesses muscle activity information from superficial muscles (Farina et al., 2002). Furthermore, muscle activity is only one contributing factor to muscle force production and resulting joint motion (Disselhorst-Klug et al., 2009). Still, EMG remains the primary peripheral sensing modality to probe information about muscle contraction that precedes limb motion. Recent advances in high-density recording and motor unit decomposition have enabled an increased signal dimensionality of EMG and understanding of the motor unit activity within superficial muscles (Merletti et al., 2008; Stango et al., 2015). However, alternative sensing modalities could provide additional information about muscle contraction, especially the contraction of multiple muscles that are within close proximity of one another and those that are not superficial. Advances are needed to enable accurate prediction of joint kinematics for multiple degrees-of-freedom during varying forms of ambulation (e.g., level ground, stair and ramp ambulation).

In comparison to EMG, which provides information about muscle activity, dynamic ultrasound imaging of skeletal muscle tissue, i.e. sonomyography, has the ability to capture real-time information about muscle deformation that is related to both fiber length and contractile velocity, two additional contributing factors to muscle force and resulting joint motion (Hallock et al., 2018). Additionally, sonomyography can access information from both superficial and deep muscles, resulting in increased spatial resolution and overall dimensionality of the signal (Rabe et al., 2021a). Multiple researchers have achieved success using sonomyography for control of multiple degrees-of-freedom in the upper limb (Shi et al., 2010; Hettiarachchi et al., 2015; Dhawan et al., 2019). In addition, previous research demonstrated improvement to classification of discrete ambulation tasks by sonomyography compared to surface electromyography (Rabe et al., 2021a). Further, sonomyography of the lower limb has been used to continuously estimate knee kinematics (Jahanandish et al., 2019b; Rabe et al., 2020a), as well as hip, knee and ankle moments during basic walking tasks and isometric contraction (Rabe et al., 2020b; Zhang et al., 2020; Zhang et al., 2021). However, the ability of sonomyography to continuously predict multiple joint kinematics during widely varying ambulation tasks encompassing level ground, ramp incline, ramp decline, stair ascent and stair descent and how the performance compares to surface EMG as well as the fusion of surface EMG with sonomyography is unknown. Identifying these important relationships would help inform the implementation and capabilities of volitional controllers for advanced lower-limb assistive technologies.

The objective of this research is to use both surface EMG and sonomyography, as well as the fusion of features from both sensing modalities, as inputs to a Gaussian process regression model for the continuous estimation of healthy subjects’ hip, knee and ankle angles and angular velocities during five ambulation tasks. Gaussian process regression is a kernel-based Bayesian technique for nonlinear regression that has been introduced as an alternative to musculoskeletal model-based solutions to inverse dynamics. In this study, regression models were trained and tested in task-invariant frameworks such that strides from all ambulation tasks were included in both the training dataset and testing dataset. We hypothesized that sonomyography would improve regression prediction of hip and knee kinematics in comparison to surface EMG, but that sensor fusion of sonomyography with surface EMG would be required for accurate performance of ankle kinematic estimation. We further hypothesized that prediction of joint angle would outperform prediction of joint angular velocity due to the increased noise and variability of the angular velocity signal—noise that is associated with the use of numerical differentiation to calculate this signal from motion capture data. Results from this research can inform strategies for implementing multiple degree-of-freedom control over powered assistive devices using features extracted from novel sensing modalities such as sonomyography and traditional sensing modalities like EMG.

Materials and Methods

A schematic overview of the methods beginning with data collection through model prediction of joint kinematics is given in Figure 1. This study was reviewed and approved by The University of Texas at Dallas institutional review board. The participants provided their written informed consent prior to participating in this study.

FIGURE 1

FIGURE 1. Schematic representation of methods. Beginning with data collection during five ambulation modes, sensing modality [sonomyography, surface electromyography (EMG) and sensor fusion] feature generation, regression model implementation, and ultimate hip, knee and ankle joint kinematic prediction.

Subjects and Data Collection

Nine able-bodied subjects (five male and four female) were asked to complete two sets of ambulation tasks including: level walk, 10° incline walk, 10° decline walk, stair ascent and stair descent. Additional subject data can be found in Table 1. Prior to two sets of experiments, subjects were equipped with reflective markers for tracking kinematics, surface EMG for recording muscle activity and ultrasound sensors for collection of sonomyography data.

TABLE 1

TABLE 1. Mean and standard deviation (SD) subject characteristics (N = 9).

First, pre-gelled, self-adhesive electrodes (H124SG Covidien, Medtronic Inc., Dublin, Ireland) were placed over the belly of eight muscles with an inter-electrode distance of 2 cm. Electrodes were connected to a Bluetooth EMG unit (Shimmer3 EMG Unit, Shimmer, Dublin, Ireland) and EMG signals were recorded from the adductor magnus (AM), biceps femoris (BF), rectus femoris (RF), vastus lateralis (VL), tensor fascia latae (TFL), medial gastrocnemius (MG), tibialis anterior (TA) and soleus (SOL) muscles at a frequency of 1,200 Hz and streamed in real-time to the same lab computer as ultrasound.

In both sets of experiments, the 128-element linear array transducer of a portable, handheld ultrasound scanner (mSonics, Lonshine Technologies Inc., Beijing, China) was affixed to the thigh of each subject (on the same limb as surface EMG) via a custom-designed probe holder. The ultrasound transducer was placed on the anterior thigh during the first set of ambulation tasks, and on the posterior thigh during the second set of ambulation tasks. Priority was given to surface EMG to ensure optimal recording from the desired muscles when placing the ultrasound transducer on the anterior and posterior thigh. For anterior thigh ultrasound trials, the transducer was placed transversely at approximately half the distance between the anterior superior iliac spine and the proximal base of the patella (above the surface EMG electrodes) to collect grayscale images of the rectus femoris (RF), vastus medialis (VM) and vastus intermedius (VI) muscles. During the posterior thigh ultrasound trials, the transducer was placed transversely over the belly of the biceps femoris (BF) and semitendinosus muscles (below the surface EMG electrodes). Correct ultrasound placement, transversely over the belly of the desired muscles (Fukunaga et al., 1997), was confirmed visually via grayscale images prior to securing the transducer in both anterior thigh and posterior thigh positions. Ultrasound transmit frequency was set to 7.5 MHz for all trials and the overall image gain was adjusted to optimize the brightness of the image and preserve image resolution at the deep muscle boundary.

Forty-two reflective markers were placed over anatomical landmarks following guidelines by Delagi et al. on the bilateral feet, shanks, and thighs, as well as trunk and pelvis for collection of kinematic data via a ten-camera Vicon system (Vicon Motion Systems, Oxford, UK) (Delagi et al., 2011). Kinematic data was recorded to the same lab computer as ultrasound and surface EMG at a sampling rate of 100 Hz (Peng et al., 2016). Following sensor placement, all subjects were asked to complete two sets of five ambulation tasks: level walk, 10° incline walk, 10° decline walk, stair ascent and stair descent. All walking tasks were completed for 1 minute at a self-selected pace on a split-belt treadmill (Bertec, Columbus, OH, United States). Stair trials were completed on a four-step stair case, beginning with stair ascent followed by stair descent, with subjects walking in a reciprocal gait pattern at a self-selected pace. Stair trials were repeated five times and walk-to-stair and stair-to-walk transition strides were included in the respective stair analyses. Hip, knee and ankle kinematic data were calculated in Visual 3D via inverse kinematics, and a custom MATLAB program was created to enable time stamping of data for synchronization of sonomyography and surface EMG with kinematic data.

Sensor Feature Generation

Raw EMG signals were first processed to remove any baseline offset, then a fourth-order Butterworth bandpass filter was applied with a low cut-off at 20 Hz and a high cut-off at 450 Hz to remove motion artifact and high frequency noise, respectively (De Luca et al., 2010; Chowdhury et al., 2013). Then, a sliding analysis window method was used for extracting six features from processed surface EMG data and has been described in detail previously (Huang et al., 2011; Rabe et al., 2021a). The six features included four time-domain features (mean absolute value, number of slope sign-changes, number of zero-crossings and waveform length), as well as the first and second coefficient of a fourth-order autoregressive model. Sliding windows of length 200 ms were used with a 50 ms overlap, generating a 20 Hz signal of six EMG features from each of the eight muscles, resulting in 48 features per time-point (Smith et al., 2011).

The changes in grayscale ultrasound image intensity are correlated to changes in muscle density, as well as muscle architecture features such as muscle thickness and fascicle dynamics. Many researchers have evaluated these ultrasound-based features of muscle morphology for correlations with muscle force production, muscle contraction, and joint motion as well as overall muscle strength and muscle fatigue (Kurokawa et al., 2001; Muraoka et al., 2005; Blazevich et al., 2006; Han et al., 2013; Panizzolo et al., 2013; Li et al., 2020). Based on previous research demonstrating these features are useful for estimation of knee kinematics from sonomyography, mean intensity and temporal intensity features were extracted from each ultrasound imaging frame, as described in detail previously (Jahanandish et al., 2019a; Jahanandish et al., 2019b; Rabe et al., 2020a; Rabe et al., 2021a; Rabe et al., 2021b). The image sequence from each trial was split by heel strikes to create an ultrasound image sequence for each stride. A spatial filter with a block size of 3 × 3 mm was used to extract mean intensity of each 3 × 3 mm block. Then, this 2-dimensional array of mean image intensity features was rearranged into 1-dimension by horizontally concatenating rows of features ranging from superficial to deep image features. The temporal features were created by taking the time derivative of each feature set between consecutive frames. Finally, the mean intensity and temporal intensity features were combined to create a single sonomyography feature set consisting of 520 features per frame.

Regression Model

The Gaussian process regression model is a Bayesian non-parametric kernel-based model that has been introduced as a real-time approximation of inverse dynamics solutions as an alternative for musculoskeletal model-based calculations (Rasmussen, 2004; Nguyen-Tuong et al., 2008; Wang et al., 2008). These models offered accurate function approximation given high-dimensional inputs at a relatively low computational demand to their inverse dynamics counterpart. In this study, we make the assumption that the distribution of joint kinematics within a subject, across multiple strides of a given task (e.g., Figure 2), can be characterized by a Gaussian distribution. Based on these preliminary analyses and previous work, we chose to implement a Gaussian process regression model with a rational quadratic kernel for solving the joint kinematics given the sonomyography, surface EMG, and sensor fusion feature sets. This model has been described in detail previously (Rasmussen, 2004; Zhang et al., 2018), but we assume the general function $\mathit{f}\left(\mathit{x}\right)$ is distributed as a Gaussian function with a mean function $\mathit{m}\left(\mathit{x}\right)$ and covariance, or kernel, function $\mathit{k}\left(\mathit{x},{\mathit{x}}^{\mathit{\prime }}\right)$

$\mathit{f}\left(\mathit{x}\right)\mathit{ }\sim \mathit{ GP}\left(\mathit{m}\left(\mathit{x}\right),\mathit{k}\left(\mathit{x},{\mathit{x}}^{\mathit{\prime }}\right)\right)(1)$

FIGURE 2

FIGURE 2. Hip, knee and ankle joint angle and angular velocity during ten strides of each ambulation task (level walk, incline walk, decline walk, stair ascent and stair descent) from a single representative subject.

The mean function reflects the expected function value at an input, $\mathit{x}$, and the rational quadratic kernel is described as

$\mathit{k}\left(\mathit{x},\mathit{x\prime }\text{|}\mathit{\theta }\right)={\mathit{\sigma }}^2\left(1+\frac{{\mathit{r}}^2}{2\mathit{\alpha }{\mathit{l}}^2}\right)(2)$

with

$\mathit{r}=\mathit{ }\sqrt{{\left(\mathit{x}-\mathit{x\prime }\right)}^{\mathit{T}}\left(\mathit{x}-\mathit{x\prime }\right)},(3)$

$\mathit{\theta }$ is the maximum a posteriori estimates, $\mathit{\sigma }$ is the signal standard deviation or amplitude parameter, and $\mathit{\alpha }$ is the non-negative parameter of the covariance or scale mixture, and $\mathit{l}$ is the length-scale (Schulz et al., 2018). A block coordinate descent method was used to solve for model parameters during training with a block size of 1,000 observations, gradient tolerance of 0.001, step tolerance of 0.001 and a maximum of 100,000 iterations to reach convergence.

Each of the three feature sets: sonomyography, surface EMG, and sensor fusion were used as inputs to separate Gaussian process regression models for subject-dependent training and testing. Leave-one-stride-out cross-validation was used to prevent over-fitting of the model. In other words, a pooled dataset containing strides from all ambulation tasks was created and one stride of each ambulation task was removed for testing, then looped through the strides such that each stride of each ambulation task was the test stride once.

Statistical Analysis

Root mean square errors (RMSE) of each ambulation task and overall root mean square errors across ambulation tasks were calculated, and descriptive statistics (mean and standard deviations (SD)) were reported. A one-way ANOVA was completed to compare RMSE across sensing modalities within ambulation tasks at the hip, knee and ankle joint (α = 0.05). When a group significant difference was found, subsequent Tukey-Kramer multiple comparisons tests were completed to determine significant differences between sensing modalities. Additionally, RMSE were normalized to the range of the measured kinematics for comparison across joints and tasks. Additionally, an adjusted coefficient of determination $\left({\mathit{R}}^2\right)$ for nonlinear regression was calculated as a goodness-of-fit measure for the predicted kinematic trajectories.

$\mathit{Adjusted }{\mathit{R}}^2=1-\frac{\mathit{N}-1}{\mathit{N}-1-\mathit{p}}\left(1-{\mathit{R}}^2\right)(4)$

where

${\mathit{R}}^2=1-\frac{\mathit{Residual Sum of Squares}}{\mathit{Total Sum of Squares}},(5)$

$\mathit{N}$ is the size of the training set, and $\mathit{p}$ is the number of variables in the regression model.

Results

Fusion of features from anterior sonomyography with surface EMG resulted in the lowest overall RMSE averaged across all ambulation tasks at the hip, knee and ankle (Table 2, Table 3, Table 4). At the hip, there was a significant difference in overall average RMSE when comparing hip angles predicted from features of EMG with anterior sonomyography, as well as both anterior and posterior sensor fusion. Additionally, there was a significant difference in overall average RMSE when comparing hip angles from features of posterior sonomyography with anterior sensor fusion. There was a significant difference in overall average RMSE when comparing the hip angular velocities from features of EMG with anterior sensor fusion. At the knee, there was a significant difference in overall knee angle RMSE from features of EMG with all other sensing modalities, in addition to features from posterior sonomyography with anterior sensor fusion. Significant differences were found between overall RMSE of knee angular velocity prediction from features of EMG with anterior sonomyography and anterior sensor fusion. There was an additional significant difference in overall RMSE of knee angular velocity prediction between posterior sonomyography and anterior sensor fusion features. Lastly, at the ankle, there were significant differences between overall RMSE of ankle angle between EMG and anterior sonomyography, anterior sensor fusion and posterior sensor fusion. There was a significant difference between RMSE of ankle angle prediction based on posterior sonomyography features and anterior sensor fusion features. There were no significant differences between RMSE of all sensing modalities prediction of ankle angular velocity.

TABLE 2

TABLE 2. Mean (SD) root mean square error (RMSE) and range-normalized RMSE (nRMSE) of hip angle and angular velocity during level walking, incline walking, decline walking, stair ascent and stair descent. Joint kinematics were predicted by Gaussian process regression models trained and tested on features from five sensing modalities: 1) surface electromyography (EMG), 2) anterior sonomyography (Ant. SMG), 3) posterior sonomyography (Pos. SMG), 4) sensor fusion of Ant. SMG with EMG (Ant. Fusion), and 5) sensor fusion of Pos. SMG with EMG (Pos. Fusion). Overall average values are mean across all ambulation tasks.

TABLE 3

TABLE 3. Mean (SD) root mean square error (RMSE) and range-normalized RMSE of knee angle and angular velocity during level walking, incline walking, decline walking, stair ascent and stair descent. Joint kinematics were predicted by Gaussian process regression models trained and tested on features from five sensing modalities: 1) surface electromyography (EMG), 2) anterior sonomyography (Ant. SMG), 3) posterior sonomyography (Pos. SMG), 4) sensor fusion of Ant. SMG with EMG (Ant. Fusion), and 5) sensor fusion of Pos. SMG with EMG (Pos. Fusion). Overall average values are mean across all ambulation tasks.

TABLE 4

TABLE 4. Mean (SD) root mean square error (RMSE) and range-normalized RMSE of ankle angle and angular velocity during level walking, incline walking, decline walking, stair ascent and stair descent. Joint kinematics were predicted by Gaussian process regression models trained and tested on features from five sensing modalities: 1) surface electromyography (EMG), 2) anterior sonomyography (Ant. SMG), 3) posterior sonomyography (Pos. SMG), 4) sensor fusion of Ant. SMG with EMG (Ant. Fusion), and 5) sensor fusion of Pos. SMG with EMG (Pos. Fusion). Overall average values are mean across all ambulation tasks.

Comparing across joints, the lowest RMSEs for both angle and angular velocity are observed at the hip and ankle, where the overall magnitude of the joint angle and angular velocity are lower (Figure 3). The normalized RMSE revealed that the hip angle and angular velocity were the lowest, there was an increase in normalized RMSE of knee angle and angular velocity, and the greatest normalized RMSE was observed when predicting ankle angle and angular velocity across most sensing modalities. Within the hip, knee and ankle joints individually, the normalized RMSE reveals that there was increased difficulty predicting joint angular velocity in comparison to joint angle for all sensing modalities.

FIGURE 3

FIGURE 3. Mean of all subjects (N = 9) root mean square error (RMSE) of hip, knee and ankle angle and angular velocity predicted by Gaussian process regression models trained by multiple feature sets during five ambulation tasks. Feature sets include surface electromyography (EMG), anterior sonomyography (Ant. SMG), posterior sonomyography (Pos. SMG), Ant. SMG sensor fusion (Ant. Fusion), and Pos. SMG sensor fusion (Pos. Fusion). RMSEs were calculated between sensor-based prediction of joint kinematics and estimated kinematics. Error bars display standard deviations for the respective RMSE. Significance bars indicate significant difference (p < 0.05) between RMSE of (*) EMG and all other sensing modalities, (a) EMG and anterior SMG, (b) EMG and anterior sensor fusion, (c) EMG and posterior sensor fusion (posterior SMG with EMG), and (d) posterior SMG and anterior sensor fusion (anterior SMG with EMG).

Hip Estimation Performance

Anterior sensor fusion resulted in the best predictive performance of the Gaussian process regression model for both hip angle and angular velocity during all ambulation tasks, as evidenced by RMSE, normalized RMSE and the adjusted R² values (Table 2; Figure 4). There were no significant differences in RMSE of hip angle and angular velocity between anterior sonomyography alone and anterior sensor fusion, although anterior sensor fusion consistently outperformed anterior sonomyography. Surface EMG resulted in the worst predictive performance for all walking and stair tasks, followed by posterior sonomyography. The regression model predicted hip angle and angular velocity best during incline walking strides, followed by level walking strides for all sensing modalities. The lowest performance (greatest RMSE and normalized RMSE, lowest adjusted R²) of regression model prediction of hip angle and angular velocity was observed during stair strides for all sensing modalities, with stair descent resulting in the greatest error.

FIGURE 4

FIGURE 4. Hip angle and angular velocity as a function of the gait cycle. Measured kinematics are displayed in gray with standard deviations in shaded regions. Predicted kinematics from Gaussian process regression models trained and tested on features from electromyography (EMG), sonomyography (SMG) and sensor fusion (Fusion) are displayed with respective standard deviations. Adjusted R² given as a goodness-of-fit metric for each sensing modality compared to the measured kinematics.

Knee Estimation Performance

Similar to the hip, anterior sensor fusion resulted in the best predictive performance of the Gaussian process regression model for both knee angle and angular velocity during all ambulation tasks (Table 3; Figure 5). There were no significant differences in RMSE of knee angle and angular velocity between anterior sonomyography alone and anterior sensor fusion. Surface EMG resulted in the worst predictive performance for all ambulation tasks, followed by posterior sonomyography. The regression model predicted knee angle during level and incline walking with the lowest error for all sensing modalities compared to decline walking and stair strides. The lowest error for knee angular velocity prediction was observed during incline walking for regression models trained from all sensing modalities, followed by decline walking, level walking, and stair strides, as evidenced by the normalized RMSE and adjusted R² values.

FIGURE 5

FIGURE 5. Knee angle and angular velocity as a function of the gait cycle. Measured kinematics are displayed in gray with standard deviations in shaded regions. Predicted kinematics from Gaussian process regression models trained and tested on features from electromyography (EMG), sonomyography (SMG) and sensor fusion (Fusion) are displayed with respective standard deviations. Adjusted R² given as a goodness-of-fit metric for each sensing modality compared to the measured kinematics.

Ankle Estimation Performance

In comparison to the hip and knee, the Gaussian process regression model performed worst at the ankle for all sensing modalities. However, comparing sensing modalities, the results remain consistent that anterior sensor fusion resulted in the lowest error of ankle angle and angular velocity, followed by anterior sonomyography alone (Table 4; Figure 6). Surface EMG resulted in the greatest error for ankle angle prediction during all ambulation tasks, with significant improvement between all other sensing modalities versus EMG during incline and decline walking. For both anterior sensor fusion and anterior sonomyography-based estimates of ankle angle, the highest performance (lowest normalized RMSE and greatest adjusted R²) was observed during decline walking, followed by stair descent, incline walking, level walking, and lastly, stair ascent. There were no significant differences between RMSE of ankle angular velocity prediction by any sensing modality during any of the five ambulation tasks.

FIGURE 6

FIGURE 6. Ankle angle and angular velocity as a function of the gait cycle. Measured kinematics are displayed in gray with standard deviations in shaded regions. Predicted kinematics from Gaussian process regression models trained and tested on features from electromyography (EMG), sonomyography (SMG) and sensor fusion (Fusion) are displayed with respective standard deviations. Adjusted R² given as a goodness-of-fit metric for each sensing modality compared to the measured kinematics.

Computational Expense

All models were trained and tested offline on a single CPU (Intel(R) Core i7-7700 at 3.60 GHz). Average time for training and testing a single fold of the leave-one-stride-out framework for the rational quadratic Gaussian process regression model are given in Table 5.

TABLE 5

TABLE 5. Mean (SD) computational time to train the Gaussian process regression model using three separate feature sets containing strides from all five ambulation tasks and test on individual strides of each ambulation task for the hip, knee and ankle.

Discussion

The objective of this research was to evaluate features of surface EMG and sonomyography, as well as fusion of these features, for the continuous prediction of hip, knee and ankle angle and angular velocity. In support of our hypothesis, sonomyography features and sensor fusion of sonomyography with surface EMG, consistently resulted in the greatest predictive performance for hip, knee and ankle angles and angular velocities in comparison to surface EMG alone. However, for all joints (hip, knee and ankle), there were no significant differences between RMSE of angle and angular velocity prediction based on the respective anterior or posterior sonomyography and sensor fusion features, and anterior sonomyography and sensor fusion resulted in relatively lower error compared to posterior sonomyography and sensor fusion. These results are surprising at the ankle, where the addition of information of muscle activity from muscles that span that joint did not significantly improve the regression performance. The increased performance, or reduced error, of anterior sonomyography in comparison to posterior sonomyography, could possibly be attributed to capturing features from three muscles on the anterior thigh (rectus femoris, vastus intermedius, and vastus medialis), as opposed to only two muscles on the posterior thigh (biceps femoris and semitendinosus). This gives further justification to the increased resolution, as well as ability to access deep muscle tissue, as probable explanations for the improved regression performance of sonomyography in comparison to surface EMG. These results are in agreement with our previous work demonstrating increased performance of sonomyography in comparison to surface EMG for ambulation mode classification, as well as high performance of sonomyography-based knee angular velocity prediction and hip, knee and ankle joint moment prediction (Rabe et al., 2020a; Rabe et al., 2021a; Rabe et al., 2021b).

For all sensing modalities, model error increased when comparing prediction of joint angle and angular velocity during walking tasks to stair ambulation tasks. This can likely be explained by the decrease in the number of stair strides in the training dataset compared to the level, incline and decline walking strides, or perhaps the increased variability of the stair ambulation kinematic trajectories. Another possible explanation is the need for additional high resolution (i.e., sonomyography) information from additional thigh muscles during the stair ambulation tasks to account for the increased variability during these tasks. Previous research showed that muscle activity of the individual quadriceps and hamstrings muscles is reduced during stair ascent and stair descent compared to level walking in healthy individuals; however, there is an opposite relationship in muscle coactivity where there is an increase in hamstrings/quadriceps coactivity during stair ambulation as opposed to level walking (Bae et al., 2009). These results point to the potential benefit of sonomyography from the anterior thigh muscles and posterior thigh muscles simultaneously to improve the prediction of hip, knee and ankle joint kinematics.

Accurate prediction of joint output kinematics from a reliable sensing modality is vital to the success of powered assistive devices. While many researchers evaluate control over multiple degrees-of-freedom in the upper-limb, research evaluating simultaneous control strategies over the hip, knee and ankle degrees-of-freedom of the lower limb is less widely reported. The first powered lower-limb assistive devices focused only on powering a single joint, such as the knee or ankle alone (Au et al., 2007; Fite et al., 2007). However, the technological advancement of powered lower-limb assistive devices has enabled the plausibility for multiple powered joints within a single device. Minimizing the complexity of the control strategy for these devices will expediate their entrance into a rehabilitation clinic setting and the hands of users. To date, researchers have relied on pre-programmed trajectories for various ambulatory tasks, such as stair ascent or ramp walking (Lawson et al., 2013; Simon et al., 2014; Lenzi et al., 2018). These strategies have proven to be successful, but require users to adapt their mechanics to accommodate the device, rather than allowing the device mechanics to adapt to the user. This can result in further injury or long-term wear to the unaffected joints (Grabowski and D’Andrea, 2013). The results from the present study are promising for continuous prediction of hip, knee and ankle kinematics that may be able to adapt to the user.

The Gaussian process regression model was chosen based on its recent success for use as a real-time approximation of inverse dynamics solutions to musculoskeletal model-based control. We believe the Gaussian process regression model is an appropriate model for the non-linear relationship between the sonomyography and surface EMG features from multiple muscles with the output of the hip, knee and ankle joints. As an alternative to this approach, Zhang et al. evaluated various ultrasound features of tibialis anterior muscle contraction and surface EMG in a musculoskeletal model-based algorithm for continuous estimation of isometric ankle dorsiflexion moment and compared this model to linear regression as well as a neural network approach (Zhang et al., 2021). While the model-based approach has many advantages, including establishing a functional, “white-box” relationship between the muscle features with the desired output, the results demonstrated there was no significant difference between the musculoskeletal model-based approach and two regression approaches for ankle moment estimation. However, these results indicated that surface EMG features, muscle fascicle length and muscle pennation angle resulted in higher accuracy of isometric ankle moment prediction in comparison to overall mean image echogenicity (of longitudinal images along the tibialis anterior muscle). In a previous study, Zhang et al. combined surface EMG muscle activation features with tibialis anterior pennation angle from ultrasound as inputs to a Hill-type neuromusculoskeletal model for prediction of isometric ankle dorsiflexion moment and found that the fusion of the two sensor data improved prediction accuracy in comparison to sole surface EMG or ultrasound alone (Zhang et al., 2020). Additionally, researchers evaluating contributions of sonomyography and electromyography in both the lower- and upper-limb, have found that both signals contribute to accurate prediction of joint and muscle mechanics (Shi et al., 2007; Ruiz-Muñoz and Cuesta-Vargas, 2014; Boyd and Liu, 2020). The present results further support fusion of sonomyography and surface EMG as an input to regression-based approaches for continuous prediction of joint-level output.

The predictive performance of the Gaussian process regression model was greatest when trained on sensor fusion features from anterior sonomyography and surface EMG. For almost all ambulation modes of all joints, anterior sensor fusion resulted in significantly improved hip, knee and ankle angel, as well as hip and knee angular velocity prediction in comparison to surface EMG. There were fewer instances of significant improvement of the prediction of hip, knee and ankle kinematics when comparing anterior sonomyography alone in comparison to surface EMG. However, there was no significant differences between the anterior sonomyography alone and the anterior sensor fusion-based predictions of hip, knee and ankle kinematics. Importantly, the lack of significant differences between sonomyography alone and sensor fusion of sonomyography with surface EMG indicates that the addition of separate features from surface EMG may not greatly improve the performance of the regression model at the hip, knee or ankle joints. This may be evident when considering the more traditional measurement of surface EMG we used in this study, and not true when considering other newer strategies for recording and decomposition of the surface EMG signal (Stango et al., 2015) Additional work is required to further evaluate the comparison of sensor fusion with sonomyography alone to determine if the burden of collecting data from a separate sensing modality is required for accurate prediction of lower-limb kinematics.

All sonomyography and surface EMG data were recorded in real-time and synchronized to the joint kinematics during each ambulation task. However, feature generation and Gaussian process regression model training and testing was completed offline. The average time to train the regression model on the anterior sonomyography fusion dataset was 23.45 s, and the average time to test the models on strides from the hip, knee and ankle was 5.4 ms. The prediction time is within the optimal reported controller delay window of 100–125 ms for myoelectric prostheses (Farrell and Weir, 2007), as well as faster than the update rates of common lower-limb device controllers (Fluit et al., 2020). However, future work is required to determine if the hip, knee and ankle angle and angular velocity prediction times and accuracies are maintained during online implementation. Given previous success in the upper-limb for online sonomyographic control of dexterous hands (Yang et al., 2020), along with an established correlation between online and offline accuracy of a powered prostheses (Hargrove et al., 2015; Lu et al., 2017), we expect the accuracy of the offline joint kinematic prediction to be comparable, and at a minimum correlate with future online results.

Limitations of the present work exist beyond the offline implementation of the regression models. The subject population included only healthy individuals, without any mobility disorders or limb loss. Additional work is required to determine how the sonomyography features will translate to individuals with gait impairments as well as limb loss. Additionally, the ability of these features and the Gaussian process regression model to predict kinematics of unknown ambulation tasks has not been addressed. Future work evaluating the ability of these features to predict hip, knee and ankle kinematics during unknown tasks will improve understanding of their feasibility for implementation in various activities of daily living, such as stepping over an obstacle. Furthermore, alternative features, such as muscle cross-sectional area and aspect ratio (Guo et al., 2010), as well as alternative models, such as more complex neural networks or musculoskeletal models, may improve performance of the surface EMG and sonomyography features for predicting kinematics of unknown (or “untrained”) tasks. To aid in the evaluation of online prediction of joint kinematics from surface EMG and sonomyographic features of muscle contraction of various subject populations including individuals with mobility disorders, steps should be taken to integrate the sensors into the assistive devices and their control schemes. Researchers have begun to develop electrodes for simultaneous surface EMG and grayscale ultrasound recording within a single system at the same muscle position. (Botter et al., 2019; Yeon et al., 2021). Furthermore, a reduction in the number of ultrasound elements, and perhaps a reduction to single amplitude ultrasound elements, could be possible. Sikdar et al. has demonstrated success in predicting dexterous finger movements with a single element ultrasound transducer in the upper limb (Sikdar et al., 2014), and sparsity analyses of grayscale ultrasound indicated a reduction to equally spaced single scanlines did not introduce significant error (Akhlaghi et al., 2020). Additionally, the transition to single element amplitude ultrasound sensors could reduce the ambiguity of the transfer function between amplitude signals and grayscale images provided by brightness modulated ultrasound. Theoretically, this will support improvement of the evaluation of task- and user-independent prediction from sonomyography.

This work evaluated a task-invariant approach for continuous prediction of hip, knee and ankle angle and angular velocity during five common, yet widely-varying, ambulation tasks using Gaussian process regression models trained and tested on features from surface EMG, sonomyography, and sensor fusion. Anterior sonomyography sensor fusion with surface EMG significantly improved almost all joint kinematics in comparison to surface EMG alone, while anterior sonomyography alone significantly improved only some of the joint kinematics (mostly at the hip and knee) in comparison to surface EMG alone. However, there were no significant differences between sonomyography alone and sensor fusion. Additionally, the results revealed that anterior sonomyography and anterior sonomyography sensor fusion gives more accurate predictive performance in comparison to posterior sonomyography and posterior sonomyography sensor fusion, respectively. Sensor fusion and the Gaussian process regression models predicted level ground and ramp ambulation kinematics with greater accuracy in comparison to stair kinematics. Larger training datasets including additional stair strides, or perhaps sonomyography data from muscles on the anterior and posterior thigh simultaneously could possibly improve stair kinematic prediction. We believe this able-bodied study is a fundamental contribution to the evaluation and potential integration of sonomyography sensors into powered assistive devices. Improved control over multiple degrees-of-freedom of powered assistive devices is critical for translation of these devices into the daily lives of users and improving the quality of life for individuals with gait impairments or limb loss.

Data Availability Statement

The raw data supporting the conclusion of this article will be made available by the authors, without undue reservation.

Ethics Statement

The studies involving human participants were reviewed and approved by the Institutional Review Board at The University of Texas at Dallas. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

KR was responsible for data collection, processing and drafting the manuscript. KR and NF designed the study, analysed the data, and contributed to review and editing the manuscript. All authors approved the submitted version of the article.

Funding

This work was supported by a National Robotics Initiative grant from the National Science Foundation (Grant # 1925343).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

The authors would like to thank their colleagues at The University of Texas at Dallas for assisting with the data collection for this work.

References

Akhlaghi, N., Dhawan, A., Khan, A. A., Mukherjee, B., Diao, G., Truong, C., et al. (2020). Sparsity Analysis of a Sonomyographic Muscle-Computer Interface. IEEE Trans. Biomed. Eng. 67, 688–696. doi:10.1109/tbme.2019.2919488

PubMed Abstract | CrossRef Full Text | Google Scholar

Au, S. K., Weber, J., and Herr, H. (2007). “Biomechanical Design of a Powered Ankle-Foot Prosthesis,” in 2007 IEEE 10th International Conference on Rehabilitation Robotics (Noordwijk, Netherlands: IEEE), 298–303. doi:10.1109/ICORR.2007.4428441

CrossRef Full Text | Google Scholar

Bae, T. S., Choi, K., and Mun, M. (2009). Level Walking and Stair Climbing Gait in Above-Knee Amputees. J. Med. Eng. Techn. 33, 130–135. doi:10.1080/03091900701404043

CrossRef Full Text | Google Scholar

Blazevich, A. J., Gill, N. D., and Zhou, S. (2006). Intra- and Intermuscular Variation in Human Quadriceps Femoris Architecture Assessed In Vivo. J. Anat. 209, 289–310. doi:10.1111/j.1469-7580.2006.00619.x

CrossRef Full Text | Google Scholar

Botter, A., Beltrandi, M., Cerone, G. L., Gazzoni, M., and Vieira, T. M. M. (2019). Development and Testing of Acoustically-Matched Hydrogel-Based Electrodes for Simultaneous EMG-Ultrasound Detection. Med. Eng. Phys. 64, 74–79. doi:10.1016/j.medengphy.2018.12.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Boyd, P., and Liu, H. (2020). “A-mode Ultrasound Driven Sensor Fusion for Hand Gesture Recognition,” in International Joint Conference on Neural Networks (IJCNN) (Glasgow: IEEE), 1–6. doi:10.1109/IJCNN48605.2020.9207492

CrossRef Full Text | Google Scholar

Chowdhury, R., Reaz, M., Ali, M., Bakar, A., Chellappan, K., and Chang, T. (2013). Surface Electromyography Signal Processing and Classification Techniques. Sensors 13, 12431–12466. doi:10.3390/s130912431

PubMed Abstract | CrossRef Full Text | Google Scholar

De Luca, C. J., Kuznetsov, M., Roy, S. H., and Roy, S. H. (2010). Filtering the Surface EMG Signal: Movement Artifact and Baseline Noise Contamination. J. Biomech. 43, 1573–1579. doi:10.1016/j.jbiomech.2010.01.027

CrossRef Full Text | Google Scholar

Delagi, E. F., Iazzetti, J., Morrison, D., and Perotto, A. O. (2011). Anatomical Guide for the Electromyographer: The Limbs and Trunk. 5th ed. Springfield, Illinois: Charles C Thomas Publisher Ltd.

Google Scholar

Dhawan, A. S., Mukherjee, B., Patwardhan, S., Akhlaghi, N., Diao, G., Levay, G., et al. (2019). Proprioceptive Sonomyographic Control: A Novel Method for Intuitive and Proportional Control of Multiple Degrees-Of-freedom for Individuals with Upper Extremity Limb Loss. Sci. Rep. 9, 9499. doi:10.1038/s41598-019-45459-7

PubMed Abstract | CrossRef Full Text | Google Scholar

Disselhorst-Klug, C., Schmitz-Rode, T., and Rau, G. (2009). Surface Electromyography and Muscle Force: Limits in sEMG-Force Relationship and New Approaches for Applications. Clin. Biomech. 24, 225–235. doi:10.1016/j.clinbiomech.2008.08.003

PubMed Abstract | CrossRef Full Text | Google Scholar

Farina, D., Merletti, R., and Enoka, R. M. (2004). The Extraction of Neural Strategies from the Surface EMG. J. Appl. Physiol. 96, 1486–1495. doi:10.1152/japplphysiol.01070.2003

CrossRef Full Text | Google Scholar

Farina, D., Merletti, R., Indino, B., Nazzaro, M., and Pozzo, M. (2002). Surface EMG Crosstalk between Knee Extensor Muscles: Experimental and Model Results. Muscle Nerve 26, 681–695. doi:10.1002/mus.10256

PubMed Abstract | CrossRef Full Text | Google Scholar

Farrell, T. R., and Weir, R. F. (2007). The Optimal Controller Delay for Myoelectric Prostheses. IEEE Trans. Neural Syst. Rehabil. Eng. 15, 111–118. doi:10.1109/tnsre.2007.891391

PubMed Abstract | CrossRef Full Text | Google Scholar

Fite, K., Mitchell, J., Sup, F., and Goldfarb, M. (2007). “Design and Control of an Electrically Powered Knee Prosthesis,” in 2007 IEEE 10th International Conference on Rehabilitation Robotics (Noordwijk, Netherlands: IEEE), 902–905. doi:10.1109/ICORR.2007.4428531

CrossRef Full Text | Google Scholar

Fluit, R., Prinsen, E. C., Wang, S., and van der Kooijvan der, H. (2020). A Comparison of Control Strategies in Commercial and Research Knee Prostheses. IEEE Trans. Biomed. Eng. 67, 277–290. doi:10.1109/tbme.2019.2912466

PubMed Abstract | CrossRef Full Text | Google Scholar

Fukunaga, T., Kawakami, Y., Kuno, S., Funato, K., and Fukashiro, S. (1997). Muscle Architecture and Function in Humans. J. Biomech. 30, 457–463. doi:10.1016/s0021-9290(96)00171-6

CrossRef Full Text | Google Scholar

Grabowski, A. M., and D’Andrea, S. (2013). Effects of a Powered Ankle-Foot Prosthesis on Kinetic Loading of the Unaffected Leg during Level-Ground Walking. J. NeuroEngineering Rehabil. 10, 49. doi:10.1186/1743-0003-10-49

CrossRef Full Text | Google Scholar

Guo, J.-Y., Zheng, Y.-P., Xie, H.-B., and Chen, X. (2010). Continuous Monitoring of Electromyography (EMG), Mechanomyography (MMG), Sonomyography (SMG) and Torque Output during Ramp and Step Isometric Contractions. Med. Eng. Phys. 32, 1032–1042. doi:10.1016/j.medengphy.2010.07.004

PubMed Abstract | CrossRef Full Text | Google Scholar

Ha, K. H., Varol, H. A., and Goldfarb, M. (2011). Volitional Control of a Prosthetic Knee Using Surface Electromyography. IEEE Trans. Biomed. Eng. 58, 144–151. doi:10.1109/tbme.2010.2070840

PubMed Abstract | CrossRef Full Text | Google Scholar

Hallock, L. A., Kato, A., and Bajcsy, R. (2018). “Empirical Quantification and Modeling of Muscle Deformation: Toward Ultrasound-Driven Assistive Device Control,” in 2018 IEEE International Conference on Robotics and Automation (ICRA) (Brisbane, QLD: IEEE), 1825–1832. doi:10.1109/ICRA.2018.8462887

CrossRef Full Text | Google Scholar

Han, P., Chen, Y., Ao, L., Xie, G., Li, H., Wang, L., et al. (2013). Automatic Thickness Estimation for Skeletal Muscle in Ultrasonography: Evaluation of Two Enhancement Methods. Biomed. Eng. Online 12, 6. doi:10.1186/1475-925X-12-6

PubMed Abstract | CrossRef Full Text | Google Scholar

Hargrove, L. J., Simon, A. M., Lipschutz, R. D., Finucane, S. B., and Kuiken, T. A. (2011). Real-time Myoelectric Control of Knee and Ankle Motions for Transfemoral Amputees. Jama 305, 1542–1544. doi:10.1001/jama.2011.465

PubMed Abstract | CrossRef Full Text | Google Scholar

Hargrove, L. J., Young, A. J., Simon, A. M., Fey, N. P., Lipschutz, R. D., Finucane, S. B., et al. (2015). Intuitive Control of a Powered Prosthetic Leg during Ambulation. JAMA 313, 2244. doi:10.1001/jama.2015.4527

PubMed Abstract | CrossRef Full Text | Google Scholar

Hettiarachchi, N., Ju, Z., and Liu, H. (2015). “A New Wearable Ultrasound Muscle Activity Sensing System for Dexterous Prosthetic Control,” in 2015 IEEE International Conference on Systems, Man, and Cybernetics (Hong Kong, China: IEEE), 1415–1420. doi:10.1109/SMC.2015.251

CrossRef Full Text | Google Scholar

Huang, H., Zhang, F., Hargrove, L. J., Zhi Dou, Zhi., Rogers, D. R., and Englehart, K. B. (2011). Continuous Locomotion-Mode Identification for Prosthetic Legs Based on Neuromuscular-Mechanical Fusion. IEEE Trans. Biomed. Eng. 58, 2867–2875. doi:10.1109/tbme.2011.2161671

PubMed Abstract | CrossRef Full Text | Google Scholar

Jahanandish, M. H., Fey, N. P., and Hoyt, K. (2019a). Lower-Limb Motion Estimation Using Ultrasound Imaging: A Framework for Assistive Device Control. IEEE J. Biomed. Heal Inf. 23 (6), 2505–2514. doi:10.1109/jbhi.2019.2891997

PubMed Abstract | CrossRef Full Text | Google Scholar

Jahanandish, M. H., Fey, N. P., and Hoyt, K. (2019b). “Prediction of Distal Lower-Limb Motion Using Ultrasound-Derived Features of Proximal Skeletal Muscle,” in 2019 IEEE 16th International Conference on Rehabilitation Robotics (ICORR) (Toronto, ON: IEEE), 71–76. doi:10.1109/ICORR.2019.8779360

PubMed Abstract | CrossRef Full Text | Google Scholar

Kannape, O. A., and Herr, H. M. (2014). “Volitional Control of Ankle Plantar Flexion in a Powered Transtibial Prosthesis during Stair-Ambulation,” in 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (Chicago, IL: IEEE), 1662–1665. doi:10.1109/embc.2014.6943925

CrossRef Full Text | Google Scholar

Kurokawa, S., Fukunaga, T., and Fukashiro, S. (2001). Behavior of Fascicles and Tendinous Structures of Human Gastrocnemius during Vertical Jumping. J. Appl. Physiol. 90, 1349–1358. doi:10.1152/jappl.2001.90.4.1349

CrossRef Full Text | Google Scholar

Lawson, B. E., Varol, H. A., Huff, A., Erdemir, E., and Goldfarb, M. (2013). Control of Stair Ascent and Descent with a Powered Transfemoral Prosthesis. IEEE Trans. Neural Syst. Rehabil. Eng. 21, 466–473. doi:10.1109/tnsre.2012.2225640

PubMed Abstract | CrossRef Full Text | Google Scholar

Lenzi, T., Cempini, M., Hargrove, L., and Kuiken, T. (2018). Design, Development, and Testing of a Lightweight Hybrid Robotic Knee Prosthesis. Int. J. Robotics Res. 37, 953–976. doi:10.1177/0278364918785993

CrossRef Full Text | Google Scholar

Li, P., Yang, X., Yin, G., and Guo, J. (2020). Skeletal Muscle Fatigue State Evaluation with Ultrasound Image Entropy. Ultrason. Imaging 42, 235–244. doi:10.1177/0161734620952683

PubMed Abstract | CrossRef Full Text | Google Scholar

Lu, Z., Chen, X., Zhang, X., Tong, K.-Y., and Zhou, P. (2017). Real-Time Control of an Exoskeleton Hand Robot with Myoelectric Pattern Recognition. Int. J. Neur. Syst. 27, 1750009. doi:10.1142/S0129065717500095

PubMed Abstract | CrossRef Full Text | Google Scholar

Mendez, J., Hood, S., Gunnel, A., and Lenzi, T. (2020). Powered Knee and Ankle Prosthesis with Indirect Volitional Swing Control Enables Level-Ground Walking and Crossing over Obstacles. Sci. Robot. 5, eaba6635. doi:10.1126/scirobotics.aba6635

PubMed Abstract | CrossRef Full Text | Google Scholar

Merletti, R., Holobar, A., and Farina, D. (2008). Analysis of Motor Units with High-Density Surface Electromyography. J. Electromyogr. Kinesiol. 18, 879–890. doi:10.1016/j.jelekin.2008.09.002

CrossRef Full Text | Google Scholar

Muraoka, T., Muramatsu, T., Takeshita, D., Kanehisa, H., and Fukunaga, T. (2005). Estimation of Passive Ankle Joint Moment during Standing and Walking. J. Appl. Biomech. 21, 72–84. doi:10.1123/jab.21.1.72

CrossRef Full Text | Google Scholar

Nguyen-Tuong, D., Peters, J., and Seeger, M. (2008). “Local Gaussian Process Regression for Real Time Online Model Learning and Control,” in Proceedings of the 21st International Conference on Neural Information Processing Systems NIPS’08 (Red Hook, NY, USA: Curran Associates Inc.), 1193–1200.

Google Scholar

Panizzolo, F. A., Green, D. J., Lloyd, D. G., Maiorana, A. J., and Rubenson, J. (2013). Soleus Fascicle Length Changes Are Conserved between Young and Old Adults at Their Preferred Walking Speed. Gait & Posture 38, 764–769. doi:10.1016/j.gaitpost.2013.03.021

CrossRef Full Text | Google Scholar

Peng, J., Fey, N. P., Kuiken, T. A., and Hargrove, L. J. (2016). Anticipatory Kinematics and Muscle Activity Preceding Transitions from Level-Ground Walking to Stair Ascent and Descent. J. Biomech. 49, 528–536. doi:10.1016/j.jbiomech.2015.12.041

CrossRef Full Text | Google Scholar

Price, M. A., Beckerle, P., and Sup, F. C. (2019). Design Optimization in Lower Limb Prostheses: A Review. IEEE Trans. Neural Syst. Rehabil. Eng. 27, 1574–1588. doi:10.1109/tnsre.2019.2927094

PubMed Abstract | CrossRef Full Text | Google Scholar

Rabe, K. G., Hassan Jahanandish, M., Hoyt, K., and Fey, N. P. (2020a). Use of Sonomyographic Sensing to Estimate Knee Angular Velocity during Varying Modes of Ambulation. Montreal, QC: IEEE, 3799–3802. doi:10.1109/EMBC44109.2020.9176674

CrossRef Full Text | Google Scholar

Rabe, K. G., Jahanandish, M. H., Boehm, J. R., Majewicz Fey, A., Hoyt, K., and Fey, N. P. (2021a). Ultrasound Sensing Can Improve Continuous Classification of Discrete Ambulation Modes Compared to Surface Electromyography. IEEE Trans. Biomed. Eng. 68, 1379–1388. doi:10.1109/tbme.2020.3032077

PubMed Abstract | CrossRef Full Text | Google Scholar

Rabe, K. G., Jahanandish, M. H., Hoyt, K., and Fey, N. P. (2020b). Use of Sonomyography for Continuous Estimation of Hip, Knee and Ankle Moments during Multiple Ambulation Tasks *. New York, NY: IEEE, 1134–1139. doi:10.1109/BioRob49111.2020.9224465

CrossRef Full Text | Google Scholar

Rabe, K. G., Lenzi, T., and Fey, N. P. (2021b). Performance of Sonomyographic and Electromyographic Sensing for Continuous Estimation of Joint Torque during Ambulation on Multiple Terrains. IEEE Trans. Neural Syst. Rehabil. Eng. 29, 2635–2644. doi:10.1109/tnsre.2021.3134189

PubMed Abstract | CrossRef Full Text | Google Scholar

Rasmussen, C. E. (2004). “Gaussian Processes in Machine Learning,” in Advanced Lectures on Machine Learning: ML Summer Schools 2003, Canberra, Australia, February 2-14, 2003, Tübingen, Germany, August 4-16, 2003, Revised Lectures. Editors O. Bousquet, U. von Luxburg, and G. Rätsch (Berlin, Heidelberg: Springer Berlin Heidelberg), 63–71. doi:10.1007/978-3-540-28650-9_4:

CrossRef Full Text | Google Scholar

Ruiz-Muñoz, M., and Cuesta-Vargas, A. I. (2014). Electromyography and Sonomyography Analysis of the Tibialis Anterior: a Cross Sectional Study. J. Foot Ankle Res. 7, 11. doi:10.1186/1757-1146-7-11

CrossRef Full Text | Google Scholar

Schulz, E., Speekenbrink, M., and Krause, A. (2018). A Tutorial on Gaussian Process Regression: Modelling, Exploring, and Exploiting Functions. J. Math. Psychol. 85, 1–16. doi:10.1016/j.jmp.2018.03.001

CrossRef Full Text | Google Scholar

Shi, J., Chang, Q., and Zheng, Y. P. (2010). Feasibility of Controlling Prosthetic Hand Using Sonomyography Signal in Real Time: Preliminary Study. J. Rehabil. Res. Dev. 47 (2), 87–98. doi:10.1682/jrrd.2009.03.0031

CrossRef Full Text | Google Scholar

Shi, J., Zheng, Y., and Yan, Z. (2007). “SVM for Estimation of Wrist Angle from Sonomyography and SEMG Signals,” in 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (United States: IEEE), 4806–4809. doi:10.1109/IEMBS.2007.4353415

PubMed Abstract | CrossRef Full Text | Google Scholar

Sikdar, S., Rangwala, H., Eastlake, E. B., Hunt, I. A., Nelson, A. J., Devanathan, J., et al. (2014). Novel Method for Predicting Dexterous Individual finger Movements by Imaging Muscle Activity Using a Wearable Ultrasonic System. IEEE Trans. Neural Syst. Rehabil. Eng. 22 (1), 69–76. doi:10.1109/tnsre.2013.2274657

PubMed Abstract | CrossRef Full Text | Google Scholar

Simon, A. M., Ingraham, K. A., Fey, N. P., Finucane, S. B., Lipschutz, R. D., Young, A. J., et al. (2014). Configuring a Powered Knee and Ankle Prosthesis for Transfemoral Amputees within Five Specific Ambulation Modes. PLoS One 9, e99387. doi:10.1371/journal.pone.0099387

PubMed Abstract | CrossRef Full Text | Google Scholar

Smith, L. H., Hargrove, L. J., Lock, B. A., and Kuiken, T. A. (2011). Determining the Optimal Window Length for Pattern Recognition-Based Myoelectric Control: Balancing the Competing Effects of Classification Error and Controller Delay. IEEE Trans. Neural Syst. Rehabil. Eng. 19, 186–192. doi:10.1109/tnsre.2010.2100828

PubMed Abstract | CrossRef Full Text | Google Scholar

Stango, A., Negro, F., and Farina, D. (2015). Spatial Correlation of High Density EMG Signals Provides Features Robust to Electrode Number and Shift in Pattern Recognition for Myocontrol. IEEE Trans. Neural Syst. Rehabil. Eng. 23, 189–198. doi:10.1109/tnsre.2014.2366752

PubMed Abstract | CrossRef Full Text | Google Scholar

Wang, J. M., Fleet, D. J., and Hertzmann, A. (2008). Gaussian Process Dynamical Models for Human Motion. IEEE Trans. Pattern Anal. Mach. Intell. 30, 283–298. doi:10.1109/tpami.2007.1167

PubMed Abstract | CrossRef Full Text | Google Scholar

Yang, X., Yan, J., Fang, Y., Zhou, D., and Liu, H. (2020). Simultaneous Prediction of Wrist/Hand Motion via Wearable Ultrasound Sensing. IEEE Trans. Neural Syst. Rehabil. Eng. 28 (4), 970–977. doi:10.1109/tnsre.2020.2977908

PubMed Abstract | CrossRef Full Text | Google Scholar

Yeon, S. H., Song, H., and Herr, H. M. (2021). Spatiotemporally Synchronized Surface EMG and Ultrasonography Measurement Using a Flexible and Low-Profile EMG Electrode. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. 2021, 6242–6246. doi:10.1109/EMBC46164.2021.9629789

PubMed Abstract | CrossRef Full Text | Google Scholar

Young, A. J., Kuiken, T. A., and Hargrove, L. J. (2014). Analysis of Using EMG and Mechanical Sensors to Enhance Intent Recognition in Powered Lower Limb Prostheses. J. Neural Eng. 11, 056021. doi:10.1088/1741-2560/11/5/056021

CrossRef Full Text | Google Scholar

Zhang, N., Xiong, J., Zhong, J., and Leatham, K. (2018). “Gaussian Process Regression Method for Classification for High-Dimensional Data with Limited Samples,” in 2018 Eighth International Conference on Information Science and Technology (ICIST) (Cordoba, Granada, and Seville, Spain: IEEE), 358–363. doi:10.1109/ICIST.2018.8426077

CrossRef Full Text | Google Scholar

Zhang, Q., Iyer, A., Kim, K., and Sharma, N. (2021). Evaluation of Non-invasive Ankle Joint Effort Prediction Methods for Use in Neurorehabilitation Using Electromyography and Ultrasound Imaging. IEEE Trans. Biomed. Eng. 68, 1044–1055. doi:10.1109/tbme.2020.3014861

PubMed Abstract | CrossRef Full Text | Google Scholar

Zhang, Q., Kim, K., and Sharma, N. (2020). Prediction of Ankle Dorsiflexion Moment by Combined Ultrasound Sonography and Electromyography. IEEE Trans. Neural Syst. Rehabil. Eng. 28, 318–327. doi:10.1109/tnsre.2019.2953588

PubMed Abstract | CrossRef Full Text | Google Scholar